OptiTrack Mocap Puts Characters and Animation at the Heart of VR

Motion capture continues to prove itself as a practical, flexible means of acquiring very accurate motion data for animation. What that data is used for after capture can range from scientific and medical research to highly creative and entertaining applications, like those featured in this article. Virtual reality is especially interesting, as traditional, linear storytelling gives way to experiences with a user-driven element that significantly changes the kind and level of control a content creator exerts over his or her project.

A few key factors are certain, however. Motion capture equipment must be extremely accurate, straightforward to use and flexible in order to make the data it captures useful to content producers. With those factors in mind, the two production companies interviewed here, Globacore and MocapNow, talk about using OptiTrack gear to track and capture motion data for VR projects, multiple-performer work, facial capture and more.

Since launching in Seattle, in late 2015, MocapNow motion capture has delivered services from full performance capture to creating custom rigs, cleaning up data and building or optimising external motion capture systems. The studio’s credits include feature films, video games and virtual reality productions, including HBO’s-winning 'Westworld' VR experience, which earned an Emmy Award. Epic Games’ 'Paragon' announcement trailer is another of their projects, plus the upcoming PlayStation VR game 'Golem' in progress at Highwire.      

Motion Data for Animators

Whether they are working with major studios or indie developers, MocapNow's approach is to be ready with a smooth, efficient motion capture process that they know will result in the best quality animation for each project. Co-owners CJ Markham and Ander Bergstrom have about 25 years of animation and motion capture experience between them,  including AAA games such as 'Halo 5' and 'Grand Theft Auto', and feature films including 'King Kong' at Weta Digital and 'Happy Feet' at Animal Logic.

While setting up a mocap stage for Rockstar London in 2007, CJ Markham encountered his first OptiTrack system. Impressed by its performance, when he was designing MocapNow's workflow later on, he outfitted both a 70 x 30ft stage with 17ft high ceilings for full body capture, and a smaller 18 x 18ft sound stage for simultaneous audio and facial capture, with OptiTrack systems.

“Because people are often using mocap for speedy turnaround at high quality, Motive software’s one-button click character calibration is huge for us. It saves at least an hour each session,” Markham said. As of last year, all OptiTrack systems equipped with Motive 2.0 software are self-calibrated with normal use after the initial traditional wand-wave at installation. The presence of markers within the space, attached either to performing players or to static props, continuously calibrates the system.

This change also means that the precision of the tracking system resists drifting over time. As well as improving the tracking quality, this is a major workflow advantage in terms of making experiences easy to operate with non-technical personnel. Importantly, associated with this is the ability to export the FBX actor file directly into content creation pipelines first, allowing animators to make edits to data before retargeting, without significant time or budget costs.

Ander Bergstrom said, “When we deliver motion files to the client, we find that delivering those motions as .FBX actor data gives animators more options to adjust and fine tune the retargeting for themselves before the motion is baked down to the skeleton, rather than doing it for them. If data goes directly to a skeleton instead, as it does with some other software, it can limit an animator’s artistic influence and control over the pipeline to fix issues.”

In the Volume

MocapNow’s performance capture setup includes 34 cameras comprising a mix of Prime 41s, Prime 17s, Prime 13s and Prime 13Ws all running through Motive software. Sessions are typically captured at 180 fps, often with five to six simultaneous performers. When clients want to see CG characters mirroring a real-time performance during sessions, MocapNow uses Motive’s plug-in to stream the data live from Motive into MotionBuilder where retargeting occurs on-the-fly and visualises for the client the effect the performance will have on the character.

In contrast to Markham who has been using Motive throughout its evolution, Ander started working with the software only recently. “The software has been easy for me to learn, which means I don’t have to manage extended ramp-up times when we bring in contractors to help with projects. They’re up and running in less than a day,” he said.

Face to Face

As significant as MocapNow's ability to work with multiple characters in the capture volume is their approach to capturing dialogue in VR. Viewing behaviour inside a VR experience is at odds with the traditional method of filming movies, cinematics or cut scenes. Normally, one participant is shot at a time and the two sets of footage are then spliced together in post during the edit to tell the story.

However, unlike linear entertainment with a fixed POV reliant on camera work, VR allows users to explore the entire environment at any moment. If VR conversation scenes are shot the usual way, the non-speaking character is entirely absent from the scene and meanwhile the viewer is likely to become distracted and compromise the integrity of the dialogue.

To overcome that problem, MocapNow has designed a process for VR experiences that captures both performers simultaneously using 24 Flex 13 cameras, Motive software and a huge number of facial markers. “We've set up a smaller capture volume inside an enclosed room built especially for audio recording with double layers of drywall, acoustic insulation and so on. We have effectively divided this volume in two by fitting it with two sets of OptiTrack cameras tracking the facial motion of each person using 3mm reflective markers, meanwhile recording sync'd audio from a microphone array.

There are several ways to sync the audio and video data. OptiTrack has a hub called the E-Sync 2 that ingests the motion data timecode and passes it into the Motive system, where it its used to sync with the timecode from an audio source. A low budget alternative to this is to place markers on a conventional clapper. At the start of the action, the clapper denotes the beginning of the audio by creating a spike in the soundwaves at the same moment as the markers' motion is recorded in Motive. Both events can be sync'd manually, again using timecode.

Ander told us that with accurate equipment and some forward planning, capturing simultaneous performances  by multiple actors, for VR or conventional projects, either facial or full body, doesn't have to be too difficult. “On our stage, we are prepared to capture up to five performers simultaneously. That number is limited or expanded by the number of cameras you have, the processing power of the capture PC you are using and, of course, the space you have to let them perform. For facial capture, the principle is the same.”

Globacore Scales Up to Large-Scale Multi-Player VR

Globacore specialises in developing new approaches to experiential marketing. Founded in 2004, the company was an early adopter of virtual reality tools and was a supporter of the original Oculus Kickstarter. Globacore's first VR project, PaperDude VR, was an internal demo that successfully used a DK1 to explore the medium's possibilities. Inspired by the 1980s arcade game 'Paperboy', participants pedal virtually through an 8-bit neighbourhood while cycling on a real, stationary bike. With the experience gained from this initial project, Globacore has continued to work with VR as the medium has evolved.    

In April 2016, Globacore developed an untethered, or free roam, large-scale VR project titled Escape Tomb VR, for the Samsung Developer Conference. An escape room experience with a Mayan Temple theme, participants solve puzzles inside the VR environment, picking up objects and moving them to various locations around a 20ft x 10ft space. OptiTrack cameras tracked the positional data of objects in the space, which was then streamed over Wi-Fi to a Galaxy S7 in the Samsung Gear HMD.  

OptiTrack was chosen for the motion capture element of the project, mainly because of the  quick calibration process and the software, which Globacore found easier to understand than the software in most mocap systems. OptiTrack’s Director of Interactive Technologies Jeff Beavers helped them put together their initial prototype and supplied real-time support during setup of the first free-roam installation.

Though Escape Tomb VR was extremely well received, the building process and customer experience revealed an area they would have to improve in their untethered projects. Since the majority of Globacore’s VR installations are for high traffic events like trade shows, the company recognised that higher throughput was needed. Making participants wait for three hours for their turn on a five-minute experience wasn't ideal. Consequently, Globacore focused on optimising its workflow for creating multiplayer experiences.

Refining the Untethered Experience

For Intel’s Developer Forum in August 2016, Globacore built an untethered two-player VR experience, Virtual Code Battle. Using a 20ft x 25ft footprint and OptiTrack cameras for tracking, the experience arms participants with a weapon to take out computer bugs in a 'Tron'-style environment. Able to roam freely within the space, communicate via headsets and see each other in the virtual environment, participants work together to destroy the bugs. 

Since then, Globacore has used each new project to extend its capabilities. From November 2016 to February 2017, the company delivered seven free roam VR experiences, using OptiTrack tracking for each one. Recently, Globacore has focused on building experiences with a 60ft x 30ft roaming area that can accommodate up to eight participants at a time. This new venture debuted at Augmented World Expo (AWE17) in Santa Clara in late May 2017 and employs a custom-built combination of OptiTrack Active and passive tracking.

OptiTrack's ability to adapt to different environments has been an advantage for Globacore. The temporary, mobile nature of their installations means that whatever gear they adopt must perform well on the road. Getting warehouse scale VR experiences up and running in a couple of hours, and training on-site teams to manage and set them up for subsequent events, are key parts of their business. After the first project, OptiTrack walked the team through the hardware and software in detail, giving them confidence to both develop new projects and set up experiences on hectic trade show floors without worrying.

The Nature of Physical Spaces

Brian Nilles, OptiTrack Chief Strategy Officer, talked further about what developers are learning about using real-time motion capture with VR experiences and how it is changing the scene for companies like Globacore. “The physical space requirements are dictated more by the specific project than the tracking system,” he said.

“A typical location-based experience has a setup area where participants put on their HMD, backpack and perhaps a weapon or controller, plus a play area, ranging from 30 to 100sqft or larger. Many of these experiences using OptiTrack systems have the cameras placed at the perimeter of the player area – much like traditional motion capture volumes.

 
“Some experiences, like those created by The Void and Nomadic, use walls and other real objects to add physical touch, and sometimes other senses, to the virtual experience. Because these walls may block the view of perimeter tracking cameras, we often use an overhead tracking system instead. An overhead grid configuration has other advantages as well. It can scale more easily to support larger, multiplayer environments, and the installation layout can be duplicated for multi-site rollouts. This means that the experience content and wall placement can be changed freely without altering the tracking system.”

Complexity

While real obstacles can be used in the space, many experiences just have wide open player areas. In fact, Brian pointed out that when viewing the virtual world from a HMD, most people find it very difficult to convince themselves to walk through virtual walls or other props, even if no physical props are used.
 
“For those cases where interacting with real world objects is allowed, only the objects that participants will move have to be tracked with the OptiTrack system,” he said. “For example, the position of a stationary wall in the physical world must match the view in the VR scene, but that can be done in CG without real-time tracking. For something like a door that will be opened physically in real time, tracking is required.

Once installed, the Optitrack system doesn't actually need to know anything about the experience itself or the way it was designed, relying instead on the extreme accuracy of its position orientation - it will track anything and everything present in the player area that has markers. The system automatically identifies those objects and their relative positions and then begins streaming data via the OptiTrack SDK. This makes the designer pretty much a free agent.

“Our engineers usually do get involved with the design of the tracking system, but only in order to optimise the number, type and location of cameras to produce a great experience and minimise cost. But after installation, even if the content is changed frequently or walls are repositioned, the system continues to perform as designed,” said Brian

Active vs Passive for VR

Noting that, for users, the calibration processes for active and passive OptiTrack systems are virtually the same, Brian explained that although Globacore has custom-designed a project that combines active and passive tracking, not many case would benefit from such a combination. “In fact, one drawback that tends to trump any other benefits is the cost. Passive tracking requires the more expensive OptiTrack cameras with strobes. Our active cameras cost about 40 percent less,” he said.

“Furthermore, the main benefit for developers of using OptiTrack Active for VR is to be able to mass produce one SKU [a single manufactured model ID] for each type of HMD, weapon and object, not every item, used in an experience. That way, the tracking system recognises and identifies the different types automatically, and only differentiates each item during use by its pulse.

“The passive approach, on the other hand, requires spatially different marker patterns for each item, so they must be fabricated individually, and then identified and managed manually in the software. Passive markers are also easily damaged, lost or stolen, so the flush-mount, nearly invisible OptiTrack Active LEDs are ideal for high-throughput, location-based experiences.”

Relationships

When talking about different types of VR, it's as well to remember the relationship between  the player, the set and the camera, or point-of-view. When motion data is recorded, the relationship often changes. OptiTrack systems track both active and passive markers, placed on people or sets, with the same accuracy and deliver a real time stream that can be recorded and processed offline.

If the users are inside the experience, then they are watching the world through their own HMDs and their POVs are constantly changing with the movement of their head. But if the data for a location-based experience were recorded, it would most likely be done in the game engine with a single POV, which could be from one of the player’s HMDs or could be a virtual camera placed anywhere in the scene.

Free roaming, different again, is interesting because it really depends on the extent of the experience video that the designer and CG artists have built. Brian said that OptiTrack encounters content that needs systems like theirs and, conversely, other experiences being built that systems can't cope with yet.  He said, “Some living room-sized VR experiences use a teleportation function to enable navigation outside their 8ft or 10ft tracking space, but translating in space using a controller can make some people queasy - I’m one of them. Not surprisingly, unfolding an environment to larger physical tracking areas makes the experience more immersive. People find actually walking to another part of the environment more compelling than teleporting.

“The game development community is just starting to explore this medium, and will forever push the boundaries of the available equipment because the storytelling and game play demand it. Last week one of our customer’s projects broke another record with the number of objects he used our system to track. Even as we continue to expand physical boundaries, the game dev masters will always take full advantage.” In that spirit, it seems that OptiTrack is happy to have their boundaries pushed.  optitrack.com