Motion capture companies Animatrik Film Design and Dimensional
Imaging talk about working together to bring specialist facial
performance capture services to VFX and game productions.
Performance Capture Pioneers Collaborate on New Mocap Experience |
Motion capture companiesAnimatrik Film DesignandDimensional Imaging, known as DI4D, are collaborating to bring specialist facial performance capture services to visual effects and video game productions in the US and Canada. |
Believable FacesAnimatrik has recently licensed DI4D’s facial performance capture software and equipment. Their team works either inside volumes at their studios in Vancouver and Toronto, or at locations to suit clients’ productions, and will now be able to carry out facial services at all locations as well. DI4D facial capture has been seen before in Microsoft’s ‘Halo 4’ and other projects. The CEO of AnimatrikBrett Inesonsaid that believable facial performance capture has always been a tough challenge to solve, and that demand from clients for hyper-realistic facial animation is what triggered their search for a new type of system. |
Colin Urquhart, CEO of DI4D, described the distinctive characteristics of facial capture and, specifically, his company’s approach to it. “In body motion capture, it is possible to capture a relatively small set of markers and use their positions to estimate the underlying skeletal pose with a relatively high degree of accuracy. However, a much higher density, and hence smaller size, of markers is required to capture the much subtler animation of the face. In turn, this demands a much higher density motion capture camera.” RadicalVarious helmet mounted camera systems have been developed to address this limitation. But because it is impractical to apply many more than a 100 markers on a face, severely limiting the fidelity of capture, other techniques have been developed that attempt to enhance the fidelity of marker based capture by referencing a set of high resolution 3D scans of the actor’s face, acquired separately. Nevertheless, as the level of realism required continues to be pushed higher, an ever larger number of high resolution 3D scans is required. |
That is why DI4D’s approach to capture, based on 3D scanning instead of tracking markers, attracted Animatrik. Brett said, “We love being able to capture the surface of the face. As a service-based company our job is to meet clients’ expectations. So, if it is a digital double we are after, using DI4D’s system means that nothing is lost, and if we do have to re-map the data, then at least we have the strongest possible data set to work with, giving us a much better chance of making the loss imperceptible.” Also, because of its untethered design it has greater potential for acting, and combines high fidelity facial capture with freedom of movement. Capture and Scanning WorkflowDuring production, theDI4D PROSystem captures sequences of high resolution, colour 3D facial performance data. The systems are supplied with one or more sets or ‘pods’ of three synchronised cameras - two monochrome cameras and one colour, capturing at up to 60 fps. The cameras stream directly to disk so that sequence length is not limited. |
The sequence data from either capture system is post processed with DI4D dense passive stereo photogrammetry software and then, if needed, a fixed topology mesh can be tracked through the sequence using DI4D optical flow tracking software. The process results in very dense, realistic facial animation that preserves the subtlety and distinctive detail of facial performances. |
Optical Flow MapsAt the same time as calculating per frame 3D models, the DI4D Processing Software calculates a denseoptical flow mapfor each image, leading to the previous and next image in the sequence. It is this dense optical flow data that is used in the DI4D Tracking Software to track the vertices of the mesh topology through a sequence. The mesh can be applied to one reference frame in the sequence, and then tracked through the sequence using the flow maps. The software has tools for correcting drift during tracking. You can output the resulting tracked set of discrete landmarks in standard motion capture C3D format, and output the tracked mesh as a vertex cache compatible with 3D software like 3ds Max, Maya or Lightwave. |
Also, if a mesh with a specified UV layout is tracked through a sequence, then a per-frame texture map can be generated using that consistent UV layout. A per frame detail map can also be generated using the same, consistent UV layout, and where colour cameras are used, a per frame colour texture map can be applied to the 3D model sequence as well. Data and Compute PowerProcessing can be carried out entirely on the CPU or can be accelerated by processing on aGPUusing theNVIDIA CUDAarchitecture. Processing can also be accelerated by distributing it over a network of processing PCs. In fact, Colin considers that the most significant industry development helping to improve facial mocap in recent years is the increase in available compute power, particularly GPU acceleration. “This change has allowed us to apply advanced computer vision algorithms to sequences of thousands of frames. Previously, it would have taken several hours to process a single frame,” he said. |
Regarding other practicalities on set, he noted that, unfortunately, it is not really possible to increase fidelity without some increase in data volumes. However, the data captured during shooting with DI4D take the form of relatively lightweight video sequences – albeit lightly compressed to maintain quality – and does not present a particular problem to users. Colin said, “The output data delivered after DI4D processing is also fairly lightweight, comprising a tracked mesh with consistent topology stored as a vertex cache, you’re your optional per frame texture and detail maps. Although the intermediate ‘raw data’ of per frame dense 3D scans and optical flow maps can become several GB per second, it only has to be stored temporarily during post processing, so again does not pose a particular problem for users on the set.” www.animatrik.com www.di4d.com |