Faceware Brings Emotion & Realism to NBA 2K16

Performance capture stage manager Anthony Tominia has been working on the NBA series team for six years, and described their efforts to improve the way they sell the emotion of basketball. “While working on 2K16, we wanted everything to be as realistic as possible,” he said. “That means everything from motion capturing the basketball action itself to representing the real life players in digital form.

“This year we’ve been trying to achieve more expressive emotions, such as better looking lip sync in our ‘My Career’ story mode in which players put together their own backstories, and distinctive reactions that mirror what actually happens on the court. We wanted to delve into the narrative movements this time and sell what’s happening on an emotional level.”

2K has used Faceware’s hardware and software for facial performance capture for several years, and used them again to bring ‘NBA 2K16’ to life, capturing the facial performances of over 30 different NBA athletes. Actions both on the court and off were recorded by carrying out full performance capture for the in-game animations and also recording dialogue for the story mode.

Once acquisition was complete, Faceware Analyzer and Retargeter applications were used to transfer the motions to the game play. Analyzer is markerless facial motion tracking software, used by following a fairly simple workflow. Through Faceware’s proprietary facial recognition system, Analyzer tracks the face and creates its own facial motion capture data, converting video of an actor’s facial performance into motion files for use in Retargeter.

Retargeter software was used to apply the facial capture data from Analyzer onto 2K16’s facial rigs through a plug-in to their 3D software. It is compatible with Autodesk products Maya, 3DS Max, Softimage and MotionBuilder. The user gains control from Retargeter’s combination of the captured poses and movement data, and can use this to produce convincing facial animation.

The 2K team could tailor this process to fit their project’s requirements very closely because Faceware’s software has a straightforward scripting interface. Anthony said, “In this case, we wrote a lot of Python scripts that allowed us to automatically solve the capture that little bit faster. There were over 232 lines spoken by each of the 30 players, and we found that we could easily batch, through Analyzer, 232 lines per player without having to touch the data.

“We did the same thing with Retargeter. In our pipeline the Python script does the retargeting, creates play blasts and saves those files immediately for review. The process doesn’t have to be manned, so we could capture during the day, and then leave the system to analyze and retarget overnight. We would come in the next morning, quickly identify which files needed the most touch up and pass those on to the team.”

The ease of use and customization of the software made it possible for them to shorten cleanup time on NBA 2K16’s animation down to three or four minutes per second of data. Faceware also helped NBA 2K team customize their hardware by building a custom camera that could capture data at a higher, non-standard frame rate the team prefers for the game.