KeenTools Builds and Tracks Faces for Nuke and Blender Artists

KeenTools, a team of software developers based in St Petersburg, started up in 2017 with a straightforward plugin called GeoTracker for Foundry Nuke, designed for object tracking. The software met with success within the Nuke community, and the same approach was applied to development of further KeenTools products for Nuke. These include FaceBuilder and FaceTracker plugins, which are used to create a face model based on photography, and then track it, without specialised tracking software or tracking skills.

Digital Media World spoke to Andrew Chernoberezhskii, a member of their small team of about 10 people who also works on marketing, about the company’s background. “Some of us have VFX and video production experience, others are programmers mostly specialised in computer vision. We started with a proof-of-concept node for Nuke called PinTool, and then gradually added new nodes like GeoTracker, FaceBuilder, FaceTracker and so on.”

Pin Tool precisely positions a 3D object into a scene, working either with stationary objects or for very simple object tracking, just by dragging control points on the surface their correct position in the frame and at the same time estimating camera focal length.

Pins are created on the mesh and dragged to shape the model, which is then continuously refined using the other views.

Making Life Easier

“Our main goal is to make the work of VFX artists easier and more pleasurable,” Andrew said. “Whenever we see a common task being done with too much effort, while it could be automated, we want to improve it. The KeenTools team is quite small and we have to plan our resources carefully, but in the end, a passion to bring new, science-based approaches to modern VFX pipelines is what drives all of our projects, no matter if they are for Nuke, Blender or any other platform.”

Face Builder has, in fact, now been updated and also extended to Blender 3D modelling and animation software, as well as Nuke. The team held a two-month open beta to collect feedback from many interested Blender users. Updates were made to both FaceBuilder versions, adding support for facial expressions.

Earlier on, FaceBuilder projects had to be based on images with a neutral face, which limited its applications. With the new release that limitation has been lifted so a user can, for example, use the same footage to both create a model and to track it, even if the video doesn’t contain a frame with a neutral expression. While the results are not quite as precise as when using neutral expression photos, the final look is much more effective than when adapting a neutral model to a smiling face.

FaceBuilder for Nuke and Blender

The FaceBuilder plugin is supplied as a node for Nuke and an add-on for Blender that can be used to create 3D model of a human face or a head with clean topology starting with a few photos captured from different angles. The images must be of the same resolution and focal length. FaceBuilder is useful whether or not you have much 3D modelling experience or expertise.

The models can be used later for rigid geometry tracking with GeoTracker, or tracking of facial gestures with FaceTracker in Nuke. You can also save and export the model as a file using Nuke's built-in WriteGeo node. In Blender, the models can be used for sculpting, animation, tracking and most other tasks. You can modify the model without leaving Blender using a procedural approach – whenever you want to tweak any details, the changes already applied to the model will be applied again.  

Using GeoTracker to create tracked geometry to serve as a particle emitter.

FaceBuilder can be used with a video as a source instead of still photos, which makes the new support for facial expressions particularly useful because often, the only imagery available to work with may be video that does not include neutral shots of an actor's face. The head models can also be exported from both Nuke and Blender for further modification in any other 3D software, and it is possible to export only the parts of the head that are needed for a project.  

Multiple Skills for Artists

The differences between FaceTracker and GeoTracker are also interesting. GeoTracker is a model- and keyframe-based object and camera tracker for Nuke that can be used with rigid and deformable 3D models - without all of the usual challenges associated with match-moving work. For instance, tracking tasks that may once have required a team of motion tracking specialists may be possible for a single artist to handle, in less time. Beyond that, compositing artists could also become skilled in tracking using GeoTracker, reducing pipeline complexity.

You can add surface masks, mesh deformation, object enhacements like colour correction, relighting and reflections. The model could also be used to apply textures to video, or produce tracked geometry to serve as a particle emitter or to replace objects.

FaceTracker is very similar but was specifically created for facial tracking in Nuke without motion capture rigs and markers. The tracking information can be later used for retouching, adding scars, relighting, face replacement, aging and de-aging and so on. As well as the model's position, it also tracks facial gestures. The tracking algorithms are stable, unlike neural networks systems, precise, allow full manual control and don't require the on-stage preparation that mocap systems depend on.

KeenTools’ upcoming plugin is TextureBuilder. TextureBuilder is a node for Nuke that can gather image data from a number of merged frames to create a texture for an object placed in those frames. The team initially developed it to work in combination with FaceBuilder, but it can work with any geometry and effectively eliminate the lengthy Project3D/FrameHold/MergeMat workflow usually needed to obtain a texture of an object or a clean plate.   keentools.io