4D Processing Pipeline
4D performance capture is a great source of data on where each millimeter of the actor’s skin is moving at each moment in time.
Different capture methods use depth cameras, infrared or color cameras to achieve that. Some methods use a single camera (monocular capture), others use multiple synchronized cameras capturing images of an actor from different views.
In this documentation we only cover the method of videogrammetry where an actor’s performance is captured using multiple synchronized cameras producing a set of image sequences for each camera. Images for each frame are then used to reconstruct a 3D scan of the actor’s face using traditional photogrammetry reconstruction software. The resulting sequences of 3D scans contain a lot of useful data but it can’t be used directly in production because each scan has a unique topology and UV.
The next step should be to convert the sequence of 3D scans into a consistent topology that can be used in production. This is where our 4D processing pipeline comes in handy. The pipeline is built around 4 standalone applications:
Wrap4D
Wrap4D is an extended version of Wrap. It contains 10 special nodes designed specifically for the processing of 4D sequences. Wrap4D takes a sequence of textured 3D scans as an input and produces a sequence of meshes with consistent topology as the output. It’s important that each frame is computed independently, which means we can process them in parallel. In order to produce good results Wrap4D requires a sparse set of markers (dots) to be painted on an actor’s face. The marker positions are used as initialization for the wrapping process and greatly increase robustness when processing extreme facial expressions. Wrap4D also uses information about eyelid and lip contours that are detected in Track.
Wrap4D includes the following nodes for 4D processing:
Track
Track is a 2D tracking software that fulfills two purposes:
Tracking markers on the actor’s face.
Detecting lip and eyelid contours using a personalized facial detector.
Node
Node is a headless version of Wrap4D with a command-line interface. It can be used in combination with Rush to run computations over a network.
Rush
Rush is a simple to use tool that allows running Wrap4D project computations on multiple machines over a network. Rush automatically distributes and manages the workload over multiple Node instances.