ImageFacialWrapping
Only in Wrap4D
Fits a facial mesh to images from calibrated cameras using reference expressions, image appearance, and stereo correspondences. Optional markers and facial detections help align facial features.
The node is used in image-based 4D processing, including head-mounted camera (HMC) capture. Each frame is computed independently, so frames of a sequence can be processed in parallel. The node does not enforce temporal consistency between frames.
Image fitting compares the rendered mesh texture with the input images. Depth fitting uses image consistency between cameras listed in Stereo Pairs to constrain depth. It does not require a scan or a depth map. Dynamic texture and illumination estimation help account for differences in expression and lighting between reference textures and target images.
Tip
Parameters marked (cm) assume geometry in centimeters. Use the same scale for the initial geometry, reference meshes, and camera positions. If you use another scale, adjust all distance parameters accordingly.
Common Use Case
Prepare at least two calibrated cameras, for example with Stereo Camera Calibration Tool, and load their images. Keep images and cameras in the same order.
Approximately align Initial Geometry to the cameras with
AlignGeomToCameras or manually.
The mesh must have UV coordinates.Connect Stabilization Mask and Sampling Mask. Add other masks where needed, as described below.
To use facial detections, connect a matching
FacialAnnotation and per-camera detections,
for example from
LoadFacialDetection. To use markers, connect both
Markers on Initial Geometry and the corresponding Screen Points.Set Deformation Type, Neutral Reference, and Stereo Pairs. For HeavyBlendWrapping or LightBlendWrapping, also supply stabilized expressions in Blendshapes Reference. If Dynamic Texture is enabled, fill in the reference mesh and texture fields on the Textures tab. Otherwise, attach a texture to Initial Geometry.
Click Compute and inspect the fit in both camera views. Refine the settings, then process the sequence. Depending on the pipeline, use
GuidableHeadStabilization,
GuidableReplace,
GuidableDeltaMush,
GuidableMesh, or
GuidableTexture for further processing.
FacialWrapping vs ImageFacialWrapping
FacialWrapping fits geometry to a target mesh. ImageFacialWrapping uses calibrated images
instead. Image quality, camera calibration, reference expressions, and visibility all affect the result.
Regions with little image information depend more strongly on the deformation model and masks.
HeavyBlendWrapping and LightBlendWrapping use reference blendshapes to guide deformation. Wrapping fits the mesh without using reference expressions to guide its shape. Dynamic textures are optional and use a separate set of mesh/texture references.
Reference Blendshapes
Neutral Reference is the neutral expression of the reference character. Blendshapes Reference is a set of reference expressions. All meshes must have the same vertex indexing and topology as Initial Geometry, and use the same scale. Align the expression meshes to the neutral mesh so that they contain expression changes rather than head motion.
Personalized references with a broad range of expressions help constrain areas that are poorly observed by the cameras. The references do not need to be decomposed into individual muscle actions: local combinations can use different parts of different expressions.
Neutral Reference is required in all deformation modes. Blendshapes Reference is required for HeavyBlendWrapping and LightBlendWrapping, and is disabled for Wrapping. References for dynamic textures are supplied separately on the Textures tab.
Note
Stabilize reference meshes before using them. The internal stabilization step does not repair an unstabilized reference set. Head motion in the references can cause artifacts in deformation, dynamic textures, and uncertainty-region prediction.
Sampling Mask
The node fits small regions of the mesh, called patches. Smaller patches allow finer adjustments and increase computation time; larger patches make the mesh less flexible. The mesh topology is unchanged.
Sampling Mask controls patch size: value 1 uses Inner Mask Radius (cm), value 0 uses
Outer Mask Radius (cm), and intermediate values blend between them. Each radius has Initial
and Final settings on the Sampling tab.
A common choice is to mark the visible face and set Inner Mask Radius (cm) lower than Outer Mask Radius (cm). This allows finer deformation on the face and more restrained movement on the sides and back of the head. This mask controls flexibility, not whether a region participates in image fitting.
Uncertainty Mask
Mark regions whose shape is poorly constrained by the images, such as the back of the head or
parts of the cheeks, chin, and nose. The node predicts the marked region from the remaining
geometry using reference expressions, similarly to
GuidableReplace.
Fully mark the region to predict with value 1, leaving an unmarked region to guide it.
Gray values do not provide proportional prediction strength. Use Uncertainty Region Weight
to control the influence of the prediction.
Prediction is available in HeavyBlendWrapping and LightBlendWrapping, and requires both a marked region and a remaining region. In HeavyBlendWrapping, the mask also limits how freely the marked region can deform, even when Uncertainty Region Weight is zero.
Tip
An inverted Sampling Mask can be a starting point for Uncertainty Mask, but inspect the result and adjust it for the actual camera coverage.
The two mask illustrations above show mask distributions. In the current visual editor, Vertex Mask displays the selected mask as a turquoise overlay.
Dynamic Texture
Dynamic Texture estimates an expression-dependent texture from stabilized reference meshes
and their corresponding textures, similarly to
GuidableTexture. This can represent changes
such as wrinkles that a neutral texture cannot describe.
Left to right: target image, neutral texture, dynamic texture.
The Textures tab has its own neutral mesh/texture pair and expression mesh/texture lists. These meshes may differ from the main deformation references, but must have matching topology, scale, and a compatible UV layout. The expression mesh and texture lists must have the same count and order. Reference textures must have the same resolution and file extension as the neutral texture.
When Dynamic Texture is disabled, the node uses the texture attached to Initial Geometry. UV coordinates are required in either case.
Tip
Use Texture Downscale to reduce texture memory and processing costs. A value of 0.5
halves both dimensions. Choose a resolution that preserves the details visible in the target
images; loading smaller source textures also reduces loading costs.
Illumination Simulation
Illumination Simulation estimates a spatial lighting correction to make the rendered texture more consistent with the target images. It can be used with either dynamic textures or the texture attached to Initial Geometry.
Left to right: neutral texture, dynamic texture, dynamic texture with illumination correction.
Lighting is estimated independently for each frame. Higher Illumination Regularization Weight values restrain the estimated lighting more strongly.
Iterations
Computation proceeds through the following levels:
Coarse-to-fine Iterations change the radius and smoothness settings.
Downsample Iterations advance through image resolutions within each coarse-to-fine stage.
Resample Iterations refresh image and geometric correspondences at the current resolution.
Outer Iterations refine the fit after each correspondence update.
Inner Iterations provide further refinement within each outer iteration.
The image width changes from Image Width Initial toward Image Width Final during fitting. Images retain their original aspect ratio. More downsample iterations make resolution changes more gradual, but increase computation time.
Pairs of Initial and Final weights control how the influence of fitting settings changes during fitting. For example, low initial image/depth weights let reference expressions, detections, and markers establish alignment before image detail gains influence. Radius and smoothness settings change at coarse-to-fine stages; fitting weights progress through the fitting stages.
Resample Iterations, Outer Iterations, and Inner Iterations set upper limits. Shape Change Threshold and Tolerance can stop fitting iterations early. Higher iteration limits do not guarantee a better result if the input data or fitting settings are unsuitable.
Note
When Coarse-to-fine Iterations is 1, the radius and smoothness settings use their
Initial values. When both Coarse-to-fine Iterations and Downsample Iterations are
1, the only image-resolution step uses Image Width Final. If Resample Iterations
is also 1, fitting weights use their Final values immediately.
Editor
The visual editor has two views. Left and Right choose a camera independently; select None for a free 3D view. This lets you inspect a camera fit alongside the shape in 3D. The computation preview provides the same display controls.
Camera view (left) and free 3D view (right).
The example uses Show = Result and Wireframe Mode = Always Show, with markers and detections
visible. Full editor screenshot.
Display controls
- Show
All overlays the result at half opacity on the target image. Basemesh shows the input mesh, Result shows the fitted mesh, and Target hides the mesh to show the image alone. Before a result is available, the input mesh is used.
- Texture
Original shows the processed Neutral Reference Texture when Dynamic Texture is enabled, or the processed texture of Initial Geometry when it is disabled. Estimate shows the latest computed texture with illumination when available, otherwise the same texture as Original. None hides the texture while keeping the geometry visible.
- Wireframe Mode
From Input inherits the input geometry’s wireframe setting. Always Show and Always Hide override it. Press X to toggle wireframe visibility.
- Vertex Mask
Displays the selected input mask as a turquoise overlay. Choose None to hide masks. This is a display control; edit the mask in its upstream node.
Camera and overlay controls
- Left / Right
Selects Cam0, Cam1, and other available cameras, or None for a free 3D view. Press F to fit the image or geometry to the active view.
- Points
Shows mesh and image markers in camera views, and mesh markers in free 3D views.
- Names
Shows marker names. Use with Points to check correspondences.
- Detection
Shows facial detections and corresponding mesh annotation in camera views.
Inputs
- Initial Geometry
GeometryMesh to deform, approximately aligned to the cameras. Requires UV coordinates and valid, connected manifold topology without isolated vertices. A texture must be attached when Dynamic Texture is disabled.- Markers on Initial Geometry
NamedPointsOnTriangle(optional) Named mesh markers corresponding to Screen Points. Supply both inputs together.- Excluded Floating Polygons
PolygonSelection(optional) Polygons excluded from image fitting. They can still deform with the rest of the mesh and follow markers placed on them. Excluded Polygons Stability Weight can resist their drift during fitting.- Cameras
Camera(multiple) At least two calibrated cameras. Camera positions and geometry must use the same coordinate system and scale. Cameras are matched to images, screen points, and detections by their order in the input lists. Facial-detection projection uses each camera’s stored Resolution; its aspect ratio must match the corresponding capture view.- Images
Image(multiple) At least two images, in camera order. Their resolutions may differ, but their aspect ratios must match. There must be at least as many cameras as images.- Screen Points
ScreenPoints(multiple, optional) Per-camera marker sets, in camera order. Only names present in both the camera’s point set and Markers on Initial Geometry contribute to fitting. The number of sets must not exceed the number of cameras.- Facial Annotation
FacialAnnotation(optional) Facial contours and polygon selections on Initial Geometry, created with
FacialAnnotation.
Required when Facial Detection is connected.- Facial Detection
FacialDetection(multiple, optional) Per-camera facial detections, in camera order, for example loaded with
LoadFacialDetection.
Required when Facial Annotation is connected. The number of detections must not exceed the
number of cameras. Matching annotation/detection contours must have the same point and semantic-point
counts; contours are matched by name.- Stabilization Mask
VertexMaskRegion used to estimate alignment to the stabilized references during fitting, similarly to
GuidableHeadStabilization. It does not lock deformation or repair reference alignment.
In Wrapping mode, stabilization uses Blendshape Reference Meshes from the Textures
tab together with the main Neutral Reference. It is inactive when Dynamic Texture
is disabled in that mode.- Sampling Mask
VertexMaskControls the radius settings inside and outside the marked region. See Sampling Mask.- Uncertainty Mask
VertexMask(optional) Marks poorly observed regions to predict from the rest of the mesh. See Uncertainty Mask.- Fixed Mask
VertexMask(optional) Restrains changes to selected vertices relative to Initial Geometry. Any nonzero mask value selects a vertex. Fixed Vertices Weight controls how strongly they are held in place. Selected vertices can still follow head motion when stabilization is active; the mask does not guarantee an exact lock in world coordinates.- Smoothing Mask
VertexMask(optional) Region controlled by Smoothness Weight on Mask Initial / Final. The rest of the mesh uses Smoothness Weight Initial / Final. Value0uses the ordinary weight,1uses the on-mask weight, and intermediate values blend them. This changes deformation smoothness, independently of the patch size controlled by Sampling Mask.- Self-Intersection Mask
VertexMask(optional) Controls self-intersection correction in marked regions. Values between0and1control its strength together with Self-Intersection Weight Initial / Final. A connected mask and a nonzero weight are required. Correction can also move unmarked vertices and does not guarantee that all intersections will be removed.
Note
All vertex masks, polygon selections, and mesh annotations must correspond to the topology of Initial Geometry. The first image belongs to the first camera, the second to the second camera, and so on. Additional cameras may provide markers or detections without an image; append them after the cameras that have images. Stereo Pairs can reference only cameras with corresponding images.
Shorter Screen Points and Facial Detection lists correspond to the first cameras in order. Marker and contour names match features, not cameras.
Output
Geometry
Wrapped geometry with the input topology and texture. When stabilization is applied, an attached transform places the geometry at the fitted position in the scene.
The dynamically estimated texture and illumination are displayed in the editor. They do not replace
the texture attached to the output geometry. Use
GuidableTexture separately when your pipeline
needs an expression-dependent output texture.
Parameters
General
- Compute
Starts computation in a preview window with two views, progress, and a Terminate button.
- Auto-Compute
Recomputes when inputs or parameters change. Disabled by default. With it disabled, click Compute after changing fitting inputs or settings. Batch rendering computes the node as needed.
- Deformation Type
HeavyBlendWrapping uses reference expressions and allows more local flexibility. LightBlendWrapping uses reference expressions with more restrained local deformation, which can help with noisy inputs. Wrapping fits the mesh without using reference expressions to guide its shape and can refine an already close fit.
- Neutral Reference
File containing the neutral reference mesh. Required in all deformation modes.
- Blendshapes Reference
List of stabilized expression mesh files. Disabled in Wrapping mode.
- Reload
Reloads the neutral and blendshape reference meshes from disk.
- Stereo Pairs
Zero-based camera-index pairs for stereo image fitting, for example
[0, 1]or[0, 1], [2, 4]. Both cameras of every pair must have input images. Leave the field empty to disable depth fitting.
Main tab
- Coarse-to-fine Iterations
Number of deformation stages. More stages make changes in radius and smoothness settings more gradual.
- Downsample Iterations
Number of image-resolution steps per coarse-to-fine stage.
- Image Width Initial / Image Width Final
Starting and final fitting widths in pixels. Larger final widths preserve finer image detail but increase computation and memory. See Iterations.
- Image Fitting Weight Initial / Image Fitting Weight Final
Strength of matching the rendered mesh texture to the target images.
- Depth Fitting Weight Initial / Depth Fitting Weight Final
Strength of stereo image consistency for Stereo Pairs. This uses image observations, rather than measured depth maps.
- Control Points Weight Initial / Control Points Weight Final
Strength of fitting mesh markers to matching Screen Points. Lower final weights can reduce the influence of noisy marker tracks once the image fit has converged.
- Smoothness Weight Initial / Smoothness Weight Final
Strength of deformation smoothness outside Smoothing Mask, or over the whole mesh when that input is absent. Larger values resist local deformation but can reduce fitting detail.
- Smoothness Weight on Mask Initial / Smoothness Weight on Mask Final
Corresponding smoothness strength inside Smoothing Mask.
Fitting tab
- Resample Iterations
Maximum number of correspondence refreshes at the current deformation stage and image resolution. Each refresh is followed by fitting iterations.
- Outer Iterations
Maximum number of fitting iterations after each correspondence update.
- Inner Iterations
Maximum number of refinement steps within each outer iteration.
- Image Fitting Trust Threshold
Controls sensitivity to differences in brightness between the rendered texture and the input images. Lower values reduce the influence of large differences and can make fitting more stable, at the cost of fitting detail. This setting also affects the handling of inaccurate markers.
- Depth Fitting Trust Threshold
Controls sensitivity to differences in brightness between stereo images. Lower values reduce the influence of large differences and can make depth fitting more stable, at the cost of detail.
- Image Fitting Normals Threshold / Depth Fitting Normals Threshold
Controls which surface orientations are used for image and depth fitting. Higher values exclude more surface areas viewed at steep angles. These values are not angles in degrees.
- Fixed Vertices Weight
Strength of the position restraint set by Fixed Mask. Larger values hold selected vertices more tightly, but do not create an exact lock.
- Excluded Polygons Stability Weight Initial / Excluded Polygons Stability Weight Final
Strength of resisting unwanted movement of excluded polygons during fitting. Larger values reduce drift but do not lock these polygons to the initial mesh or guarantee rigid motion.
- Uncertainty Region Weight
Final strength of reference-based prediction in the marked Uncertainty Mask region. Prediction requires a marked and an unmarked region, and is unavailable in Wrapping mode.
- Self-Intersection Weight Initial / Self-Intersection Weight Final
Strength of self-intersection correction. Requires Self-Intersection Mask. Set both weights to zero to disable correction.
- Self-Intersection Trust Threshold
Controls how strongly deep self-intersections affect fitting. Lower values reduce their influence and can make fitting more stable.
- Resolution Distance (cm)
Desired separation between intersecting surfaces. Zero aims at contact; larger values request a gap.
- Max Collision Distance (cm)
Maximum distance at which intersecting surface regions can be matched for correction.
- Shape Change Threshold
Stops a fitting step when changes to the mesh become small enough. Units follow the geometry scale. Larger values can reduce computation but leave small corrections unfinished. Later deformation stages can still run.
- Tolerance
Controls when a fitting step is considered accurate enough to stop. Larger values can reduce computation time at the cost of fitting accuracy.
Detection tab
- Detection Eyes Weight Initial / Detection Eyes Weight Final
Strength of fitting detected eyelid contours.
- Detection Outer Lips Weight Initial / Detection Outer Lips Weight Final
Strength of fitting detected outer-lip contours.
- Detection Inner Lips Weight Initial / Detection Inner Lips Weight Final
Strength of fitting detected inner-lip contours. Accurate annotation, detections, and suitable mouth geometry are particularly important for fitting this contour.
- Semantic Lips Weight Initial / Semantic Lips Weight Final
Strength of fitting corresponding semantic points on lip contours.
- Semantic Eyes Weight Initial / Semantic Eyes Weight Final
Strength of fitting corresponding semantic points on eye contours.
- Semantic Other Weight Initial / Semantic Other Weight Final
Strength of fitting corresponding semantic points on other facial contours.
Deformation tab
- Blendshape Compression Type
Sparse uses selected reference expressions. Dense compresses information from the full reference set into the requested number of components.
- Num Components
Number of components used to represent reference expressions. Higher values can retain more expression variation but increase computation and may reduce robustness.
- Blendshapes Constraint Weight
Discourages exaggerated deformation when using reference expressions. Higher values can improve stability but may make extreme expressions harder to fit. Available only for Sparse compression in HeavyBlendWrapping or LightBlendWrapping.
- Blendshapes Regularization Weight
Controls how strongly the fit favors the neutral reference. Larger values reduce expression changes.
- Stabilization Sampling Radius (cm)
Patch radius used during stabilization.
- Stabilization Smoothness Weight
Strength of consistency between neighboring patches in the stabilization fit.
- Stabilization Regularization Weight
Limits excessive use of reference expressions while estimating head alignment.
Blendshape compression, component count, and blendshape constraint/regularization controls are disabled in Wrapping mode. Texture references remain independent of these controls.
Sampling tab
- N Nodes To Use
Controls how broadly local deformation is spread across the mesh. Higher values can restrain local changes and increase computation time. Adjust the radius and smoothness settings first when changing local flexibility.
- Post-sampling Smoothing Iterations
Number of iterations used to smooth transitions between patches.
- Inner Mask Radius (cm) Initial / Inner Mask Radius (cm) Final
Starting and final radius values inside Sampling Mask. Smaller values allow more local deformation.
- Outer Mask Radius (cm) Initial / Outer Mask Radius (cm) Final
Starting and final radius values outside Sampling Mask. Larger values can restrain poorly observed regions.
Textures tab
- Dynamic Texture
Estimates an expression-dependent texture from the reference mesh/texture pairs below. When disabled, fitting uses the texture attached to Initial Geometry.
- Illumination Simulation
Estimates lighting correction for either the dynamic texture or the input geometry texture.
- Neutral Reference Mesh
Stabilized neutral mesh for dynamic-texture estimation.
- Neutral Reference Texture
Texture corresponding to Neutral Reference Mesh.
- Blendshape Reference Meshes
Stabilized expression meshes for dynamic-texture estimation.
- Blendshape Reference Textures
Expression textures matching the count and order of Blendshape Reference Meshes. Use the same resolution and file extension as Neutral Reference Texture.
- Reload
Reloads the texture reference meshes and textures from disk.
- Texture Color Space
Autodetect from File selects the color space from the file extension. Linear reads values as linear; sRGB converts sRGB values to linear values. Choose an explicit setting when the file’s contents differ from the extension’s default.
- Grayscale Method
Selects From Red, From Green, From Blue, or Lightness for converting input images and textures to the grayscale representation used by fitting.
- Texture Downscale
Scale factor for texture dimensions.
1keeps the original size;0.5halves width and height.- Sample Radius (cm)
Patch radius for dynamic-texture estimation. Smaller patches allow more local variation.
- Illumination Estimator Sampling Radius (cm)
Patch radius for illumination simulation. Smaller patches allow more local lighting variation.
- Illumination Estimator Smoothness Weight
Strength of smoothness between neighboring lighting patches.
- Illumination Regularization Weight
Controls how strongly the estimated lighting is restrained in each frame.
The texture reference fields, color-space, grayscale, and downscale controls are disabled when Dynamic Texture is off. Sample Radius (cm) is enabled when either Dynamic Texture or Illumination Simulation is on. The three illumination-estimator settings are enabled only when Illumination Simulation is on.
When Dynamic Texture is off, the stored Grayscale Method still affects input images and the input geometry texture, and Texture Downscale still affects that texture, even though these controls are disabled.
Troubleshooting
Temporal jitter in geometry
Since frames are fitted independently, noisy images, detections, or marker tracks can produce jitter. Check those inputs first. To restrain unstable deformation, try increasing Smoothness Weight Final (or Smoothness Weight on Mask Final in a selected region), or increase the appropriate Inner Mask Radius (cm) Final / Outer Mask Radius (cm) Final.
More local jitter on the left; a smoother example on the right.
For poorly observed regions, review Uncertainty Mask and Uncertainty Region Weight in a
blendshape deformation mode. Stronger smoothing can suppress real expression detail as well
as jitter.
TemporalSmoothing can help with residual temporal noise after fitting.
Artifacts in dynamic textures or illumination
Texture artifacts on the left; a cleaner example on the right.
Check the scale and stabilization of the reference meshes, UV compatibility, and Sample Radius (cm) on the Textures tab. A radius that is too small can allow unstable local texture variation. Check illumination settings separately if the artifact appears only with Illumination Simulation.
Incorrect texture detail (left); cleaner result (right).
Also check that Blendshape Reference Meshes and Blendshape Reference Textures describe the same expressions in the same order. A mismatch can associate texture changes with the wrong shape.
Poor image quality
Noisy image on the left; cleaner image on the right.
Noise, blur, and inconsistent exposure can weaken image fitting and increase jitter. Improve the source images where possible. Increasing the final radius values or smoothness can restrain noisy local motion. Lower Image Fitting Trust Threshold or Depth Fitting Trust Threshold can reduce the influence of noisy image regions, but cannot recover missing image detail.
Interior of the mouth is broken
Damaged mouth on the left; a better fit on the right.
Compare the projected Facial Annotation with Facial Detection on a neutral frame. They should
follow the same anatomical features. Correct inaccurate detections with
EditFacialDetection
or fix the source tracking. If the inner-lip fit remains unstable, reduce
Detection Inner Lips Weight Initial / Final and inspect the mouth socket.
Mouth socket has a hole
Inner-lip contour fitting needs suitable mouth-socket geometry. It identifies the boundary between opposing lips, or between a lip and the mouth socket. A hole visible through the open mouth can interfere with this contour detection.
Open mouth socket on the left; closed socket on the right.
Use a closed mouth socket, or ensure that any opening is not visible from the fitting cameras.
FillHoles can close holes, but changing topology requires updating all corresponding reference
meshes, vertex masks, polygon selections, annotation, and mesh markers.
Visible hole on the left; closed surface on the right.
The fit is not accurate enough
Check calibration, initial alignment, reference-expression coverage, and detections first. To allow more local fitting, reduce the relevant final patch radius or smoothness weight. A larger Image Width Final preserves more image detail. In a blendshape mode, increasing Num Components can retain more reference variation. Change one setting at a time and check poorly observed regions for new artifacts.
Bad eye fitting
Check that annotation and detection follow the same eyelid feature on a neutral frame. Place the annotation along the eyelash attachment line, and track that same line in the images. Using the boundary of the sclera in one input and the eyelash line in the other creates a systematic mismatch. Inspect both camera views before changing eye-fitting weights.
Cannot wrap an extreme expression
Check whether the reference set contains the required shape variation. In a blendshape mode, an
insufficient reference set can prevent the fit from reaching an extreme expression.
Manually clean up an example with
Brush, stabilize it, and add it to Blendshapes Reference.
For generic references, retarget the example to the reference character before adding it.
If it is also added to the dynamic-texture set, supply the matching texture in the same list position.