1Universitat Politècnica de Catalunya (UPC)2Institut de Robòtica i Informàtica Industrial, CSIC-UPC
Accepted for Oral presentation at the DAGM German Conference on Pattern Recognition (GCPR 2026)Acceptance rate46.99%*Oral rate20.51%*Not officially confirmed
01 — Overview
Sculpt an animatable avatar with words.
From a short monocular video and a free-form prompt such as “make her nose very narrow”, ManifoldSplat returns an animatable 3D Gaussian head avatar of the same person with the requested, localized change, in about a minute and a half on a consumer GPU.
Text-driven editing of animatable head avatars. ManifoldSplat produces localized, semantically meaningful shape changes through natural-language prompts while preserving identity, photorealism and animatability. (a) Unedited avatar; (b) “make face bigger”; (c) “adjust lips to be very big”; (d) “big nose”.
Abstract
High-fidelity 3D head avatars have reached near-photorealistic quality. While recent methods enable text-driven manipulation, they struggle to provide fine-grained localized control, often entangling features or lacking geometric consistency. Modifying geometry through natural language currently requires slow per-prompt optimization or compromises identity and rigging.
We present ManifoldSplat, the first end-to-end framework for language-guided semantic shape editing of animatable 3D Gaussian Splatting avatars reconstructed from monocular videos. By performing edits within the structured FLAME manifold rather than directly optimizing an unstructured Gaussian cloud, we strictly preserve identity and animation. We introduce DeltaRegion, a per-region disentangled Conditional Variational Autoencoder (CVAE) delivering feed-forward shape deltas, alongside a refining stage to recover view-consistent details. ManifoldSplat reconstructs and edits an avatar in ~90 seconds on a consumer GPU, rendering at ~800 FPS. Extensive evaluations demonstrate our approach sets a new state-of-the-art in localized prompt alignment, geometric coherence, and identity preservation.
Method
Edits are performed inside FLAME’s identity space, a structured manifold of plausible head geometry, rather than on the unstructured Gaussian cloud. Changing only the identity coefficients β changes who the subject is without altering how they move.
Overview of ManifoldSplat.Top: the pipeline turns a monocular video and a prompt into an animatable edited avatar via a frozen DeltaRegion edit followed by Gaussian training and warped refining. Bottom: DeltaRegion, our text-conditioned shape-editing CVAE; the region gate zeroes out non-selected latent blocks before decoding. Click to enlarge.
Backbone
Mesh-bound Gaussian avatar
Following RGBAvatar, each Gaussian is anchored to a FLAME triangle and expressed in its tangent–bitangent–normal frame, with learned blendshape offsets driven by the tracked expressions. Any change of the mesh, including a new identity β, propagates to the Gaussians without full re-optimization.
Stage 1 · Shape editing
DeltaRegion
A conditional VAE fuses a frozen Flan-T5-Large prompt encoding with an MLP embedding of the source shape β0 via cross-attention. Its latent is split into one block per facial region (face, nose, lips, ears, neck, eyes); a gate predicted from the prompt alone zeroes out the blocks of non-named regions, so the decoded Δβ ∈ ℝ300 stays local by construction.
At inference the posterior is dropped and the scaled prior mean is decoded: one deterministic feed-forward pass, with edit strength controlled by qualifiers (“slightly”, “very”) or by a continuous prior-scale knob.
Stage 2 · Gaussian refining
Targeted fixes for edited geometry
(a) Surface consistency: an L2 penalty on TBN-local positions and blendshape offsets keeps splats on the mesh surface, so they follow it when the shape changes.
(b) Smooth color activation: the sigmoid-like SH-to-color mapping is replaced by ½(tanh(v)+1) inside the CUDA rasterizer, avoiding harsh color transitions at zero cost.
(c) Warped refining: the shared FLAME topology gives a closed-form per-pixel warp from the original to the edited mesh, producing view-consistent pseudo-ground-truth; a short re-optimization (~30 s) with frozen colors and an anchor loss realigns appearance cues to the new geometry.
02 — Results
Localized edits that keep the person, and the performance.
Compare each unedited avatar with its edited counterpart. Both are driven by the same pose and expression sequence, so the edited avatar stays fully animatable.
Qualitative results
Identity
Edit prompt
Drag across the video to move the divider: the left side shows the unedited avatar, the right side the edited one. With the viewer focused, use ←→ to move the divider and Space to play or pause.
Unedited“bigger lips”
Full-avatar editing comparison on NeRSemble against InstructPix2Pix and GaussianAvatar-Editor, for “small lips” and “long chin”. InstructPix2Pix produces 2D artifacts and identity drift; GaussianAvatar-Editor preserves structure but barely realizes the requested geometric change; ManifoldSplat delivers a localized edit while preserving identity and surrounding appearance.
Shape editing comparison. Meshes are color-coded by per-vertex displacement from the neutral identity for two ear edits. DESCRIBE3D spreads displacement over face and neck, Semantify drags the bust and face along, and only DeltaRegion (ours) concentrates the deformation on the named region.
Fine-grained local edits. Three subjects and three nose-region prompts, ranging from a global elongation (“long nose”) to localized tweaks of the tip (“upturned tip nose”) and base (“deviated nose base left”). The normal-coded FLAME insets confirm the change is confined to the named region while hair, eyes, mouth and skin remain untouched.
Quantitative results
Full-avatar editing comparison (Table 1 of the paper) on the pre-processed NeRSemble subset released by GaussianAvatar-Editor: 18 natural-language edits per subject covering all six regions, with each subject’s front camera held out for evaluation.
Method
CLIP-D ↑
ArcFace ↑
TV2D ↓
InstructPix2Pix
0.0432
0.5212
0.0210
GaussianAvatar-Editor
0.0467
0.7600
0.0126
ManifoldSplat (Ours)
0.0490
0.9319
0.0122
CLIP-D measures prompt alignment, ArcFace identity preservation, and the anisotropic 2D total variation TV2D serves as a no-reference image-quality proxy. ManifoldSplat tops every metric: since every edit is constrained to FLAME’s 300-D identity subspace, the edited identity is valid by construction (highest ArcFace), and because edits never pass through a per-frame 2D denoiser, they avoid the high-frequency residue of diffusion-based editors (lowest TV2D).
The synthetic corpus used to train DeltaRegion: 580,500 supervised triplets(β0, t, β1), each pairing a source FLAME identity β0 and a natural-language edit prompt t with the target identity β1 = β0 + Δβ that realizes the edit, localized to one of six facial regions.
What is inside
No public corpus pairs free-form prompts with identity-preserving FLAME shape deltas, so we built one. Each edit is authored once as a prototype: a sparse set of FLAME vertices in the target region, each with a 3D displacement vector, plus a canonical prompt fragment (e.g. “hawk-like”). Because all FLAME identities share the same topology, each prototype can be projected onto any source identity, turning a small library of prototypes into a combinatorially large supervised set.
300source identities (MICA fits of LYHM subjects)
129edit prototypes over 6 regions
3edit magnitudes: 0.5×, 1.0×, 1.5×
5prompt variants per (edit, magnitude)
300 × 129 × 3 × 5 = 580,500 rows. For every (subject, edit, scale) tuple, Δβ is the FLAME 2020 shape-coefficient change whose induced vertex motion best matches the (scaled) prototype displacements on that subject’s mesh. It is obtained by iterative optimization (Adam, 1,500 steps, lr 10−2, ℓ2 weight 10−6, warm-started from the analytical least-squares solution) with an outside-region penalty (weight 102) that prevents the deformation from leaking into unrelated areas.
# Download, verify and extract (≈1.41 GB once extracted)
wget -O ManifoldSplat_Dataset.tar.xz "https://zenodo.org/records/24223049/files/ManifoldSplat_Dataset.tar.xz?download=1"
wget -O ManifoldSplat_Dataset.tar.xz.sha256 "https://zenodo.org/records/24223049/files/ManifoldSplat_Dataset.tar.xz.sha256?download=1"
sha256sum -c ManifoldSplat_Dataset.tar.xz.sha256
tar -xJf ManifoldSplat_Dataset.tar.xz
File format
Each *_edit_data.pth is a torch.saved Python dict of flat, row-aligned fields (row i of every field describes the same triplet). It contains only tensors, strings, numbers, lists and dicts, so it loads with the safe torch.load(..., weights_only=True).
Key
Type · shape
Content
source_shapes
torch.float32 · (N, 300)
Source identity β0: FLAME 2020 shape coefficients of the subject before editing.
target_shapes
torch.float32 · (N, 300)
Edited identity β1 = β0 + Δβ. The edit itself is target_shapes - source_shapes.
attributes
list[str] · N
Facial region of the edit: one of face, nose, lips, ears, neck, eyes (constant within a file; the gate label used to train DeltaRegion).
edit_names
list[str] · N
Prototype identifier, e.g. hawk, close_set, thick_both, chin_cleft (see the full list below).
scales
list[float] · N
Edit magnitude applied to the prototype displacements: 0.5, 1.0 or 1.5.
source_subjects
list[str] · N
Identifier of the source subject (LYHM scan ID, e.g. 130925160744).
edit_prompts
list[str] · N
Natural-language prompt for the row, e.g. “make the nose much more hawk-like”.
metadata
dict
num_entries, num_subjects (300), num_edits, num_scales (3), num_prompt_variants (5), attribute_name, scales ([0.5, 1.0, 1.5]), edit_names (ordered list of the file’s prototypes), reg_weight (1e-06) and method ("iterative").
Row ordering
Rows are laid out as edit → scale → subject → prompt variant, so row i of a file is
i = ((e · 3 + s) · 300 + j) · 5 + v
e: index of the edit in metadata["edit_names"];
s: index of the scale in metadata["scales"];
j: subject index; the 300 subjects, their order and their β0 are identical in all six files;
v: prompt variant (0–4). The five prompts of an (edit, scale) pair are shared by all 300 subjects.
Prompts
Prompts are deterministic templates around the prototype’s canonical fragment, with a magnitude qualifier drawn at random per (edit, scale): subtly, slightly, a little, a bit, somewhat for 0.5×; no qualifier for 1.0×; very, significantly, much more for 1.5×. Variant v uses the v-th template of the edit’s family (basic, archetype, asymmetric or custom), e.g. for basic edits:
make the {region} {qualifier} {edit}
{qualifier} {edit} {region}
give the {region} a {qualifier} {edit} shape
adjust the {region} to be {qualifier} {edit}
change the {region} to be {qualifier} {edit}
Example rows from nose_edit_data.pth
Row
edit_names
scales
source_subjects
edit_prompts
0
small
0.5
130925160744
make the nose subtly small
1
small
0.5
130925160744
subtly small nose
5
small
0.5
130926112610
make the nose subtly small
1,500
small
1.0
130925160744
make the nose small
4,499
small
1.5
131218100933
change the nose to be much more small
142,500
hawk
1.5
130925160744
make the nose much more hawk-like
How to use it
Load a region file, slice it by prototype and magnitude, or merge all six regions into a single PyTorch dataset of (prompt, β0, β1, region) samples, exactly the supervision used to train DeltaRegion.
ℹ️
Only FLAME coefficients are distributed (no scans, images or meshes). To turn β into meshes, feed the 300 coefficients to a FLAME 2020 layer; the model files require registration at flame.is.tue.mpg.de and their use is subject to the FLAME license. Source identities are MICA fits of the Liverpool-York Head Model (LYHM). Expression, pose, eyelids and texture are never modified. When using the dataset, please cite ManifoldSplat as well as FLAME, MICA and LYHM.
quickstart.py
import torch
from torch.utils.data import Dataset, ConcatDataset
ROOT = "ManifoldSplat_Dataset"
REGIONS = ["face", "nose", "lips", "ears", "neck", "eyes"] # DeltaRegion gate order# 1) Load one region (plain tensors / lists / dicts -> safe with weights_only=True)
data = torch.load(f"{ROOT}/nose/nose_edit_data.pth", map_location="cpu", weights_only=True)
meta = data["metadata"]
print(meta["num_entries"], meta["num_subjects"], meta["num_edits"], meta["scales"])
# 193500 300 43 [0.5, 1.0, 1.5]
src = data["source_shapes"] # (N, 300) float32, FLAME 2020 identity betas (beta_0)
tgt = data["target_shapes"] # (N, 300) float32, edited betas (beta_1)
delta = tgt - src # (N, 300) localized shape delta (delta_beta)# 2) Direct indexing: rows are ordered edit -> scale -> subject -> prompt variant
E, S, J, V = meta["num_edits"], len(meta["scales"]), meta["num_subjects"], meta["num_prompt_variants"]
def row(edit, scale, subject_idx, variant):
e, s = meta["edit_names"].index(edit), meta["scales"].index(scale)
return ((e * S + s) * J + subject_idx) * V + variant
i = row("hawk", 1.5, subject_idx=0, variant=0)
print(data["edit_names"][i], data["scales"][i], data["source_subjects"][i])
print(data["edit_prompts"][i])
# hawk 1.5 130925160744# make the nose much more hawk-like# 3) All six regions as a single PyTorch dataset of (prompt, beta_0, beta_1, region) samples
class ShapeEditTriplets(Dataset):
def __init__(self, path):
d = torch.load(path, map_location="cpu", weights_only=True)
self.src, self.tgt, self.prompts = d["source_shapes"], d["target_shapes"], d["edit_prompts"]
self.region = REGIONS.index(d["metadata"]["attribute_name"])
def __len__(self):
return len(self.prompts)
def __getitem__(self, i):
return {"prompt": self.prompts[i], "source": self.src[i],
"target": self.tgt[i], "region": self.region}
full = ConcatDataset([ShapeEditTriplets(f"{ROOT}/{r}/{r}_edit_data.pth") for r in REGIONS])
print(len(full)) # 580500
Edit prototypes (edit_names) by region · hover a key for its prompt phrase
The full code for ManifoldSplat will be available soon. We are currently cleaning up the repository and preparing the release.
If you need the code sooner (for instance, to run comparisons for an upcoming submission), please feel free to open an Issue. Knowing there is active demand will help us prioritize and expedite the release process!
The shape-editing model can be used today from 🤗 Hugging Face to edit any FLAME 2020 identity from a text prompt.
deltaregion.py
# pip install torch transformers safetensors sentencepiece
import torch
from transformers import AutoModel
model = AutoModel.from_pretrained("a-canela/DeltaRegion", trust_remote_code=True).eval()
# Edit the neutral (mean) FLAME identity
out = model.edit("make the nose long")
edited_betas = out.pred_shape # (1, 300) FLAME 2020 shape coefficients
print(model.region_probabilities(out.gate)) # [{'face': ..., 'nose': 0.99, ...}]# Edit your own identity (e.g. FLAME shape from MICA / a face tracker)
source_betas = torch.zeros(300) # replace with your subject's 300 coefficients
out = model.edit(["make the lips thick", "make the ears very protruding"],
source_shape=source_betas)
# Control the edit strength continuously
weaker = model.edit("make the nose long", prior_scale=0.6)
stronger = model.edit("make the nose long", prior_scale=1.8)
05 — Acknowledgments
Acknowledgments.
This work has been partially supported by project PID2023-151184OB-I00, funded by the Ministerio de Ciencia e Innovación (MCIN) / Agencia Estatal de Investigación (AEI) / 10.13039/501100011033, by the European Social Fund (ESF) “Investing in your future” and by ERDF, EU; and by the industrial doctorate 2021 DI 00102 funded by the Government of Catalonia.
06 — Cite
Citation.
If you find ManifoldSplat, DeltaRegion or the dataset useful in your research, please consider citing:
BibTeX
@inproceedings{canela2026manifoldsplat,
title={ManifoldSplat: Language-Guided Semantic Shape Editing of 3D Gaussian Head Avatars},
author={Canela, Antonio and Sanchez-Riera, Jordi},
booktitle={DAGM German Conference on Pattern Recognition (GCPR)},
year={2026}
}