Edit3r

Instant 3D Scene Editing from Sparse Unposed Images

Jiageng Liu*, Weijie Lyu*, Xueting Li, Yejie Guo, Ming-Hsuan Yang

Instruction-driven 3D Gaussian scene editing without per-scene optimization.

Feed-forward 3D editing Recoloring pretext Editor agnostic
Paper Code
Edit3r system overview and edited 3D Gaussian results

Edit3r turns sparse unposed views and inconsistent 2D edits into a coherent edited 3D Gaussian scene in about 0.5 seconds.

Abstract

We present Edit3r, a feed-forward framework that reconstructs and edits a 3D scene in a single pass from sparse unposed images and a text prompt. Instead of optimizing a scene-specific representation, Edit3r directly predicts instruction-aligned 3D Gaussians and resolves cross-view conflicts introduced by stochastic 2D editors. The revised training setup reformulates editing as a SAM2-based recoloring pretext task, pairing one recolored reference view with raw auxiliary views so the model learns to align disparate observations without true multi-view edited supervision. At inference, Edit3r supports diverse 2D editors and renders coherent edited novel views. The paper also introduces DL3DV-Edit-Bench, a 100-edit benchmark over 20 real scenes and four edit types.

Overview & Demo

End-to-end Edit3r walkthrough: instruction -> 3D Gaussian edit -> rendered result.

Edit3r training pipeline

Training pipeline: SAM2 recoloring creates consistent supervision, asymmetric inputs expose cross-view conflicts, and a frozen LRM supplies 3D geometry regularization.

Training Recipe

  • Recoloring pretext replaces noisy multi-view edited labels with deterministic, view-consistent supervision.
  • Asymmetric training pairs combine one recolored reference view with raw auxiliary views.
  • Pose-free Gaussian prediction lifts sparse unposed views into a canonical 3D Gaussian scene.
  • 2D + 3D losses combine CLIP, LPIPS, low-frequency MSE, center matching, and Chamfer geometry regularization.
Edit3r backbone architecture

Backbone overview: a shared ViT encoder, cross-view decoder, and DPT-based Gaussian heads predict 3D centers and rendering attributes.

Training pipeline animation: recolor supervision, asymmetric pairs, reconstruction, and mixed 2D/3D losses.

SAM2 masks and recoloring example

SAM2 Recoloring

SAM2 discovers object masks, filters reliable regions, and applies region-wise color transforms with shared parameters across views.

Recoloring supplies a controlled training signal that teaches Edit3r to fuse edited evidence while preserving scene structure.

Benchmark: DL3DV-Edit-Bench

100 text-driven edits over 20 indoor/outdoor DL3DV test scenes, covering Add / Remove / Modify / Global instructions.

  • Grounding-DINO extracts object labels and region proposals.
  • An LLM synthesizes candidate prompts; five valid edits are kept per scene.
  • Metrics include CLIPt2i, C-FID, C-KID, no-reference image quality, sharpness, and epipolar consistency.
Generated prompts and edited image examples

Generated prompts and edited image examples used to build DL3DV-Edit-Bench.

Quantitative & Ablations

Method Time (s, lower) CLIPt2i (higher) C-FID (lower) C-KID (lower)
GaussCtrl325.530.227135.00.091
EditSplat584.460.241174.10.122
NoPoSplat0.610.253180.60.125
Edit3r (Ours)0.510.266171.30.116
Training Variant CLIPt2i (higher) C-FID (lower) C-KID (lower)
Edit3r0.266171.30.116
w/o Recolor0.243215.00.141
w/o 3D Loss0.237278.40.182
w/o SAM0.248179.60.127
w/o R-Drop0.252183.10.130

Edit3r is the fastest method and reaches the strongest text alignment; ablations confirm that recoloring supervision, 3D losses, SAM2 augmentation, and random drop all improve consistency.

Editor-Agnostic Inference

Edit3r decouples 2D editing from feed-forward reconstruction, so the same 3D fusion pipeline can consume edits from IP2P, FLUX, GPT-Image-1, or Gemini.

Different editors trade off fidelity, localization, and artifacts, while Edit3r keeps the reconstruction and rendering path unchanged.

Edit3r results with IP2P, FLUX, GPT, and Gemini image editors

The revised paper highlights modular 2D editor support during inference.

Qualitative Comparisons

Qualitative comparison against EditSplat, GaussCtrl, and NoPoSplat

Edit3r keeps edits stable across viewpoints while optimization-based baselines are slower and feed-forward reconstruction without editing supervision can blur inconsistent evidence.

One-pass Inference

At inference, all input views are edited by a 2D image editor, then Edit3r performs a single feed-forward pass to produce edited 3D Gaussians and novel-view renderings.

Inference-time editing example

Example inference flow: edited input views produce per-view Gaussians, which are fused into final rendered views.

3D Regularization Matters

Failure cases without 3D regularization

Without explicit 3D constraints, Gaussian centers can drift and separate into layered structures; center and geometry losses stabilize the canonical 3D scene.

Edited Examples

Additional DL3DV-Edit-Bench examples show object insertion, removal, appearance edits, and global stylization across multiple views.

More qualitative editing results

Example Videos

Takeaways

  • Edit3r unifies sparse-view reconstruction and text-driven 3D scene editing in one feed-forward pass.
  • SAM2-based recoloring provides consistent supervision when true multi-view edited ground truth is unavailable.
  • Asymmetric training and 3D regularization teach the model to resolve edited-view conflicts in Gaussian space.
  • DL3DV-Edit-Bench evaluates real scene-level Add / Remove / Modify / Global edits with speed, fidelity, and consistency metrics.