Humanoid Everywhere: Large-Scale Real-to-Sim Environment and Data Generator for Humanoid Loco-Manipulation

TL;DR

Abstract

A fundamental question in robot learning is what data matters most for training generalizable humanoid policies. Addressing this requires frameworks that balance scalability, efficiency, generality, and visual fidelity. We present Humanoid Everywhere, an automated Real2Sim data generator for humanoid loco-manipulation. From smartphone captures, our pipeline reconstructs photorealistic scenes via 3D Gaussian Splatting (3DGS) and populates them with foundation-model-generated interactive assets, bypassing manual modeling and per-object scanning. Central to this is a Compositional Gaussian Simulator coupling contact-rich physics with efficient rendering to enable whole-body locomotion and dexterous interaction. Guided by text instructions, this auto-generative pipeline continuously outputs fully annotated trajectories—including states, actions, and multi-view RGB-D. A per-Gaussian relighting module further multiplies visual diversity at sub-linear cost without trajectory re-simulation. By unifying real-world reconstruction, generative assets, and scalable physics, Humanoid Everywhere provides an open-ended data source for Vision-Language-Action (VLA) models, establishing a data flywheel for continual real-world scaling.

1 capture

smartphone walkthrough → interactive simulation

8,000+

fully labeled trajectories (9 scenes, 40 tasks)

16–20 FPS

photorealistic rendering, ~10–30× faster than path tracing

M × N

rollouts × lighting variants at zero extra trajectory cost

What we want to achieve

Goal

From one smartphone walkthrough to labeled robot data. A user captures a scene with an iPhone; the pipeline reconstructs a collision mesh (PolyCam) and a photorealistic 3DGS scene, applies text-prompted image editing with object-level 3D reconstruction to compose new interactive assets, and generates physics-grounded task trajectories for the humanoid—fully automatically.

Turn a single smartphone capture of any real environment into an open-ended, fully labeled supply of contact-rich humanoid loco-manipulation data. A short walkthrough video becomes a photorealistic, physics-grounded interactive simulation in which a humanoid robot walks, reaches, grasps, and manipulates objects—continuously emitting trajectories with exact analytical labels (joint states, SE(3) poses, contact forces, synchronized multi-view RGB-D with instance segmentation) ready for Vision-Language-Action policy training.

The generator is human-out-of-the-loop: only three lightweight steps require an operator—a short capture, choosing which objects to reconstruct, and a text-prompted scene edit (e.g., “add a coffee mug and a laptop”). Everything else—reconstruction, asset generation, physics estimation, grasp synthesis, and trajectory generation—runs automatically. Because new captures and prompts regenerate data exactly where a deployed policy is weakest, the system naturally closes the loop as a post-deployment data flywheel toward an “Internet for Robots”.

Why this matters

Motivation

Robot learning lacks an “Internet for Robots”: a scalable supply of labeled interaction data. General-purpose humanoid manipulation remains bottlenecked by how data is sourced and annotated. Contemporary efforts fall into two dominant paradigms, each facing a structural limitation:

Physics-based simulation

Yields cheap, repeatable, explicitly labeled rollouts—but synthetic assets require unscalable manual CAD authoring, and standard domain randomization discards the subtle depth, material, and geometric cues crucial for precise real-world manipulation.

Generative video world models

Visually faithful, but physics is implicit: no guarantees on contact forces, collision geometry, or friction—phasing through hands, clipping geometry—plus noisy state recovery and latency bounds that make large-scale rollout generation impractical for contact-rich tasks.

Real2Sim via 3DGS: bridging both worlds. We resolve this trade-off with a dual representation: 3D Gaussian Splatting supplies real-world visual richness, while co-registered collision meshes provide the structured geometry required for accurate physical interaction. Explicitly decoupling the visual representation (anisotropic Gaussians) from the geometric one (triangle meshes for collision and rigid-body control) yields both real-world appearance and physically accurate, high-throughput dynamics—explicit MuJoCo physics for reliable grasp planning and whole-body control, 3DGS for visual fidelity.

Contributions

Automated, human-out-of-the-loop data generator

Fuses real-world grounding (3DGS + mesh), web-scale foundation models (editing, reconstruction, pose, physics), and MuJoCo simulation—replacing manual teleoperation with scalable trajectory generation.

Multiplicative scaling with exact labels

A single capture expands multiplicatively (scenes × objects × rollouts × lighting) at sub-linear marginal cost, and every frame carries exact analytical labels: joint states, SE(3) poses, and synchronized multi-view RGB-D with instance segmentation.

Compositional Gaussian Simulator

By explicitly decoupling physics from rendering, the engine populates scenes with unlimited generated assets and re-renders identical rollouts under randomized illumination—scaling visual diversity without additional capture or simulation compute.

How it works

Pipeline

Overview of the Humanoid Everywhere framework. Starting from a monocular walkthrough video, the pipeline operates in two primary stages: (1) 3D asset reconstruction and generation—a photorealistic 3DGS base scene is reconstructed, and user-provided text prompts guide 2D image editing for object insertion; inserted objects are extracted and converted into manipulable assets via single-image 3D reconstruction and depth-calibrated 6-DoF pose estimation. (2) Physics integration and Gaussian simulation—generated meshes are decomposed into convex collision geometries, a VLM infers plausible physical parameters, and the compositional 3DGS renderer coupled with MuJoCo dynamics generates diverse contact-rich trajectories with zero-cost visual augmentation.

1Scene Capture & Dual Reconstruction

Scene capture and dual reconstruction: VGGT camera poses and point cloud, PolyCam/TSDF collision mesh, 3DGS training

A user records a smartphone walkthrough (RGB frames, LiDAR depth optional). VGGT jointly predicts per-frame camera poses and a dense point cloud, while PolyCam or TSDF fusion produces a geometric collision mesh with accurate support surfaces. The aligned mesh initializes training of a photorealistic 3DGS scene; an optional single-step Difix3D+ refinement removes floaters and sharpens renderings.

2Image-Guided Object Generation

Following a segment–calibrate–reconstruct–place–validate pipeline, a rendered view is edited with a text prompt (e.g., Nano Banana) to insert or rearrange objects. Each inserted object is segmented by SAM 2 and lifted by SAM-3D into a textured mesh and Gaussian splat; FoundationPose estimates the full SE(3) pose and a VLM critic corrects residual yaw, seating each asset on its support surface to form the scene layout.

Image-guided object generation: text-prompt editing, SAM-3D reconstruction, VLM physical parameters, SE(3) layout estimation

3Simulation-Ready Assets

A VLM (Qwen3-VL) infers each object's simulation properties from texture, name, and size: material category, volume-consistent mass, and sliding/torsional/rolling friction. An object joins the asset pool only after a closed-loop stability check—instantiated in MuJoCo with its convex collision geometry (CoACD decomposition) and verified to rest without interpenetration, sliding, or jitter. For each object, BODex synthesizes ~20 grasp candidates (wrist SE(3) pose + 7 joint angles for the Unitree Dex3 hand), filtered for reachability from standing.

4Compositional Gaussian Simulator

Compositional Gaussian Simulator: independent Gaussian sets for background, objects, tables, robot links, composed with MuJoCo physics and per-Gaussian relighting

The scene is represented as distinct Gaussian sets for the background, articulated robot links, and manipulable objects. MuJoCo simulates contact-rich dynamics, applying time-indexed SE(3) poses to rigidly transform each set. A unified illumination pipeline combines per-Gaussian relighting (albedo recovery + Blinn–Phong shading) with shadow mapping before final 3DGS rasterization. Within this simulator, an AMO whole-body policy handles bipedal balance and locomotion (legs + waist, 15 DOF) while a motion planner drives the upper body through rest → pre_reach → reach_table → grasp → lift—every rollout densely auto-labeled with exact action labels rather than inferred ones.

Per-Gaussian dynamic shading and cast shadows comparison: 3DGS only, +Shading, +Shadow+Shading

Per-Gaussian dynamic shading & cast shadows. Naive composition (left) leads to static, mismatched illumination that fails as assets move. Our simulator dynamically re-shades each Gaussian per frame (middle) and integrates cast shadows (right), visually grounding the robot and objects with coherent illumination.

Zero-cost lighting domain randomization: varying light direction, color, and intensity

Zero-cost lighting domain randomization. Because relighting decouples physics from rendering, each physics rollout is re-rendered under randomized illumination—varying light direction, color, and intensity—multiplying visual diversity (M rollouts × N lighting variants) at marginal rendering cost, without re-simulating the trajectory.

Only 3 of 9 pipeline stages need a human

Pipeline stage	Agent / tool	Input
Scene capture	iPhone video	User
Camera calibration	VGGT	Auto
3DGS training	gsplat	Auto
Reconstruction target selection	operator picks objects	User
Collision mesh	TSDF / PolyCam + ICP	Auto
Scene editing	text-to-image	User
Object generation	SAM-3D + CoACD + Qwen3-VL	Auto
Grasp synthesis	BODex	Auto
Trajectory generation	motion planner	Auto

Only capture, object selection, and the text edit require an operator—the other six stages run automatically with zero manual 3D modeling or trajectory annotation.

See it in action

Video Demos

Real2Sim: from smartphone capture to interactive simulation

From a smartphone capture of a real scene to a photorealistic, physics-grounded interactive simulation with humanoid loco-manipulation.

Captured 3DGS scenes

Photorealistic 3DGS reconstructions rendered from smartphone captures across diverse real environments—offices, kitchens, meeting rooms, markets, and more—each ready to serve as an interactive simulation environment.

Manipulation: generated assets & egocentric trajectories

Left: from a captured view, text-prompted editing inserts objects that are segmented, pose-estimated (FoundationPose), and lifted into meshes and 3DGS assets. Right: the resulting simulation-ready objects support contact-rich manipulation trajectories rendered from the robot's egocentric view.

Loco-grasp in a captured bedroom scene

The Unitree G1 humanoid walks to the target and executes a contact-rich grasp inside the reconstructed 3DGS environment.

Experiments

Results

Feasible, diverse data — end to end

Top: captured 3DGS scenes and locomotion trajectories across diverse real environments. Middle: from a text-edited image we generate manipulable 3D assets and physics-valid manipulation trajectories, rendered with correct novel-view appearance and geometry from egocentric head/wrist views. Bottom: each rollout is replayed through the Gaussian simulator at zero additional trajectory cost under domain randomization—varied distractor layouts, scene lighting, and shading— multiplying one trajectory into many visually diverse, fully labeled locomotion and manipulation samples.

Scalability: many scenes and objects at low marginal cost

Dataset summary: tasks per scene, trajectory distribution, 9 scenes, 40 tasks, 8000 trajectories

Tasks per scene, trajectory distribution, and dataset summary. Data collection currently spans 9 scenes (bedroom, living room, kitchen, office, meeting room, market, factory, outdoor, lab basement), 40 tasks, and 200 trajectories per task — 8,000 trajectories in total, and remains extensible through flexible capture sources.

Adding new environments and objects is cheap in human effort: only capture, object selection, and the text edit involve the operator, while reconstruction, asset generation, physics estimation, grasp synthesis, and trajectory generation run automatically. Objects are populated consistently with each scene's real-world scenario—lamps and thermos in the bedroom, kettles and bowls in the kitchen, laptops and plants in the office, bottles and fruit in the market.

All trajectory data—synchronized multi-view RGB sequences, proprioceptive states, and action tokens—exports directly to the standardized LeRobot dataset format, immediately consumable by modern VLA architectures on a Unitree G1 humanoid (29-DOF upper body, 12-DOF dexterous hands) with head-stereo and egocentric wrist cameras.

Efficiency: high-throughput photorealistic rendering

Renderer	ms / frame	FPS	Per trajectory
Isaac Sim (path-traced)	~500–1700	~0.6–2	~3–10 min
Ours (3DGS)	50–62	16–20	18–22 s
Ours (3DGS + refiner)	106–118	~9	38–42 s

Rendering throughput per frame and per 360-frame trajectory at 320×240 on a single NVIDIA A6000 Ada GPU. The 3DGS rasterizer sustains substantially higher throughput than path-tracing alternatives while remaining photorealistic.

Render speed (FPS) versus number of Gaussians, with real-time threshold at about 3 million Gaussians

Render speed scales predictably with scene complexity, reaching real-time (30 FPS) at ~3M Gaussians. Because rendering is decoupled from physics, generating N relit variants requires only N rendering passes over a single physics rollout—scalable visual diversification at minimal additional compute.

Data quality: high-quality by construction

Exact labels

Every rollout is simulated, so joint states, per-link and object SE(3) poses, contact forces, and instance segmentation are read directly from the engine—no pose-estimation noise from video or human demonstrations.

Physical validity

Each object passes a closed-loop stability check before entering the asset pool, so contacts and rest poses are simulation-consistent throughout every generated trajectory.

Photorealism

Per-Gaussian shading and cast shadows keep composited objects and the robot illumination-coherent with the captured scene, narrowing the visual domain gap relative to synthetic-mesh rendering.

Toward unbounded scaling

User capture is not even strictly required: an input image—captured or generated—can be expanded by a video world model (e.g., Genie 3) into a complete 3DGS scene, then populated with unlimited generated objects in varied layouts and loaded into the simulator for trajectory generation. This removes per-scene capture effort entirely and points toward generating fully labeled data without bound.

BibTeX

@inproceedings{wang2026humanoideverywhere,
  title     = {Humanoid Everywhere: Large-Scale Real-to-Sim Environment and Data Generator
               for Humanoid Loco-Manipulation},
  author    = {Wang, Junying and Chen, Xinjie and Zhan, Yunkai and Kim, Junsoo and Wei, Songlin
               and Guan, Xulang and Guizilini, Vitor and Zou, Chuhang and Mao, Jiageng and Wang, Yue},
  booktitle = {RSS 2026 Workshop on Data-Centric Robotics},
  year      = {2026}
}