with Embodiment Dreaming
Overview of EgoX. EgoX treats human, real-robot, and simulated demonstrations as unified egocentric trajectories, pretrains one shared manipulation representation, and fine-tunes it on a new embodiment–task domain. Its model, MXT+, routes each embodiment's observations through modality tokenizers into a shared Transformer trunk and decodes actions with embodiment-specific action experts, while a training-only dreaming expert predicts future embodiment latents. Embodiment dreaming and cross-embodiment pretraining both improve the average task progress score and success rate.
MXT+ Architecture and EgoX Embodiments. Top: embodiment-specific modality tokenizers map each embodiment’s observations into tokens that a shared Transformer encoder-decoder trunk fuses across embodiments. Embodiment action experts decode the trunk output into embodiment-specific action chunks, while embodiment-dreaming experts predict future embodiment latents supervised by EMA copies of the corresponding modality tokenizers. Bottom: the embodiments used in EgoX. Each panel marks the unified egocentric frame ℱego and the end-effector-link keypoints.
Choose an embodiment
Select a model to filter videos. Use its expand icon for a larger 3D view.
G1-Dex5
xArm7-Dex4
G1-Sim-Dex3
LocoMan-Dex2
R1Lite-Gripper
Z1-Tool
Policy rollouts
Across all tasks and embodiments, we show successful rollouts of MXT+ with pretraining, illustrating each task setup and what the method can do. On R1Lite-Gripper Kitchen-Bread, we also compare it with MXT+ without pretraining, π0.5, and ACT, marking each rollout as a success or failure. All videos are played at 1× speed.
Does egocentric cross-embodiment pretraining outperform training from scratch?
We compare MXT+ with and without cross-embodiment pretraining against ACT and π0.5 on 13 task–embodiment pairs. Under ID evaluation, pretraining yields particularly large gains on G1-Dex5 Sort, LocoMan-Dex2 Scoop, and xArm7-Dex4 Scoop. The benefit becomes even more pronounced under OOD evaluation, with substantial improvements on G1-Dex5 Kitchen-Bread, G1-Dex5 Insert, and Z1-Tool Scoop.
Normalized task progress score, mean ± SEM. Real robots: 20 trials per pair (13 ID, 7 OOD). Simulation: 50 trials per pair (32 ID, 18 OOD). Hover over a bar for its value.
Main Results. Cross-embodiment-pretrained MXT+ generally improves task progress score across 13 task–embodiment pairs, with substantially larger gains under OOD evaluation.
Does pretraining benefit from embodiment diversity?
We vary the pretraining mixture: no pretraining, human-only, human plus other robots, and full cross-embodiment data, which also includes the target robot's demonstrations on other tasks. Every variant is fine-tuned on the same target-embodiment, target-task demonstrations.
Normalized task progress score for each pretraining mixture, mean ± SEM (shaded). Each mixture adds to the previous one. Hover over a line or a legend entry to trace one setting.
Pretraining Data Composition. Every pretraining mixture improves average OOD performance over no pretraining, and the gains grow as more embodiments are added. Full cross-embodiment pretraining performs best, improving the average task progress score by 0.12 in-distribution and 0.23 out-of-distribution, with OOD gains on all six task–embodiment pairs.
Does embodiment dreaming improve cross-embodiment learning?
Embodiment dreaming asks the shared trunk to predict future embodiment latents built from egocentric image features, robot joint configurations, and end-effector joint configurations. EMA copies of MXT+'s own modality encoders provide stable targets. The objective encourages embodiment-aware dynamics modeling during pretraining and fine-tuning; dreaming heads and EMA teachers are discarded at deployment.
We compare MXT+ with MXT, its counterpart without the dreaming objective, on seven task–embodiment settings across five embodiments. The gains hold across tasks and embodiments, and are particularly pronounced on G1-Dex5.
Normalized task progress score pooled over ID and OOD trials, mean ± SEM.
Embodiment Dreaming. The predictive objective improves task progress score across all seven reported task–embodiment settings while adding no deployment-time computation.
Does latent geometry become more similar across embodiments?
We compute Gromov–Wasserstein (GW) distances between embodiment-specific distributions of shared-trunk latents extracted from Scoop demonstrations, and report the percentage change in GW for each embodiment pair. Negative values indicate greater geometric similarity.
mean GW across pairs, lower for 9 of 10 pairs. MXT+ with vs. without cross-embodiment pretraining.
mean GW across pairs, lower for all 10 pairs. MXT+ vs. MXT, both pretrained and fine-tuned.
Latent Geometry. Pretraining and embodiment dreaming both make the internal distance structure of embodiment-specific latents more similar for the same manipulation task.
Does modular design enhance cross-embodiment transfer?
We compare MXT+ with a coarse variant that merges modality-specific inputs and actions, each with and without cross-embodiment pretraining. The modular design consistently outperforms the coarse variant, with particularly pronounced gains on the higher-dimensional xArm7-Dex4 and G1-Sim-Dex3.
Normalized task progress score. Arrows go from w/o to w/ pretraining, with the change above each column; violins show each mean's uncertainty (from its SEM). Hover over a marker for its value.
Modular Design. After pretraining, the coarse variant scores zero on Kitchen-Cleanup, whose first step requires a 3D velocity command for locomotion, a small but critical modality within G1's 51-dimensional action space.
@article{egox2026,
title={EgoX: Egocentric Cross-Embodiment Manipulation with Embodiment Dreaming},
author={Niu, Yaru and Chen, Binghong and Yao, Changwei and Li, Shangtao and
Fang, Zhenlong and Senthilkumaran, Revanth Krishna and Zhou, Shuai and
Mao, Yuemin and Ching, Kalie and Chao, Po-Ju and Luo, Yudong and Yu, Peiqi and
Zhang, Heng and Zhang, Hao and Chen, Bingqing and Qiu, Chen and
Tseng, H. Eric and Liu, Changliu and Francis, Jonathan and Zhao, Ding},
year={2026}
}