EgoX
Egocentric Cross-Embodiment Manipulation
with Embodiment Dreaming
Yaru Niu1,*, Binghong Chen1,*, Changwei Yao1,*, Shangtao Li1,*, Zhenlong Fang4, Revanth Krishna Senthilkumaran1,2, Shuai Zhou1, Yuemin Mao1, Kalie Ching1,2, Po-Ju Chao1, Yudong Luo1, Peiqi Yu1, Heng Zhang1, Hao Zhang1,3, Bingqing Chen2, Chen Qiu2, H. Eric Tseng3, Changliu Liu1, Jonathan Francis1,2, Ding Zhao1 1Carnegie Mellon University 2Bosch Center for Artificial Intelligence 3The University of Texas at Arlington 4University of California, Berkeley *Equal contribution
EgoX framework spanning human and robot egocentric data, cross-embodiment pretraining, target-embodiment fine-tuning, and manipulation tasks

Overview of EgoX. EgoX treats human, real-robot, and simulated demonstrations as unified egocentric trajectories, pretrains one shared manipulation representation, and fine-tunes it on a new embodiment–task domain. Its model, MXT+, routes each embodiment's observations through modality tokenizers into a shared Transformer trunk and decodes actions with embodiment-specific action experts, while a training-only dreaming expert predicts future embodiment latents. Embodiment dreaming and cross-embodiment pretraining both improve the average task progress score and success rate.

Abstract
Scaling robot manipulation requires learning from demonstrations collected across humans and heterogeneous robots. These sources cover diverse objects, scenes, tasks, and interaction strategies, but differences in morphology, sensing, and control interfaces make it difficult to learn effectively from their combined data. We introduce EgoX, an egocentric cross-embodiment framework for robot manipulation. Its central design is to share manipulation structure at the level of semantically corresponding modalities, while preserving embodiment-specific sensing and control interfaces. EgoX stores human, real-robot, and simulated demonstrations as unified egocentric observation-action trajectories, pretrains a Modularized Cross-Embodiment Transformer with Embodiment Dreaming (MXT+) on the mixture, and fine-tunes it on target-embodiment demonstrations. MXT+ pairs embodiment-specific modality tokenizers and action experts with a shared Transformer trunk, and its training-only embodiment dreaming objective predicts future embodiment latents from egocentric visual and proprioceptive features, so that the trunk captures each embodiment's dynamics. Across 13 task–embodiment pairs spanning five tasks and six robot embodiments, including a simulated humanoid, cross-embodiment pretraining raises average normalized task progress from 0.63 to 0.80 over training from scratch (0.53 to 0.77 out of distribution). Broader pretraining mixtures strengthen transfer, and embodiment dreaming further raises task progress from 0.63 to 0.78 across seven settings (0.56 to 0.74 out of distribution) and reduces mean cross-embodiment latent-geometry distance by 38.52%.
Method
EgoX has three stages: (i) collect human and robot demonstrations as unified egocentric observation-action trajectories organized by corresponding modality types, (ii) pretrain MXT+ on mixed cross-embodiment data using action prediction and embodiment dreaming, and (iii) fine-tune the shared representation on target-embodiment tasks. Embodiment-specific modality tokenizers and action experts preserve heterogeneous sensing and control interfaces around a shared Transformer trunk. Dreaming experts predict future egocentric visual and proprioceptive latents during training and are discarded at deployment.
MXT+ architecture with embodiment-specific modality tokenizers, a shared Transformer trunk, action experts, and embodiment-dreaming experts, above the EgoX embodiments

MXT+ Architecture and EgoX Embodiments. Top: embodiment-specific modality tokenizers map each embodiment’s observations into tokens that a shared Transformer encoder-decoder trunk fuses across embodiments. Embodiment action experts decode the trunk output into embodiment-specific action chunks, while embodiment-dreaming experts predict future embodiment latents supervised by EMA copies of the corresponding modality tokenizers. Bottom: the embodiments used in EgoX. Each panel marks the unified egocentric frame ℱego and the end-effector-link keypoints.

Embodiments & Tasks
EgoX is evaluated on five manipulation tasks across six robot embodiments (five real robots and one simulated humanoid). Egocentric human demonstrations provide an additional pretraining embodiment.

Choose an embodiment

Select a model to filter videos. Use its expand icon for a larger 3D view.

Left Right
Human-Dex5

Human-Dex5

G1-Dex5

G1-Dex5

xArm7-Dex4

xArm7-Dex4

G1-Sim-Dex3

G1-Sim-Dex3

LocoMan-Dex2

LocoMan-Dex2

R1Lite-Gripper

R1Lite-Gripper

Z1-Tool

Z1-Tool

Explore in 3D

Drag to rotate · scroll or pinch to zoom

Policy rollouts

Across all tasks and embodiments, we show successful rollouts of MXT+ with pretraining, illustrating each task setup and what the method can do. On R1Lite-Gripper Kitchen-Bread, we also compare it with MXT+ without pretraining, π0.5, and ACT, marking each rollout as a success or failure. All videos are played at 1× speed.

G1-Dex5 | InsertID
MXT+ (w/ pretrain)
G1-Dex5 | InsertID
MXT+ (w/ pretrain)
G1-Dex5 | InsertID
MXT+ (w/ pretrain)
G1-Dex5 | InsertID
MXT+ (w/ pretrain)
G1-Dex5 | InsertID
MXT+ (w/ pretrain)
G1-Dex5 | InsertOOD
MXT+ (w/ pretrain)
G1-Dex5 | InsertOOD
MXT+ (w/ pretrain)
G1-Dex5 | InsertOOD
MXT+ (w/ pretrain)
G1-Dex5 | InsertOOD
MXT+ (w/ pretrain)
G1-Dex5 | SortID
MXT+ (w/ pretrain)
G1-Dex5 | SortID
MXT+ (w/ pretrain)
G1-Dex5 | SortID
MXT+ (w/ pretrain)
G1-Dex5 | SortOOD
MXT+ (w/ pretrain)
G1-Dex5 | ScoopID
MXT+ (w/ pretrain)
G1-Dex5 | ScoopID
MXT+ (w/ pretrain)
G1-Dex5 | ScoopOOD
MXT+ (w/ pretrain)
G1-Dex5 | ScoopOOD
MXT+ (w/ pretrain)
xArm7-Dex4 | ScoopID
MXT+ (w/ pretrain)
xArm7-Dex4 | ScoopID
MXT+ (w/ pretrain)
xArm7-Dex4 | ScoopOOD
MXT+ (w/ pretrain)
xArm7-Dex4 | ScoopOOD
MXT+ (w/ pretrain)
xArm7-Dex4 | ScoopOOD
MXT+ (w/ pretrain)
R1Lite-Gripper | SortID
MXT+ (w/ pretrain)
R1Lite-Gripper | SortID
MXT+ (w/ pretrain)
R1Lite-Gripper | SortOOD
MXT+ (w/ pretrain)
G1-Dex5 | Kitchen-BreadID
MXT+ (w/ pretrain)
G1-Dex5 | Kitchen-BreadID
MXT+ (w/ pretrain)
G1-Dex5 | Kitchen-BreadID
MXT+ (w/ pretrain)
G1-Dex5 | Kitchen-BreadOOD
MXT+ (w/ pretrain)
G1-Dex5 | Kitchen-BreadOOD
MXT+ (w/ pretrain)
xArm7-Dex4 | SortID
MXT+ (w/ pretrain)
xArm7-Dex4 | SortID
MXT+ (w/ pretrain)
xArm7-Dex4 | SortID
MXT+ (w/ pretrain)
xArm7-Dex4 | SortID
MXT+ (w/ pretrain)
xArm7-Dex4 | SortOOD
MXT+ (w/ pretrain)
xArm7-Dex4 | SortOOD
MXT+ (w/ pretrain)
xArm7-Dex4 | InsertID
MXT+ (w/ pretrain)
xArm7-Dex4 | InsertID
MXT+ (w/ pretrain)
xArm7-Dex4 | InsertID
MXT+ (w/ pretrain)
xArm7-Dex4 | InsertOOD
MXT+ (w/ pretrain)
xArm7-Dex4 | InsertOOD
MXT+ (w/ pretrain)
xArm7-Dex4 | InsertOOD
MXT+ (w/ pretrain)
R1Lite-Gripper | Kitchen-BreadID
MXT+ (w/ pretrain)
R1Lite-Gripper | Kitchen-BreadID
MXT+ (w/ pretrain)
R1Lite-Gripper | Kitchen-BreadID
MXT+ (w/ pretrain)
R1Lite-Gripper | Kitchen-BreadID
MXT+ (w/ pretrain)
R1Lite-Gripper | Kitchen-BreadID
MXT+ (w/ pretrain)
R1Lite-Gripper | Kitchen-BreadID
MXT+ (w/ pretrain)
R1Lite-Gripper | Kitchen-BreadOOD
MXT+ (w/ pretrain)
R1Lite-Gripper | Kitchen-BreadOOD
MXT+ (w/ pretrain)
R1Lite-Gripper | Kitchen-BreadOOD
MXT+ (w/ pretrain)
R1Lite-Gripper | Kitchen-BreadOOD
MXT+ (w/ pretrain)
R1Lite-Gripper | Kitchen-BreadOOD
MXT+ (w/ pretrain)
R1Lite-Gripper | Kitchen-BreadOOD
MXT+ (w/ pretrain)
R1Lite-Gripper | Kitchen-BreadID
MXT+ (w/o pretrain)
R1Lite-Gripper | Kitchen-BreadID
MXT+ (w/o pretrain)
R1Lite-Gripper | Kitchen-BreadID
MXT+ (w/o pretrain)
R1Lite-Gripper | Kitchen-BreadID
MXT+ (w/o pretrain)
R1Lite-Gripper | Kitchen-BreadID
MXT+ (w/o pretrain)
R1Lite-Gripper | Kitchen-BreadID
MXT+ (w/o pretrain)
R1Lite-Gripper | Kitchen-BreadOOD
MXT+ (w/o pretrain)
R1Lite-Gripper | Kitchen-BreadOOD
MXT+ (w/o pretrain)
R1Lite-Gripper | Kitchen-BreadOOD
MXT+ (w/o pretrain)
R1Lite-Gripper | Kitchen-BreadOOD
MXT+ (w/o pretrain)
R1Lite-Gripper | Kitchen-BreadOOD
MXT+ (w/o pretrain)
R1Lite-Gripper | Kitchen-BreadOOD
MXT+ (w/o pretrain)
R1Lite-Gripper | Kitchen-BreadID
π0.5
R1Lite-Gripper | Kitchen-BreadID
π0.5
R1Lite-Gripper | Kitchen-BreadID
π0.5
R1Lite-Gripper | Kitchen-BreadID
π0.5
R1Lite-Gripper | Kitchen-BreadID
π0.5
R1Lite-Gripper | Kitchen-BreadID
π0.5
R1Lite-Gripper | Kitchen-BreadOOD
π0.5
R1Lite-Gripper | Kitchen-BreadOOD
π0.5
R1Lite-Gripper | Kitchen-BreadOOD
π0.5
R1Lite-Gripper | Kitchen-BreadOOD
π0.5
R1Lite-Gripper | Kitchen-BreadOOD
π0.5
R1Lite-Gripper | Kitchen-BreadOOD
π0.5
R1Lite-Gripper | Kitchen-BreadID
ACT
R1Lite-Gripper | Kitchen-BreadID
ACT
R1Lite-Gripper | Kitchen-BreadID
ACT
R1Lite-Gripper | Kitchen-BreadID
ACT
R1Lite-Gripper | Kitchen-BreadID
ACT
R1Lite-Gripper | Kitchen-BreadID
ACT
R1Lite-Gripper | Kitchen-BreadOOD
ACT
R1Lite-Gripper | Kitchen-BreadOOD
ACT
R1Lite-Gripper | Kitchen-BreadOOD
ACT
R1Lite-Gripper | Kitchen-BreadOOD
ACT
R1Lite-Gripper | Kitchen-BreadOOD
ACT
R1Lite-Gripper | Kitchen-BreadOOD
ACT
Human-Dex5 | Kitchen-Bread
Human demonstration
Human-Dex5 | Kitchen-Bread
Human demonstration
Human-Dex5 | Kitchen-Bread
Human demonstration
Human-Dex5 | Kitchen-Bread
Human demonstration
Human-Dex5 | Kitchen-Bread
Human demonstration
Human-Dex5 | Sort
Human demonstration
Human-Dex5 | Sort
Human demonstration
Human-Dex5 | Insert
Human demonstration
Human-Dex5 | Insert
Human demonstration
Human-Dex5 | Scoop
Human demonstration
Human-Dex5 | Scoop
Human demonstration
LocoMan-Dex2 | ScoopID
MXT+ (w/ pretrain)
LocoMan-Dex2 | ScoopID
MXT+ (w/ pretrain)
LocoMan-Dex2 | ScoopID
MXT+ (w/ pretrain)
LocoMan-Dex2 | ScoopOOD
MXT+ (w/ pretrain)
LocoMan-Dex2 | ScoopOOD
MXT+ (w/ pretrain)
LocoMan-Dex2 | ScoopOOD
MXT+ (w/ pretrain)
G1-Sim-Dex3 | Kitchen-BreadID
MXT+ (w/ pretrain)
G1-Sim-Dex3 | Kitchen-BreadID
MXT+ (w/ pretrain)
G1-Sim-Dex3 | Kitchen-BreadID
MXT+ (w/ pretrain)
G1-Sim-Dex3 | Kitchen-BreadID
MXT+ (w/ pretrain)
G1-Sim-Dex3 | Kitchen-BreadOOD
MXT+ (w/ pretrain)
G1-Sim-Dex3 | Kitchen-BreadOOD
MXT+ (w/ pretrain)
G1-Sim-Dex3 | Kitchen-BreadOOD
MXT+ (w/ pretrain)
G1-Sim-Dex3 | Kitchen-CleanupID
MXT+ (w/ pretrain)
G1-Sim-Dex3 | Kitchen-CleanupID
MXT+ (w/ pretrain)
G1-Sim-Dex3 | Kitchen-CleanupID
MXT+ (w/ pretrain)
G1-Sim-Dex3 | Kitchen-CleanupID
MXT+ (w/ pretrain)
G1-Sim-Dex3 | Kitchen-CleanupOOD
MXT+ (w/ pretrain)
G1-Sim-Dex3 | Kitchen-CleanupOOD
MXT+ (w/ pretrain)
G1-Sim-Dex3 | Kitchen-CleanupOOD
MXT+ (w/ pretrain)
G1-Sim-Dex3 | Kitchen-CleanupOOD
MXT+ (w/ pretrain)
Z1-Tool | ScoopID
MXT+ (w/ pretrain)
Z1-Tool | ScoopID
MXT+ (w/ pretrain)
Z1-Tool | ScoopID
MXT+ (w/ pretrain)
Z1-Tool | ScoopID
MXT+ (w/ pretrain)
Z1-Tool | ScoopOOD
MXT+ (w/ pretrain)
Z1-Tool | ScoopOOD
MXT+ (w/ pretrain)
Z1-Tool | ScoopOOD
MXT+ (w/ pretrain)
Z1-Tool | ScoopOOD
MXT+ (w/ pretrain)
Z1-Tool | ScoopOOD
MXT+ (w/ pretrain)
Z1-Tool | ScoopOOD
MXT+ (w/ pretrain)
Results
We study five questions: (1) does egocentric cross-embodiment pretraining improve downstream manipulation over training from scratch, (2) how do the diversity and composition of pretraining embodiments affect performance, (3) does embodiment dreaming improve cross-embodiment transfer, (4) does latent geometry become more similar across embodiments, and (5) does modular design enhance cross-embodiment transfer? Evaluations cover ID and OOD settings using the normalized task progress score.
1

Does egocentric cross-embodiment pretraining outperform training from scratch?

We compare MXT+ with and without cross-embodiment pretraining against ACT and π0.5 on 13 task–embodiment pairs. Under ID evaluation, pretraining yields particularly large gains on G1-Dex5 Sort, LocoMan-Dex2 Scoop, and xArm7-Dex4 Scoop. The benefit becomes even more pronounced under OOD evaluation, with substantial improvements on G1-Dex5 Kitchen-Bread, G1-Dex5 Insert, and Z1-Tool Scoop.

Normalized task progress score, mean ± SEM. Real robots: 20 trials per pair (13 ID, 7 OOD). Simulation: 50 trials per pair (32 ID, 18 OOD). Hover over a bar for its value.

Main Results. Cross-embodiment-pretrained MXT+ generally improves task progress score across 13 task–embodiment pairs, with substantially larger gains under OOD evaluation.

2

Does pretraining benefit from embodiment diversity?

We vary the pretraining mixture: no pretraining, human-only, human plus other robots, and full cross-embodiment data, which also includes the target robot's demonstrations on other tasks. Every variant is fine-tuned on the same target-embodiment, target-task demonstrations.

Normalized task progress score for each pretraining mixture, mean ± SEM (shaded). Each mixture adds to the previous one. Hover over a line or a legend entry to trace one setting.

Pretraining Data Composition. Every pretraining mixture improves average OOD performance over no pretraining, and the gains grow as more embodiments are added. Full cross-embodiment pretraining performs best, improving the average task progress score by 0.12 in-distribution and 0.23 out-of-distribution, with OOD gains on all six task–embodiment pairs.

3

Does embodiment dreaming improve cross-embodiment learning?

Embodiment dreaming asks the shared trunk to predict future embodiment latents built from egocentric image features, robot joint configurations, and end-effector joint configurations. EMA copies of MXT+'s own modality encoders provide stable targets. The objective encourages embodiment-aware dynamics modeling during pretraining and fine-tuning; dreaming heads and EMA teachers are discarded at deployment.

We compare MXT+ with MXT, its counterpart without the dreaming objective, on seven task–embodiment settings across five embodiments. The gains hold across tasks and embodiments, and are particularly pronounced on G1-Dex5.

Normalized task progress score pooled over ID and OOD trials, mean ± SEM.

Embodiment Dreaming. The predictive objective improves task progress score across all seven reported task–embodiment settings while adding no deployment-time computation.

4

Does latent geometry become more similar across embodiments?

We compute Gromov–Wasserstein (GW) distances between embodiment-specific distributions of shared-trunk latents extracted from Scoop demonstrations, and report the percentage change in GW for each embodiment pair. Negative values indicate greater geometric similarity.

Pretraining · lower triangle −21.30%

mean GW across pairs, lower for 9 of 10 pairs. MXT+ with vs. without cross-embodiment pretraining.

Embodiment dreaming · upper triangle −38.52%

mean GW across pairs, lower for all 10 pairs. MXT+ vs. MXT, both pretrained and fine-tuned.

Latent Geometry. Pretraining and embodiment dreaming both make the internal distance structure of embodiment-specific latents more similar for the same manipulation task.

5

Does modular design enhance cross-embodiment transfer?

We compare MXT+ with a coarse variant that merges modality-specific inputs and actions, each with and without cross-embodiment pretraining. The modular design consistently outperforms the coarse variant, with particularly pronounced gains on the higher-dimensional xArm7-Dex4 and G1-Sim-Dex3.

Normalized task progress score. Arrows go from w/o to w/ pretraining, with the change above each column; violins show each mean's uncertainty (from its SEM). Hover over a marker for its value.

Modular Design. After pretraining, the coarse variant scores zero on Kitchen-Cleanup, whose first step requires a 3D velocity command for locomotion, a small but critical modality within G1's 51-dimensional action space.

BibTeX
@article{egox2026,
  title={EgoX: Egocentric Cross-Embodiment Manipulation with Embodiment Dreaming},
  author={Niu, Yaru and Chen, Binghong and Yao, Changwei and Li, Shangtao and
          Fang, Zhenlong and Senthilkumaran, Revanth Krishna and Zhou, Shuai and
          Mao, Yuemin and Ching, Kalie and Chao, Po-Ju and Luo, Yudong and Yu, Peiqi and
          Zhang, Heng and Zhang, Hao and Chen, Bingqing and Qiu, Chen and
          Tseng, H. Eric and Liu, Changliu and Francis, Jonathan and Zhao, Ding},
  year={2026}
}