Large-Scale Physical AI Data Collection: Egocentric, Stereo, MONO and UMI
Field Motion collects physical AI data at large scale, including egocentric video, stereo capture, MONO (monocular) video and UMI demonstrations. These are not four interchangeable datasets: viewpoint, camera configuration and action representation each solve a different part of the robot-learning problem.
For robotics teams choosing a collection partner, the useful question is not simply “How many hours can you record?” It is “Which observations and actions will those hours contain, and can our training pipeline use them?” This guide explains the formats and how to scope a collection program without confusing human activity footage with robot-action supervision.
What is egocentric data collection?
Egocentric data collection records activity from the participant's perspective. A head-mounted or body-mounted camera can show what a person sees while reaching, grasping, sorting or using tools. The viewpoint reveals hand–object interaction and task context that a distant camera may miss.
Egocentric describes the viewpoint, not the number of cameras. An egocentric rig can be monocular or stereo. A first-person video can support visual representation learning, action understanding and human-demonstration pretraining, but pixels alone do not provide a robot's joint commands, forces or calibrated end-effector actions.
MONO versus stereo: what changes?
| Format | What it supplies | What to validate |
|---|---|---|
| Egocentric | First-person task observations; can use one or two cameras. | Hand visibility, occlusion, viewpoint consistency and task coverage. |
| MONO / monocular | One camera stream; a simpler recording configuration. | Motion blur, exposure and whether downstream learning requires metric depth. |
| Stereo | Two views with a known camera relationship, enabling geometric depth estimation under suitable conditions. | Calibration, baseline, synchronized exposure and difficult reflective or textureless surfaces. |
| UMI | A handheld manipulation interface for demonstrations with a compatible action representation. | Trajectory tracking, gripper signals, coordinate frames and robot compatibility. |
A MONO program may be appropriate when visual diversity and broad task understanding are priorities. Stereo can add geometric information when the intended pipeline can consume it. Neither choice is universally superior: extra cameras also create extra calibration, storage and quality-control obligations. Derived depth must be labeled as an estimate, not treated as measured ground truth.
What is UMI, and how is it different from ordinary video?
UMI stands for Universal Manipulation Interface. The original research introduces handheld grippers and a collection-to-policy system designed to transfer human demonstrations to robot manipulation. Its design includes relative-trajectory action representations and inference-time latency matching. See the original UMI project for the hardware and method.
UMI is not shorthand for any video of a person holding an object. A usable UMI collection needs the appropriate equipment, tracking outputs, gripper information and processing workflow. Transfer still depends on the robot, task, sensing and policy. General first-person footage and UMI demonstrations can complement one another, but should not be pooled as though they carry identical action labels.
What large-scale collection needs beyond more recording hours
- A coverage plan. Specify task families, object variations, environments, participants and success criteria before recording begins.
- A capture specification. Agree on viewpoints, resolution, frame rate, timestamps, calibration records and any trajectory or gripper streams.
- Episode-level labels. Define task instructions, start/end boundaries, outcomes and how interruptions or failed attempts are represented.
- Quality gates. Check hand visibility, dropped frames, synchronization, tracking failures and duplicate episodes; report accepted data separately from raw footage.
- Responsible collection. Establish participant consent, permitted training uses, location permissions and handling of bystanders or identifying information.
- A training integration pilot. Load a small batch into the intended learner before committing to production volume.
Diversity should be designed, not inferred from total hours. Repeating one task in one room can produce a large archive with narrow coverage. The DROID project is a useful reference for real-world robot data collection across varied environments, rather than a guarantee that any similarly sized dataset will generalize.
How to scope collection with Field Motion
Field Motion's work spans large-scale collection, annotation and development for physical AI. For egocentric, stereo, MONO or UMI projects, start with the intended model, robot embodiment, tasks, environments and required output signals. Capture formats, volume, acceptance checks and delivery requirements should be agreed for the specific project rather than assumed from a general format name.
Send a collection brief to info@fieldmotion.ai, or review our data modalities. For the broader training picture, read physical AI training-data requirements and the recent shifts in robotics datasets.
Frequently asked questions
Can egocentric video be both MONO and stereo?
Yes. Egocentric is a viewpoint; MONO and stereo describe camera configurations. Choose the combination based on the observations the model needs.
Does UMI replace robot demonstration data?
Not universally. UMI provides a structured human-demonstration pathway for compatible manipulation tasks. Robot-specific adaptation and evaluation remain necessary.
Does Field Motion offer large-scale collection?
Yes. Field Motion performs large-scale physical AI data collection, including egocentric, stereo, MONO and UMI formats. Contact the team to scope the requirements of your project.
Sources
- Universal Manipulation Interface — original project and collection workflow (2024).
- Universal Manipulation Interface — research paper (2024).
- DROID — Distributed Robot Interaction Dataset (2024).
Published by Field Motion. Research context reviewed October 24, 2026. This is a collection-design guide, not a claim of affiliation with the cited projects.