Free guide · Robot training data How robot training data works
The types of robot training data, who buys it, what it costs, and how to collect it well. Written by a founder who spent two years building a robotics data company.
By Sam Padilla · Published
- Robot training data comes in four main forms: teleoperation, egocentric video, wearable capture, and simulation.
- Only a handful of companies buy it at scale, and the people who approve those purchases number in the low hundreds worldwide.
- Eidon AI sold video with arm tracking at about $60 per hour, before hardware, logistics, and QA costs.
- Buyers want a steady supply, not a one-off dataset, so the hard part is running a pipeline.
- Prove your data works with a controlled comparison before you scale it.
What is robot training data?
Robot training data is the recorded experience a robot learning model trains on. Usually that means video of a task being done, paired with signals that describe how the body moved: joint angles, hand poses, gripper commands, forces. From many examples, a model learns which actions lead to which results, and then produces actions of its own.
This guide comes from direct experience. Sam Padilla co-founded Eidon AI, which collected 1,274 hours of egocentric household video with arm tracking and released it as an open dataset on Hugging Face that has since passed 83K downloads. Eidon shut down after more than two years in the data space. What follows is what its team learned, plus what Sam heard from others in the field afterward.
What types of robot training data are there?
Four types cover most of what labs buy and use. They differ in cost, in how closely they match a real robot, and in how easily they scale.
Teleoperation
A person controls a real robot through a leader arm, a VR rig, or a similar interface, while the system records what the robot sees and does. The data matches the target robot exactly, because that robot produced it. It is also the slowest and most expensive type to collect, since every hour needs a robot, an operator, and a setup. It suits fine-tuning a policy on one specific robot and task.
Egocentric video
People wear a camera and do ordinary tasks. The footage is cheap to gather and easy to scale, so models can pick up broad visual and manipulation knowledge from it. Video alone does not record forces, joint angles, or exactly how a hand moved. After two years in the market, Sam’s view is that video-only egocentric data is a commodity unless you can deliver it at very large scale.
Motion capture and gloves
Wearable sensors record the body alongside the video. Eidon’s tracker used IMUs to capture seven degrees of freedom of arm motion, and its glove tracked 19 degrees of freedom across the fingers and wrist. Hand and finger data is worth more because dexterity is where much of robotics breaks. It is also much harder to manufacture, calibrate, and support once it is in the field.
Simulation and synthetic data
Simulators generate new demonstrations from a small set of real ones, replaying and varying them across new object positions and tasks. NVIDIA’s GR00T work produced about 6,500 human-hours of equivalent data this way, and NVIDIA reported that the mixed corpus improved GR00T N1 by 40% over real data alone. It turns a logistics problem into a compute problem. Whether the gains hold from simulation to a real robot is still debated.
Serious programs usually mix types, because each one adds value at a different scale. The T-Rex project, from researchers at Berkeley, NVIDIA, and Stanford, combined about 23,000 hours of human demonstrations with only 100 hours of richer, tactile-synchronized robot data. The video taught broad manipulation, and the small tactile set tied that knowledge to contact and force.
Who buys robot training data?
Very few companies. Robot builders and robotics model labs buy it to train their systems, and inside each one only a few people can approve a purchase that runs to hundreds of thousands of dollars. Sam’s estimate is that the buyer network is a couple of hundred people worldwide. At that size, marketing does little. You either have the relationship or you do not.
Because access is so hard, brokers and aggregators sit between sellers and labs. Eidon worked with Defined AI, which already had relationships with large labs and brokered deals with them. That worked, and it came with the usual costs of an intermediary: slower communication, filtered requirements, thinner margins, and less direct feedback from the people using the data.
Buyers also hold the leverage. The supplier market is crowded, so a lab that does not like your volume or specs can go to another of the dozen suppliers waiting. Sam’s advice to sellers is to avoid exclusivity deals.
How much does robot training data cost?
Eidon sold 1,000-hour checkpoints of video-plus-tracker data at about $60 per hour. Data with richer signals, such as finger-level dexterity, could reach $100 per hour. That looks fine on paper, but the sticker price overstates what the seller keeps. These costs come out of every hour:
- Collector pay, roughly 10% of the final hour price
- Hardware bill of materials
- Loss, breakage, and replacement
- Customs
- Assembly time
- QA and rejected recordings
The seller usually pays all of it up front, before any buyer has committed. Three things make the unit economics harder still.
- Requirements differ by buyer. One lab wants final joint angles from your kinematic chain, and another wants raw IMU measurements. If you collect one and do not preserve the other, you cannot recover it later.
- Data goes stale. Once a customer trains on a batch, that batch is of little use to them. Labs want a recurring pipeline and often will not treat you as a serious supplier until you can show one.
- Buyers want volume. Eidon reached 3,000 hours in about five months from the day it shipped its first trackers. The labs it spoke to asked for about 3,000 hours every month, preferably every week.
How do you design a robot data collection program?
Treat it as an operations problem first. The sensors matter, but programs mostly succeed or fail on logistics.
Start from the buyer’s data shape
Decide what the data must contain before you build anything: which sensors, which tasks, which robot body. Then get samples in front of prospective buyers early. A sample that reaches a lab’s engineers before you ship hardware is cheap insurance against building the wrong thing.
Choose rigs you can support
Pick hardware you can manufacture, calibrate, and repair at the volume you plan to reach. Easy rigs are easy for competitors to match. Harder ones create a moat, but they are harder to manufacture, calibrate, support, and deploy. Sam’s main regret is that Eidon chose its simpler tracker over the glove, which was harder to build and produced more valuable data. Even the tracker cost thousands of dollars and months to deploy at scale, so the easy choice was not a cheap one.
Recruit and manage contributors
Collector pay is a small share of the hour price, but managing collectors is most of the work: recruiting, communicating, paying, and moderating them. Eidon deployed its tracker to more than 50 collectors across several countries. It also used a third-party logistics company, BeeWork, which received kits and coordinated collectors against Eidon’s standards. That helped, but it raised the per-hour cost and did not remove the edge cases that only appear once hardware is in the field.
Put QA gates on every recording
Write the standard down and check every recording against it. For first-person video the checks are visible hands, consistent lighting, clean point-of-view framing, task compliance, and enough duration to be usable. With hardware, add a calibration check. Catch problems at the gate and tell the collector what went wrong, because every rejected recording is a cost you have already paid for.
Deliver in the format buyers train on
Ask what format the training team uses. LeRobot, the Hugging Face dataset format, and RLDS, a Google format built around episodes made of steps, are two common ones for robot learning. Ship raw sensor streams alongside any derived values, such as raw IMU readings next to joint angles, and include calibration data, task labels, and device identifiers.
How do you prove robot training data improves a model?
Buyers will ask, and a controlled comparison is the answer.
- Fix a baseline policy and a task set, with held-out object positions, environments, and lighting.
- Train or fine-tune with and without your data, using the same recipe both times.
- Compare success rates on the held-out set, with enough trials to see past the noise.
The GR00T result above is framed this way, as a mixed corpus against real data alone. Test under change, too. Published task completion rates often drop by several percentage points when the object configuration, environment, or lighting shifts slightly, and a dataset that only helps in the conditions it was collected in is worth less to a buyer.
If you cannot run training yourself, give buyers a sample and make it easy to test. Sam’s advice is to socialize samples fast, so engineers can judge whether the data is useful before you commit to scale.
Should you build or buy robot training data?
Buy while you are still learning what data you need. A purchased dataset lets you test a hypothesis in weeks, without running a collection operation. Build once you know the data shape, need it to match your own robot, and need it on a steady schedule.
The pull toward building is strong, because data is more valuable the closer it matches the robot that will use it, including the sensor stack, degrees of freedom, and tasks. Full-stack robotics companies such as Mimic Robotics, Sunday Robotics, and Genesis AI have each announced data collection gloves built for their own robots. Teams that start by buying often end up building.
If you build, everything in the sections above lands on your team: collector operations, hardware logistics, and QA. If you buy, ask the seller for a sample from the same pipeline, how every recording is checked, which raw signals are kept, which formats they deliver, and how many hours a month they can supply.
Where is the robot data market going?
Sam puts the window for new third-party collection companies at one to three years. The goal of every robotics lab is a data flywheel, where deployed robots collect their own data and feed it back into the model. Once labs can meet their own needs, through internal teams or fleet data, the need for outside collectors falls away. Third-party collectors are like the safety drivers in a Waymo before it is allowed to operate on its own: they matter, for a limited time.
In Bet Hard or Get Out, Sam argues that sellers still have three bets with decent odds: scale, which means hundreds of thousands of hours a month; specificity, which means more information per hour, such as tactile and force data; and synthetic data. Mid-sized datasets that are neither unusual nor huge sit in an awkward middle with no clear path.
Decoupled Robotics looks at the buyer side. Robot companies feel pressure to build every layer in-house, partly because specialists are not yet reliable enough to depend on, and specialists cannot get reliable without customers. Sam sees data collection as one of the clearest places where plain commercial agreements can work, and wants the field to decouple. Data is also only one bottleneck, and evaluation and generalization get far less attention.
Sources and further reading
- Robotics Data is a Broken Business on X. What two years of building a robotics data company taught Sam.
- Robotics Data: Bet Hard or Get Out on X. Where data companies can still win.
- Decoupled Robotics on X. Why robot companies keep building everything themselves.
- Eidon tracker-pov dataset on Hugging Face, with 83K downloads.
- The Robotics Map, a map of more than 275 robotics companies that Sam built with Varun Nair.
Questions about robot training data
What is egocentric video data for robotics?
It is first-person footage from a camera a person wears, usually on the head or chest, while they do everyday tasks like loading a dishwasher or folding laundry. Models learn how hands and objects interact from it. Video alone carries no force or joint data, so buyers pay more for video with synchronized tracking.
How much robot training data do you need?
There is no single number. It depends on the model and the task. For reference, the T-Rex project trained on about 23,000 hours of human demonstrations plus 100 hours of tactile-synchronized robot data. Labs that Eidon AI talked to asked for around 3,000 hours of new data every month.
How much does robot training data cost?
Eidon AI sold 1,000-hour checkpoints of video-plus-tracker data at about $60 per hour, and richer data with finger-level tracking could reach $100 per hour. Hardware, logistics, and QA costs come out of that price, and collector pay is roughly 10% of it.
Where can I buy robot training data?
You can buy directly from collection companies or through brokers and aggregators that already have relationships with large labs. Open datasets, like Eidon’s tracker-pov set on Hugging Face, are a free place to start. The Robotics Map lists 275+ companies, and the By Crux network at network.bycrux.com introduces buyers to collectors.
Can synthetic data replace real robot data?
So far it extends real data rather than replacing it. Synthetic pipelines start from a small set of real demonstrations and vary them across new objects and positions. NVIDIA reported that a mixed corpus improved GR00T N1 by 40% over real data alone. Whether those gains carry over to a real robot is still debated.
What format should robot training data come in?
Whatever your model trains on. LeRobot and RLDS are two common formats for robot learning data. Ask the buyer or the training team first, and keep the raw sensor streams next to any derived values, because you cannot rebuild signals you did not record.
Need help with a robotics data program?
We advise robotics and physical AI teams on data strategy, capture hardware, collection operations, quality assurance, and evaluation. If you are weighing a build or a buy, we can talk it through.