Robotics

August 5, 2026·Bryan Lai

TechnicalMLRobotics

Physical AI and Robotics Field Guide

How to read this guide

Physical AI here means AI that perceives and acts through a physical body inside a feedback loop. It is a useful umbrella for robot learning and embodied AI, not a separate replacement for robotics, control, or mechanical engineering.

Start with the complete mental map below, then read sections 0 through 4 in order:

1. Section 0 gives the whole map.
2. Section 1 explains a robot as a physical feedback system.
3. Section 2 introduces geometry, dynamics, control, estimation, planning, and learning.
4. Section 3 explains the body those methods control and how land, air, water, and space change the system.
5. Section 4 connects the body to software and fleet infrastructure.

Read sections 5 through 9 before making model, data, simulation, evaluation, or safety decisions. Sections 10 onward map deployments, organizations, economics, timelines, hype checks, and the career path.

Evidence labels:

• Verified: a primary source establishes the narrow claim.
• Vendor reported: the company or customer publishes the number, but no independent audit is public.
• Directional: several credible sources point the same way, but the exact magnitude is uncertain.
• Unproven: public evidence does not establish the claim.

Frontmatter grades classify source type (A: primary/standard, B: vendor technical, C: independent report, D: founder claim/demo). Evidence labels classify how strongly available sources establish a specific claim, so the two systems do not map one-to-one. For example, a grade B product page verifies what a vendor states but leaves an uncorroborated performance claim Vendor reported.

Flowchart color key: slate = goal or human input; green = learned intelligence or passed gate; indigo = planning or orchestration; blue = control or software; orange = data; amber = hardware or the physical world; purple = safety; yellow = decision or provisional status; red = blocked, failed, or unproven.

Visual First

Complete mental map

This graph is the field guide compressed into one view. Follow arrows from left to right. Vertically stacked boxes are parallel concepts, not a sequence. The detailed sections define, test, and source each node. The long arrows between groups show reading order, not a serial build plan. Scan unfamiliar acronyms as labels first; sections 1 through 9 and Appendix B define them.

0. Answer first

The shortest accurate field map

Robotics is accelerating, but it has not reached a ChatGPT-like product moment.

• What is already real: industrial arms, autonomous mobile robots, surgical robots, drones, robotaxis in bounded operating domains, learned locomotion, task-specific imitation learning, and early paid humanoid workflows.
• What is improving quickly: vision-language-action models, diffusion and flow policies, cross-embodiment pretraining, synthetic data, tactile sensing, whole-body control, and low-cost robot hardware.
• What remains hard: dexterous contact, long-horizon reliability, recovery, calibration, battery and thermal limits, application integration, safety evidence, and economics outside a narrow workflow.
• What the demos hide: resets, human interventions, teleoperation, cherry-picked runs, restricted objects, fixed lighting, maintenance, integration work, and the denominator behind "success."
• Where near-term value accrues: narrow automation plus the infrastructure that makes it dependable: integration, evaluation, observability, teleoperation escalation, safety workflows, data quality, and field service.
• Best career entry for a strong software engineer: robotics software, simulation and evaluation, robot data infrastructure, perception, fleet reliability, or deployment engineering.
• Weakest risk-adjusted startup entry: another capital-intensive general humanoid manufacturer with no unique customer workflow, manufacturing edge, or deployment data.

One diagram to hold in your head

An action chunk is a short sequence of commands predicted together. A state estimator combines noisy measurements into the robot's best current state estimate. World state here means that current estimate, not a learned world model that predicts possible futures. Mechanical compliance means yielding under contact instead of behaving like a perfectly rigid machine.

The arrows show responsibility and information flow, not one serial clock. The learned policy, state estimator, controller, and safety functions keep running between slower reasoner updates.

A vision-language-action model, or VLA, occupies one part of this stack. It does not replace the robot body, state estimation, motion planning, low-level control, independent safety functions, or operations.

The central distinction

A demo shows that a policy can complete a task at least once. A deployment shows that the whole system can repeat useful work safely, economically, and with an acceptable amount of human rescue.

Seven durable rules

1. Judge the workflow, not the checkpoint. A better model can still lose to bad tooling, a weak gripper, poor calibration, or a slow reset.
2. Use the cheapest embodiment that solves the job. A fixed arm, conveyor, gantry, or wheeled base often beats a biped.
3. Measure interventions per operating hour. A polished success video has no denominator.
4. Simulation is an accelerator, not an exemption from reality. Contact, wear, latency, and people expose the residual gap.
5. Data quality means context and outcomes, not raw hours. Calibration, synchronization, failures, and recovery labels matter.
6. Keep learned behavior behind an independent safety boundary. A model's confidence is not a certified stop function.
7. Orders, funding, capacity, and manufactured units are not productive deployment. Ask what is delivered, operating, paid for, and measured.

1. First principles: what a robot is

1.1 A robot closes a loop

A web service can return an error without changing the room. A robot can drop a part, hit a fixture, overheat a motor, or injure a person. The system therefore cannot only "predict an action." It must repeatedly:

1. Observe the world and itself.
2. Estimate what state it is actually in.
3. Choose a goal or action.
4. Check whether the action is feasible and safe.
5. Execute through motors and mechanisms.
6. Measure what happened.
7. Correct, recover, stop, or ask for help.

This is a closed-loop system. Open-loop behavior executes a command without using the result to correct itself. Open-loop control is cheap, but it fails when friction, payload, timing, object pose, or hardware condition differs from the assumption.

1.2 The state is never directly known

The robot does not receive "the true world." It receives noisy measurements:

• Camera pixels depend on lighting, exposure, occlusion, and calibration.
• Encoders, the angle sensors on robot joints or motors, measure position but can miss flex, backlash, or external contact.
• Force sensors drift and saturate.
• Maps become stale.
• Network messages arrive late or out of order.
• Objects move while the robot is thinking.

The robot therefore maintains an estimate of hidden state. State estimation is not a side feature. It is the foundation on which planning and control depend.

1.3 Physical systems have hard constraints

Software can often retry. Physical systems pay for every attempt:

• Mass creates inertia.
• Torque and current create heat.
• Friction changes with material, dust, temperature, and wear.
• Batteries lose voltage and capacity.
• Gearboxes have backlash and finite life.
• Cables snag.
• Cameras lose line of sight.
• A collision can permanently change calibration.

The important engineering question is not "can the model output this action?" It is:

Can this body execute the action within its force, speed, reach, thermal, power, collision, and safety limits?

1.4 Robotics runs at several time scales

The layers cannot all run at cloud-model speed.

Layer	Typical order of response	Main job	Common location
Motor drive and electrical protection	microseconds to milliseconds	Current, commutation, overcurrent, encoder processing	Motor drive or embedded controller
Joint and whole-body control	milliseconds	Track motion, balance, regulate force and compliance	Real-time computer on robot
Perception and learned action policy	tens to hundreds of milliseconds	Understand observations and produce short action chunks	Onboard GPU or nearby edge computer
Motion and task planning	hundreds of milliseconds to seconds	Find paths, choose skills, sequence subgoals	Onboard or edge computer
Fleet learning and analytics	minutes to days	Aggregate failures, retrain, evaluate, roll out	Data center or cloud

These are orders of magnitude, not universal specifications. Figure's Helix architecture is one public example of a slower semantic layer paired with a faster visuomotor layer. Safety and balance cannot wait for a remote model round trip.

1.5 Software-to-robotics mapping

Software concept	Robotics analogue	Important difference
Request	Task command	The environment changes while the request runs
Database state	World and robot state estimate	State is partial, noisy, and time-sensitive
API contract	Coordinate frame, message, timing, and units contract	A meter/radian or frame error can damage hardware
Timeout	Missed control deadline	Late can be as wrong as incorrect
Retry	Regrasp, replan, reset, or ask for help	Repeated attempts can worsen the physical scene
Rollback	Safe stop, known pose, prior policy, or human takeover	Physical state cannot always be undone
Incident	Collision, drop, unsafe motion, deadlock, or degraded throughput	Can involve injury and equipment damage
Observability	Time-synchronized sensor, state, action, fault, and video logs	Logs must preserve physical timing and calibration
SRE	Fleet reliability and field operations	Includes maintenance, spare parts, and site procedures

2. The prerequisite technical foundations

This section assumes basic linear algebra: vectors, matrices, coordinate systems, and matrix multiplication, plus basic trigonometry and derivatives. If those are unfamiliar, use Modern Robotics chapters 2 and 3 before continuing.

2.1 Coordinate frames and rigid transforms

Robotics constantly asks: where is object A relative to robot B at time T?

A coordinate frame is an origin plus three axes. A rigid transform contains:

• Translation: where one frame's origin is.
• Rotation: how its axes are oriented.

The mathematical home is SE(3), the special Euclidean group for 3D rigid-body poses and the rules for composing their rotation and translation. ROS 2 systems commonly use TF2 to maintain a time-indexed tree of frames.

Typical frames include:

If an object pose is expressed in the camera frame but interpreted in the robot base frame, the numbers can look valid while the motion is catastrophically wrong.

2.2 Kinematics

Forward kinematics asks:

Given joint positions, where is the hand?

Inverse kinematics asks:

Given a desired hand pose, what joint positions can produce it?

Inverse kinematics can have:

• No solution because the target is out of reach.
• One solution near a singular configuration.
• Several solutions with different collision or joint-limit consequences.

A Jacobian maps small joint changes to hand motion and relates joint torques to end-effector forces. Near a singularity, the arm cannot move the hand freely in some direction without very large joint speeds.

2.3 Dynamics

Kinematics describes geometry. Dynamics describes forces and motion.

A simplified robot equation is:

inertia * acceleration
+ velocity-dependent effects
+ gravity
+ friction
= commanded torque
+ external contact force

You need dynamics when payload, speed, balance, contact, or energy matters. A path that is geometrically valid can still be dynamically impossible.

2.4 Control

A controller turns a desired state into commands that reduce error.

The error can be the difference between desired and measured position, velocity, force, or another controlled quantity over time.

• PID, proportional-integral-derivative control: simple feedback using present, accumulated, and changing error.
• LQR, linear-quadratic regulation: optimizes a quadratic model around an operating point.
• Model predictive control (MPC): repeatedly solves a finite-horizon constrained optimization problem.
• Impedance control: makes the robot behave like a virtual spring and damper, useful for safe contact.
• Force control: regulates contact force rather than only position.
• Whole-body control: coordinates many joints while honoring balance, contact, and task priorities.

The core lesson:

Position control says where to go. Contact-rich work also needs to control how hard the robot pushes and how it yields.

2.5 State estimation

State estimation combines an imperfect process model with noisy sensors.

• Kalman filters work well for approximately linear Gaussian systems.
• Extended and unscented Kalman filters handle nonlinear models approximately.
• Particle filters represent multiple hypotheses.
• Factor graphs optimize many measurements and constraints together.
• Simultaneous localization and mapping (SLAM) estimates a map and the robot's location in it.

Calibration belongs here:

• Camera intrinsics: how pixels map to rays.
• Camera extrinsics: the fixed coordinate transform between the camera and robot.
• Hand-eye calibration: relation between camera and end effector or base.
• Time calibration: whether measurements refer to the same physical moment.

2.6 Planning

Planning operates at several levels:

• Task planning: choose the sequence of symbolic actions.
• Motion planning: find a collision-free path through configuration space, the set of all joint-angle combinations the robot can occupy.
• Trajectory optimization: find a smooth, dynamically feasible time-parameterized motion.
• Grasp planning: choose contact geometry and approach.
• Task-and-motion planning: connect symbolic steps with physical feasibility.
• Behavior trees or state machines: execute, monitor, retry, and recover.

MoveIt 2 provides manipulation planning and collision tooling on ROS 2. Nav2 provides navigation, localization, planning, control, and behavior-tree orchestration for mobile robots. Neither replaces low-level motor control or application safety.

2.7 Learning

Learning is useful when explicit models or rules are too hard to write:

• Perception under visual variation.
• Grasping deformable or diverse objects.
• Mapping language to tasks.
• Learning recovery from experience.
• Producing multimodal action sequences.

Learning is weakest where the system requires hard guarantees outside its training distribution. The practical architecture therefore mixes learned and engineered components.

3. Hardware: the body that makes the model real

3.1 The hardware stack

3.2 Actuators and transmissions

An actuator converts energy into motion.

• Electric motors dominate modern mobile and collaborative robots.
• Hydraulics offer high force density but add pumps, fluid, noise, maintenance, and leak risk.
• Pneumatics are cheap and compliant but difficult to control precisely.

Motors are often paired with a transmission:

• Harmonic drives: compact, high reduction, precise, but can be costly and wear-sensitive.
• Cycloidal drives: robust and high torque.
• Planetary gears: common, efficient, and compact.
• Belts and cables: light and remote the motor, but introduce compliance and maintenance.
• Tendons: useful in hands, but hard to route, tension, and service.
• Direct and quasi-direct drive: low backlash and good force control, but demand larger motors and more current.

Critical specifications:

• Continuous torque, not only peak torque.
• Speed-torque curve.
• Backlash and compliance.
• Thermal duty cycle.
• Efficiency.
• Brake behavior when power is lost.
• Service life and replacement procedure.

For humanoids, the actuator is also a supply-chain and cost problem. A 2026 McKinsey supply-chain analysis estimates that actuators account for 40% to 60% of a humanoid bill of materials and identifies several concentrated inputs:

• Planetary roller screws for high-load linear joints have a narrow supplier base, long lead times, and limited substitutes.
• Harmonic and cycloidal reducers trade compactness, shock tolerance, life, and cost differently.
• High-torque motors depend on neodymium-iron-boron permanent magnets; the analysis estimates China controls about 69% of rare-earth mining and 90% of magnet processing and refining.
• Torque sensors and high-durability linear guides can become qualification and production bottlenecks.

These are directional industry estimates, not a universal bill of materials. The exact share changes with robot size, joint architecture, payload, and production volume. A software plan can still fail after simulation if the required actuator cannot be sourced, cooled, repaired, or produced consistently.

3.3 Sensors

Sensor	What it measures	What commonly goes wrong
Joint encoder	Joint angle or motor rotation	Offset, wraparound, lost index, gearbox flex
Current sensor	Motor current, a proxy for torque	Temperature dependence and drivetrain friction
Inertial measurement unit (IMU)	Angular velocity and acceleration	Bias, vibration, drift
RGB camera	Appearance	Lighting, blur, occlusion, privacy
Depth camera or lidar	Geometry and range	Reflective, transparent, dark, or outdoor surfaces
Force-torque sensor	Wrench at wrist or joint	Drift, overload, coordinate-frame mistakes
Tactile sensor	Local contact and pressure	Durability, wiring, calibration, coverage
Microphone	Sound and speech	Noise, privacy, weak spatial grounding

Vision tells the robot what contact might happen. Touch and force tell it what contact is happening.

3.4 End effectors

The end effector often determines whether a task is economically possible.

• Parallel-jaw gripper: robust and easy to control.
• Suction: excellent on suitable surfaces, poor on porous or irregular objects.
• Tool changer: lets one arm use several specialized tools.
• Dexterous hand: flexible, but expensive in sensing, control, maintenance, and data.
• Application fixture: changes the environment so a simpler gripper succeeds.

Open research hands have lowered the entry cost without removing the maintenance problem:

Open hand	Public hardware boundary	What it is useful for
LEAP Hand	Four fingers with 16 actuated joints; the paper reports a $1,000 build cost	Direct-drive dexterous-learning experiments
ORCA Hand	17 degrees of freedom, tendon drive, integrated fingertip touch sensing, and material cost below CHF 2,000	Anthropomorphic manipulation and long-run hardware experiments
Yale OpenHand	Open family of underactuated hand designs	Grasping research where simpler mechanics are valuable

The ORCA paper reports about 2,000 uninterrupted grasp cycles over 7 hours 17 minutes, but also reports tactile-skin and wiring degradation after several thousand cycles. Low purchase cost does not mean zero service cost.

Do not ask "how human-like is the hand?" first. Ask:

1. What object families must it handle?
2. What pose and force accuracy are required?
3. Can the environment or object presentation be redesigned?
4. What failure can the gripper detect?
5. How quickly can it be replaced?

3.5 Power, heat, and endurance

Battery capacity is only one limit. Endurance also depends on:

• Robot mass.
• Joint efficiency.
• Duty cycle.
• Compute power.
• Cooling.
• Payload and walking.
• Idle behavior.
• Charging or battery-swap process.

More compute and torque create heat. Thermal limits can reduce performance long before the battery is empty.

Measure endurance under the intended payload, ambient temperature, motion profile, and cooling configuration. Report productive minutes, derating, cooldown, and interventions per operating hour. A no-load demo runtime does not establish a useful duty cycle.

3.6 Embodiment choice

Humanoid shape is valuable when the environment cannot be changed and is already designed around human reach, tools, aisles, and stairs. It is not automatically the best robot shape.

3.7 Hardware bottlenecks that software people underestimate

1. Cable routing and connector failures.
2. Calibration drift after impacts or maintenance.
3. Gripper wear and contamination.
4. Thermal derating.
5. Battery and charging logistics.
6. Spare parts and service time.
7. Tolerances across nominally identical robots.
8. Sensor occlusion and vibration.
9. Safe brakes and power-loss behavior.
10. Resetting the scene after failure.

3.8 Operating environments: land, air, surface water, underwater, and space

An embodiment is the robot's body. An operating environment is the physical medium, weather, communications, rules, and support system around that body. A wheeled warehouse robot and an autonomous mining truck are both ground robots, but their allowed terrain, localization, maintenance, and failure responses are different.

An operational design domain (ODD) is the exact set of conditions in which the robot is allowed to operate, plus its fallback when those conditions fail. Section 9.2 expands the definition.

Choose the operating environment before choosing the autonomy stack or model. The same feedback loop applies everywhere, but the hardest constraint changes:

3.8.1 One comparison table

Environment	Common bodies	Dominant physical constraint	State and perception	Communications	Operational burden	Work with strong public evidence
Structured land	Fixed cells, autonomous mobile robots, mobile manipulators	Contact, traction, collision, floor geometry	Encoders, lidar, cameras, IMU, mapped localization	Facility network with local fallback	Congestion, charging, maintenance, scene reset	Manufacturing, warehouse transport, and surgical workflows
Outdoor land	Haul trucks, tractors, off-road vehicles, quadrupeds	Soil, slope, slip, dust, rain, vibration	GNSS and inertial fusion, lidar or vision, traversability estimation	Private radio, cellular, or intermittent edge operation	Weather, seasonal use, remote service, changing terrain	Mining haulage, targeted spraying, survey, and bounded inspection
Air	Multirotors, fixed-wing aircraft, vertical-takeoff-and-landing aircraft	Lift, mass, wind, aerodynamic coupling, energy density	IMU, GNSS, barometer, airspeed, cameras, lidar, or radar	Command link, telemetry, cellular, or satellite with lost-link behavior	Airspace authorization, weather gates, batteries, launch and landing	Mapping, inspection, defense, agriculture, and bounded delivery
Surface water	Autonomous or uncrewed surface vessels and autonomous ships	Waves, currents, wind, roll, pitch, and heave	GNSS and inertial navigation, radar, Automatic Identification System (AIS), cameras, lidar, and sonar	VHF, cellular near shore, satellite offshore	Remote operations center, maritime traffic, weather, corrosion, biofouling	Hydrography, ocean science, surveillance, and selected commercial survey
Underwater	Remotely operated vehicles, autonomous underwater vehicles, gliders	Pressure, buoyancy, drag, currents, turbidity, corrosion	IMU, Doppler velocity log, depth, sonar, cameras, and acoustic position fixes	Tether, low-rate acoustic link, short-range optical link, or no live link	Support vessel, launch, recovery, recharge, leak prevention, navigation drift	Tethered intervention, science survey, pipeline and cable inspection, long-duration sensing
Space	Rovers, free flyers, landers, orbital servicers	Light-time, vacuum, radiation, thermal cycling, gravity regime	Cameras, IMU, joint sensors, star trackers, and mission-specific navigation	Scheduled relay or delayed direct link	Qualification, fault recovery, energy, launch, and little or no repair	Mars rover autonomy, International Space Station robots, landing systems, and geosynchronous life extension

The table is not a ranking. It says which constraint must be solved before a model matters.

3.8.2 Land robotics

Land robotics is several markets, not one:

• Structured indoor: factories, warehouses, hospitals, and labs can be mapped, marked, fenced, and redesigned around the robot.
• Road: autonomous-driving systems inherit traffic law, public exposure, high speed, and a very large validation burden.
• Off-road field: mines, farms, construction sites, forests, and disaster zones add deformable terrain, dust, weather, weak connectivity, and changing maps.

Outdoor localization should never be reduced to "add GPS." Open sky can support precise Global Navigation Satellite System (GNSS) corrections, but buildings, trees, pit walls, and terrain can block or reflect signals. The state estimator must fuse inertial, wheel, visual, lidar, radar, and map evidence appropriate to the site. The planner also needs a traversability estimate, meaning whether the body can cross the terrain without slipping, tipping, digging in, or exceeding clearance.

The strongest outdoor deployment evidence comes from highly bounded work. The specialized-system deployment table records commercial Komatsu mine-haul scale across North America, South America, Australia, and Europe. Its pattern matters here: the trucks repeat one job on controlled haul roads with fleet-management and service systems.

The near-term opportunity is not one universal land policy. It is better site-specific autonomy:

• Dust- and weather-aware perception.
• Traversability and slip estimation.
• Brownfield integration with dispatch, work orders, and safety zones.
• Remote assistance that exposes the right context to one operator.
• Long-run evidence by terrain, season, payload, and software revision.

3.8.3 Aerial robotics

Air changes the first principle: the robot must continuously produce enough lift to remain safe. Every extra battery, computer, sensor, guard, and package adds mass. Added battery can therefore increase stored energy while also increasing the power needed to fly.

The common forms make different trades:

• Multirotor: can hover and maneuver precisely; flight time is usually the binding constraint.
• Fixed wing: efficient over distance; cannot hover and needs a launch and landing plan.
• Hybrid vertical-takeoff-and-landing aircraft: combines hover and efficient cruise, but adds transition modes and more failure cases.

Production flight software keeps the fast safety-critical loop on a flight controller. PX4 documents a pipeline from sensors and estimators through guidance, navigation, cascaded controllers, control allocation, and actuators. Its controller diagrams show position, velocity, attitude, and angular-rate loops. A Linux companion computer can run perception, route planning, or ROS 2, but it sends bounded setpoints to the flight controller instead of replacing the fast stabilization loop.

MAVLink is a message protocol for vehicle, ground-station, and companion-computer communication. It is not a planner or safety case. PX4 and ArduPilot are autopilot stacks; ROS 2 may sit beside them for higher-level autonomy.

The operational boundary matters as much as the code:

• Wind, precipitation, temperature, visibility, and icing.
• Payload, takeoff mass, reserve energy, and landing site.
• GNSS quality and obstacles.
• Command-link and navigation loss behavior.
• Airspace, people, property, and other aircraft.

In the United States, the Federal Aviation Administration Part 107 path governs many small commercial drone operations. Night, operations over people, and operations over moving vehicles can run without a waiver only when the rule's conditions are met; controlled airspace can separately require airspace authorization. Beyond visual line of sight generally needs specific authority under the current Part 107 framework. The FAA proposed Part 108 in August 2025 to normalize some routine BVLOS operations. By this guide's cutoff, a final rule was under executive review but had not been published or taken effect, so current Part 107 authority still governed operations. Remote ID applies to drones that must be registered, subject to defined exceptions and authorizations. Rules differ by country and mission.

Commercial scale is real but bounded. The specialized-system deployment table records Zipline's vendor-reported delivery and distance denominators across approved networks. They do not establish unrestricted drone delivery.

The missing infrastructure is less glamorous than another vision model:

• Detect-and-avoid evidence and airspace integration.
• Weather-aware mission authorization.
• Battery-health and reserve-energy prediction.
• Dock inspection, charging, and maintenance.
• Flight-log replay tied to software, hardware, and regulatory identity.
• Fleet supervision that handles lost link without silently broadening risk.

3.8.4 Surface-water robotics

An autonomous surface vessel (ASV) or uncrewed surface vessel (USV) operates on the water surface. A large commercial ship with remote or autonomous functions may fall under the International Maritime Organization's Maritime Autonomous Surface Ship (MASS) framework.

The surface looks two-dimensional on a map, but the hull moves in six degrees of freedom. Waves add roll, pitch, and heave while wind and current change the planned ground track. Cameras and lidar move with the hull; radar and AIS have their own clutter, coverage, and trust limits.

A practical stack combines:

• GNSS and inertial navigation.
• Radar, AIS, cameras, and sometimes lidar for traffic.
• Sonar for bathymetry or underwater targets.
• Global weather-aware routing and local collision avoidance.
• Low-level heading, speed, and station-keeping control.
• VHF or cellular near shore and satellite communications offshore.
• A remote operations center with explicit loss-of-link and safe-state rules.

Maritime collision avoidance must account for the International Regulations for Preventing Collisions at Sea. This is not only a perception benchmark. The system must interpret encounters, make legible maneuvers, retain a responsible human and legal chain where required, and produce evidence after an incident.

The IMO MASS Code is a non-mandatory code for covered cargo ships that took effect on 2026-07-01. It addresses navigation, connectivity, remote operations, fire safety, search and rescue, risk assessment, cybersecurity, and human oversight. IMO's stated roadmap targets a mandatory code for adoption by 2030 and entry into force in 2032. Small research, defense, and national-water USVs may follow different rules.

The mature surface use cases are narrower than "autonomous shipping":

• Ocean and weather sensing.
• Hydrographic and cable-route survey.
• Maritime-domain awareness.
• Defense exercises and bounded operations.
• Support and communications relay for underwater vehicles.

The specialized-system deployment table records NOAA's named 2025 Atlantic hurricane USV mission. It proves field use and real-time data delivery, not fully unattended merchant shipping.

Fully unattended global merchant shipping remains a much harder claim because it combines autonomy with port operations, maintenance, international law, cybersecurity, search and rescue, and liability.

3.8.5 Underwater robotics

Underwater robotics begins with a communications break. Ordinary GPS and high-rate radio do not work at useful depth. A vehicle either keeps a tether, uses a low-rate acoustic link, uses a short-range optical link, or completes parts of the mission without live communication.

Platform	Human control	Endurance shape	Best fit	Main cost or failure
Remotely operated vehicle (ROV)	Real-time pilot over a tether	Power and duration can come from the support system	Inspection, intervention, sampling, repair	Support vessel, tether management, pilot team
Autonomous underwater vehicle (AUV or UUV)	Mission-level supervision, no continuous pilot	Battery-limited untethered mission	Mapping, search, inspection, sensing	Navigation drift, energy, launch and recovery
Underwater glider	Sparse supervision	Slow buoyancy-driven missions lasting much longer than powered AUV missions	Wide-area ocean sensing	Low speed, small payload, limited intervention
Hybrid or resident vehicle	Switches between autonomous and remotely assisted modes	Intended for repeated local missions or docking	Persistent inspection and intervention	Dock reliability, recharge, communications, service

Woods Hole Oceanographic Institution shows the boundaries clearly: its Jason ROV gives shipboard scientists real-time access through a tether, while the Sentry AUV explores autonomously without real-time human control.

An underwater state estimate commonly combines:

• IMU and depth.
• Doppler velocity log (DVL), which estimates motion relative to the seabed or water.
• Sonar and camera observations.
• Ultra-short-baseline (USBL) or long-baseline (LBL) acoustic position fixes.
• A GNSS fix only after surfacing or through a surface relay.

Sonar remains central because turbidity and darkness can make optical perception useless. Acoustic communications travel farther than light in many underwater conditions, but they have much lower bandwidth and much higher delay than ordinary local networks. The mission must tolerate missing updates and delayed measurements.

Launch, recovery, and recharge are part of the robot, not logistics after the robot:

Industrial autonomy is already more than a lab idea. The specialized-system deployment table records Oceaneering's named TotalEnergies pipeline-inspection pilot. It establishes a customer and industrial task, but not resident fleet economics.

Ulysses: what the linked video establishes

The Ulysses video is useful because it shows a startup treating the surface vessel, underwater vehicle, payload modules, manufacturing process, and recovery system as one product. It is an edited company profile, not an independent sea-trial report.

Public claim	Evidence status as of 2026-07-20	What it does and does not establish
Mako performs a tetherless autonomous test in Lake Tahoe	Vendor reported: shown in the edited company video	Shows a freshwater test and a mission run; no raw telemetry, intervention log, open-ocean sea state, or repeated-trial denominator is public
Mako has modular battery, compute, thruster, sensor, and payload sections	Vendor reported: shown in the video and described on the technology page	Establishes the modular design concept; does not establish every payload's qualification
Up to 12 hours endurance, 5,000 ft depth, 12 nmi range, and 250 lb payload	Vendor reported: technology page, which labels the system "Coming 2026"	Pre-production public specifications; no independent qualification report was found
Up to 72 hours endurance and 200 lb payload	Vendor reported: home page	Conflicts with the technology page; the configuration that explains the difference is not public
One Mako can be manufactured on two machines in about 12 hours	Unproven: founder statement at about 9 minutes in the linked video	A second public interview says 24 to 26 hours on current machines; neither statement is a production audit
Leviathan surface vessel plus Kraken launch, recovery, and recharge system	Directional: product architecture plus April 2026 coverage that says Kraken was in testing and the systems were moving from prototype to production	Establishes the intended integrated system; zero-human operations and production fleet performance remain unproven
A $18 million Series A plus an earlier $1 million seed	Verified: independently reported by Defense Daily and described by a16z	Establishes access to capital; does not establish reliability, margin, or productive fleet size
Tens of thousands of underwater robots and global remote operations centers	Unproven: founder vision in the video	Aspirational; no public deployment evidence establishes this scale

The important thesis is not "the ocean has found its SpaceX." It is:

A low-cost underwater vehicle only changes the market if launch, recovery, recharge, payload qualification, weather tolerance, and remote operations also become repeatable and cheaper.

3.8.6 Space robotics

Space robots use the same estimation, planning, control, and fault-management ideas under a different time and repair model. NASA's Mars Relay Network reports about 3 to 22.4 minutes of one-way light time between Earth and Mars, depending on orbital position. A human therefore cannot joystick a Mars rover through hazards in real time. The vehicle executes uploaded plans, performs local navigation, and enters safe states when it cannot continue.

Space also changes hardware:

• Vacuum removes convective cooling.
• Radiation can corrupt or damage electronics.
• Thermal cycles are severe.
• Lunar, Martian, orbital, and microgravity contact dynamics differ.
• Launch vibration and mass budgets constrain every part.
• Repair may be impossible for an unserviceable deep-space asset after launch.

Evidence must distinguish flown systems from planned missions. Northrop Grumman's Mission Extension Vehicles have docked with commercial geosynchronous satellites and provided life extension. The more capable Mission Robotic Vehicle was scheduled for 2026-07-21, one day after this guide's verification cutoff. Docking and propulsion service are therefore flown; broad robotic repair and assembly are not yet equivalent commercial evidence.

3.8.7 What to measure by environment

Environment	Minimum domain-specific evidence
Structured land	Accepted units per shift; interventions per operating hour; congestion waits; charging; safety stops; mean time to repair
Outdoor land	Terrain, slope, weather, payload, GNSS condition, stuck or slip events, remote assists, seasonal window
Air	Flight hours and missions; payload and route; wind and weather; aborts; lost-link events; reserve energy; airspace authority
Surface water	Nautical miles and mission days; sea state; traffic encounters; remote-operator interventions; satellite-link loss; port and maintenance time
Underwater	Depth and water condition; distance and mission duration; navigation drift; acoustic-link availability; launch and recovery attempts; support-vessel days
Space	Mission phase and environment; communication windows; safe-mode events; energy and thermal state; autonomous distance or task count; survival duration

Never compare "autonomy hours" across these rows without the environment, mission, intervention rule, and accepted output. An hour of warehouse transport, an hour of flight, an hour below the surface, and an hour on Mars do not carry the same difficulty, cost, or failure exposure.

After choosing an environment, use section 4.3 for the software split, section 8.5 for the report fields, section 9.2 for the full ODD, section 10.5 for deployment evidence, and section 16.4 for a portfolio branch.

4. Software infrastructure: from firmware to fleet

4.1 The deployed stack

Layer	Responsibilities	Representative tools
Firmware and drives	Motor commutation, encoder reading, current loops, electrical protection	Microcontroller unit (MCU), field-programmable gate array (FPGA), real-time operating system (RTOS), EtherCAT industrial Ethernet, or Controller Area Network (CAN)
Hardware abstraction	Expose joints, sensors, and commands with units and limits	Robot SDK, ros2_control
Middleware	Typed messages, discovery, services, actions, lifecycle, time	ROS 2 and DDS
Frames and state	Coordinate transforms, robot model, joint state	TF2, URDF, robot_state_publisher
Estimation and perception	Localization, tracking, segmentation, pose, calibration	OpenCV, Point Cloud Library (PCL), SLAM, learned perception
Planning and skills	Navigation, manipulation, task execution, recovery	Nav2, MoveIt 2, behavior trees
Learned policy	Observation-to-action or subgoal generation	LeRobot, OpenVLA, openpi, GR00T
Safety	Limits, scanners, interlocks, safe stop, safe speed	Safety-rated programmable logic controller (PLC), drive safety functions, certified sensors
Fleet and operations	Jobs, maps, deployments, alerts, teleoperation, maintenance	Vendor fleet manager or custom platform
Data and learning	Logging, lineage, curation, training, evaluation, release	MCAP, Parquet, object storage, orchestration, model registry

In this table, Data Distribution Service (DDS) is ROS 2's underlying publish-subscribe middleware family; URDF describes robot links and joints; and TF2 maintains the live coordinate-frame tree.

As of 2026-07-20, Lyrical Luth is the newest ROS 2 long-term support release and is supported through May 2031. Jazzy Jalisco remains supported through May 2029. Kilted Kaiju is the shorter-lived 2025 release and reaches end of life in December 2026. Use the release supported by the robot vendor; for a new standalone project, start with Lyrical.

4.2 ROS 2 is middleware, not the whole robot

ROS 2 provides:

• Nodes and executors.
• Topics, services, and actions.
• Discovery and quality-of-service settings.
• Lifecycle management.
• Message definitions and tooling.
• A large package ecosystem.

ROS 2 does not automatically provide:

• Hard real-time behavior.
• Safe motor control.
• Correct coordinate frames.
• Deterministic networks.
• A complete application.
• Safety certification.

The default production question is not "do we use ROS?" It is:

Which responsibilities run in ROS 2, which run below it in real-time controllers, which run above it in application services, and which safety functions remain independent?

4.3 MoveIt 2, Nav2, and LeRobot solve different problems

• MoveIt 2: kinematics, collision scenes, motion planning, trajectories, and manipulation interfaces.
• Nav2: localization, mapping interfaces, global and local planning, control, recovery, and navigation behavior trees.
• LeRobot: robot-learning datasets, policies, recording, training, evaluation, and rollout through Python hardware abstractions.

LeRobot does not replace ROS 2. MoveIt 2 does not directly drive motors. Nav2 does not solve manipulation. A deployed system can use all three, only one, or vendor-native equivalents.

LeRobotDataset v3 stores multimodal time series with consolidated Parquet data, MP4 video, episode metadata, tasks, statistics, and dataset information; see the current format documentation.

Other operating environments split responsibilities in the same way:

Environment	Vehicle or real-time layer	Higher-level autonomy and operations	Representative evidence and tools
Ground	Motor controller, ros2_control, safety controller	Nav2, MoveIt 2, Autoware, task application, fleet manager	ROS 2, rosbag2 or MCAP, Gazebo, Isaac
Air	Flight controller runs state estimation, stabilization, mission modes, and failsafes	Companion computer runs perception, avoidance, or ROS 2; ground station and fleet service supervise	PX4 architecture, ArduPilot, MAVLink, ULog or DataFlash
Surface water	Marine controller or autopilot runs heading, speed, propulsion, and local failsafes	Route planning, collision avoidance, payload processing, and remote operations center	MOOS-IvP, ROS 2, marine autopilots, mission replay
Underwater	Vehicle controller runs thrusters, depth, attitude, leak response, and basic modes	Sonar perception, mission behaviors, acoustic coordination, surface supervision	ArduSub, BlueOS, MOOS-IvP, mission and sonar logs
Space	Mission-specific flight software runs guidance, navigation, control, fault detection, and safe mode	Delayed ground planning plus bounded onboard autonomy	Hardware-in-the-loop, mission telemetry, and the open NASA Astrobee research stack

These are representative stacks, not universal standards. PX4, ArduPilot, MOOS-IvP, and ROS 2 can support more than one vehicle type. The architecture boundary matters more than the logo: hard real-time control and local fault response stay below slower perception, planning, fleet, and cloud systems.

4.4 Real-time and distributed-systems rules

Robot software inherits distributed-system failures plus physical deadlines:

• Clock drift.
• Message delay and loss.
• Stale transforms.
• Sensor and command rate mismatch.
• Queue buildup.
• Priority inversion.
• CPU and GPU contention.
• Driver reset.
• Partial network partition.

Every command should carry or imply:

• Coordinate frame.
• Units.
• Timestamp.
• Validity duration.
• Sequence or correlation identity.
• Limits and expected rate.
• Failure and cancellation semantics.

4.5 Observability for robots

A useful incident record contains:

Record group	Required fields
Identity and site	robot_id; site_id; map and map version
Release identity	Software, firmware, model, dataset, and configuration versions
Task context	Task and subgoal
Sensor context	Sensor timestamps, calibration versions, and raw or compressed observations
Decision trace	Estimated state, planned command, and executed command
Safety and assistance	Safety-state transitions and human interventions
Result and condition	Outcome, failure category, and maintenance state

The log must let an engineer reconstruct:

1. What the robot saw.
2. What it believed.
3. What it intended.
4. What it commanded.
5. What the hardware did.
6. Why safety or recovery intervened.

Fleet observability is not ordinary application logging with video attached. Robot records are multimodal, time-synchronized, large, and often created where network bandwidth is limited. A production data plane therefore needs:

• Edge buffering and resumable upload.
• Search across release identity, task, site, outcome, and time.
• Synchronized replay of video, 3D, joint state, commands, and safety events.
• Event annotation that can become a curated training or regression episode.
• Retention and access controls for people, homes, factories, and customer data.

Foxglove is one commercial example of this product shape: edge capture, cloud or on-premises synchronization, petabyte-scale search, evaluation, and 3D replay. Its customer and scale claims are vendor reported. The durable point is the layer, not the vendor: incident replay and training-data curation must share timestamps, schemas, and release identity.

4.6 Release engineering for robots

A safe release path is:

Software-in-the-loop uses robot software with a simulated plant. Hardware-in-the-loop uses a real controller with simulated or partial hardware.

Rollback must include software, model, configuration, calibration, and compatible firmware. A model rollback alone can fail if the surrounding interface changed.

5. Robot learning: what the model families actually do

Before comparing models, define the training unit: a robot demonstration is a time-aligned episode containing observations, robot state, commanded and measured action, task, outcome, calibration and version identity, and any intervention. Section 6 expands this contract and its quality gates.

5.1 Behavior cloning

An action is the command a policy outputs in a fixed convention, such as joint-position deltas, an end-effector pose, or a gripper command.

Behavior cloning learns:

observation + task -> demonstrated action

It is supervised learning on demonstrations. It is practical because it avoids unsafe exploration and complex reward design.

Main failure: covariate shift. A small error moves the robot into a state that was absent from the demonstrations. The next prediction is worse, and errors compound.

Mitigations:

• Collect diverse states, not only perfect runs.
• Add corrective and recovery demonstrations.
• Query a human when uncertainty rises.
• Train on autonomous failures.
• Use receding-horizon action chunks: predict a short sequence, execute part, observe again, and replace the remaining plan.
• Add planning or constraints around the policy.

5.2 ACT and action chunking

Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware (ACT) predicts a short sequence of actions rather than one action at a time. This:

• Reduces effective decision frequency.
• Smooths behavior.
• Models coordinated bimanual motion.
• Still depends on representative demonstrations and feedback.

ACT remains a strong baseline. A foundation model should beat it on the target distribution before the extra complexity is justified.

Action chunks do not remove model latency. If the next chunk is unavailable when the current chunk ends, the robot pauses or jumps between incompatible motions. Real-Time Chunking is an inference-only method for diffusion and flow policies:

1. Start generating the next chunk while the current chunk executes.
2. Freeze the prefix that will already have executed when inference finishes.
3. Inpaint the remaining actions with soft guidance from the previous chunk.
4. Keep the low-level controller running while policy inference happens asynchronously.

The paper evaluates 12 dynamic simulation tasks and six real bimanual tasks with pi0.5. It reports operation with more than 300 ms of inference delay and a 20% faster match-lighting motion than synchronous chunking. This is evidence for one latency-handling method, not proof that a large policy meets every robot's real-time budget.

5.3 Diffusion and flow policies

Diffusion Policy represents actions as a distribution and iteratively denoises a candidate action sequence. This handles tasks with several valid motions better than a single averaged prediction.

Flow matching, used by the pi0 family, learns a continuous transformation from noise to action. Both are ways to model multimodal continuous behavior.

Strengths:

• Smooth action chunks.
• Several valid strategies.
• Strong task-specific manipulation performance.

Costs:

• Inference latency and compute.
• More moving parts in deployment.
• No automatic safety guarantee.
• Still sensitive to observation and action conventions.

5.4 Reinforcement learning

Reinforcement learning, or RL, learns from reward through interaction. Simulation here means a software environment that approximates the robot, task, sensors, and physics. Section 7 explains its tools and transfer limits; simulation-only scores remain incomplete until paired with physical evaluation.

RL is strong when:

• Simulation can generate many safe trials.
• Reward is measurable.
• Reset is cheap.
• Exploration can be bounded.

This explains success in locomotion and game-like tasks. Real-world manipulation makes exploration, resets, wear, and reward design expensive.

Useful patterns:

• Train in simulation, then adapt on hardware.
• Start from imitation, then improve with RL.
• Use human intervention to avoid catastrophic states.
• Use offline RL on logged data.
• Use RL to learn recovery, not only nominal success.

Two 2025 examples show the post-training shift:

• RECAP, RL with Experience and Corrections via Advantage-conditioned Policies, and pi*0.6 combine demonstrations, teleoperator corrections after policy mistakes, a learned value function, and on-robot experience. Physical Intelligence reports more than 2x throughput on some hard tasks, over 90% espresso success, an all-day espresso run, 50 novel garments handled in a new home, and 59 factory packaging boxes assembled without interruption. These are vendor-run results on a closed 5B VLA, not independent deployment audits.
• SimpleVLA-RL applies group-relative policy optimization with a binary task-completion reward. With one demonstration per LIBERO task, the authors' OpenVLA-OFT implementation rises from 17.3% to 91.7% on LIBERO-Long. Simulation-only RL also raises average real-robot success from 17.5% to 38.5% across four dual-arm tasks, with 50 physical trials per task. The LIBERO gain is simulation-scoped benchmark evidence, not field reliability.

The shared lesson is narrower than "RL solves robotics." Demonstrations define the behavior, corrections cover policy-induced states, and autonomous experience optimizes recovery and throughput. Real hardware still limits how much exploration is safe and affordable.

5.5 Vision-language-action models

A VLA connects visual and language representations to robot actions.

What internet pretraining contributes:

• Object and scene semantics.
• Language following.
• Visual features.
• Some commonsense relations.

What it does not directly contribute:

• Robot-specific action coordinates.
• Contact force.
• Joint limits and dynamics.
• Calibration.
• Safety.
• Reliable recovery.

Training the action head can also damage the representation it is attached to. Knowledge Insulation studies a failure mode where gradients from a newly initialized continuous action expert slow training and weaken language following in the pretrained vision-language backbone. Its recipe trains the backbone with discrete Frequency-space Action Sequence Tokenization, or FAST, tokens and general vision-language data, trains the action expert on continuous flow matching, and stops the action expert's gradients from flowing into the backbone. The result is a useful design principle, not a universal law: preserve pretrained semantic features while adapting a smaller module for fast continuous control.

5.6 World models

A useful operational definition:

A world model predicts task-relevant future state conditioned on actions.

Representations include:

• Pixels or video.
• Latent state.
• 3D or 4D scene structure.
• Motion fields.
• Physics-informed state.

Uses include:

• Generate synthetic experience.
• Propose visual subgoals.
• Score candidate actions.
• Plan through imagined futures.
• Detect likely success or failure.

The critical boundary:

A visually plausible future can still be physically impossible.

The 2026 world-model survey for manipulation emphasizes that value depends on what is predicted and how a controller consumes it, not on calling a video generator a "world model."

V-JEPA 2 is a concrete example of the promise and boundary. Meta pretrained the 1.2B-parameter representation on more than 1 million hours of video and 1 million images, then action-conditioned it with 62 hours of DROID robot data. Meta reports 65% to 80% zero-shot pick-and-place success on new objects in new lab environments using model-predictive control and visual subgoals. This is short-horizon planning evidence from a vendor research team, not proof of persistent long-horizon memory or production autonomy.

5.7 The likely frontier architecture

The core robot stack shows the full hierarchy: slow semantic reasoning, a fast learned policy, motion planning, real-time engineered control, and an independent safety envelope.

This architecture is not proof that one specific company has solved deployment. It is a sensible division of responsibilities.

5.8 Current model map

Model or family	Main contribution	Public access	Evidence boundary
RT-1	Transformer policy trained across many tasks	Paper	Research platform
RT-2	Vision-language pretraining transferred into action tokens	Paper; weights closed	Google evaluation
Open X-Embodiment / RT-X	Cross-lab, cross-embodiment data and transfer	Data and paper	Heterogeneous data quality and licenses
Diffusion Policy	Action-sequence diffusion	Open	Strong practical baseline, not a generalist reasoner
Octo	Open generalist diffusion policy	Open	Research and fine-tuning
OpenVLA	Open 7B VLA pretrained on a 970,000-episode subset of Open X-Embodiment	Open	Research baseline
OpenVLA-OFT	Faster fine-tuning and action chunking	Open	Simulation benchmark details and limits are in section 8.4
pi0	Vision-language model (VLM) plus flow-based action expert	Selected weights and code	Lab and vendor evaluations
pi0.5	Open-world generalization program	Public base and named expert checkpoints through openpi	Vendor evaluation
pi*0.6 and RECAP	Demonstrations, corrections, and real-world RL for throughput and recovery	Paper and vendor demos; weights closed	Multi-hour runs are vendor reported
pi0.7	Multimodal context, strategy metadata, failures, subgoal images	Paper and demos; no public pi0.7 checkpoint found as of 2026-07-20	Strong frontier evidence, vendor run
GR00T N1.7	Open 3B VLA with human-video pretraining and deployment tooling	Code: Apache 2.0; weights: NVIDIA Open Model License; gated Hugging Face access is required for the Cosmos-Reason2-2B backbone	Repository says GA; the original release tag says Early Access
Gemini Robotics	VLA plus a separate embodied-reasoning model	Limited VLA access	Not one monolithic policy
Gemini Robotics-ER 1.6	Spatial reasoning, planning, success detection, and tool use	Preview API	Text-output VLM, not low-level motor control; not for safety-critical use
Helix and Helix 02	Closed hierarchy: S2 semantic reasoning, S1 full-body targets at 200 Hz, and S0 balance and contact control at 1 kHz	Closed	Figure vendor evidence; no independent uptime or intervention denominator
V-JEPA 2	Video-pretrained predictive representation used for zero-shot model-predictive control	Code, checkpoints, and paper	Short-horizon lab manipulation; Meta-reported evaluation
TRI Large Behavior Models	Controlled pretraining and fine-tuning study with 1,800 real trials and more than 47,000 simulation rollouts across 29 tasks	Paper and project	Gains required target-task fine-tuning; no consistent zero-shot advantage
Generalist GEN-0	Proprietary cross-embodiment scaling program	Blog and demos; no public weights	Corpus scale, scaling law, and 7B threshold are company claims without independent reproduction

5.9 What future models need

1. Explicit memory and progress state.
2. Better geometry and contact representations.
3. Tactile and force conditioning.
4. Cross-embodiment adapters.
5. Learning from failures and interventions.
6. Uncertainty that changes behavior.
7. Built-in help-seeking and safe refusal.
8. Long-horizon evaluation without hidden resets.
9. Onboard small policies paired with larger slower reasoners.
10. Continual learning with regression protection and rollback.

6. Robot data: the episode is the unit

6.1 A robot episode

Training episodes are derived from fleet logs. Every transformed field should trace to a source signal or a documented filter, and raw incident evidence must remain replayable.

An image label is not enough. A useful robot episode contains:

Episode group	Required contents
Task definition	Task and context
Observations	Camera, depth, and audio observations
Robot state and execution	Robot state, commanded action, and measured execution
Physical interaction	Force and contact
Time and space	Timestamps and coordinate frames
Provenance	Calibration, hardware identity, and software, model, and configuration versions
Result	Outcome, failure reason, and human intervention

The exact unit is usually a time-aligned episode, not a frame.

6.2 Collection methods

Method	Strength	Weakness
Leader-follower teleoperation	Robot-native, accurate action labels	Hardware, operator fatigue, reset cost, style bias
VR teleoperation	Flexible and intuitive	Latency, retargeting, weak force feedback
Kinesthetic teaching	Natural for compliant arms and contact	Slow, body-specific, human dynamics leak into data
Scripted or planner-generated data	Cheap and repeatable	Limited diversity and can encode planner bias
Human egocentric video	Large semantic and environment diversity	No direct robot action, joint, or force labels
Simulation	Scale, rare events, automatic labels, safe exploration	Reality gap and simulator bias
Autonomous fleet data	Real failure distribution	Requires deployed robots and good telemetry
Human intervention	High-value boundary and recovery examples	Selection bias and privacy or labor concerns

6.3 Data quality gates

Reject or quarantine episodes with:

• Missing or non-monotonic timestamps.
• Dropped frames or rate drift.
• Calibration mismatch.
• Wrong units or coordinate frame.
• Stale task text.
• Safety override without label.
• Unclear success criterion.
• Scene reset contamination.
• Duplicate episode.
• Hardware fault.
• License or consent ambiguity.

Add quality dimensions:

• Task diversity.
• Object diversity.
• Environment diversity.
• Operator diversity.
• Strategy diversity.
• Failure and recovery coverage.
• Contact coverage.
• Outcome confidence.

Before collection scales:

1. Define the episode contract: start state, task, success, failure, interruption, reset, required signals, and safety procedure.
2. Run a small physical pilot before recording thousands of episodes. Fix camera placement, operator ergonomics, reset steps, and mechanical bottlenecks first.
3. Measure leader-follower delay, camera frame stability, dropped observations, and cross-stream drift. Do not infer synchronization from matching nominal rates.
4. Separate failed work from bad data. Keep a failed or recovered task when its streams and labels are valid; reject telemetry with occlusion, corrupt timing, calibration mismatch, or a broken protocol.
5. Preserve raw recordings and link every derived dataset to the conversion code, filters, and configuration.

This operational sequence follows the Trossen teleoperation guide. Trossen sells collection hardware, so treat product choices as vendor guidance; the episode-contract and synchronization checks are broadly applicable.

There is no universal demonstration count. NVIDIA's GR00T N1.7 FAQ gives task-shaped starting ranges: about 100 trajectories for simple fixed pick-and-place, 500 or more for complex multi-step scenes, and 2,000 or more for high-degree-of-freedom humanoid tasks. It recommends starting with about 100 teleoperated demonstrations, then using human-gated dataset aggregation: run the policy, intervene on failures, and add the corrections. These are vendor guidelines, not guaranteed sample-complexity laws.

6.4 Formats and storage

Format	Best use	Boundary
MCAP	Time-stamped multimodal robot logs; the default rosbag2 storage plugin since ROS 2 Iron when --storage is omitted	Container format, not a semantic quality standard; SQLite3 remains available
rosbag2	ROS 2 topic recording and playback	Tied to message schemas and middleware conventions
LeRobotDataset v3	Parquet signals, MP4 video, episode and task metadata	Robot-learning format, not full fleet observability
RLDS	Episodic reinforcement and imitation learning data	TensorFlow-centered ecosystem
Parquet plus object storage	Analytics and large-scale signals	Requires a separately governed video and schema contract

The production answer is often two representations:

1. A lossless or high-fidelity incident log for replay.
2. A training-optimized dataset derived from it with lineage.

6.5 Important public datasets

Dataset	Verified public scope	Important boundary
Open X-Embodiment	Source corpus with more than 1 million trajectories across 22 embodiments	Many institutions, formats, policies, and licenses; individual model mixtures use subsets
DROID	76,000 demonstrations, about 350 hours, 564 scenes, 86 tasks, 50 collectors	Primarily one Franka hardware family
BridgeData V2	60,096 trajectories across many environments and objects	Includes scripted and teleoperated data
RH20T	More than 110,000 manipulation sequences with multimodal sensing	Split license: RH20T-C allows commercial use; RH20T-NC is noncommercial only; check the scene subset
AgiBot World Beta	More than 1 million trajectories and about 2,976 hours; the beta card reports 2,976.4 hours	Noncommercial licensing limits some uses
RoboMIND 2.0	More than 310,000 dual-arm trajectories across six embodiments and 739 tasks, totaling more than 1,000 hours; includes 12,000 tactile and 20,000 mobile-manipulation trajectories	Corpus metadata and author evaluations do not prove transfer to a new embodiment or site

Counts describe corpus size. They do not prove that a model trained on the corpus transfers to a new robot or customer.

Proprietary training mixtures are much larger than most public datasets, but their counts are not independently auditable:

Training mixture	Reported composition	What the evidence supports
TRI Large Behavior Models	About 468 hours of internal bimanual teleoperation, 45 simulation hours, 32 Universal Manipulation Interface (UMI) hours, and about 1,150 curated Open X-Embodiment hours	Blind A/B physical trials show 3-5x lower target-task data needs in challenging settings after fine-tuning; gains rise smoothly with more pretraining data
pi0	More than 10,000 hours; 9.1% of the weighted mixture comes from open OXE, Bridge V2, and DROID data	Broad pretraining plus 5 to 100 or more hours of high-quality task-specific post-training beats either stage alone in the reported tasks
Generalist GEN-0	Company claims 270,000 hours, growing by 10,000 hours per week, across homes, warehouses, and workplaces	Vendor reports power-law improvement and a model-size transition near 7B parameters; no public dataset, checkpoint, or independent reproduction verifies the claim

Hours alone hide control rate, episode length, embodiment, operator skill, failure coverage, and repeated scenes. Compare action timesteps, task and site diversity, rights, and target-task improvement before comparing headline hours.

6.6 The data flywheel

Section 4.6 governs release promotion across simulation, hardware, site, and fleet gates. This loop governs which field evidence to collect and learn from.

The moat is not "we have many hours." It is:

• Exclusive workflow access.
• High-quality failure labels.
• Fast recollection.
• Clear rights to use the data.
• Evaluation tied to customer outcomes.
• A deployment channel that produces the next data.

6.7 Is robot data labeling the next Scale AI?

Generic labeling alone: unlikely. The full data and evaluation loop: plausible.

Generic labor is exposed to:

• Falling teleoperation cost.
• Open datasets.
• Customer insourcing.
• Embodiment-specific transfer limits.
• Automated segmentation and quality scoring.
• Buyers caring about policy improvement, not hours delivered.

A stronger company owns the quality and lineage gates in section 6.3 and the recollection and regression loop in section 6.6. It also proves that deployment feedback improves throughput, uptime, or intervention rate.

6.8 The 100,000-year gap and the commercial bootstrap

Ken Goldberg's 2025 Science Robotics editorial compares internet-scale vision-language pretraining with roughly 100,000 human-years of reading and viewing, while the largest reported robot teleoperation corpus was on the order of one year. The arithmetic is a scale analogy, not a forecast that useful robots literally require 100,000 years.

His proposed bridge is good old-fashioned engineering, or GOFE: estimation, kinematics, planning, controls, and safety make a narrow commercial system useful before a general end-to-end policy exists. Paid work then produces real failure, intervention, and adjacent-task data.

This commercial bootstrap explains who pays for the first useful system. It extends, rather than duplicates, the release gates in section 4.6 and the learning loop in section 6.6.

The loop does not make every log valuable. It creates the right to observe real failures and recollect targeted episodes while customer value funds continued operation.

6.9 Data rights and deal structure

Before collection, contractually define:

• Ownership of raw logs, derived datasets, labels, embeddings, and trained weights.
• Exclusive versus non-exclusive use.
• Rights to failures, interventions, and future recollection at the same site.
• Whether the operator, robot vendor, model vendor, or customer can sublicense.
• Retention, deletion, geographic transfer, and breach obligations.
• Consent and privacy boundaries for workers, bystanders, homes, and audio.
• Per-dataset license compatibility inside a mixed corpus.

Data collected while doing paid work can be more valuable than a generic data farm because it contains the real workflow and failure distribution. It can also be unusable for model training if the contract, worker consent, or customer confidentiality terms do not grant that right. "We generated the data" does not prove "we may train on it."

7. Simulation, synthetic data, and sim-to-real

7.1 What simulation is good at

• Training locomotion with many parallel agents.
• Navigation and collision testing.
• Automatic labels.
• Dangerous or rare scenarios.
• Regression testing.
• Domain randomization.
• Initial policy learning before hardware exists.
• Reproducing selected field failures.

7.2 What simulation struggles with

• Friction and stick-slip.
• Deformable objects.
• Cables, cloth, liquids, food, and granular material.
• Tactile sensing.
• Wear and backlash.
• Camera and depth artifacts.
• Human unpredictability.
• Exact reset and fixture tolerances.

Photorealism does not guarantee physics realism. Physics realism does not guarantee the sensor and operational distribution matches the site.

7.3 Tool map

Tool	Main strength	Best fit
MuJoCo	Fast, accurate rigid-body simulation and contact	Control, manipulation, RL, research
MuJoCo Playground	GPU-accelerated robot learning workflows	Parallel RL and sim-to-real experiments
NVIDIA Isaac Sim	Photorealistic sensors, USD assets, NVIDIA ecosystem	Synthetic data and integrated simulation
Isaac Lab	Robot-learning environments on Isaac Sim	RL, imitation, large parallel training
Gazebo	ROS integration and general robotics simulation	System integration and education
Drake	Dynamics, optimization, planning, control	Model-based robotics and verification
ManiSkill	Manipulation benchmarks and GPU parallelism	Learning and benchmark research
RoboCasa	Large household manipulation scenes	Data generation and household tasks
Habitat	Embodied navigation and 3D environments	Navigation and embodied AI research
Genesis	New GPU simulation platform with broad ambition	Research; benchmark claims require independent checking
Newton	Emerging open physics engine collaboration	Track as an evolving platform, not a settled default
CARLA	Open urban-driving simulation with sensors, traffic, and scenario tooling	Autonomous-driving research and system integration
PX4 SITL with Gazebo	Runs PX4 flight code against simulated aircraft and sensors	Aerial flight-stack integration and regression
ArduPilot SITL	Runs Plane, Copter, Rover, boat, and underwater autopilot code without vehicle hardware	Multi-domain autopilot development and log-driven testing
MOOS-IvP	Modular mission autonomy and helm behaviors for marine vehicles	Surface and underwater autonomy development
HoloOcean	Underwater scenarios, sensors, dynamics, and multi-agent interfaces	Underwater autonomy research and integration
NASA Astrobee	Open flight software and simulation for an International Space Station free flyer	Space-robotics software and ISS free-flyer simulation; communication delay must be injected outside the simulator

7.4 Sim-to-real failure taxonomy

Mitigations:

• System identification.
• Domain randomization.
• Better sensor models.
• Paired real and simulated evaluation.
• Real-world fine-tuning.
• Residual learning around a model-based controller.
• Field-log replay.
• Uncertainty and fallback.

7.5 World models are not automatically simulators

A generated video can help propose a subgoal or score a plan. It cannot replace physical validation unless its prediction error is shown to correlate with action success on the target task.

Ask:

1. Is the model action-conditioned?
2. What state does it predict?
3. Does prediction quality correlate with real control success?
4. Can search exploit its errors?
5. What is the inference latency?
6. What real-world validation closes the loop?

8. Evaluation, reliability, and the demo-to-deployment gap

8.1 The evaluation ladder

Stage	What it proves	What it does not prove
Offline loss	Model fits held-out data	Closed-loop task success
Simulation benchmark	Behavior in the simulator distribution	Physical transfer
Selected physical demo	One task can work under chosen conditions	Repeatability
Controlled physical trials	Physical feasibility and repeatability on a named setup	Site uptime and economics
Perturbation trials	Robustness to named changes	Open-world robustness
Long-duration run	Recovery, drift, thermal, and intervention behavior	Another site or robot revision

Commercial evidence continues in section 10.3 after these technical evaluation rungs.

8.2 Metrics that matter

Every metric needs a task definition, denominator, environment, and window.

• Task success rate: completed trials / initiated trials under a named condition.
• Intervention rate: human assists / operating hour or task.
• Autonomy time: time operating without human action.
• Cycle time: end-to-end time per completed unit of work.
• Throughput: completed work units per hour or shift.
• Recovery success: failures recovered without human assist / recoverable failures.
• Mean time between operational failures: operating time / failure count.
• Mean time to repair: maintenance time / repair event.
• Safety stop rate: safety-triggered stops / operating hour.
• Damage or drop rate: harmful events / handled units.
• Cost per successful unit: all-in operating cost / accepted output.

8.3 Why small error rates compound

If a ten-step task has an independent 95% success probability at each step:

0.95 ^ 10 = 0.60

Only about 60% of complete tasks succeed under that simplifying assumption. Real failures are often correlated, so independence is optimistic. Long-horizon systems need recovery, not only higher nominal step accuracy.

8.4 Benchmark traps

• Train/test object overlap.
• Same lab, lighting, and camera.
• Simulator-only success.
• Hidden reset labor.
• Excluding timeouts.
• Reporting best checkpoint.
• Small trial count.
• No confidence interval.
• Human-selected episodes.
• Different action rates or proprioception inputs.
• Language test that only selects among familiar objects.

OpenVLA-OFT's 97.1% is useful evidence on the LIBERO simulation suites. It is not a 97.1% production reliability claim.

Several published results show why one score is not enough:

• The robomimic study found that the validation-selected checkpoint could perform 50% to 100% worse than the best physical policy. Offline validation is a weak checkpoint selector for closed-loop behavior.
• LIBERO-Plus perturbs object layout, camera viewpoint, robot initial state, language, lighting, background, and sensor noise. The paper reports drops from about 95% to below 30% under modest camera and initial-state changes, and finds that some policies continue after language is removed or changed.
• LIBERO-PRO changes manipulated objects, initial states, instructions, and environments. Models above 90% on the original benchmark fall to 0.0% in its full combined generalized setting. Individual perturbations are not universally zero: the paper reports pi0.5 at 38% on LIBERO-Goal under position changes. The combined collapse is evidence of benchmark memorization, not a claim that every deployment fails.
• An independent SO-101 evaluation reports that the tested policies averaged about 84% on the easiest pen-transfer task but 4% on color sorting, with low recovery rates after errors. Cheap hardware makes independent checks possible; it does not make the test distribution complete.

8.5 A minimum credible robot report

Publish:

1. Exact task and operating domain.
2. Robot and end effector.
3. Model, data, and configuration versions.
4. Number of trials and selection rule.
5. Success definition.
6. Failures and timeouts.
7. Intervention policy.
8. Cycle time distribution.
9. Perturbations.
10. Raw continuous runs.
11. Safety events.
12. Maintenance and reset assumptions.
13. Environment-specific conditions and recovery evidence: terrain and weather on land; wind, payload, and lost link in air; sea state and traffic on the surface; depth, water condition, navigation drift, and launch or recovery underwater; communication windows and safe-mode events in space.

8.6 Evaluation systems for different questions

Evaluation system	Question it answers	Verified public evidence	Boundary
Fixed simulation benchmark	Does the policy fit a repeatable task suite?	LIBERO, RoboCasa, ManiSkill, and EmbodiedBench	Simulator distribution and benchmark design limit the claim
Perturbation benchmark	Which named changes break the policy?	LIBERO-Plus and LIBERO-PRO	Covers selected perturbations, not the open world
EmbodiedBench	Can a multimodal reasoner plan and control across navigation and manipulation simulations?	1,128 tasks, four environments, and 24 evaluated models; the paper's best model averaged 28.9%	Evaluates simulated embodied agents, not a physical VLA fleet
AutoEval	Can repeated physical trials run with automatic success scoring and resets?	Three tabletop cells, up to 500 episodes per 24 hours, and more than 99% lower human supervision time	Learned reset and success systems create their own error modes
RoboArena	Which policy is preferred across diverse real tasks and sites?	Seven policies, seven universities, and 612 double-blind pairwise comparisons	Pairwise ranking depends on evaluator task selection and preference criteria
Blind randomized physical trials	Does a candidate beat a baseline on the named physical distribution?	TRI used 1,800 physical trials with sequential statistical testing	Expensive and still scoped to the tested robots and tasks
Continuous deployment telemetry	Does it produce accepted work over hours, shifts, sites, and revisions?	Throughput, intervention, failure, repair, safety, and cost records	Requires operating access and honest denominator definitions

Use a portfolio. A repeatable simulator supports iteration; perturbation tests locate brittleness; automated and distributed physical trials improve sample size and diversity; continuous operations test economics and recovery. None is a substitute for all the others.

9. Safety, assurance, and regulation

9.1 Safety is an application property

"The model is safe" is not a sufficient claim. Risk depends on:

• Robot mass, speed, force, and reach.
• End effector and payload.
• People and access control.
• Environment and escape paths.
• Failure detection.
• Braking and power-loss behavior.
• Task and foreseeable misuse.
• Maintenance.
• Software and model change process.

For a dynamically stable legged robot, removing power can create a fall hazard. A safe stop may require bounded deceleration, balance, controlled lowering, and then energy isolation. "Emergency stop" is a required function, not a promise that every embodiment can freeze instantly without creating a second hazard.

9.2 Operational design domain

The operational design domain, or ODD, defines where the system is allowed to operate:

• Site and mapped area.
• Floor condition and slope.
• Lighting and visibility.
• Object classes.
• Allowed people and proximity.
• Speed and payload.
• Network state.
• Temperature and contamination.
• Human supervision.
• Terrain, traction, weather, and GNSS quality for outdoor land.
• Wind, precipitation, visibility, airspace, and reserve energy for flight.
• Sea state, current, traffic, port, and satellite-link conditions on surface water.
• Depth, pressure rating, turbidity, acoustic-link state, and recovery window underwater.
• Radiation, thermal, lighting, gravity, and communication windows in space.

When the ODD is violated, the robot should degrade, stop, isolate energy, or request help.

9.3 Layered production architecture

Use mechanical and electrical risk reduction before software where possible:

1. Remove the hazard by design.
2. Guard or isolate it.
3. Limit energy, speed, force, and reach.
4. Detect abnormal conditions.
5. Stop safely.
6. Add procedures, training, and protective equipment.

A control barrier function, or CBF, represents a safe set of physical states. A runtime filter can solve for the smallest change to a learned command that keeps the next state inside that set. This is useful for constraints that can be measured and modeled, such as distance, joint limits, or support stability. It does not cover an unknown hazard that the sensors or barrier definition omit.

CBF-RL is a 2025 research example that uses a CBF filter and barrier-shaped reward during training so a humanoid policy learns safer behavior. Its Unitree G1 results are research evidence. Removing the runtime filter after training does not turn the learned policy into an independent, safety-rated protection layer.

NVIDIA Halos for Robotics is a June 2026 commercial example of the same separation principle: safety-aware compute, operating software, runtime supervision, and an inspection process around AI components. Agility is the first announced adopter for Digit. This is a vendor announcement before final third-party certification, not evidence that Halos or Digit is already certified for every stated standard or application.

9.4 Learned-system hazards

• Distribution shift.
• Hallucinated object or free space.
• Overconfident action.
• Prompt or instruction ambiguity.
• Sensor spoofing or occlusion.
• Delayed remote reasoning.
• Model or dataset regression.
• Unsafe recovery loop.
• Operator overtrust.
• Cyber compromise.

The response is not one "AI safety score." It is a safety case with hazards, controls, verification, residual risk, change management, and incident evidence.

SafeVLA-Bench shows why task success and safety must be separate outputs. It adds task-specific temporal safety specifications to LIBERO and RoboCasa-365 and reports:

• Success rate.
• Safety-specification satisfaction.
• Successful-but-unsafe rate.
• Worst-violation severity.

The paper finds 13% to 15% unsafe-episode rates for high-success LIBERO baselines and at least one active safety violation in 36% to 56% of successful RoboCasa-365 rollouts. These are simulator-side proxies with published thresholds, not hardware certification limits.

9.5 Standards map as of 2026-07-20

Standard or rule	Scope	Current boundary
ISO 10218-1:2025	Industrial robot design	Robot as partly completed machinery
ISO 10218-2:2025	Industrial robot applications and cells	Integration, commissioning, operation, maintenance, decommissioning
ISO/TS 15066	Collaborative industrial robot applications	Complements industrial robot standards
ISO 13482	Personal care robots	Non-medical personal care scope
ISO/CD 25785-1	Dynamically stable industrial mobile robots with legs, wheels, or other active-stability locomotion	Committee draft at ISO stage 30.60, close of comment period; not a published standard
ISO 13849-1	Safety-related control systems	Machinery control-system design
IEC 62061	Functional safety of machinery control systems	Machinery sector
IEC 61508	Functional safety framework	Broad electrical/electronic/programmable systems
ISO/IEC TR 5469:2024	AI inside safety-related functions, non-AI functions protecting AI-controlled equipment, and AI used to develop safety functions	Published technical report and process guidance; not a stand-alone product certification
ISO/PAS 8800:2024	Safety and AI in road vehicles	Automotive, not a general humanoid standard
FAA 14 CFR Part 107	U.S. civil small uncrewed-aircraft operations	Mission conditions determine whether standard rules, airspace authorization, or a waiver applies
FAA Remote ID	Identification broadcasts for U.S. drones that must be registered	Defined exceptions and authorizations exist; Remote ID is not permission for the mission itself
COLREGs	International collision-prevention rules for vessels	Autonomous navigation must still satisfy applicable maritime traffic rules and responsibilities
IMO MASS Code	Remotely controlled and autonomous cargo ships covered by SOLAS	Non-mandatory from 2026-07-01; IMO targets mandatory-code adoption by 2030 and entry into force in 2032
EU Machinery Regulation 2023/1230	Machinery placed on the EU market	Applies from 2027-01-20 after corrigendum; the uncorrected Official Journal text shows 2027-01-14
EU AI Act 2024/1689	AI systems placed on or used in the EU, with obligations based on role and risk classification	A robot is not automatically high-risk; classification depends on the AI component, product safety role, and use case
OSHA robotics standards page	U.S. workplace context	OSHA states there is no single robotics-specific federal standard

Where a robot's AI system falls within the EU AI Act's high-risk scope, Article 10 requires governance for training, validation, and test data, including provenance, collection, preparation, assumptions, bias examination, gap identification, and fitness for the intended context. This is a lifecycle and evidence obligation, not a generic instruction to collect more data.

9.6 Change control

Treat a change to any of these as potentially safety-relevant:

• Model weights.
• Dataset mixture.
• Prompt.
• Action normalization.
• Control rate.
• Camera or lens.
• Calibration.
• Robot firmware.
• Payload.
• End effector.
• Site map.
• Allowed object set.

Run impact analysis and regression before rollout. Log the complete release identity on every episode.

10. Commercial reality

10.1 Maturity map

Category	2026 status	Why
Fixed industrial arms	Mature	High repetition, engineered cells, established safety and service
Autonomous mobile robots	Mature in bounded facilities	Mapped floors, fleet orchestration, human-aware navigation
Autonomous mining haulage	Commercial at scale on controlled mine sites	Fixed routes, managed traffic, large vehicle economics, and site-level operations
Agricultural guidance and task automation	Commercial for selected crops and tasks	Valuable seasonal work; field variability, weather, and service still narrow the ODD
Surgical robots	Mature in defined procedures	Strong workflow integration and trained human authority
Drones	Commercial at scale in selected inspection, mapping, defense, and delivery operations	Bounded missions and airspace rules
Surface and underwater survey robots	Established in specialized science and industrial work	Mission planning and sensing work; launch, recovery, weather, and intervention remain expensive
Space robotics	Operational for mission-specific exploration and propulsion-based servicing	Deep qualification and bounded missions; cost, communication delay, and lack of repair prevent generality
Robotaxis	Commercial in bounded cities	Huge sensing, mapping, operations, and safety investment
Learned manipulation	Early commercial and strong research	Works on bounded object and task distributions
Humanoid factory and logistics work	Early paid deployments and pilots	Useful narrow tasks; fleet economics remain opaque
General household robot	Unproven	Long-tail tasks, safety, privacy, cost, and service
Human-level general physical intelligence	Research goal	No credible delivery date

10.2 Industrial scale is already large

The International Federation of Robotics reports:

• 542,000 industrial robots installed during calendar 2024.
• More than 500,000 annual installations for the fourth straight year.
• 4.664 million industrial robots operating at the end of 2024.
• Asia accounted for 74% of 2024 installations.

These are industrial robots, not humanoids.

The IFR service-robot report reports almost 200,000 professional service robots sold during 2024 from its sample of 294 suppliers. Of those sampled sales, 102,900 were transportation and logistics robots, mainly indoor goods transport and handling. The sample is not a census of every supplier.

For a regional demand signal, the Association for Advancing Automation reports 36,766 robots worth $1.25 billion ordered in North America during calendar 2025 from a consistent reporting cohort. Orders are not installations, active robots, or productive utilization.

10.3 Deployment evidence ladder

Following the technical evaluation rungs in Section 8.1, this ladder measures commercial status:

Do not collapse these categories into "deployed."

10.4 Named humanoid evidence

Program	Public status	What the evidence proves	What remains unknown
Figure 02 at BMW Spartanburg	Completed production pilot	Figure reports an 11-month deployment, more than 1,250 hours, more than 90,000 parts, and contribution to more than 30,000 X3 vehicles	Fleet economics, intervention rate, and independent uptime
Figure 03 at BMW Spartanburg	Logistics sequencing deployment starting	BMW describes the next bounded use case and says Figure 03 will start at Spartanburg	Long-duration outcome is not yet public
Figure 03 package-sorting run	Company demonstration, not a named customer deployment	Sherwood reports 249,560 packages over a 200-hour livestreamed run, with robots rotating for charging every three to four hours; a human won the initial 10-hour comparison by about 190 packages	Customer economics, independent failure telemetry, and whether the same result transfers to another line
Agility Digit at GXO	Paid multi-year robots-as-a-service at one site	GXO's agreement and Agility's 100,000-tote milestone establish commercial activity	Site count, intervention, maintenance, cost per tote, network scale
Apptronik Apollo at Mercedes-Benz	Customer pilot program	Mercedes-Benz describes evaluation in manufacturing logistics	Sustained production metrics
Walden Robotics at a North American Toyota plant	Company-reported production work since February 2026	Walden says the wheeled semi-humanoid moved from pilot to production tasks in under two months; Toyota's CTO is quoted in the release	Public task counts, intervention, uptime, customer-confirmed economics, and cross-site repeatability
AgiBot G2 at Longcheer Technology in Nanchang	Vendor and customer-representative report of live tablet-test-line work in April 2026	AgiBot reports multiple robots, up to 310 units per hour, more than 99% success, and more than 140 cumulative hours at announcement	Independent intervention and downtime audit, paid terms, and cross-site repeatability
Tesla Optimus	Internal program and demonstrations	Material investment and development	Verified external paid fleet and published operating metrics
1X NEO	Consumer program, preorders, and teleoperation-assisted development	Consumer interest and a human-assistance path	Delivered autonomous household service at scale
Chinese humanoid makers	Manufacturing and deployment programs	Fast hardware iteration and lower-cost supply chain	Comparable public uptime, intervention, and unit-economic data

10.5 Non-humanoid businesses are the benchmark

The field should compare humanoids with existing alternatives:

• Industrial arms and custom cells.
• Conveyors and sorters.
• AMRs.
• Surgical systems.
• Warehouse automation.
• Drones.
• Robotaxis.
• Human labor plus ergonomic tools.

The question is not "can a humanoid do it?" It is:

Does this system beat the best alternative on total cost, throughput, safety, flexibility, and deployment time?

The strongest public deployment denominators remain specialized systems:

Operator	Reported operating scale	Evidence boundary
Amazon Robotics	Amazon reports more than 1 million robots across more than 300 facilities	Company-reported fleet count; establishes scale, not one robot type's standalone return
Intuitive Surgical	Its audited 2025 Form 10-K reports $10.0647 billion revenue and about 3.153 million da Vinci procedures during 2025	Regulated financial and procedure evidence; the surgeon remains in authority and the workflow is highly controlled
Locus Robotics	Locus reports more than 6 billion robot-assisted picks across more than 350 sites	Vendor-reported cumulative work; not independently audited margin
Chef Robotics	Chef reports 100 million production servings across more than a dozen customer facilities	Vendor-reported production count; no public site-level margin or intervention denominator
Komatsu FrontRunner	Komatsu reports 1,000 commissioned ultra-class autonomous haul trucks and more than 11.5 billion metric tons of material moved	Vendor-reported mine-haulage scale across North America, South America, Australia, and Europe; does not expose site-level intervention or margin
Zipline	Zipline reports more than 2 million commercial deliveries and more than 125 million autonomous commercial miles	Vendor-reported aerial delivery scale in approved networks; does not prove unrestricted operations
NOAA 2025 Atlantic hurricane USV mission	NOAA documents seven Oshen C-Stars and one Chance MC40 collecting and transmitting ocean and weather observations for operational forecasts	Named public-science mission with remotely supervised USVs; not commercial fleet economics or unattended shipping
Oceaneering Freedom	Oceaneering and TotalEnergies report an autonomous underwater pipeline-inspection pilot covering more than 120 km	Named industrial task and customer; still a pilot, not a resident multi-site fleet or audited unit economics
Northrop Grumman SpaceLogistics	MEV-1 and MEV-2 docked with commercial satellites and provide mission extension	Flown orbital rendezvous, docking, and life extension; broader robotic repair and assembly remain a later system

These systems win through a bounded workflow, integration, service, and repeated use. Their form factor is secondary to the complete operating system around the task.

11. Companies, labs, and where to pay attention

11.1 Model and research organizations

Organization	Main bet	Evidence to watch
Physical Intelligence	Generalist VLA policies, flow matching, deployment feedback, context steering	Open releases, cross-site evaluation, customer operations
Google DeepMind Robotics	VLA execution plus embodied reasoning, planning, and success detection	Trusted-tester results and API boundaries
NVIDIA GEAR and Isaac	Models, human video, simulation, synthetic data, edge deployment	Independent real-robot results outside NVIDIA stack
Toyota Research Institute	Careful large behavior model studies and robot learning	Target fine-tuning efficiency and long-duration transfer
Walden Robotics	Deployment-first large behavior models on wheeled semi-humanoids	Customer-confirmed production metrics and cross-site replication
Generalist AI	Large proprietary real-manipulation corpus and cross-embodiment scaling	Public checkpoints, independent corpus audit, and customer outcomes
Figure AI	Proprietary full-body hierarchy on humanoids	Customer metrics, intervention, uptime, and safety
Skild AI	Proprietary cross-embodiment "robot brain"	Named deployments and reproducible technical evidence
Meta FAIR	Joint embedding predictive architectures and tactile/open hardware research	Control results, not only representation benchmarks
World Labs and world-model startups	Spatial world generation and prediction	Action-conditioned physical validity

11.2 Academic and public research centers

• MIT CSAIL and Russ Tedrake: control, manipulation, planning, learning, Drake.
• Berkeley Robot Learning: imitation, RL, manipulation, datasets.
• Stanford IRIS: robot learning and generalization.
• CMU Robotics Institute: perception, autonomy, manipulation, field robotics.
• UPenn GRASP: robotics, perception, control, multi-robot systems.
• ETH Robotic Systems Lab: legged robots and field systems.
• UC San Diego ManiSkill: simulation and manipulation.
• Toyota Research Institute Robotics: home and assisted robotics research.
• Woods Hole Oceanographic Institution: underwater vehicles, ocean sensing, and long-duration field science.
• Monterey Bay Aquarium Research Institute: marine operations, ROVs, AUVs, ASVs, and ocean observatories.
• NASA Jet Propulsion Laboratory Robotics: planetary mobility, manipulation, autonomy, and mission systems.

11.3 Hardware and deployment companies

Humanoids:

• Agility Robotics.
• Apptronik.
• Boston Dynamics.
• Figure AI.
• 1X.
• Tesla.
• Unitree.
• UBTECH.
• AgiBot.
• Fourier Intelligence.

Other high-signal categories:

• Industrial: ABB, FANUC, KUKA, Yaskawa, Universal Robots.
• AMR and warehouse: Amazon Robotics, Locus, Symbotic, Geek+, GreyOrange.
• Manipulation: Dexterity, Ambi Robotics, Covariant lineage.
• Surgical: Intuitive Surgical.
• Autonomous driving: Waymo.
• Outdoor land and heavy equipment: Komatsu, Caterpillar, John Deere, Built Robotics, Field AI.
• Aerial delivery and inspection: Zipline, Wing, Skydio.
• Surface-water autonomy: Saildrone, Oshen, Fugro, Ocean Infinity, Sea Machines, Kongsberg Maritime.
• Marine autonomy and operations: Oceaneering, Exail, Saab, Blue Robotics, Anduril, Ulysses.
• Space robotics: NASA/JPL, Northrop Grumman SpaceLogistics, Astrobotic, Intuitive Machines, ispace, GITAI.
• Robot data and observability: Foxglove, Formant, Alloy.
• Robot data collection: XDOF and other data foundries.

This is a watch list, not an endorsement or a maturity ranking. Watch accepted work, intervention, fleet age, maintenance, permits, and customer outcomes more than announcement volume.

12. Economics, incentives, supply, and startup opportunities

12.1 The customer equation

annual value
  = labor and injury cost avoided
  + throughput and quality value
  + flexibility value

annual cost
  = lease or depreciation
  + integration
  + maintenance and spare parts
  + energy
  + downtime
  + supervision and teleoperation
  + safety and regulatory compliance
  + domain operations such as airspace, support vessel, ground station, or launch and recovery
  + workflow change

buy when risk-adjusted annual value > risk-adjusted annual cost

A robot price is not total cost of ownership. A wage is not total human labor cost. Compare complete systems over the same work definition and utilization window.

12.2 Why robots stay in pilot purgatory

• The task is too variable.
• The gripper is wrong.
• Integration takes longer than expected.
• Human exceptions dominate.
• Reset and recovery are manual.
• Launch, landing, docking, or recovery needs more people and equipment than the business case assumed.
• Weather, positioning, or communications make the usable operating window too small.
• The robot cannot meet cycle time.
• Safety controls remove the promised speed.
• Maintenance and spares are weak.
• The customer has no internal owner.
• The vendor measures technical success, not business value.

Fast development is not durable deployment. Amazon said Blue Jay moved from concept to a South Carolina production test in just over a year and could handle about 75% of item types there. The same official page now says Amazon stopped using Blue Jay in operations by February 2026 while retaining its underlying technology. A production test can still be discontinued.

12.3 Supply and demand

Oversupplied:

• Humanoid announcements.
• Polished general-purpose demos.
• Generic VLA checkpoints.
• Unverified total-addressable-market forecasts.
• Manufacturing-capacity targets.
• Preorders and memoranda presented as deployments.

Undersupplied:

• Brownfield integration with warehouse management systems (WMS), manufacturing execution systems (MES), enterprise resource planning (ERP), and safety systems.
• Reliable task-specific end effectors.
• Multi-vendor fleet operations.
• Failure recovery and human escalation.
• Long-duration evaluation.
• Dataset lineage and quality.
• Safety-case and change-control tooling.
• Field engineers who understand software and machinery.
• Service networks and spare-parts operations.
• Outdoor autonomy evidence by terrain, weather, and season.
• Drone airspace, detect-and-avoid, dock, battery-health, and flight-log infrastructure.
• Marine remote operations, launch and recovery, subsea docking, corrosion management, and mission replay.
• Space-grade fault management, delayed operations, qualification, and servicing interfaces.

12.4 Startup wedge map

These rows are not ordered. They are thesis prompts, not scored market sizing:

Wedge	Buyer	Moat	Fatal risk
Brownfield robot orchestration and integration	Factory, warehouse, integrator	Connectors, site knowledge, workflow history	OEMs bundle enough functionality
Fleet reliability and remote-assist control plane	Robot operator and RaaS vendor	Cross-fleet failure and recovery data	Vendors keep systems closed
Robot-data quality, provenance, and evaluation	Model lab, OEM, large operator	Failure-linked data and regression history	Data formats commoditize
Safety and regulatory-compliance workflow	OEM, integrator, operator, insurer	Audit trail, application templates, accepted evidence	Slow sales and liability
Vertical robot application	Specific industry workflow	Distribution, domain integration, outcomes	Pilot purgatory and weak margins
Application-specific end-effector kit	Integrator and operator	Mechanics, recipes, distribution	Commodity pressure
Independent real-robot evaluation network	Buyers, labs, insurers	Cross-site benchmark and trusted results	Labs resist external evaluation
General humanoid OEM	Large strategic customers	Manufacturing, fleet data, service, brand	Extreme capital and time before repeatability

12.5 The likely billion-dollar categories

No category is guaranteed. This is a thesis list, not a revenue forecast:

• Robot operations platform: job orchestration, telemetry, incident replay, teleoperation, maintenance, release control.
• Data and evaluation system: capture, synchronization, calibration, lineage, failure mining, regression, deployment feedback.
• Vertical automation company: owns one painful workflow and sells completed work, not a robot.
• Safety assurance infrastructure: hazards, evidence, model-change impact, cyber, incident records.
• End-effectors and tactile systems: reliable application-specific manipulation.
• Simulation-to-field infrastructure: turns real failures into reproducible scenarios and gates releases.
• Embodiment adaptation: maps policies and data across bodies while preserving semantics and safety limits.
• Mission operations and robotic logistics: docks, charging, launch and recovery, remote operations, maintenance, and compliance for field robots.

Weak theses:

• Generic annotation labor with no deployment loop.
• "Foundation model for every robot" with no exclusive data or customer.
• A humanoid clone differentiated by a demo.
• A world model measured only by video quality.
• A marketplace that assumes robot data is fungible across embodiments.

12.6 Funding is a signal, not proof

Large rounds establish investor willingness to finance the field. They do not establish:

• Product reliability.
• Revenue quality.
• Gross margin.
• Paid fleet size.
• Safety.
• General intelligence.

Prefer official financing announcements, but distinguish cash received, committed capital, milestone-contingent amounts, and press-reported valuations. Treat "in talks," secondary-market marks, and database estimates as unverified.

Selected financing announcements show where investors are placing large bets:

Organization	Announced financing as of 2026-07-20	What it proves and does not prove
Figure AI	More than $1 billion of committed Series C capital at a stated $19 billion post-money valuation, September 2025	Investor commitments for hardware, compute, and data collection; the announcement does not prove all cash was received or establish fleet economics
Skild AI	About $1.4 billion at a valuation above $14 billion, January 2026	Investor demand for a cross-embodiment model; capability and revenue claims remain company reported
Physical Intelligence	CapitalG confirms it led the Series B; Bloomberg reports $100 million at a $1.6 billion post-money valuation, November 2025	Investor and round are investor-confirmed; amount and valuation are press-reported, and customer economics are undisclosed
Apptronik	More than $135 million total Series A, February 2026	Capital to scale Apollo production and pilots; no valuation or repeatable deployment result disclosed
Generalist AI	$100 million round, with Bloomberg reporting a $1 billion post-money valuation, June 2026	Capital for model, data, and compute scale; no independent validation of the proprietary corpus
Walden Robotics	$100 million seed at a stated $1.1 billion post-money valuation, July 2026	Capital plus a company-reported Toyota production relationship; operating denominators remain private
NEURA Robotics	Up to $1.4 billion Series C, June 2026; CNBC reports the full amount is milestone-contingent	Investor appetite for a full-stack robotics platform; "up to" is not cash received and does not prove deployment or unit economics
Foxglove	$10 million Series B, November 2025	Capital for multimodal robot-data infrastructure; customer and performance claims are vendor reported
XDOF	$10 million launch financing, June 2026	Investor demand for collection, cleaning, tooling, and annotation; named buyers and vendor margins are not public
Ulysses	$16 million across an $1 million seed and $18 million Series A, with the Series A led by a16z, April 2026	Capital for modular underwater and surface systems; public evidence does not yet establish the proposed fleet scale, zero-human operations, or unit economics

Do not compare aggregate "robotics funding" totals without matching category scope. Trackers variously include autonomous vehicles, drones, industrial automation, hardware, and model labs.

13. Geography and China's role

13.1 China's verified industrial position

The International Federation of Robotics reports that China accounted for 54% of global industrial-robot installations in 2024 and that Chinese manufacturers held 57% of the Chinese domestic market that year. This is strong evidence of manufacturing and deployment scale in industrial robotics.

For humanoids, Interact Analysis estimates that global 2025 production exceeded 20,000 units, more than 90% came from Chinese vendors, and only around 10% of produced units entered real-world applications. The rest primarily served research, data collection, and entertainment. This is a market-research estimate, not an audited census, but it shows why manufactured, shipped, and productively deployed must remain separate counts.

13.2 Structural strengths

• Dense motors, drives, batteries, electronics, and machining supply chains.
• Fast prototype-to-production cycles.
• Large domestic manufacturing base.
• Policy and financing support.
• Lower hardware prices.
• Access to many factory workflows.

13.3 Open questions

• Productive utilization versus shipped units.
• Long-duration autonomy.
• Cross-site software quality.
• Safety and certification.
• Service economics.
• Export restrictions and geopolitical risk.
• Comparable public metrics across companies.

13.4 Other ecosystems

• United States: frontier AI, venture capital, autonomous driving, warehouse, defense, and research labs.
• Europe: industrial automation, safety engineering, machine builders, research, and integrators.
• Japan: industrial robotics, precision manufacturing, aging-population demand, and service-robot research.
• South Korea: electronics, automotive manufacturing, and strategic robot investment.

The likely winners combine software, manufacturing, deployment access, and service. No region owns all four.

14. Timelines: how many more years?

14.1 Define the capability before predicting it

14.2 Scenario ranges

Capability	Bull case	Base case	Bear case
More narrow paid humanoid workflows	2026-2028	2026-2030	Pilots remain isolated
Repeatable multi-task industrial cells	2028-2031	2030-2035	Later 2030s
General industrial humanoid across many sites	Early 2030s	Mid-to-late 2030s	Not economically dominant
Useful general household robot	Around 2030	2030s or later	2040s or specialized devices instead

These are scenarios, not consensus forecasts.

The environment-specific picture is different because narrow field robots already work:

Environment	Real by the 2026-07-20 cutoff	Plausible 2028-2031 direction	Still unproven
Outdoor land	Controlled mine haulage, precision guidance, selected spraying, survey, and inspection	More mixed fleets, better traversability, remote assist, and task automation on bounded sites	One policy that safely handles arbitrary terrain, weather, machinery, and public roads
Air	Mapping, inspection, agriculture, defense, and approved delivery networks	More beyond-visual-line-of-sight networks, automated docks, detect-and-avoid evidence, and fleet supervision	Unrestricted all-weather dense-urban flight without material ground infrastructure or regulatory limits
Surface water	Ocean sensing, hydrography, surveillance, survey, and bounded remote operation	More multi-vessel supervision, endurance, satellite integration, and surface-to-subsea teaming	Fully unattended global merchant shipping across ports, weather, law, maintenance, and emergencies
Underwater	Tethered intervention, autonomous survey, science, pipeline and cable inspection	More hybrid vehicles, resident docks, repeated local missions, and lower-cost surface support	General manipulation at depth or tens of thousands of vehicles operating with negligible human and vessel support
Space	Rover local navigation, robotic arms, rendezvous, docking, and satellite life extension	More NASA-planned lunar surface logistics, inspection, and servicing demonstrations	Routine economical repair, refueling, and large-scale assembly across arbitrary spacecraft

These are directional scenarios. Regulation, insurance, support infrastructure, maintenance, and customer economics can move slower than the autonomy software.

Public estimates disagree because they predict different capabilities:

Forecast or claim	Exact scope and boundary
Sergey Levine: about a five-year median, or roughly 2030 from the interview date	A machine that autonomously runs a household as well as a human housekeeper; personal forecast from an optimistic foundation-model builder
Rodney Brooks: more than ten years	First profitable humanoid deployment with even minimal dexterity; argues tactile data is missing and expects wheeled, specialized bodies
Ken Goldberg: not in two, five, or even ten years	Broad humanoid and dexterity timeline skepticism; outperforming human surgeons is one example in the interview
Interact Analysis: commercial inflection after 2032	Market analyst forecast for large-scale real-world humanoid applications, conditional on autonomy, reliability, return on investment, and regulation

Metaculus centers near October 2030 for its specific operational definition as of 2026-07-20. Its wide distribution demonstrates uncertainty, not scientific consensus.

14.3 What would move the timeline forward

• Durable dexterous hands and tactile sensing.
• Large improvement in intervention-free operating hours.
• Cheap service and repair.
• Better recovery.
• Cross-site evaluation.
• Clear safety standards and evidence.
• Customer workflows with repeatable economics.
• Deployment data feeding the next model.

14.4 What would slow it

• Hardware reliability plateau.
• Data transfer failing across embodiments.
• Safety incidents.
• Capital contraction.
• Weak unit economics.
• Teleoperation cost hidden inside "autonomy."
• Regulation or insurance barriers.
• Customers choosing simpler automation.

15. How to detect hype

Start with the minimum credible robot report, then ask:

1. Is this simulation, lab hardware, pilot, contract, or sustained production?
2. How many robots and continuous operating hours produced the result?
3. Who reports the number, and is there a named customer?
4. What integration, maintenance, and post-failure work were required?
5. What is the fully loaded cost per accepted work unit?
6. Does a simpler machine solve the same job?

Red-flag translations:

Claim	Translate it into a question
"Autonomous"	How many interventions per operating hour?
"General-purpose"	Which tasks, objects, sites, and embodiments were held out?
"Production-ready"	What are uptime, cycle time, maintenance, and safety-stop rates?
"Human-level"	Which measured task distribution and human baseline?
"Cheaper than labor"	What is fully loaded cost per accepted unit of work?
"Simulation solved it"	What paired physical evaluation validates the simulator ranking?
"Data moat"	Which rights, environments, failures, outcomes, and recollection loop are unique?
"Safe around people"	Which application risk assessment, limits, tests, and incident history?
"Thousands of robots"	Manufactured, shipped, delivered, active, or productively operating?

16. Breaking into robotics from software and data

16.1 Best entry routes

For someone already strong in backend, infrastructure, data, deployment, and observability:

Route	Transferable advantage	Missing robotics skills
Robot data and evaluation	Schemas, pipelines, lineage, analytics, testing	Time synchronization, calibration, episodes, physical metrics
Simulation infrastructure	Distributed compute, CI, environments, observability	Dynamics, assets, sensor models, sim correlation
Fleet reliability	SRE, incident response, rollout, telemetry	Robot fault modes, maintenance, safety states
Robotics platform software	APIs, distributed systems, C++, deployment	ROS 2, real-time boundaries, frames, hardware
Perception	ML and data science	Geometry, calibration, sensor fusion
Robot learning	Deep learning and experimentation	Control, imitation, action spaces, real hardware
Controls and planning	Algorithms and systems	Dynamics, optimization, real-time C++
Frontier VLA research	ML research	Publications, large-scale training, robotics depth

Highest-probability route:

Enter through data, simulation, platform, evaluation, or fleet operations. Build enough mechanics, control, and hardware knowledge to avoid treating the robot like a web API.

16.2 What to learn first

1. Linear algebra and 3D transforms.
2. Kinematics and Jacobians.
3. Feedback control.
4. State estimation and calibration.
5. Motion planning.
6. ROS 2 and robot software.
7. Simulation.
8. Imitation learning.
9. Safety and evaluation.
10. VLAs and world models.

Do not start with a giant VLA checkpoint. Without frames, control, and evaluation, you cannot tell whether a failure is the model, calibration, controller, or hardware.

16.3 Twelve-month curriculum

Months 1-2: math and mechanics

• Work through Modern Robotics.
• Learn transforms, twists, forward and inverse kinematics, Jacobians, and basic dynamics.
• Implement a planar arm and a 3D pose library.

Deliverable:

• A small simulator that plots reachable workspace, singularities, and inverse-kinematics solutions.

Months 3-4: control and estimation

• Study Underactuated Robotics.
• Implement PID, LQR, and a Kalman filter.
• Learn impedance control conceptually and in simulation.

Deliverable:

• A controller comparison with disturbances, latency, saturation, and plots.

Months 5-6: ROS 2 and planning

• Complete ROS 2 Lyrical tutorials, or use Jazzy when the target robot vendor requires it.
• Learn TF2, URDF, topics, services, actions, lifecycle, quality of service (QoS), and rosbag2.
• Build with MoveIt 2 or Nav2.
• Use the Lyrical ros_gz stack with Gazebo Jetty, or the simulator named by the target robot's official MoveIt or Nav2 tutorial.

Deliverable:

• A simulated robot with launch files, transforms, planning, replayable logs, and automated tests.

Months 7-8: simulation and evaluation

• Choose MuJoCo or Isaac Lab.
• If you choose MuJoCo, use MuJoCo Playground for a reproducible MJX training and sim-to-real path. The paper reports state- and vision-based policies across six physical platforms in under eight weeks; this accelerates iteration but does not remove paired hardware evaluation.
• Build scene randomization.
• Define task success and failure categories.
• Run at least 100 seeded trials.

Current local gates:

Simulator path	Published requirement
MuJoCo Playground	Python 3.10 or newer and a CUDA 12 JAX GPU path for documented accelerated training; the repository does not present CPU or Mac training as the equivalent path
Isaac Lab	Ubuntu 22.04 or Windows 11, at least 32 GB RAM and 16 GB GPU memory, Python 3.11 for Isaac Sim 5.x, and a recommended Linux NVIDIA driver of 580.65.06 or newer

Deliverable:

• A reproducible benchmark report with confidence intervals and failure videos.

Months 9-10: imitation learning

• Use current LeRobot policy documentation as the executable path. The Hugging Face Robotics Course is a companion: as of 2026-07-20, its welcome, introduction, and classical-robotics units are released, while reinforcement learning, imitation learning, and foundation-model units remain marked "Coming Soon."
• Use a public dataset, or complete the SO-101 bring-up gate before recording teleoperated data.
• Use a LeRobot source environment and install its training extra, train ACT through lerobot-train, and add the diffusion extra only for a Diffusion Policy comparison.
• Add correction and recovery episodes.
• Compare against a scripted or planner baseline.

Deliverable:

• A baseline table that says where learning helps and where it does not.

Months 11-12: integrated deployment loop

Choose one capstone, not all three: a simulation-only policy evaluation; an SO-101 ACT, Diffusion Policy, or SmolVLA loop; or the full NVIDIA sim-to-real workshop.

• Fine-tune an open policy such as OpenVLA, pi0 or pi0.5, SmolVLA, or GR00T if hardware, compute, and model-access gates fit. pi0.7 remains paper-and-demo only unless Physical Intelligence releases checkpoints.
• Follow NVIDIA's SO-101 sim-to-real learning path after calibration and simulation basics. It covers teleoperation, domain randomization, simulated evaluation, real deployment, and actuator-gap tuning. The current workshop pins Isaac Sim 5.1.0, Isaac Lab 2.3.0, LeRobot 0.4.3, and GR00T N1.6; it requires Docker and an Ada- or Blackwell-generation NVIDIA GPU. Run hf auth login before downloading its models. Treat this as a pinned workshop track: do not mix its LeRobot data or commands with a newer host installation without an explicit conversion.
• For a custom robot, use the GR00T NEW_EMBODIMENT guide with GR00T's LeRobot v2 schema, meta/modality.json, and a declared modality configuration. Convert LeRobot v3 data with NVIDIA's supplied script. NVIDIA recommends at least one GPU with 40 GB or more VRAM for N1.7 fine-tuning; full visual or language-backbone tuning can require 80 GB or more per GPU.
• Add uncertainty or failure detection.
• Add human-assist escalation.
• Version data, model, config, and calibration.
• Run simulation regression before hardware.

Pick the model path from the hardware you actually have:

Policy path	Published local gate
SmolVLA	Current LeRobot requires Python 3.12 or newer and the smolvla extra; the model runs on a CPU and trains on one consumer GPU or a MacBook, but no quantified memory floor is published
OpenVLA	Python 3.10; LoRA fine-tuning needs about 27 GB GPU memory at minimum, while the documented default batch needs about 72 GB; the full fine-tuning example uses eight A100 GPUs
openpi	More than 8 GB for inference, 22.5 GB for LoRA, and 70 GB for full fine-tuning; Ubuntu 22.04 is the supported operating system
GR00T N1.7	Ubuntu 22.04 or newer, Python 3.12 and CUDA 12.6 or newer on a discrete GPU; at least 16 GB for inference and 40 GB for default fine-tuning; visual or language-backbone tuning needs 80 GB or more; downloading the Cosmos-Reason2-2B backbone requires Hugging Face login and license acceptance

Use a separate uv or conda environment for each branch. OpenVLA, Isaac Lab, current LeRobot, and GR00T do not share one Python and CUDA contract.

Deliverable:

• One end-to-end project with raw runs, interventions, safety limits, regression tests, and an honest failure taxonomy.

16.4 Four portfolio projects

1. Manipulation benchmark

   • Simulated pick-and-place.
   • 100 or more fixed-seed trials.
   • Position, object, lighting, and distractor perturbations.
   • Failure categories and confidence intervals.

2. Robot data validator

   • MCAP or LeRobot ingestion.
   • Timestamp, frame, calibration, rate, schema, and outcome checks.
   • Episode viewer and quality report.

3. Policy comparison

   • Scripted or planned baseline.
   • ACT.
   • Diffusion Policy or open VLA.
   • Same tasks, data split, hardware, and metrics.

4. Failure and recovery loop

   • Detect failure.
   • Stop safely.
   • Request human correction.
   • Save the corrected episode.
   • Retrain.
   • Gate with a frozen regression suite.

Add one environment branch to one project. Do not build five shallow demos:

Branch	Simulation-first project	Required failure and evidence
Outdoor land	Nav2, CARLA, or ArduPilot Rover route with terrain or visibility changes	GNSS degradation, obstacle, slip or blocked route; publish replay, recovery, and task denominator
Air	PX4 or ArduPilot SITL mission with a companion planner	Wind, sensor fault, low energy, or lost link; publish the autopilot log, failsafe transition, and landing result
Surface water	MOOS-IvP or ArduPilot boat mission with traffic encounters	Current, moving vessel, or lost communications; publish closest approach, rule decision, and safe fallback
Underwater	HoloOcean or ArduSub mission with sonar and dead reckoning	DVL dropout, acoustic delay, or navigation drift; publish uncertainty, abort condition, and recovery state
Space	NASA Astrobee simulation plus an externally injected command delay	Stale command, blocked path, or low energy; publish local decision, safe mode, and delayed-ground timeline

Before training a physical-arm portfolio project, publish one calibration repeatability check: place the leader and follower in the same physical pose several times and show that joint readings remain consistent within a declared tolerance.

16.5 Hardware ladder

You do not need a humanoid.

1. Laptop plus simulation.
2. Webcam, AprilTag, and small mobile base or servo kit.
3. Low-cost arm such as SO-101.
4. Dual-arm leader-follower setup if the project needs bimanual data.
5. Research or industrial arm only when the workflow justifies the cost and safety process.

SO-101 bring-up gate

The low-cost path spans two repositories:

1. Use TheRobotStudio SO-ARM100 repository for printing, mechanical assembly, and the hardware bill of materials.
2. Use the LeRobot SO-101 guide for motor, calibration, teleoperation, recording, and policy workflows.
3. Use the LeRobot source path with Python 3.12 or newer. Install its core_scripts and feetech extras, identify each serial port with lerobot-find-port, and configure one motor at a time with lerobot-setup-motors. On Linux, the current LeRobot source install defaults to CUDA 12.8 PyTorch wheels with a 570.86 or newer driver unless you select another backend.
4. Calibrate leader and follower with stable IDs through lerobot-calibrate. Those IDs select the saved calibration in later runs.
5. Check repeatable joint state, motion direction, range, current limits, stop, and leader-follower tracking before recording demonstrations.
6. Before training, add the training extra plus the selected policy extra, such as diffusion or smolvla.

Do not debug a VLA until the same physical pose produces consistent state and the leader-follower pair can repeat a slow scripted motion.

The full NVIDIA workshop additionally uses wrist and external cameras, a standardized lightbox and props, and an Ada- or Blackwell-GPU Docker host. A leader arm is needed when collecting your own teleoperated demonstrations.

16.6 Portfolio quality bar

Publish:

• Every field in the minimum credible robot report.
• Problem and task boundary.
• System diagram.
• Bill of materials.
• Reproducible setup.
• Data schema.
• Baseline.
• Latency and compute profile.
• Model and data licenses.
• What did not work.

One integrated artifact beats ten disconnected notebooks.

16.7 Job titles to search

• Robotics software engineer.
• ROS 2 engineer.
• Manipulation engineer.
• Motion planning engineer.
• Controls engineer.
• Perception or SLAM engineer.
• Simulation engineer.
• Robot data engineer.
• Robot learning engineer.
• Autonomy evaluation engineer.
• Fleet reliability engineer.
• Field robotics engineer.
• Flight-software or aerial-autonomy engineer.
• Marine robotics or subsea-autonomy engineer.
• Guidance, navigation, and control engineer.
• Mission-systems or remote-operations engineer.
• Deployment engineer.
• Functional safety engineer.
• Teleoperation systems engineer.

Current postings repeatedly ask for C++, Python, ROS 2, Linux, simulation, motion planning, calibration, sensor integration, debugging, and real-world deployment. Research roles add PyTorch or JAX, imitation learning, RL, and publications.

16.8 Communities

• ROS Discourse.
• Robotics Stack Exchange.
• LeRobot GitHub.
• MoveIt GitHub.
• Nav2 GitHub.
• PX4 GitHub and forums.
• ArduPilot community.
• MOOS-IvP.
• Blue Robotics community.
• NASA Astrobee GitHub.
• ICRA, RSS, CoRL, IROS, NeurIPS robotics workshops.
• RoboCup and local robotics groups.

Contribute a bug fix, driver, benchmark, or reproducible negative result. That is stronger evidence than a certificate.

17. A personal operating system for staying current

Weekly

• Read one paper deeply.
• Watch one unedited hardware run.
• Review one customer or regulator source.
• Track one open-source repository release.
• Write one claim with its evidence boundary.

Monthly

• Re-run one benchmark.
• Add one failure case to a personal taxonomy.
• Compare one vendor claim with a customer source.
• Update the model, dataset, and standards tables.
• Build or repair something physical.

Quarterly

• Re-rank startup opportunities.
• Refresh deployment evidence.
• Recheck which models have open weights.
• Recheck standards status.
• Publish one integrated project update.

Source order

1. Paper, model card, repository, standard, regulator, or customer.
2. Independent technical evaluation.
3. Credible reporting.
4. Vendor release.
5. Founder interview.
6. Forum as a lead, never final proof.

Appendix A. Claim adjudication

Claim	Verdict as of 2026-07-20	Reason
"Robotics has reached its ChatGPT moment"	Unproven	Models improved, but there is no broad, low-friction, mass-market product equivalent
"Locomotion is solved"	False as a universal claim	Strong on many platforms and bounded terrains; endurance, payload, safety, and unfamiliar terrain remain system problems
"Data is the only bottleneck"	False	Hardware, dexterity, integration, reliability, safety, service, and economics also bind
"More robot data always helps"	False	Mixture quality, embodiment mismatch, labels, and negative transfer matter
"Generic robot labeling is the next Scale AI"	Weak thesis	Full data, quality, evaluation, and deployment feedback is stronger
"Gemini Robotics-ER 1.6 is a motor policy"	False	The preview API outputs text for physical reasoning, planning, and tool use
"GR00T N1.7 is unavailable"	False	Code and weights are public; release metadata contains GA versus Early Access inconsistency
"ISO 25785-1 is a finished humanoid-only standard"	False	It is an unpublished committee draft at ISO stage 30.60 and covers dynamically stable industrial mobile robots with legs, wheels, or other locomotion
"BMW proves fleet-scale humanoid economics"	False	It proves a meaningful bounded production pilot, not fleet economics
"GXO proves network-wide humanoid scale"	False	It proves a paid one-site workflow and cumulative tote volume
"A shipped robot is a productive deployment"	False	Delivery, active operation, paid work, and economic value are different states
"Simulation removes the need for real data"	False	The residual gap is largest in contact-rich and operational details
"General household robots will arrive in 2030"	Unproven	2030 is one forecast center under a specific definition, not consensus
"Large funding rounds prove product maturity"	False	Capital is an input; deployment evidence is the output
"An autonomous drone can fly any route without human or regulatory approval"	False	Autopilot capability and legal operating authority are separate; airspace, people, weather, lost-link behavior, and jurisdiction still constrain the mission
"An autonomous underwater vehicle removes the ship and human-operations cost"	False as a universal claim	Untethered motion can remove continuous piloting, but launch, recovery, communications, recharge, weather, maintenance, and support vessels can still dominate cost
"Ulysses has publicly proven 72-hour Mako endurance"	Unproven	Its home page says up to 72 hours while its technology page says up to 12 hours; the configuration and independent qualification evidence are not public

Appendix B. Glossary

• Action chunk: a short sequence of future commands predicted together.
• Actuator: component that converts energy into motion.
• AIS: Automatic Identification System, a maritime broadcast system for vessel identity, position, course, and related data.
• AMR: autonomous mobile robot, usually a wheeled robot that navigates a facility.
• ASV: autonomous surface vessel, a robot that operates on the water surface.
• AUV: autonomous underwater vehicle, an untethered underwater robot that executes a mission without continuous piloting.
• ArduPilot: open autopilot software for aircraft, rovers, boats, and underwater vehicles.
• Backlash: mechanical free play when motion reverses.
• Behavior cloning: supervised learning from demonstrated actions.
• BlueOS: browser-based software environment for configuring and extending Blue Robotics vehicles.
• BVLOS: beyond visual line of sight, a flight in which the remote pilot cannot keep the aircraft in unaided visual view.
• Calibration: estimating parameters that connect sensors, frames, timing, and hardware.
• Closed loop: behavior that measures results and corrects future actions.
• Configuration space: all possible robot joint configurations.
• Controller: algorithm that turns desired state into low-level commands.
• Mechanical compliance: physical or controlled yielding under contact.
• Regulatory compliance: satisfying applicable laws, standards, and assurance obligations.
• Covariate shift: the robot enters states not represented in training data.
• DDS: data distribution service, the middleware family used by ROS 2.
• Degrees of freedom: independent coordinates the robot can move.
• Diffusion policy: policy that generates actions through iterative denoising.
• DVL: Doppler velocity log, an acoustic sensor that estimates underwater velocity relative to the seabed or water.
• Embodiment: a robot's body, sensors, actuators, and action interface.
• End effector: tool or gripper at the end of a robot arm.
• Episode: time-aligned sequence of observations, states, actions, and outcomes.
• Flight controller: real-time computer that estimates vehicle state, stabilizes motion, runs modes, and sends commands to aircraft actuators.
• Flow matching: generative method that learns a continuous path from noise to data.
• Forward kinematics: compute end-effector pose from joint positions.
• Functional safety: engineering discipline for safety functions that depend on control systems.
• GNSS: Global Navigation Satellite System, the family that includes GPS and other satellite navigation systems.
• Impedance control: control that shapes the relationship between motion and force.
• Intervention: human action that changes, rescues, or completes robot behavior.
• Inverse kinematics: compute joint positions for a desired end-effector pose.
• Jacobian: local mapping between joint motion and task-space motion.
• Kinematics: geometry of motion without forces.
• Localization: estimate the robot's pose in a map.
• Manipulation: physically changing objects with a robot.
• MAVLink: a message protocol for communication among vehicles, ground stations, and companion computers.
• MOOS-IvP: modular autonomy software and behavior arbitration used mainly in marine robotics.
• Motion planning: find a feasible collision-free path.
• IMU: inertial measurement unit, which measures angular velocity and acceleration.
• LQR: linear-quadratic regulation, a model-based optimal-control method near an operating point.
• ODD: operational design domain, the conditions in which operation is allowed.
• Policy: mapping from observations and goals to actions.
• PID: proportional-integral-derivative feedback control.
• PX4: open flight-control and autopilot software for drones and other vehicles.
• Proprioception: sensing the robot's own state.
• RaaS: robots as a service, usually a recurring contract for operating capability.
• Receding-horizon execution: repeatedly plan a short future, execute only part of it, observe again, and replan.
• Recovery: behavior that returns from failure to progress or a safe state.
• ROV: remotely operated vehicle, commonly an underwater robot piloted through a tether.
• ROS 2: robotics middleware and ecosystem, not a complete robot operating system.
• SE(3): special Euclidean group for 3D rigid-body rotation and translation.
• Safety envelope: independent constraints that limit or stop dangerous behavior.
• Sim-to-real: transferring behavior learned or tested in simulation to hardware.
• SLAM: simultaneous localization and mapping.
• State estimation: infer hidden state from models and noisy measurements.
• Tactile sensing: sensing contact and pressure over a surface.
• Teleoperation: a human remotely controls the robot.
• Trajectory: time-indexed path with position, velocity, and sometimes acceleration.
• URDF: XML format describing robot links, joints, geometry, and related properties.
• USV: uncrewed surface vessel, a surface-water vehicle without people onboard; autonomy level varies.
• VLA: vision-language-action model, which maps multimodal inputs to robot actions.
• VLM: vision-language model, which processes visual and language inputs.
• World model: action-conditioned predictor of task-relevant future state; not the same as a current state estimate.

Appendix C. High-value source library

The curated library is grouped by prerequisite and use. It favors primary sources and durable documentation over news volume.

C.1 Foundations, mechanics, control, and planning

1. Modern Robotics textbook and course
2. MIT Underactuated Robotics
3. MIT Robotic Manipulation
4. Planning Algorithms by Steven LaValle
5. Probabilistic Robotics
6. Reinforcement Learning: An Introduction
7. UC Berkeley CS 185/285 Deep Reinforcement Learning
8. Stanford CS223A Introduction to Robotics
9. Drake tutorials
10. Controls Engineering in the Python Control Systems Library

C.2 Robot software, middleware, and operations

1. ROS 2 Lyrical documentation and Jazzy LTS documentation
2. ROS 2 Lyrical tutorials
3. ROS 2 design documents
4. ROS 2 quality of service concepts
5. TF2 concepts
6. ros2_control documentation
7. rosbag2 repository
8. MCAP
9. MoveIt 2 documentation
10. MoveIt 2 repository
11. Nav2 documentation
12. Nav2 repository
13. BehaviorTree.CPP
14. LeRobot repository
15. LeRobot documentation
16. LeRobotDataset v3
17. OpenCV
18. Point Cloud Library
19. GTSAM factor-graph library
20. ORB-SLAM3 repository

C.3 Simulation and synthetic environments

1. MuJoCo documentation
2. MuJoCo repository
3. MuJoCo Playground
4. NVIDIA Isaac Sim
5. Isaac Lab
6. Gazebo
7. Drake
8. ManiSkill
9. RoboCasa
10. AI Habitat
11. BEHAVIOR-1K
12. Genesis documentation
13. Newton physics engine
14. PyBullet
15. Brax
16. MimicGen
17. SIMPLER

C.4 Robot-learning papers and model projects

1. RT-1 paper
2. RT-2 paper
3. RT-2 overview
4. PaLM-E paper
5. SayCan project
6. Open X-Embodiment and RT-X
7. RoboCat
8. Octo project
9. Octo paper
10. OpenVLA project
11. OpenVLA paper
12. OpenVLA-OFT
13. Diffusion Policy
14. Diffusion Policy paper
15. ACT and ALOHA
16. Mobile ALOHA
17. pi0
18. pi0 paper
19. pi0.5
20. pi0.6 and RECAP
21. pi0.7
22. pi0.7 paper
23. openpi repository
24. NVIDIA GR00T N1 paper
25. NVIDIA Isaac GR00T repository
26. GR00T N1.7 model collection
27. Gemini Robotics models
28. Gemini Robotics launch
29. Gemini Robotics-ER 1.6 API
30. Gemini Robotics-ER 1.6 model card
31. Figure Helix
32. Figure Helix 02
33. TRI Large Behavior Models study
34. V-JEPA 2
35. NVIDIA Cosmos
36. Google DeepMind Genie 3
37. DreamerV3 paper
38. SmolVLA
39. Skild Brain technical overview
40. World Models for Robotic Manipulation survey
41. Vision-Language-Action survey of datasets and benchmarks
42. Foundation Models in Robotics review

C.5 Datasets, benchmarks, and evaluation

1. Open X-Embodiment
2. DROID
3. BridgeData V2
4. RH20T
5. AgiBot World Beta
6. RoboMIND 2.0
7. LeRobot datasets
8. RLDS
9. RoboNet
10. Ego4D
11. EPIC-KITCHENS
12. CALVIN benchmark
13. LIBERO benchmark
14. LIBERO-Plus
15. SIMPLER
16. EmbodiedBench
17. SafeVLA benchmark
18. NIST humanoid robot benchmark initiative
19. ManiSkill benchmark
20. RoboCasa benchmark
21. BEHAVIOR benchmark
22. Independent pi0 evaluation
23. Data Scaling Laws in Imitation Learning
24. Grounding Sim-to-Real Generalization in Robotic Manipulation

C.6 Safety, assurance, and regulation

1. ISO 10218-1:2025
2. ISO 10218-2:2025
3. ISO/TS 15066
4. ISO 13482
5. ISO/CD 25785-1
6. ISO 13849-1
7. IEC functional safety
8. IEC machinery safety
9. ISO/PAS 8800:2024
10. OSHA robotics standards
11. EU Machinery Regulation 2023/1230
12. EU AI Act
13. NIST AI Risk Management Framework
14. UL 3300 service, communication, information, education, and entertainment robots

C.7 Industry statistics and deployment evidence

1. IFR World Robotics 2025 industrial statistics
2. IFR professional service robot statistics
3. Association for Advancing Automation 2025 North American orders
4. BMW and Figure 03 logistics project
5. Figure 02 BMW production report
6. GXO and Agility multi-year agreement
7. Agility Digit 100,000-tote milestone
8. Mercedes-Benz and Apptronik
9. Boston Dynamics electric Atlas
10. Figure AI
11. Agility Robotics
12. Apptronik Apollo
13. 1X NEO
14. Tesla AI and robotics
15. Unitree
16. UBTECH humanoid robots
17. AgiBot
18. Fourier Intelligence
19. Amazon one-million-robot milestone
20. Locus Robotics
21. Symbotic investor relations
22. Intuitive Surgical Q2 2026 earnings release
23. Waymo safety
24. Zipline
25. Skydio
26. Field AI

C.8 Skeptics, forecasts, and community signals

1. Rodney Brooks on humanoid dexterity
2. IEEE Spectrum humanoid scaling reality check
3. Russ Tedrake on the robotics acceleration
4. Sergey Levine interview
5. Ken Goldberg on the robot-data gap
6. Metaculus household-robot forecast
7. Goldman Sachs humanoid forecast
8. Morgan Stanley humanoid forecast
9. Bessemer robotics and physical AI thesis
10. ROS Discourse
11. Robotics Stack Exchange
12. The Robot Report
13. IEEE Robotics and Automation Society
14. Robotics: Science and Systems proceedings
15. Conference on Robot Learning

C.9 Land, aerial, marine, and space robotics

Land:

1. Komatsu FrontRunner 1,000-truck announcement
2. John Deere autonomy
3. CARLA simulator

Air:

1. PX4 architecture
2. PX4 controller diagrams
3. PX4 simulation with Gazebo
4. ArduPilot documentation
5. ArduPilot software-in-the-loop simulator
6. MAVLink overview
7. FAA commercial drone operators and Part 107
8. FAA Remote ID
9. FAA proposed BVLOS rule
10. FAA Part 108 final-rule executive review
11. Zipline two-million-delivery announcement

Surface water and underwater:

1. Woods Hole underwater-vehicle taxonomy
2. MOOS-IvP autonomy software
3. ArduSub
4. BlueOS
5. HoloOcean
6. COLREGs
7. IMO MASS Code adoption and roadmap
8. NOAA 2025 Atlantic hurricane USV mission
9. Oceaneering and TotalEnergies Freedom AUV pilot
10. Ulysses video profile
11. Ulysses technology page
12. a16z investment note on Ulysses
13. Second Ulysses manufacturing interview
14. Tectonic Defense Ulysses financing and product-status coverage

Space:

1. NASA Mars Relay Network
2. NASA Astrobee repository
3. Northrop Grumman SpaceLogistics
4. NASA Moon to Mars surface-mobility phases

Appendix D. Research method and limits

Breadth

The operating-environment research pass screened 1,069 unique OpenAlex works published from 2024-01-01 through 2026-07-20:

Search lane	Titles and abstracts screened
Ground mobile and field robotics	200
Aerial robotics	200
Surface-water robotics	150
Underwater robotics	115
Space robotics	64
Cross-domain autonomy, fleet, safety, and sim-to-real	100
General marine robotics	100
Agricultural field robotics	80
Hazardous-environment and inspection robotics	60
Total	1,069

The nine saved lanes contain 1,069 unique OpenAlex work IDs, no cross-lane duplicates, and no publication dates outside the stated window. The underwater query retrieved 115 of 250 requested works and the space query retrieved 64 of 100 requested works. Five records carried placeholder-like abstract text.

Screening means title and abstract review, not full-paper reading. Official documentation, regulators, mission sources, customer or joint releases, and independent reporting then verified the claims retained in the body. The guide does not treat the 1,069 search results as 1,069 authoritative sources.

Depth

Targeted verification emphasized:

• Original papers and model cards.
• Official repositories and release metadata.
• Dataset cards and licenses.
• Standards-body status.
• Named customer releases.
• Industry installation statistics.
• Independent technical criticism.
• Vehicle-stack documentation for land, aerial, surface-water, underwater, and space systems.
• Regulators and international rulemaking for aerial and maritime operations.
• The linked Ulysses transcript checked against current company specifications and independent financing coverage.

Limits

• Vendor operating metrics are not independently audited unless stated.
• Private-company revenue, valuation, and shipment figures change quickly and are often reported through secondary databases.
• Standards can change stage.
• Closed models cannot be independently reproduced.
• Forecasts express assumptions, not facts.
• The research corpus is relevance-ranked and English-heavy; it is not a systematic-review sample and may underrepresent non-English deployments.
• Military and classified field-robotics performance is not publicly auditable.
• The Ulysses video is edited company media. It can show a test and the stated architecture, but it cannot establish fleet reliability or economics.
• This guide is broad enough to orient decisions, not a replacement for a textbook, lab work, or an application-specific safety assessment.