Quick answer
Stanford researchers connected an ordinary AI coding agent to a real robot through a simple visual interface, with no extra robot training, and it succeeded in 24 of 25 trials across five physical tasks. In his latest video, Nate B Jones explains why this matters: it suggests that making AI better at general jobs like coding and computer use may also make it better at controlling machines. That idea, known as the Bitter Lesson, is also behind OpenAI’s recent robotics reshuffle. The big catch is speed: robots still think too slowly for many real-world jobs.
Key takeaways
- 24 of 25: the VIA project ran Codex Astra 6 on a Stretch 4 robot across lamp, shelf, towel, wipe and basket tasks.
- No robot-specific training: the agent used a visual interface and simple tools, much like using a computer.
- Look, act, check: the agent places a virtual gripper, moves the real one, then checks what happened.
- The Bitter Lesson: general methods that keep improving with more computing power tend to beat hand-built expertise over time.
- Speed is the bottleneck: a robot that needs constant supervision may cost more attention than it saves.
In this guide
- What did Stanford’s coding agent do with a robot?
- How does VIA work?
- What does the wipe test show?
- What is the Bitter Lesson, and why does it matter here?
- Why did OpenAI reorganise its robotics team?
- What still holds robots back?
- What does this mean for you?
- Watch: the source video
- Frequently asked questions
- Related posts
🛠️ Practical guides
Ready to put this into practice? These step-by-step guides answer two common follow-up questions:
- What Is the Bitter Lesson in AI, and How Should It Change What You Build? — what the Bitter Lesson means for the AI workflows and tools you build.
- How to Measure Whether an AI Agent Really Saves You Time — a simple scorecard to check an agent saves more time than it takes to supervise.
This is our plain-English summary of Nate B Jones’s video on the Stanford robotics study, with notes on what it means if you use AI agents in your business, even if you’ll never own a robot.
What did Stanford’s coding agent do with a robot?
The project is called VIA, short for Visual Interface Agent, from a Stanford team including Hengyuan Hu, Jeannette Bohg and Dorsa Sadigh. They took off-the-shelf coding agents, Codex and Claude Code, and gave them a way to see and control a robot.
On a real Stretch 4 mobile robot, using Codex Astra 6, the agent completed 24 of 25 attempts across five tasks:
| Task | What the robot had to do | Result |
|---|---|---|
| Lamp | Find a lamp on a cabinet and pull its chain | 4/5 |
| Shelf | Move between a toy and a drawer, remembering where they were | 5/5 |
| Towel | Fold a towel neatly | 5/5 |
| Wipe | Wipe a mark off a table | 5/5 |
| Basket | Open a basket lid, retrying if it slipped | 5/5 |
In simulation, the researchers also found results improved with the underlying model: from 38% to 100% across Codex tiers and from 43% to 85% across Claude Code tiers.
📌 Note: This is a small study: 25 real-world attempts. Nate says he has seen several similar results, but it’s an early signal, not proof that any AI can run any robot.
How does VIA work?
VIA treats controlling a robot like using a computer. The agent looks at a screenshot showing the camera view with a virtual “target” gripper drawn on top. It uses simple tools, similar to clicks and drags, to move that target where it thinks the real gripper should go.
- Look: the agent views the scene and places the virtual gripper, for example on a drawer handle.
- Adjust: if it looks a bit high or too far forward from another camera angle, it moves the target.
- Act: once happy, it tells a simple controller to move the real gripper there.
- Check: it looks again. Did it catch the handle? Does it need to adjust?
The agent doesn’t work out every motor movement. It picks the target, and the controller handles the mechanics. Nate compares it to how we move our own arms: we choose where to reach, not which muscles to fire.
What does the wipe test show?
Nate’s favourite example is the wipe task. While wiping, the robot’s own gripper blocked the camera view, so the agent moved the gripper away to check whether the table was clean. In one attempt, holding the wipe wasn’t working, so it dropped it and pushed it with a closed gripper instead.
That’s the behaviour you want from any AI agent: try something, check whether it worked and adjust. It’s the same loop good agents use on a computer, only here the results are physical.
💡 Tip: The same principle helps with software agents. Give them a way to check their own work, and feed back what failed. See Give AI Your Failed Attempts: How Fable 5.2 Got It Right First Time.
What is the Bitter Lesson, and why does it matter here?
Richard Sutton, a pioneering AI researcher, wrote The Bitter Lesson in 2019. His point: researchers keep building human expertise into AI systems, and keep getting overtaken by general methods that simply improve with more computing power, data and search. He traced it through chess, Go, speech recognition and computer vision.
For robotics, the idea is that some of a robot’s intelligence may come “for free” as AI gets better at everything else: understanding images, using tools, working through long tasks and noticing mistakes.
🛠️ Practical guide: What does the Bitter Lesson mean for the tools and workflows you build? Read What Is the Bitter Lesson in AI, and How Should It Change What You Build?.
Why did OpenAI reorganise its robotics team?
OpenAI confirmed it moved its robotics researchers into its main research group, with applied robotics and hardware moving to its applied organisation. It says robotics is still part of its plans, and that general frontier models are becoming more useful for robotics work.
Nate notes there may be several reasons, and reporting has mentioned financial pressure. But the technical logic fits the Bitter Lesson: if improving the general model also improves robotics, it makes sense for robotics to sit close to that work.
What still holds robots back?
Two things, according to Nate:
- The harness matters. A November study (Nate calls it “JustRUA”) kept the model the same and significantly improved results just by improving the software around it. You can’t plug an agent into anything and expect magic.
- Speed. A separate October study found that in robot trials, roughly 90% of the time went on the model thinking, and successful runs took over an hour. The real world doesn’t pause while AI deliberates.
Nate’s practical test for any robot is simple: how long did the whole attempt take, how often did someone step in, and what happened when it made a mistake? A robot that finishes eventually while someone watches over it may use more attention than it saves.
🛠️ Practical guide: The same test works for software agents. See How to Measure Whether an AI Agent Really Saves You Time.
Using AI agents in your business? Before you add more, check they’re actually saving time. Our guide How to Measure Whether an AI Agent Really Saves You Time gives you a simple scorecard.
What does this mean for you?
You probably aren’t buying a robot arm. But the lessons carry over to any AI agent:
- Watch for knock-on effects. Nate’s advice is to “look around corners”: an improvement in one area, such as computer use, can show up somewhere unexpected.
- Don’t over-build. Complex custom workarounds may be overtaken by the next general model. Keep them simple and easy to replace.
- Build in checking. The best results come when an agent can see what happened and adjust.
- Measure the whole job. Count your supervision time, not just the agent’s output.
Your next step: pick one AI agent or automation you use and time the next run from start to finish, including every time you step in.
Watch: the source video
What this video covers: Nate B Jones explains Stanford’s VIA study, where a coding agent controlled a real robot with no extra training, and what Richard Sutton’s Bitter Lesson means for robotics.
Video: Nate B Jones (AI News & Strategy Daily), “Stanford Hooked A Coding Agent To A Robot. It Went 24 For 25 with No Extra Training.”, published 18 November 2026. No sponsor. Study details checked against the VIA project page. This summary and the practical advice are our own; the predictions are Nate’s.
Frequently asked questions
What is VIA in robotics?
VIA (Visual Interface Agent) is a Stanford research project that lets off-the-shelf coding agents such as Codex and Claude Code control a robot through a visual interface and a few simple tools, without robot-specific training.
Did the robot learn the tasks first?
No. The point of the study is that the agent completed the tasks zero-shot, using general abilities it already had, though the model will have seen general information about the world during its original training.
Does this mean household robots are coming soon?
Not necessarily. Speed and supervision are still big problems, and 25 real-world attempts is a small test. It’s a promising direction rather than a finished product.
What is the Bitter Lesson?
It’s Richard Sutton’s 2019 observation that general AI methods that scale with more computing power tend to beat carefully hand-built approaches over time.
Why does this matter for small businesses?
It shows that general AI agents keep gaining new abilities. The practical lessons are to avoid over-building around today’s limits, to let agents check their work and to measure the time they really save.
Related posts
- AI Desk Robot Assistant: How Mark Kashef Rebuilt a $99 Robot With Claude Fable 5.2
- GPT-6.2 Astra Review: Cheaper and Efficient, but Is It Worth Switching?
- AI Agent Cost: How to Tell If Your Agent’s Work Is Worth the Bill
- Give AI Your Failed Attempts: How Fable 5.2 Got It Right First Time
