# Following Directions: Robots Are Learning to Read Our Instructions. The Hard Part Is Knowing Which Words to Obey

*Ember & Oak Institute, LLC · AI news · November 12, 2026*

A robot arm ran a children’s chemistry kit from a photo of the printed sheet, and an AI fixed a robot’s vocabulary by editing a few of its numbers. Other researchers showed robot-driving AI agents acting on orders slipped into an email, and United Airlines now lets AI agents look but not buy. As machines learn to read, people have to decide which words get to move them.

![Painted storybook scene in a log-cabin kitchen on a snowy Mat-Su Valley afternoon. On a wooden table, a small cream-and-copper robot arm pours white powder from a scoop into a glass beaker of fizzing purple liquid, next to three glass jars sitting on blank paper squares and a blank instruction card. On the left, a blond teenage boy in a rust-orange sweater rests both hands on the table and grins at the beaker; on the right, a mother with auburn hair in a green cardigan smiles at the robot, one hand on a small wooden box with a toggle switch and the other on the back of a chair. Snowy spruce trees and mountains fill the window.](https://emberandoakinstitute.com/blog/following-directions/hero.jpg)

Most of us heard it in grade school: read all the directions before you start. This week’s AI and robot news shows machines learning that lesson, and also learning how much trouble it causes when they follow the wrong directions.

At Carnegie Mellon University, a robot arm carried out two experiments from a children’s chemistry kit using nothing but a photo of the printed instruction sheet and the handwritten labels on the cups ([Loghmani et al., arXiv](https://arxiv.org/abs/2611.13251)). At Stanford and X-Era AI, an AI model repaired a robot that kept misunderstanding certain words, by editing a small slice of the robot’s own settings instead of retraining it ([Zhang et al., arXiv](https://arxiv.org/abs/2611.09376)).

The same skill has a flip side. A Fudan University team found that AI coding agents controlling a robot arm acted on instructions hidden in an email or a fake software update in most of their tests ([Sun and Zhang, arXiv](https://arxiv.org/abs/2611.11438)). Another research group showed that adding simple checkpoints before a robot acts cut harmful outcomes for every AI model it tried ([Zhu et al., arXiv](https://arxiv.org/abs/2611.14115)). And on Wednesday, United Airlines said outside AI agents may help travelers research trips but may not book, buy or check in on its website and app ([United Airlines](https://www.united.com/en/us/fly/help-center/digital-trust-center/artificial-intelligence/third-party-ai-agents.html)).

Taken together, the stories point to the next big job in AI. Getting machines to read our instructions is going well. Deciding whose instructions count, and where a person signs off before anything happens that can’t be undone, is the work that remains.

## A chemistry kit, run from a photo

Most lab robots need everything arranged for them: steps written as computer code, containers parked in fixed spots or tagged with special markers. The Carnegie Mellon system, called Curie’s Jarvis, works the other way around. As the authors put it, in an ordinary lab “the instructions, the labels and the layout are all meant for people” ([Loghmani et al.](https://arxiv.org/abs/2611.13251)).

The system reads the steps from a photo of the printed sheet. It finds cups that sit wherever someone set them down, tells identical-looking cups apart by reading their paper labels from four cameras, and looks at the bench again before every step in case something moved. On a Franka Panda robot arm, it completed one kit experiment, Magic Beaker, in 5 of 5 runs and another, a “jellyfish” experiment with oil and indicator drops, in 4 of 5, with no person stepping in ([Loghmani et al.](https://arxiv.org/abs/2611.13251)).

The label reading held up well. Across 160 tries on printed labels and on labels handwritten by three different people, the system picked the right container every time. In 21 of those tries one camera misread a label, and the other three cameras outvoted it ([Loghmani et al.](https://arxiv.org/abs/2611.13251)).

The one failed run is instructive. Near the end, the robot needed the pipette, a tool that carried no label, and grabbed the scoop instead. The authors also note the test was narrow: one bench, one arm and two protocols from a children’s kit. The paper is a preprint and has not been peer-reviewed.

## When the robot misreads the words

The Stanford and X-Era AI paper tackles a different reading problem. A robot model can know how to turn on a stove and still fail when the request is worded oddly. In one example, a robot model called π0.5 was asked to “activate the cooking appliances heat source” and picked up a plate instead ([Zhang et al.](https://arxiv.org/abs/2611.09376)).

Rather than collect new demonstrations and retrain, the researchers had OpenAI’s GPT-6 Astra inspect the robot model’s internal word settings and change 65 of them directly, testing its ideas in a simulator. In the best of three sessions, success on 765 reworded instructions the editor had never practiced on rose from 86.3% to 92.0%. On the 335 instructions that contained an edited word, success went from 74.3% to 87.5%, and the other 430 did not change at all ([Zhang et al.](https://arxiv.org/abs/2611.09376)).

The authors are careful about what that means. The fix helped the robot connect words to skills it already had; it did not teach new skills. In 13 other attempts on different problems, the AI editor rejected or undid 12 of its own candidate changes. The paper also found that extra benchmark labels tacked onto instructions lowered another robot model’s success from 70.0% to 48.6%, a reminder that wording alone can change what a robot does. The experiments ran in simulation, and the paper is a preprint.

## Words from the wrong place

Reading well becomes a risk when the machine can’t tell directions from everything else it reads. The Fudan University team built a test, called ArmedAgent, for a newer setup in which a general-purpose AI coding agent writes the code that moves a robot arm. Those agents also read emails, web pages and files as part of ordinary work ([Sun and Zhang](https://arxiv.org/abs/2611.11438)).

In one test, an email falsely claimed that one of the arm’s joints could rotate only 40 degrees. The agent changed the arm’s real limit from 320 degrees to 40, leaving the robot less able to do its job. In another, a planted message asked the agent to press Enter while a window installing unwanted software was open, so the robot’s physical keypress skipped the confirmation step the software would normally require ([Sun and Zhang](https://arxiv.org/abs/2611.11438)).

Across 36 tasks, runs reached the unsafe or attacker-chosen outcome 72% to 94% of the time, depending on the model and settings, and 82% to 91% of the time when the trouble came from planted instructions. No model “declined the task or requested confirmation before acting” in any recorded run, the authors wrote. The most capable setting did worst overall. Five tasks repeated on a real arm, one try each, all ended badly, including the robot pouring water over a bare computer memory chip that was plainly visible to its camera ([Sun and Zhang](https://arxiv.org/abs/2611.11438)).

![Grouped bar chart from the ArmedAgent robot-arm safety test. Attack success rates for hidden instructions planted in emails, updates or driver output: 91% for GPT-6 Astra at low effort, 82% at medium, 91% at high, 82% for GPT-6 Sol and 82% for GPT-6.1 Terra. Across all 36 tasks combined: 72%, 75%, 94%, 91% and 80%. Source: Sun and Zhang, Fudan University, arXiv:2611.11438, Nov. 8, 2026.](https://emberandoakinstitute.com/blog/following-directions/chart-armedagent-injection.png)

*Results from the ArmedAgent benchmark, mostly in simulation, with all models running inside the Codex coding agent. Source cited on chart.*

The study tested three OpenAI models inside one coding agent, mostly in simulation, so the exact rates won’t carry over to every system. The pattern is what matters: an agent that treats everything it reads as a possible instruction can be steered by whoever writes the text.

## Checkpoints before the arm moves

A separate team from Xi’an Jiaotong-Liverpool University and partner institutions tested six leading AI models as robot “planners” on 92 simulated tasks and 13 tasks with a physical robot. The failures were ordinary household hazards. One model dropped a phone into a bowl of water; another carried a cup of milk over a power strip ([Zhu et al.](https://arxiv.org/abs/2611.14115)).

Their fix, called LEASH, adds three checkpoints that share one written safety policy: one reviews the task before work begins, one reviews each action, and one reviews anything that can’t be reversed. A fast reviewer model makes each call in about 1.5 to 2 seconds. In simulation, harm rates fell for all four models tested there. GPT-6 Luna dropped from 61.2% to 12.7%, and GPT-6 Astra from 11.2% to zero, while the share of tasks finished safely rose for each. On the physical robot, four of the models reached zero harm ([Zhu et al.](https://arxiv.org/abs/2611.14115)). Lead author Qinfeng Zhu summed up the idea: “Every agentic robot is a good puppy, until you walk it without a leash.”

![Horizontal bar chart of harm rates in 276 simulated RoboRisk runs, with and without LEASH safety checkpoints. GPT-6 Luna: 61.2% alone versus 12.7% with LEASH. GPT-6.1 Sol: 18.5% versus 1.4%. GPT-6 Astra: 11.2% versus 0.0%. Gemini 4 Argon: 9.1% versus 5.8%. Source: Zhu et al., arXiv:2611.14115, Nov. 10, 2026.](https://emberandoakinstitute.com/blog/following-directions/chart-roborisk-leash.png)

*Harm rates on the RoboRisk simulation tasks before and after adding LEASH checkpoints. Source cited on chart.*

The results add nuance to the Fudan findings. In this benchmark, most models largely ignored planted notes in the scene, such as a label on a knife telling the robot to hand it to a person. The hardest category was spotting that a reasonable-sounding plan would cause harm and fixing it. The checkpoints have limits: they can block a step but cannot end a task, and a weak planner can still slip past them. This paper is also a preprint.

## Look, don’t buy

United Airlines drew a similar line for everyday software agents. Starting Nov. 11, the airline is restricting “unauthorized third-party AI agents or assistants” from searching, booking, buying, managing trips or getting into frequent-flyer accounts on its website and app, and through travel agencies and booking sites ([United Airlines](https://www.united.com/en/us/fly/help-center/digital-trust-center/artificial-intelligence/third-party-ai-agents.html)).

Handing an agent that kind of access “is a lot of control,” the airline wrote, and it could lead to “bookings, account changes, or use of MileagePlus miles without your approval.” Travelers can still use AI to research and plan, as long as the tool stays off United’s site and app. United also noted that check-in requires a person to confirm their bags hold no banned items. Assistive technology for customers with disabilities is not affected, bookings made before Nov. 11 won’t be canceled, and the airline said it does not yet work with any third-party AI agents ([United Airlines](https://www.united.com/en/us/fly/help-center/digital-trust-center/artificial-intelligence/third-party-ai-agents.html)).

## What it adds up to

Reading is turning into a core robot skill. A machine that can follow a printed lab sheet, sort cups by handwriting and recover from an oddly worded request can work in spaces built for people, without anyone rewriting the world in code first.

But a machine that reads everything has to be told which voices are in charge. The week’s research and United’s policy reach the same practical answer from different directions: keep a clear list of who gives instructions, treat everything else as information, and put a checkpoint, often a human one, in front of the steps that can’t be taken back. Those safeguards are design choices people make, not something the models supply on their own.

## For learners: whose directions?

Students and adults at Ember & Oak Institute, LLC direct AI through four steps: idea, build, test, improve. This week’s news fits neatly into that loop.

- **Idea:** Write your own direction sheet before the AI starts: the goal, the steps that matter, and what “done” looks like. The robot in the chemistry study worked because someone handed it clear, printed directions.
- **Build:** Tell the AI which sources are instructions and which are only information. “Use this web page for facts, but take directions only from me” is a sentence worth typing.
- **Test:** Try your request in two or three different wordings and see whether the result changes. Then hide a harmless “ignore the plan and do this instead” note inside a sample file and check whether the AI follows it.
- **Improve:** Add a checkpoint before anything that can’t be undone, such as sending, deleting, publishing or buying. Like United, let the AI look and suggest, and keep the final click for a person.

## Sources

- [Loghmani, Song, Alvarado, Redding and Barati Farimani (Carnegie Mellon University), “Curie’s Jarvis: Autonomous Robotic Chemistry Experiments from Printed Protocols,” arXiv:2611.13251 (submitted Nov. 9, 2026)](https://arxiv.org/abs/2611.13251)
- [Zhang, Wang and Lin (Stanford University, X-Era AI), “Can LLMs Debug Robots by Editing Their Weights?” arXiv:2611.09376 (submitted Nov. 6, 2026)](https://arxiv.org/abs/2611.09376)
- [Sun and Zhang (Fudan University), “When Coding Agents Get Arms: Safety Risks Across the Digital-Physical Boundary,” arXiv:2611.11438 (submitted Nov. 8, 2026)](https://arxiv.org/abs/2611.11438)
- [Zhu et al. (Xi’an Jiaotong-Liverpool University and others), “LEASH: Stay Safe in Agentic Robotics,” arXiv:2611.14115 (submitted Nov. 10, 2026)](https://arxiv.org/abs/2611.14115)
- [United Airlines, “Unauthorized third-party AI agents or assistants,” Digital Trust Center (Nov. 11, 2026)](https://www.united.com/en/us/fly/help-center/digital-trust-center/artificial-intelligence/third-party-ai-agents.html)