Painted storybook scene in a log-cabin room on a snowy Mat-Su Valley morning. A girl with long light-blond braids and a mustard-yellow sweater smiles as she places a blue wooden tile into a colorful grid puzzle on a wooden table. Behind her, a bearded father in a green flannel shirt holds a mug and smiles at her work. On the right, a small round cream-and-copper robot sits quietly beside a closed notebook, watching and waiting. Through the window are snowy spruce trees and a white mountain.

AI news

Practice and Patience: New Tests Ask Whether AI Helpers Get Better on the Job, and Whether They Know When to Stay Quiet

Two new benchmarks grade AI the way we grade a new coworker. One found that note-keeping AI agents barely improved with repeat work, while a model that trained on its practice got better even at puzzles it had never seen. The other found top models speak up well when help is needed, but often butt in when silence is the right call.

November 13, 2026 by Ember & Oak Institute

Picture a new hire in their first month. You want two things from them: that they get a little better each time the same job comes back around, and that they know when to jump into a conversation and when to let it be. This week, two research teams built tests that grade AI helpers on exactly those two habits. The results suggest today’s AI is quick on its feet but still shaky on both.

The tests matter because AI is shifting from tools you ask to helpers that run in the background: agents that keep notes, watch a team’s chat and step in on their own. A one-time quiz can’t show whether a helper improves with practice or has good judgment about interrupting. These new benchmarks try to measure both, and they point to a useful lesson for anyone, young or grown, who works alongside AI.

Notes are not the same as learning

Unify, a London AI company, released Continual-ARC, a test built from ARC puzzles, the colored-grid riddles where you study a few examples and work out the hidden rule. The twist is that puzzles come back again and again with fresh inputs, on a schedule the team compares to an office worker’s week, “where the report you wrote in March is due again in June,” according to Unify’s write-up. A remembered answer never fits the new grid. What’s worth carrying forward is the rule.

After the stream of puzzles, each AI is frozen and tested twice: on fresh versions of puzzles it already met, which measures memory, and on 100 puzzles it never saw, which measures whether it actually got better at the skill.

The researchers tested popular agent “harnesses,” software that wraps an AI model and lets it keep notes and write itself instructions. “OpenClaw, Hermes Agent and Prime Agent don’t meaningfully learn from experience,” Unify’s Daniel Lenton wrote on X. Run as shipped on OpenAI’s GPT-6 Luna, none saved a measurable share of effort over the puzzle stream, Unify reported. One harness never saved a lesson for use beyond a single puzzle in 600 sessions. Another wrote 47 to 84 notes per run but read a note named for the puzzle in front of it on at most 1% of return visits.

Clear instructions helped. When the team told the harnesses exactly what to keep (one small program per puzzle, run first when that puzzle returns), every harness saved a measurable share. But notes only helped with puzzles the AI had already seen. In the paper’s words, “No harness-based method gains on unseen tasks.”

What did help on new puzzles was changing the model itself. The team trained a small open model, Qwen3.5-9B, to imitate the moves of a stronger “teacher” model as it played. That trained version used 57% less effort than the untrained model on practiced puzzles and 29% less on the 100 new ones, solving about 49 of them to the untrained model’s 19, according to Unify. “A harness’s notes are hindsight,” the team wrote: a note can repeat a rule, but it can’t make the model better at working out the next one.

Bar chart from Unify’s Continual-ARC benchmark. An open AI model, Qwen3.5-9B, worked through a long stream of returning puzzles, then was tested. On fresh versions of puzzles it had practiced, keeping notes (teacher-written notes plus expert guidance) saved 0.34 of the effort versus starting fresh, while training the model’s own weights saved 0.57. On 100 puzzles it had never seen, the notes saved −0.05, slightly worse than starting fresh, while the trained model saved 0.29. Source: Unify blog and arXiv:2611.14505.

There are limits. The results come from ARC puzzles only, the trained model learned with a teacher’s help, and Unify, which built the test, notes that a smarter harness could still prove the team wrong.

Knowing when not to help

The second test comes from Deepnote, a data-notebook company. “An AI teammate should know when to help without being asked,” the benchmark page opens. To test that, Deepnote built PageMantle, a made-up software company with 71 employees and six years of linked records across 11 services, including chat, customer files, billing and documents. All of it is open for anyone to download and run.

Twelve leading AI models each listened in on 50 real-feeling Slack conversations. In each one, the AI had to dig through company records, decide whether anyone needed help, and then pick what to say, whom to tell and where to post it. Some situations called for stepping in, such as warning a team not to use a customer’s logo without approval. In 10 of them the problem was already handled, so the right move was to say nothing. Two separate AI graders scored each run from 0 to 10 against a reference answer.

OpenAI’s GPT-6 Astra led at 8.97, followed by GPT-6.1 Sol at 8.65 and Anthropic’s Claude Fable 5.2 at 8.17. But the pattern across the board was the headline. “Models are good at helping and bad at not helping,” Deepnote wrote. When help was needed, a message went out 9 times out of 10. When silence was right, models held back only 6 times out of 10, and speaking out of turn cost about 1.8 times as much as staying quiet.

Horizontal bar chart from Deepnote’s Agent Initiative Benchmark comparing 12 AI models. Green bars show how often each model spoke up when help was needed, out of 40 tasks; orange bars show how often it stayed quiet when silence was right, out of 10 tasks. Most models spoke up on 37 to 40 of 40 tasks but stayed quiet on only 4 to 8 of 10. GPT-6 Astra: 40/40 and 8/10. GPT-6.1 Sol: 39/40 and 8/10. Claude Opus 5.5: 40/40 and 8/10. GPT-6 Luna, DeepSeek V4.1 Flash and Mistral Large 4: 4/10 quiet. Gemini 4 Argon stayed quiet 10/10 but spoke up on only 23/40; Grok 4.7: 24/40 and 6/10. Source: Deepnote, deepnote.com/research/benchmark.

More digging didn’t mean better answers, either. Google’s Gemini 4 Argon and xAI’s Grok 4.7 made the most tool calls, about 35 and 36 per task, and finished last. “Searching more is not the same as finding more,” the page says. The most expensive models weren’t reliably the best. Deepnote notes its own limits: a 50-task panel, a single snapshot in time, one effort setting, and replies only through Slack.

What it adds up to

Put the two tests side by side and a theme emerges. Being good once isn’t the same as being a good helper over time. Today’s AI agents can solve a hard problem on the spot, and they’re eager to pitch in. What they struggle with is the slower, quieter stuff people learn on the job: turning practice into real skill, and having the judgment to stay out of the way. Notes and eagerness can look like learning and helpfulness. These tests are built to tell the difference.

For learners: practice, then patience

Our students direct AI rather than hand-code, using a simple loop: idea, build, test, improve. These studies suggest two upgrades to that loop.

Idea: Before you build, decide what your AI helper should remember between sessions, and when it should stay quiet. Write both down as plain rules.
Build: Give the AI a specific “keep this” instruction, like Unify’s one-program-per-puzzle rule, instead of hoping it saves the right notes on its own.
Test: Bring back an old task with new details and see if the helper does better the second time. Then test a situation where nothing needs doing. A good helper says “nothing needed” instead of inventing work.
Improve: Check whether the helper ever rereads its notes. If not, change the instructions and test again. And keep your own practice log, because people get better by practicing, not just by taking notes.

Sources

Back to the blog