{
  "title": "Practice and Patience: New Tests Ask Whether AI Helpers Get Better on the Job, and Whether They Know When to Stay Quiet",
  "slug": "practice-and-patience",
  "url": "https://emberandoakinstitute.com/blog/practice-and-patience.html",
  "date": "2026-11-13",
  "category": "AI news",
  "tags": [
    "artificial intelligence",
    "AI agents",
    "continual learning",
    "benchmarks",
    "AI assistants",
    "AI news"
  ],
  "summary": "Two new benchmarks test AI helpers like new coworkers. Unify's Continual-ARC found note-keeping agent harnesses barely improved on returning puzzles and never on new ones, while training a small model's weights helped on both. Deepnote's Agent Initiative Benchmark found 12 top models spoke up when needed about 9 times in 10 but held back when silence was right only about 6 times in 10.",
  "key_facts": [
    {
      "claim": "Continual-ARC (Unify; Lenton, Arif, Mahmood; arXiv:2611.14505, submitted Nov. 10, 2026) serves ARC puzzles that return with fresh inputs, then freezes the learner and tests it on fresh versions of met puzzles (memory) and 100 never-seen puzzles (generalization).",
      "sources": [
        "https://arxiv.org/abs/2611.14505",
        "https://www.unify.ai/blog/why-we-built-continual-arc"
      ]
    },
    {
      "claim": "Lenton on X (Nov. 11): 'OpenClaw, Hermes Agent and Prime Agent don't meaningfully learn from experience.'",
      "sources": [
        "https://x.com/DanielLenton1/status/2120535187587268777"
      ]
    },
    {
      "claim": "On GPT-6 Luna, none of the three harnesses as shipped saved a measurable share; Prime Agent never marked a lesson global in 600 sessions; OpenClaw wrote 47–84 notes per run and read one named for the puzzle on at most 1% of returning visits; told a keep-one-program-per-puzzle procedure, every harness saved 0.17–0.62.",
      "sources": [
        "https://www.unify.ai/blog/why-we-built-continual-arc"
      ]
    },
    {
      "claim": "Paper abstract: 'No harness-based method gains on unseen tasks, whereas weight-based learning shows substantial gains.'",
      "sources": [
        "https://arxiv.org/abs/2611.14505"
      ]
    },
    {
      "claim": "Qwen3.5-9B trained (DAgger) to imitate GLM-5.3 Flash saved 0.57 on the seen test (84% to 36% of budget) and 0.29 on the held-out test, solving about 49 of 100 new puzzles vs 19 untrained. Teacher-written notes plus expert guidance saved 0.34 seen and -0.05 held-out (QETE).",
      "sources": [
        "https://www.unify.ai/blog/why-we-built-continual-arc",
        "https://continual-arc.unify.ai/"
      ]
    },
    {
      "claim": "Unify quote: 'A harness's notes are hindsight.' Unify says a harness could still prove them wrong.",
      "sources": [
        "https://www.unify.ai/blog/why-we-built-continual-arc"
      ]
    },
    {
      "claim": "Deepnote Agent Initiative Benchmark (Nov. 13, 2026): 12 frontier models, 50 workplace situations (40 need help, 10 need silence) in synthetic company PageMantle (71 employees, six years of records, 11 services); two LLM evaluators score 0–10 against a reference; open-sourced on GitHub and Hugging Face.",
      "sources": [
        "https://deepnote.com/research/benchmark",
        "https://github.com/deepnote/agent-initiative-benchmark"
      ]
    },
    {
      "claim": "Scores: GPT-6 Astra 8.97, GPT-6.1 Sol 8.65, Claude Fable 5.2 8.17 ... Gemini 4 Argon 5.60, Grok 4.7 4.51. Helped/stayed-silent counts per results table (chart).",
      "sources": [
        "https://deepnote.com/research/benchmark"
      ]
    },
    {
      "claim": "Deepnote: message went out 9 of 10 times when help was needed; held back only 6 of 10 when silence was right; speaking out of turn cost about 1.8x as much as staying quiet; Argon and Grok made ~35 and ~36 tool calls and finished last; 'Searching more is not the same as finding more.'",
      "sources": [
        "https://deepnote.com/research/benchmark"
      ]
    },
    {
      "claim": "Example task: warning the team not to use an unapproved customer logo.",
      "sources": [
        "https://x.com/jakubjurovych/status/2121256499552915586"
      ]
    },
    {
      "claim": "Deepnote limitations: 50-task panel, fixed snapshot, one effort level (high), replies only through Slack.",
      "sources": [
        "https://deepnote.com/research/benchmark"
      ]
    }
  ],
  "charts": {
    "chart-speak-or-stay-quiet.png": "https://deepnote.com/research/benchmark",
    "chart-notes-vs-practice.png": [
      "https://www.unify.ai/blog/why-we-built-continual-arc",
      "https://arxiv.org/abs/2611.14505"
    ]
  }
}