Painted storybook scene in a cozy Mat-Su Valley cabin kitchen with birch trees and snowy mountains outside the window: a girl with light-brown braids kneels at a wooden table beside a homemade marble run of blocks and cardboard ramps, pointing at one ramp as she explains how it works, while her smiling mother listens with a mug in hand and a small white-and-copper robot leans in to watch.

AI news

Knowing Isn’t Understanding: AI Is Handing Out Answers Faster Than People Can Make Sense of Them

A big classroom study found students’ homework grades rose with AI while their exam scores fell, an AI-designed protein campaign passed only a sliver of its designs through the lab, and mathematicians are asking what a solved problem is worth if no one understands the proof. The lesson for young builders: the answer is not the point.

October 25, 2026 by Ember & Oak Institute

Machines produced answers by the hundreds of thousands this week. A classroom study, a biology lab and a room full of worried mathematicians all ran into the same harder question: does anyone actually understand them?

A working paper tracking nearly 27,000 secondary students in China found that homework scores rose after students started using AI, while their closed-book exam scores fell. Anthropic reported that its Claude model designed 318,484 candidate proteins against scorpion toxins, and lab tests found that 17 of them did what they were designed to do. And Cornell mathematician Steven Strogatz wrote in The Atlantic that his field is “in crisis” because AI is knocking down long-standing problems faster than people can absorb them. Taken together, the stories point the same way. Getting an answer is becoming cheap. Understanding it, and checking it, is still the expensive part, and it still falls to people.

Better homework, worse exams

Researchers David Strömberg of Stockholm University and Ziteng Lei and Yanhui Wu of the University of Hong Kong followed 26,811 students in grades 7 through 12 in one county in central China for 30 months, across nine subjects. By June 2025, about 80% of them had started using AI chatbots, mostly the Chinese apps Doubao and DeepSeek. The study was released as a CEPR discussion paper in June and spread widely on X this week.

Once students started using AI, their homework scores rose 18% and the time they spent on homework dropped from about 64 minutes to 45. Within six months, their scores on monthly closed-book exams fell 20%. On the high-stakes entrance exams for high school and college, scores fell 24% and 18%, with the full effect showing up after about two years. The biggest losses came in social science, around 27%, and the smallest in Chinese, around 9%. Younger students, boys and high achievers lost the most, the authors wrote.

“But for learning, completing homework is not the end goal,” they wrote. About four in five AI users behaved as if they were handing the work off rather than using AI as a tutor. The authors also found a hopeful exception: “when AI students spend the same amount of time on homework as non-AI students, they learn as much.”

Bar chart from a study of 26,811 students in grades 7 through 12 in central China. After students began using AI, homework scores rose 18% and time spent on homework fell 30%, while monthly closed-book exam scores fell 20%, high-school entrance exam scores fell 24% and college entrance exam scores fell 18%.

Source: Strömberg, Lei and Wu, “The Generative AI Learning Penalty,” CEPR Discussion Paper 21577, and VoxEU column. Chart: Ember & Oak Institute, LLC.

The study has limits. It has not been peer reviewed, it covers one region with its own school system, and the authors caution that the effect sizes cannot be applied directly to other places. They also warn that simply requiring more homework time may not fix the problem. Their conclusion: “The essential question in AI adoption in education is not technological, but organisational and institutional.”

318,484 designs, 17 that held

In the lab, the same gap between answer and proof played out with proteins. Working from one prompt and four follow-ups, an internal Anthropic research model spent 64 hours using 12 design programs to generate 318,484 protein designs meant to latch onto five scorpion neurotoxins, the company said. Claude picked 180 for testing, and the team added 31 more with its help. Two outside labs, Adaptyv Bio and Twist Bioscience, then tested all 211.

Seventeen met Anthropic’s criteria for a binder, each against a single toxin. None of the 151 designs meant to grab more than one toxin at once did so, according to the company’s technical report. Scorpions sting more than 1.2 million people a year and kill more than 3,250, mostly children in places with limited medical care, Anthropic noted. Still, it was blunt about what it had not shown: “we do not claim to have produced an antivenom.” The binders have not yet been tested to see whether they block venom in cells or animals.

Horizontal bar chart on a log scale showing Anthropic’s scorpion-toxin binder campaign: Claude generated 318,484 protein designs, 211 were lab-tested, 17 met the binding criteria against a single toxin, and 0 of the 151 multi-toxin designs met the criteria for more than one toxin.

Source: Anthropic, “How Claude designed protein binders against scorpion neurotoxins” and technical report, Oct. 20, 2026. Chart: Ember & Oak Institute, LLC.

Outside scientists called the work a real step and warned against reading too much into it. Susana Vázquez Torres of Spain’s national cancer research center, CNIO, one of two experts Anthropic asked to review it, said “binding itself is no longer the main bottleneck.” Andreas Hougaard Laustsen-Kiel of the Technical University of Denmark told The Verge: “It is easy-ish to discover a good binder against snake and scorpion toxins… The hard part is to translate it into a medicine.”

A bottle of cognac, still corked

Mathematicians are wrestling with a stranger version of the same problem. Many AI-produced math results can be checked by computer, New Scientist reported, but they often arrive in a form that is hard for people to understand. In his Atlantic essay, published Oct. 25, Strogatz describes a Cornell town hall held after OpenAI said in early October that its latest model had solved close to 400 well-known open problems. One of his colleagues, the logician Justin Moore, had saved a bottle of cognac since 2010 to open when a problem he loved was finally solved. Now that an AI has claimed the solution, Moore told Strogatz he will open it “once a group of peers I trust all agree that the proof is correct, and once we collectively understand the proof.”

“The object of our desire, as mathematicians, is a mixture of knowing and understanding,” Strogatz wrote. Some colleagues are racing to use AI to clear old problems so new work can begin. Others are pulling back. The Association for Human Mathematics, formed after OpenAI released hundreds of AI-written math papers this month, lists 1,008 members who have pledged not to work for commercial AI companies or publish AI-generated math. Of those, 654 have also promised to keep AI out of their research entirely. “I’m not comfortable putting out things that I don’t understand, that I can’t vouch for,” one of them, Steven Sivek of Imperial College London, told New Scientist.

Show how you built it

Game makers are taking a more practical approach. Epic Games’ AI MegaJam, a four-day contest running Nov. 12 to 15, lets teams build Unreal Engine or Fortnite games with “unlimited agents.” But one of its award categories, called Glass Box, goes to the best dev log “showing how you built your game.” The jam’s rules page advises entrants to “keep notes as you go” and says that “a finished, fun game beats an ambitious broken one.” Using AI is optional, and the contest is open only to developers 18 and older, but the idea works at any age: if you can explain how a thing was made, you understand it.

What it adds up to

In all four stories, AI made producing an answer cheap: a finished worksheet, a list of protein designs, a proof, a playable game. In all four, the value moved to the step after that: the closed-book exam, the lab bench, the mathematician reading the proof, the dev log. The students who kept spending real time on their homework, even with AI in hand, kept learning. The proteins that counted were the ones that held up when tested. A proof only feels solved once people understand it.

At Ember & Oak Institute, LLC, learners don’t hand-code. They direct AI tools through a simple loop: idea, build, test, improve. This week’s news shows that the loop only works if the learner stays in it. The AI can build almost anything you ask for. Testing it, explaining it and deciding what to improve is the part that grows you.

Try it at home: Have your learner build something small with an AI helper, such as a marble-run plan, a quiz game or a science-fair chart. Then close the laptop and ask them to teach it back to you: what each part does, why it works, and one way it could break. Anything they can’t explain becomes the next round of “test” and “improve.” Keep the same amount of time you would have spent without AI, and spend it on understanding instead of typing.

Sources

Back to the blog