A team of physicists sat down to grade an AI's homework and ended up rewriting their own answer key. That's the short version of a new evaluation of GPT-6 Astra, and it matters more than the score alone suggests, because AI solving physics problems at research level was supposed to be years away. This run came in at 49 out of 50 — for less than the price of a coffee per question.
What happened
The test is called TheoPhysEval: 50 unpublished, research-level problems in theoretical physics. "Unpublished" is the key word, because it means the AI couldn't simply have read the answers somewhere online. According to the authors, a typical problem would take a narrow specialist up to a full working day.
GPT-6 Astra, running inside Codex with its tools and web search switched on, solved 49 of those 50 problems. That's 98%. The average cost per problem came in under $3 each. The agent worked without a single human hint.
Then the human reviewers did their job. In several problems they marked Astra's answer as wrong and went back to their own reference solutions to prove it. Instead, they found their own reference answers were flawed. Once the experts corrected those mistakes, Astra's answers were confirmed as correct. You can read the paper yourself here: TheoPhysEval on arXiv, or in the full HTML version.
Why the graders got it wrong
Picture a teacher circling an answer in red, then discovering the answer key in the textbook was misprinted. The student was right. The key wasn't.
That's an uncomfortable place for a benchmark to land — and to their credit, the authors say so out loud rather than hiding it. They also flag the obvious caveat: these are hard computations using methods physicists already know. Executing a known playbook flawlessly is not the same as inventing a new one, and the paper is clear that building genuinely new physics theories takes more. But an assistant that pushes experts to re-check their own gold-standard answers has already earned a second look. The Decoder's write-up put it bluntly: the machines solved the problems, and the humans had to issue corrections.
What it means for you
You're probably not doing quantum field theory at your desk today. That's fine — the pattern is what transfers. When a tool can grind through a long, technical, multi-step task and get 4 out of 5 of them right cheaply, the useful skill stops being "can I do this by hand?" and becomes "can I tell when the answer is wrong?" Here's what that looks like in daily life.
At home
Think about the tasks that pile up because they're fiddly rather than hard. Comparing three mortgage offers across 20 years of payments. Untangling a health insurance explanation of benefits. Rebuilding a household budget in a spreadsheet. Hand a clean description of the problem to an AI with tools turned on, and you get a first draft in minutes instead of an evening. You'll still check the numbers — but checking is far faster than building from scratch.
At work
Most office jobs are full of research-grade-it's-not problems: pulling figures out of a 90-page report, reconciling two conflicting datasets, drafting a technical summary for people who don't share your background. This study is a decent signal that such work is increasingly automatable end to end, including the boring parts in the middle. The catch is the same one the physicists hit — if you can't verify the output, you're just hoping.
In business
Small teams feel this first. A three-person company can now produce competitor analysis, pricing models, first-pass legal summaries and internal documentation at a pace that used to require a hire. The cost math matters here: under $3 per hard task, run dozens of times a week, is cheaper than almost any alternative. Treat the model as a fast junior analyst whose work always gets reviewed before it goes out the door.
For students and study
This is the scenario that worries teachers, and it should — but it's also the one with the most upside. Use the AI as a tutor that shows its reasoning, not as an answer vending machine. Ask it to solve a problem, then try to poke holes in the solution. If you can't find the hole, you've probably learned the method. If you're testing the waters and don't want to pay for anything yet, MyKreaTool rounds up free AI tools you can try without entering a card.
For creative work
Long technical chains show up in creative jobs too: researching a documentary, building a consistent world for a novel, checking historical details across 40 pages of script. An agent that can search the web and run calculations can assemble that scaffolding quickly, leaving you the part that actually needs a human — deciding what's interesting.
For making money
Every delivery bottleneck you remove becomes capacity you can sell. Freelancers who used to quote two weeks for a technical audit can now quote three days and take on more clients, or keep the same client list and raise the quality. The risk isn't that AI replaces you; it's that someone with AI, better judgment and a sharper eye for errors undercuts you.
How to try it right now
You don't need a research budget. Work through these steps in order, cheapest option first.
1. Start free. Open a free AI chatbot — ChatGPT's free tier works fine as a starting point — and paste in a problem where you already know the answer. Watch how it reasons, not just what it concludes. If you want a broader menu of no-cost options, browse the free AI tools at MyKreaTool.
2. Turn the tools on. The big jump in this study came from giving the model a code interpreter and web search, not from a clever prompt. In most chat apps that's a toggle in the interface. A model that can run code can check its own arithmetic — which is exactly where long answers fall apart.
3. Read the actual benchmark. The problems and the story of the corrected answer keys are in the arXiv paper and the HTML version. Even skimming the introduction tells you what kind of task this approach suits: long, technical, method-driven work with a verifiable answer.
4. Try the real thing. The setup here was GPT-6 Astra running inside Codex with tools and search enabled. If you have access, give it one problem from your own work that would normally eat half a day.
5. Budget against the benchmark. Under $3 per problem on research-level physics is the reference point. If your tasks run at that scale, the economics work — and if a single run costs more, you're probably feeding it too little context.
6. Verify, then verify again. Before you trust anything, redo the check the physicists did: look for the answer you expect, and be genuinely open to finding out your own number was the wrong one.
Upsides and what changes
The headline upside is price. Under $3 per hard problem means you can run a real experiment today instead of waiting for a budget cycle. The second upside is speed on multi-step work that used to require a person to hold the whole chain in their head.
But the real shift is in how we judge expertise. For decades, the bottleneck was producing the answer. Now the bottleneck is knowing which answer to trust. That's arguably a better use of a specialist's time anyway — the experts in this study spent their hours auditing, not grinding.
And there's a quieter upside. The benchmark's answer key got better because the AI disagreed with it. That's a healthy loop: the tool finds the weak spot, the human fixes it, everyone's standard goes up.
Limitations
Be careful with the hype here. The authors themselves stress that TheoPhysEval tests difficult calculations using known methods — it does not test whether an AI can invent new physics, and that gap is huge. A benchmark with 50 unpublished problems is small, and the fact that reference answers needed correcting means the test itself was, at least partly, graded against a moving target. There's also the general rule that any single benchmark on any single model is a snapshot, not a verdict: results like this get retested, contested and revised all the time. On top of that, the run that scored 98% wasn't a bare chatbot — it was an agent with web search and code execution, which is a different product from the one in your browser tab. Your mileage on a Tuesday afternoon with a messy real-world question will be worse than 49 out of 50. Treat this as a strong signal about direction, not a guarantee about your specific task.
Conclusion
Here's the part worth remembering: the AI didn't just answer the physics questions. It answered them so well that the humans went back and found their own errors. That's a different kind of milestone than a leaderboard score, because it says the tool is now good enough to be a check on expert work — not a replacement for the expert, but a second opinion worth having.
Your one action today: pick a single task from your work that would normally take two or three hours, hand it to an AI with tools and web search switched on, and then check the output line by line — the way those physicists did. You might find a mistake in the answer. You might find one in your own.
Comments 0