Ai2 finds a novelty judge can fall from 93.5 percent to 40.9 percent when the prompt changes

Watch

Primary

On October 27, 2026, the Allen Institute for AI released a study of models that score whether a research idea is new. For GPT-5.4, one prompt scored 93.5 percent on pairs of human and generated ideas. A shorter prompt on the same pairs scored 40.9 percent. A simple Claude Opus 4.6 prompt cost $2.59 and beat two purpose-built judges that cost $30.66 and $82.05.

At a glance

Scores and costs are from Ai2's October 27 post. Macro-F1 is Ai2's measure of how well a judge separates novel and non-novel ideas.
Human ideas, higher novelty	154, top 10 percent of an ICLR 2026 area, reviews agree
Human ideas, lower novelty	145, bottom 10 percent, reviews raise originality concerns
GPT-5.4, hinted prompt	93.5 percent on human-plus-generated pairs
GPT-5.4, short prompt	40.9 percent on the same pairs
Simple Opus 4.6 judge	0.854 macro-F1, $2.59
Idea Novelty Checker	0.642 macro-F1, $30.66
Sakana AI Scientist check	0.772 macro-F1, $82.05
GPT-5.6 Sol with search	0.78 macro-F1, about 10 times the cost of Opus 4.6 at 0.89

How to read this claim

Measurement — The generated ideas are labeled less novel by assumption. Ai2 says those labels are weak.
Bill — Simple Opus 4.6 judge: 0.854 macro-F1, $2.59
Door — No second lab has rerun the benchmark.

Keep the habit · field card with prices and worked checks

What changed

Novelty checks had been validated mostly on human papers. This study measures the same judges on generated ideas and shows the prompt can reverse the result.

What it introduces

A public set of human and generated research ideas, plus a measured example of a novelty judge flipping when the prompt changes.

Who benefits Anyone using a model to accept or reject a research idea, and anyone comparing two idea generators by asking a third model which one is more novel.

Try this

If a system cites a novelty score, ask which prompt and which kind of idea. Treat a score on human papers as unproven for generated ideas. Ai2's cheaper baseline beat the purpose-built judges, so a higher bill is not evidence of a better judge.

What we do not know

The generated ideas are labeled less novel by assumption. Ai2 says those labels are weak.
The expert review covered 30 pairs, chosen toward the strongest judge's errors, not a random sample of the whole set.
The judges are the six models named in the study. A newer model is not in the table.
No second lab has rerun the benchmark.

Evidence

Primary Study. The accuracy swing and the costs are from Ai2's October 27 study. The labels on AI-generated ideas are Ai2's assumption, which the post says are weak.

Independent These are Ai2's runs. No second lab has rerun the pairs.

Sources

Study Primary · October 27, 2026

Still unclear? · Start here · ranks, prices, unknown The generated ideas are labeled less novel by assumption. Ai2 says those labels are weak.

The record

What happened

Ai2 published Old Ideas, Novel Problems on October 27, 2026, with a benchmark and a pipeline for testing novelty judges. The human-only set uses review text, not acceptance, as the novelty signal. An annotator agreed with the extracted signal in 47 of 50 checks. A second set pairs those 154 human ideas with 154 ideas from a basic generator that had no literature search. Ai2 treats the generated ideas as less novel, and says those labels are weak. An expert who reviewed 30 of those pairs, weighted toward the strongest judge's mistakes, preferred the human idea every time. Across six judges, the largest swings came from the prompt. Claude Opus 4.6 moved 21 points on the human-only set from the same kind of prompt change. A simple Claude Opus 4.6 setup, low reasoning and three votes, reached 0.854 macro-F1 for $2.59. Idea Novelty Checker scored 0.642 at $30.66. Sakana's AI Scientist novelty check scored 0.772 at $82.05. Both dedicated systems used Claude Opus 4.6.

Why it matters

Labs are using one model to decide whether another model's research idea is new. Ai2's result is that the decision depends on the wording. A judge that looks reliable on human papers can fall below chance on generated ideas when the prompt no longer hints that one idea is novel. Spending more on search did not fix it: GPT-5.6 Sol with extra reasoning and live arXiv search scored 0.78 macro-F1, 11 points behind a Claude Opus 4.6 setup at 0.89, at about ten times the cost.

When Ai2 compared ideas from two generators, about a quarter to a third of verdicts changed across prompt variants, averaged over six judges. That count does not require knowing which generator is actually more novel.

Dropping a warning not to treat incremental work as novel helped some judges and hurt others. Ai2 says no prompt was better for every model.

The six judges were GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6. Those are the models in the study, not a claim about newer releases.