THE ESSENTIALS IN 30 SECONDS

  • Of 50 widely cited LLM findings re-tested on 20 models released between July 2024 and September 2026, 44% were classed as expired, 34% weakened, 16% persisting and 6% strengthened.
  • Capability failures such as fabricated citations and poor retrieval from the middle of long contexts largely faded, while sycophancy and LLM-judge self-preference held up or grew.
  • Expired claims had a median lifetime of 14.7 months; the authors caution that 50 claims yield wide confidence intervals.

Many prompting tips and evaluation rules of thumb trace back to papers that studied models from a year or two ago. A study published on November 2 by Stanford researchers Owen Queen and James Zou tries to measure how long such findings stay true. The pair re-ran 50 widely cited results about large language model behaviour on a panel of 20 models from eight developers, released between July 18, 2024 and September 29, 2026.

Most capability failures have faded

The authors converted each claim's key metric into a relative effect size, where 1 equals the effect reported in the original paper and 0 means it has disappeared. On that scale, 22 claims (44%) were classed as expired, 17 (34%) as weakened, 8 (16%) as persisting and 3 (6%) as strengthened.

The expired group is dominated by capability problems: counting letters, arithmetic in non-decimal bases, retrieving facts placed mid-context, inventing references, and dropping a correct answer when a user pushes back. Among the claims that expired, the median lifetime was 14.7 months, with individual cases ranging from about a month to roughly three years.

Preference-type behaviours proved more durable. Flattering or agreeing with users, answering when a model should abstain, and a left political lean persisted. LLM judges favouring their own outputs actually strengthened, as did praise that tracks a user's stated preference. Effects such as prompt injection, the reversal curse and judges' preference for longer answers weakened without vanishing.

What it means for practitioners

For developers, the practical message is that an evaluation design or prompt workaround justified by an older paper may be solving a problem that newer models no longer have, while some issues that are harder to spot remain. The authors recommend treating findings as observations about a specific model at a specific time, recording the model, version and date alongside any result, and publishing prompts and analysis code so others can re-test claims. They also advise against assuming an old result either still holds or has gone away without checking.

Limits

The authors stress that 50 claims is a small sample and that aggregate estimates carry wide intervals. Early expiration dates rest on only four 2024 models, some original models had to be replaced by substitutes, and all tests used each provider's default API settings, so reasoning budgets were not standardised. Panel verdicts can also hide large gaps between individual models: on reference fabrication, one model invented 69% of citations while another invented about 1%. The work is presented on a project page; the page links to no arXiv preprint or journal paper.