936 AI Agents Wrote 936 Papers — Then Built Their Own Journal

Human Verified

Share to: , and

Small robot scientists working at desks around a giant glowing open book in a dark laboratory

From August to November, 936 AI agents working in 35 labs published 936 studies on a shared preprint server. The agents debated designs, retracted their errors, and founded a journal that re-runs code before accepting claims. The experiment, published by alphaXiv, tests whether machines can develop the norms of science.

How the Deployment Worked

The agents worked through a preprint server and a communication workspace, following a five-step routine: critique prior work, announce a hypothesis, run experiments and debate, link claims to artifacts, then freeze and share. A claim ledger tied every number in a paper to the file and field it came from.

Human operators chose the hardware, the API budgets and the broad themes, while the agents chose their own hypotheses and specific designs. Over twelve weeks the agents exchanged 9,500 messages and opened 229 chatrooms themselves, including topical hubs that grew as participation widened across the 35 labs.

Self-Correction in Public

One agent announced a 13.2-percentage-point improvement, then a preregistered control showed the comparison was flawed, and the agent retracted its headline before publishing. Among the 62 Stage 1 agents, 20 corrected themselves in public, and every correction came before the agent’s first upload, at a median lead of 21.4 hours.

A Journal That Re-Runs the Code

The agents founded the Journal of Agent Research, an agent-run venue where editors test referee objections against the data authors uploaded, instead of reading the paper. Text-only review has a weakness: a reviewer can flag a concern without establishing whether the flaw survives in a corrected version of the work.

In 16 flawed versions and their corrected revisions, text-only referees flagged 11 of 16 flawed papers and 10 of 16 corrected ones, barely telling repaired papers from flawed. Running 276 objections against the data overturned conclusions for 77 of 156 major objections, 49 percent, against 24 of 120 minor ones.

How Agent Papers Measured Up

The authors also compared research quality directly. They matched 202 agent papers with contemporaneous human arXiv preprints and collected 808 judgments from two language-model judges, in both presentation orders. On a ten-point rubric, agent papers scored higher on experimental rigor, 9.02 against 7.00, and on claim calibration, 9.32 against 7.23.

Human papers scored higher on novelty, 7.45 against 7.05, and on significance, 7.34 against 7.04. The pooled overall advantage for agent papers came entirely from Lab 1, and Labs 2 through 35 did not differ significantly from the matched human papers, a concentration the authors state plainly in the paper.

Our Read

The numbers matter less than the mechanism. Shared records let agents inspect, challenge and reuse earlier work, and executable review — running the objection against the data — changed judgments in half the major cases. That finding survives the caveats, and it is why the experiment matters beyond its scores.

Replit’s 32-agent swarm showed teamwork beating a stronger solo model on a coding benchmark, but this claim differs: a shared workspace can produce scientific norms. The limits are real: the study is descriptive, ratings are language-model judgments rather than expert peer review, and some practices were designed into the onboarding.

Why It Matters

If verification infrastructure holds up, agent communities could complement researchers in experimentation, replication and auditing, which is the division of labour the authors propose. The practical stake is concrete: alphaXiv has opened the agent preprint server publicly, so anyone can watch, inspect or join the next round of agent science.

Background

The closest prior work is AgentRxiv, a 2025 study asking whether agents could build on earlier papers to improve one benchmark task, reporting an 11.4 percent improvement on MATH-500. OpenAI’s research-acceleration report made a similar case for agents finishing tasks, but the alphaXiv deployment is the first at this scale.

Source: alphaXiv paper

Share to: , and

Faheem R. Khattak

Senior automation engineer and AI marketing journo covering LLMs, algorithms, analytics, core updates, Bing search, and technical SEO.

0 Comments

Leave a Reply

Optional Take Tag:

🛡️ Spam protection: Honeypot + Time-Trap Active

This site uses Akismet to reduce spam. Learn how your comment data is processed.