I have published a new pre-print on ResearchGate: Two Decades After the Robot Scientist: A Survey of AI Scientists, Autonomous Research, and Why Validation Still Sets the Pace
It has 83 pages, 340 references, and an expanded inventory of 62 research-agent systems and capabilities.
The paper was basically written by AI. I provided the prompts, direction, framing, and many many rounds of feedback. ChatGPT and Claude did almost all the literature synthesis, drafting, restructuring, and editing.
I then checked samples of the cited literature, bibliographic records, numbers, and the correspondence between references and claims. I did not independently verify every claim or every one of the 340 references. The paper states this explicitly.
The survey covers AI systems that participate in scientific research: searching the literature, generating hypotheses, designing experiments, writing and executing code, analysing results, operating laboratory equipment, writing papers, and reviewing them.
If you work in AI, research methods, scientific infrastructure, or recommender systems, I recommend reading it.
For two reasons.
- To see what AI can already do in scientific research.
- To understand AI Scientists and Autonomous Research. I expect this to become one of the defining topics of science over the coming years.
From the Robot Scientist to Autonomous Research
The idea of an AI scientist is older than ChatGPT.
In 2004, the Robot Scientist generated hypotheses about yeast gene function, selected experiments, executed them using laboratory automation, and interpreted the results. Earlier systems such as DENDRAL and BACON had already automated parts of scientific reasoning.
Current systems add foundation models, software tools, literature access, code generation, persistent memory, and increasingly complete experimental environments.
The survey traces this development through systems such as The AI Scientist, AI Scientist-v2, Google Co-Scientist, Coscientist, Robin, Virtual Lab, ERA, AlphaEvolve, scientific workbenches, self-driving laboratories, and industrial research agents.
Our earlier AI4Research overview started collecting many of these systems. The new survey puts them into a common historical and methodological structure.
What counts as an AI Scientist?
The survey does not call every AI tool used in science an AI scientist.
An AI scientist makes several research decisions and uses scientific evidence from earlier steps to choose later actions.
A literature assistant can summarize papers. A coding agent can implement an experiment. A bounded scientific agent can solve a particular research task. An AI scientist connects several such decisions into a research loop.
A simplified version is:
read → propose → experiment → evaluate → decide what to try next
This is where Autonomous Research becomes interesting. The system does not wait for a human to specify every next step. Evidence from one experiment affects what happens next.
The number of systems is growing quickly
The survey’s expanded inventory contains 62 systems.
It identifies 4 introductions in 2023, 13 in 2024, 29 in 2025, and another 16 by 21 September 2026. These numbers describe the selected survey inventory, not every research agent that exists.
The systems also cover very different forms of research. Some work mainly with literature. Others generate and execute code. Some interact with laboratory equipment. Others operate inside industrial experimentation systems.
Google reports AI-generated recommender-system changes that reached production at YouTube. Meta reports agents coordinating distributed recommender experiments that run for days across servers.
In recommender systems, we have been working in the same direction with AutoRecLab. The current AutoRecLab implementation takes a natural-language experiment description, derives requirements, generates code, executes it, checks the implementation, and revises it. In the reported evaluation, 8 of 9 implementation runs succeeded. A representative run cost about $1 in model/API fees.
Verification is the hard part
The survey’s central argument is that research generation is becoming easier to scale than research validation.
AI can generate hypotheses, code, experiments, analyses, figures, reviews, and manuscripts quickly. Each output still needs a test that fits the scientific claim.
EXP-Bench illustrates the current gap. Agents were asked to reconstruct experimental work from 51 AI papers. Across 461 tasks, the reported complete executable-experiment success rate was about 0.5%.
We found related problems in our evaluation of Sakana’s AI Scientist. Five of twelve runs failed at the coding stage. Runs that executed could still contain problems in experimental design or interpretation.
Other applications perform better. In a randomized ICLR 2025 field trial, reviewers who received AI critique revised 26.6% of their reviews, compared with 9.4% in the control group.
These results should not be combined into one “AI Scientist success rate”. The tasks are different and so are the tests.
Code execution establishes that code ran. Researchers still need to check whether the code implemented the intended experiment. A correctly implemented experiment then needs to support the conclusion reported in the paper.
This becomes harder when agents can generate hundreds or thousands of candidate results.
The scientific contract
The survey proposes a scientific contract for describing and evaluating a research run.
The contract records the research question, the actions delegated to the agent, the data and tools it can access, the evaluator, stopping rules, human responsibilities, and the conclusions that the resulting tests can support.
It also records whether the agent could inspect or influence the evaluator.
This is particularly relevant for autonomous systems. An agent that repeatedly queries a final benchmark may gradually optimize for that benchmark. The final test then becomes part of the search process.
Exploration and confirmation therefore need different mechanisms. Protected data, independent code, formal verification, new measurements, replication, or domain review may be required depending on the research claim.
The relevant unit of progress is eventually a result that survives these checks.
Who is going to read all these papers?
My survey illustrates a challenge: Successful Autonomous Research will produce more scientific output.
More hypotheses lead to more experiments. More experiments lead to more results. More results lead to more papers, reports, and data.
Human reading and reviewing do not scale at the same rate.
The survey therefore treats publication and review capacity as part of the scientific infrastructure around AI Scientists.
AI can also help here. Systems can check references, code, reporting requirements, statistics, manuscript consistency, and some classes of research errors before human review. In the ICLR experiment mentioned above, AI critique changed how human reviewers assessed papers.
I expect scientific publishing to adapt. More checks may happen automatically before a human reviewer sees a manuscript. Research records may become more machine-readable. Claims may be linked more directly to code, data, and experimental evidence.
That would not be the first large change in how science handles information.
Printing changed access to books. Computers changed calculation and simulation. The Internet changed access to papers and data. Search engines changed literature discovery.
AI Scientists and Autonomous Research will change the research process again.
A topic worth following
I expect AI Scientists and Autonomous Research to become a central research topic of the coming years.
The reason is practical. If AI systems can search a larger literature, generate more hypotheses, run more experiments, and analyse more results than researchers could process manually, then the organization of scientific work changes.
The interesting questions then concern autonomy, validation, scientific value, cost, human responsibility, publication, and the design of research infrastructure.
That is what the survey tries to organize.
And yes, AI helped write it
The survey was, for practical and research purposes, written by AI. I supplied the prompts, direction, framing, and feedback. ChatGPT and Claude produced most of the text and literature synthesis.
The complete 83-page survey took me less than ten hours to produce.
I checked samples of the cited literature, bibliographic records, numbers, and links between references and claims. I did not independently re-evaluate all 340 sources or verify every statement. The declaration in the paper makes that limitation explicit.
Regardless of how it was produced, I recommend reading the survey.
First, it is an example of what current AI can produce with limited human time.
Second, the subject itself is worth understanding. AI Scientists and Autonomous Research are moving from isolated demonstrations toward systems that participate in larger parts of the research process.
Read the pre-print on ResearchGate.
If you read it, I would be very interested to hear what you think: which parts are convincing, which parts are wrong, what is missing, and whether the proposed view of verification and scientific contracts is useful.
0 Comments