Joeran Beel
University of Siegen, Germany
Abstract
AI scientists can plan and execute experiments, analyse results, and produce findings supported by laboratory assays, clinical cohorts, or formal verification. But can they defend a methodological choice against relevant alternatives they never considered? No, not fully. Research in recommender systems (RecSys), machine learning (ML), information retrieval (IR), and natural language processing (NLP) involves intertwined decisions about data, splitting, baselines, candidate sets, tuning, metrics, and inference. Human expertise and foundation-model knowledge provide selective views of those alternatives. I propose that research communities build research atlases: shared, source-grounded, continuously maintained maps that strive to include every documented methodological decision point, method family, and consequential variant within a field, including rare and disputed approaches. Atlases catalogue options, conditions, and dependencies; AI scientists use them to construct experimental designs suited to their questions and to identify alternatives their initial plans missed. A shared atlas also supports repeated discussion between planning and critical agents: they can challenge a choice, examine the same documented alternatives, and revise a design. A study-specific record can preserve the alternatives, objections, reasons, revisions, and claim boundaries for independent assessment. Agents can assist both in building these maps from the literature and in navigating them at research time. I call for an initial atlas of offline RecSys evaluation and direct comparisons against strong retrieval and equally curated plain text. The aim is autonomous research capable of better methodological decisions, and more comprehensive defences of those decisions, than individual researchers or today’s agents can routinely provide.
Preprint and PDF
Keywords: AI scientists; autonomous research; autoresearch; AI4Science; research atlas; methodological design space; experimental design; research description language; recommender-system evaluation
1 Introduction
AI scientists are beginning to achieve remarkable scientific results. Systems such as the AI Scientist and Agent Laboratory can formulate research plans, run computational experiments, and produce reports (Lu et al., 2024; Schmidgall et al., 2025; Yamada et al., 2025). Several recent achievements have also been supported by evidence beyond the agents’ own accounts. Robin proposed ripasudil for dry age-related macular degeneration, and laboratory assays found increased retinal pigment epithelial-cell phagocytosis (Ghareeb et al., 2026). Co-Scientist independently proposed a hypothesis about bacterial gene transfer consistent with findings from an experimental study not yet published at the time (Gottweis et al., 2026). SPARK identified clinically relevant associations in an analysis covering 18 patient cohorts and more than 5,400 patients (Trost et al., 2026). Claude formalised the existing proof of Fermat’s Last Theorem in 11 days, with Lean checking the resulting formalisation (Castelvecchi, 2026). These achievements deserve recognition, and they also sharpen an important distinction: an external test can support a scientific result without establishing that the methods chosen to obtain it were the most appropriate among the relevant alternatives.
Can AI scientists defend their methods? Not fully. An agent may select a plausible procedure, implement it correctly, explain its choice, and receive approval from another agent. Yet neither can comprehensively defend that choice against alternatives both overlooked. In offline recommender evaluation, Krichene and Rendle (2020) show that scoring recommendations against sampled candidate items instead of the full catalogue can change algorithm comparisons even when the ranking metrics are calculated correctly. A reproducible experiment may therefore support a narrower or different claim than its authors intend. The omitted alternative leaves no trace in the code or execution log; a reviewer unfamiliar with it can approve the study without ever recognising the omission. Existing evaluation, replication, and verification mechanisms remain valuable, but they do not automatically establish that the original methodological design was well chosen.
Humans face the same challenge, because expertise is necessarily selective, especially when a study crosses subfields or adopts an unfamiliar protocol. In a many-analysts study, 29 teams analysed the same data and question but arrived at substantially different results through different analytical decisions; differences in expertise and peer-rated analysis quality did not explain the variation (Silberzahn et al., 2018). Kapoor and Narayanan compiled evidence from earlier investigations of data leakage affecting at least 294 papers across 17 fields (Kapoor and Narayanan, 2023). These findings illustrate how consequential methodological decisions can be; they do not measure the number of alternatives researchers failed to consider. As autoresearch advances toward increasingly autonomous research, our ambition should extend beyond reproducing the methodological breadth of an individual expert.
I propose that research communities build research atlases so AI scientists can make better methodological decisions and defend them more comprehensively. I define a research atlas as a shared, versioned, source-grounded map of a field’s methodological decision points, documented alternatives, applicability conditions, dependencies, and supporting evidence. Its explicit goal is to include every documented method family and consequential variant, whether common, rare, disputed, historical, or newly introduced. AI scientists would consult the atlas before choosing an experimental design, discover alternatives absent from their initial plans, and assess which methods fit their questions and intended claims. Planning and critical agents could challenge each other against the same documented alternatives and revisit their choices before execution. Recording the options considered, objections, reasons, and revisions would make these discussions assessable and encourage more considered method selection. AI agents can help communities compile and maintain the atlas from the literature and navigate it during research. I call for a first atlas of offline RecSys evaluation and for its integration into planning and review, with applications extending to ML, IR, NLP, and other disciplines.
2 The Problem: A Methodological Design Space Too Large to Know
The methodological design space of an experiment is vast, hierarchical, and interdependent. Wicherts and colleagues catalogued 34 researcher degrees of freedom in psychological studies and explicitly considered their list non-exhaustive; work on multiverse analysis and ML underspecification demonstrates how defensible choices can change conclusions (D’Amour et al., 2022; Steegen et al., 2016; Wicherts et al., 2016). Even the comparatively narrow problem of joint algorithm selection and hyperparameter optimisation spans numerous methods and conditional parameter spaces (Thornton et al., 2013). Data splitting in RecSys evaluation illustrates the branching. One must choose whether interactions, users, items, sessions, or time periods are separated, then among random holdout, leave-one-out (LOO), cross-validation, grouped separation, global temporal splitting, and rolling or expanding windows. LOO itself may hold out a random or a user’s last interaction; temporal procedures differ in cutoffs, target events, validation construction, and training-window updates. A user-wise LOO split can include training interactions occurring after another user’s test event. Gusak et al. (2025) show that LOO and global-temporal variants can change evaluations and model rankings, while Cañamares et al. (2020) show how offline-evaluation choices can yield opposing conclusions. Similar branches arise in candidate-set construction, baseline tuning, metric implementation, and statistical analysis (Krichene and Rendle, 2020; Shehzad and Jannach, 2023; Tamm et al., 2021; Urbano et al., 2019). Selecting an apparently ordinary protocol can therefore require examining a substantial network of consequential alternatives.
Foundation models cannot be assumed to know that network comprehensively. Their training exposes them to large quantities of scientific text, but it does not produce an explicit, independently audited inventory of methods, conditions, and dependencies. A model may accurately explain LOO and global temporal splitting while overlooking a less familiar temporal target-selection variant; competence with the options recalled gives no indication of which others were missed. The gap matters most for rarely documented approaches. Language models show weaker recall of long-tail factual knowledge (Kandpal et al., 2023); that evidence concerns facts rather than measured omissions in methodological planning, but it motivates particular attention to methods reported only once or twice. AInstein examines AI-generated research solutions based on pretrained knowledge, while LitLLMs investigates LLM-based literature retrieval and synthesis (Agarwal et al., 2025; Mishra et al., 2025). These capabilities are valuable, yet search results depend on the queries issued and provide no systematic account of branches never queried. Even a capable research agent can retrieve convincing support for its chosen method without finding a consequential alternative. A reviewing agent may share this blind spot, particularly when both rely on the same foundation model or overlapping training literature; research on correlated model errors and AI oversight documents the broader risk of non-independent judgements (Goel et al., 2025; Kim et al., 2025). More capable language models improve search and reasoning, but neither larger pretraining corpora nor a second model automatically provides inspectable methodological coverage.
Human expertise offers critical judgement, but it does not solve the coverage problem. Researchers know many options within their specialisations, yet cannot routinely survey every documented variant when moving into neighbouring fields or combining unfamiliar protocols. A supervisor can independently search for missing alternatives, but doing so for each experiment recreates a substantial part of the work delegated to the agent. The underlying problem is shared: both the person planning a study and the person reviewing it tend to evaluate choices against the methodological alternatives they already know. Better research decisions require wider access to alternatives during planning and a shared reference whenever those choices are challenged or revised.
3 The Vision: Research Atlases for Better Science
3.1 Mapping the Entire Methodological Design Space
I propose that research communities systematically map their methodological design spaces. A research atlas would strive to include every documented decision point, method family, and scientifically consequential variant in a defined field. The target is ambitious and concrete at once. The atlas catalogues the components of experimental designs and the relationships among them; AI scientists compose complete pipelines for their research questions. For example, LOO with a randomly selected interaction and LOO with a user’s last interaction are distinct methodological variants because they encode different temporal assumptions. Individual cutoff dates or training-window lengths can be represented as parameters of a temporal procedure. The same distinction applies across fields: options merit separate entries when they materially differ in their procedure, assumptions, applicability, or evidential consequences. Documenting the available components does not require precomputing every combination or every numerical parameter setting.
Comprehensive coverage is central to the proposal, so a method reported in one publication deserves a place in the inventory just as a method reported in a thousand does; its inclusion establishes that the option is documented, without implying equal scientific merit. An atlas should preserve uncommon, unsuccessful, historical, and contested approaches alongside widely accepted ones. Each entry links the option to its sources, conditions of applicability, dependencies, incompatible combinations, resource implications, and disputed evidence. Stable identifiers and public versions allow corrections to improve the shared record. A sufficiently broad scholarly index offers a plausible basis for approaching this goal. For example, if a system with Google Scholar’s literature breadth could reliably recognise RecSys experiments and analyse their full texts, it could systematically extract documented methodological families and rare variants for expert checking (Martín-Martín et al., 2021). Independent literature and expert audits should continually challenge its coverage. The practical task is to discover and verify overlooked methods, while incorporating new ones as the field develops.
An entry is a short structured record. For the global temporal split with successive prediction targets, that record gives the option a stable identifier and a name; states the decision point it answers, namely how training and evaluation data are separated in time; places it inside the global-temporal family; and defines the procedure in one or two sentences: a single cutoff separates the periods, and each post-cutoff interaction is predicted in turn from the history available at that moment. It then records what a reader needs in order to judge the option. Applicability: questions about repeated prediction after a common point in time. Dependencies: a choice of cutoff, and a validation scheme built on the same timeline. Incompatibilities: user-specific cutoffs, which break the common timeline this option assumes. Cost: evaluating successive targets and updating user histories can increase computation relative to a single-target holdout, depending on the implementation. Evidence: the publications that introduce, use, or criticise it, each as a passage that can be opened and read. Disputed claims stay attached with attribution rather than being resolved into one verdict, and each entry carries a version and a correction history. Multiply that record by every documented option at every decision point in a field, link the records by their dependencies, and the result is an atlas.
Geographic information systems provide a useful model, because a geographic atlas maps local roads as well as motorways, their connections, access restrictions, and changing conditions. A navigation system uses that shared map to find a suitable route for a particular journey. Similarly, a research atlas connects choices about data preparation, splitting, optimisation, evaluation, and inference, exposing the options relevant to a particular question. A traveller choosing between two known roads may overlook a better third route; a scientist choosing between two familiar methods can overlook a consequential third alternative. The map expands the choices visible before the decision and provides a common reference for examining the decision afterwards. Which route is appropriate still depends on the destination; in science, it also depends on assumptions, available resources, and the claims the evidence is intended to support.
3.2 From the Map to Better Experimental Designs
The primary benefit is better methodological planning. An AI scientist consults the atlas before committing to its experimental protocol. It identifies the decision points activated by the research question, follows their conditional branches, compares relevant alternatives, and selects a defensible combination of methods. A broad map does not oblige an agent to run every possible experiment. It equips the agent to discover alternatives it would otherwise miss and to evaluate their fit before committing resources. This is where autonomous research can surpass the methodological breadth of an individual researcher: agents can navigate a collectively curated repertoire of methods across many stages of an experiment, repeatedly and at low marginal effort.
In practice, the agent queries the relevant parts of the atlas instead of inspecting every entry. A research question and a few constraints identify which decision points apply, and the atlas returns the decision points and documented options, each linked to the literature supporting its description. Unfolding one option exposes the sub-decisions it creates and the ones it rules out, so the agent can descend only the branches its question activates and stop where further detail stops mattering. Unfamiliar options appear alongside familiar ones, with the conditions needed to judge their relevance. The agent then composes a pipeline from the components it has inspected, and records which it rejected.
Consider an AI scientist evaluating a sequential recommender for next-item prediction after a common deployment cutoff. Its initial plan follows the familiar LOO protocol. That choice reflects published practice: among 75 sequential-recommendation papers from RecSys, SIGIR, and CIKM in 2022–2024, 77.3% used LOO, while 16% used global temporal splitting and 6.7% used a global temporal split in a next-item setting (Gusak et al., 2025). The atlas draws attention to the global-temporal family and to variants that differ in how post-cutoff targets and validation periods are constructed. A user-wise LOO protocol may draw on events that would not have existed at the common deployment cutoff. For the stated deployment-oriented question, a global temporal split with appropriately constructed prediction targets and validation provides a closer match between protocol and claim. The agent changes its initial design because it has encountered a relevant alternative and understood its implications.
An agent reproducing a historical LOO benchmark faces the same map and a different answer. The atlas presents the global-temporal alternatives, but the appropriate choice is to retain the published LOO protocol and limit the interpretation to that benchmark’s conditions. The atlas supports both outcomes: a changed method when the original choice does not fit the question, and a retained method when it does. Its benefit lies in making the comparison informed and explicit. This benefit begins with the methodological map alone, before any further layer is added. A separate, later project could link the map to structured descriptions of completed studies and their results; such a historical corpus is not a prerequisite for using the atlas to design new experiments.
3.3 From Better Decisions to Defensible Research
A shared research atlas makes methodological criticism an ongoing dialogue. A planning agent might propose LOO; a critical agent can then ask why a global temporal split was overlooked, pointing to the same atlas entries, applicability conditions, and sources. The planner can revise the protocol, defend LOO for a benchmark reproduction, or question the critic’s interpretation. Each response may raise further decisions, and discussion can resume when new evidence emerges. Requiring such exchanges should help agents reconsider familiar choices and select methods better suited to their research questions. A study-specific decision record should preserve the alternatives actually examined, objections, responses, reasons, revisions, and the claims each choice is intended to support. The atlas remains the shared reference against which this evolving account can be assessed.
The shared atlas gives the exchange a stable vocabulary and a source-linked reference. Without it, a critic and planner can still consult the literature, but they may repeatedly have to reconcile names, reconstruct variants, and establish their applicability. With a shared map, the critic names an option, cites the entry, and points to the condition under which it applies. The planner can adopt it, explain why its applicability condition does not hold, or challenge the entry’s reading of the sources. Each of those responses can be examined by a third party, and human supervisors can consult the same documented alternatives and sources, then assess the reasons the agents recorded.
The decision record can initially be a structured, source-linked account attached to the experiment. A future research description language (RDL) could standardise such exchanges and eventually support machine-readable descriptions of entire papers. Defining that language is a separate research challenge; the atlas is useful before it exists. When an initial record is time-stamped and fixed before results are observed, it can also support preregistration (Bakker et al., 2020). Later objections and revisions should remain traceable rather than overwrite the original rationale.
The dialogue complements other scientific checks rather than replacing them. One check asks whether the computations ran correctly, a second whether the run followed the agreed design, a third whether the reported findings trace back to the analyses behind them. All three take the design as given, and all three can pass while the study answers the wrong question. The atlas supports the check that comes first: do the chosen methods fit the research question, and was a consequential alternative overlooked? A reviewer may challenge the plan before execution and revisit it when new results appear. Because both participants can consult the atlas and its sources, each can examine methods initially unfamiliar to the other. Figure 1 illustrates their iterative exchange without prescribing an agent architecture or RDL implementation.
method choices
suggest alternatives
reasons and revisions
review results
The proposal is practical now for a reason that did not hold earlier. AI agents can assist on both sides of the atlas. They can search literature collections for methods and variants, propose source-linked entries, detect candidate inconsistencies, and help experts curate updates. Once the map exists, agents can consult it repeatedly, following more branches than an individual researcher can reasonably inspect at every decision. An inventory too large for a human to check at every decision is exactly the right size for an agent. Developing an initial option map is also a more focused undertaking than formalising entire scientific papers: the first release needs decision points, options, source links, and enough conditions to support navigation. Community validation and continued maintenance matter, but the effort can be shared and reused across many studies. AutoRecLab offers one possible experimental testbed for atlas-assisted planning (Baumgart et al., 2026); the atlas itself should serve many research agents and human users.
4 Related Work
Four research traditions supply the pieces an atlas needs: scientific maps, catalogues of methodological alternatives, machine-readable experiment descriptions, and agent-assisted knowledge extraction. Each supplies something the proposal depends on, and none supplies the object itself.
Mapping science is an established idea. The ISI Atlas of Science described research fronts in 1981, while Börner’s Atlas of Science mapped the structure and evolution of scientific knowledge (Börner, 2010; Garfield, 1981). Evidence and gap maps organise defined study populations, interventions, outcomes, and gaps; EviAtlas makes such maps navigable (Haddaway et al., 2019; Saran and White, 2018). These projects demonstrate the value of shared research maps, and my proposal shifts the principal object of mapping to the detailed methodological alternatives available when designing a new experiment.
Methodological alternatives and design rationale have important predecessors. MacLean and colleagues’ Questions–Options–Criteria framework represented design alternatives and their justifications (MacLean et al., 1991). Multiverse analysis and work on researcher degrees of freedom examine how different defensible analytical choices affect outcomes (Steegen et al., 2016; Wicherts et al., 2016). In RecSys, FEVR classifies evaluation facets and Cañamares et al. (2020) analyse the effects of different offline-evaluation options (Zangerle and Bauer, 2023). These approaches provide substantial conceptual and empirical foundations, which a research atlas would assemble such knowledge into a comprehensive, source-grounded, continually maintained inventory that AI scientists can consult across an entire experimental design.
Machine-readable experiments and formal requirements already exist. Robot Scientist Adam used formal scientific representations, and OBI, ISA, and SED-ML model aspects of investigations or simulation experiments (Bandrowski et al., 2016; King et al., 2009; Rocca-Serra et al., 2010; Waltemath et al., 2011). Experiment Databases, OpenML, ML-Schema, and ORKG structure research tasks, runs, and scholarly information (Brack et al., 2022; Publio et al., 2018; Vanschoren et al., 2014, 2012). Croissant Tasks supports machine-actionable ML evaluation descriptions (Benjelloun et al., 2026). These systems supply building blocks for structured scientific workflows, to which the atlas adds a shared reference for discovering and assessing the alternatives from which an experimental protocol may be chosen, before and after execution.
AI-assisted knowledge extraction and scientific agents provide enabling techniques. Work on scholarly knowledge graphs and machine-assisted schema construction addresses the extraction, organisation, and revision of structured scientific information (Arda et al., 2026; Brack et al., 2022; D’Souza et al., 2026; Sadruddin et al., 2026). AutoML searches alternatives within defined tasks and evaluation procedures, while the AI Scientist, Agent Laboratory, and AutoRecLab automate broader stages of research planning and experimentation (Baumgart et al., 2026; Lu et al., 2024; Schmidgall et al., 2025; Thornton et al., 2013). Data-to-paper and Scientist-One link analyses and claims to inspectable evidence (Ifargan et al., 2025; Sinha et al., 2026); ReviewerToo studies AI-assisted peer review (Sahu et al., 2025). My proposal connects these developments through a common, field-wide methodological map: the same curated alternatives inform the agent’s initial choice and a reviewer’s later assessment.
Across these four traditions, one object remains unbuilt. Science maps chart published literature; design-rationale frameworks record the alternatives a single project weighed; experiment formats describe a study that has already been designed; extraction systems produce schemas rather than method inventories. None of them maps the methodological options a field has documented, at the granularity at which an experiment is actually designed, in a form an agent can query before choosing. Building that object is what I propose.
5 Alternative Views
Five objections deserve an answer, beginning with the one that concerns the architecture itself.
A shared atlas can propagate shared errors. If an atlas omits a relevant method, a planning agent and reviewer consulting it may both overlook that method. Yet planning and review already rely on overlapping model knowledge and scientific literature. Long-tail knowledge is less reliably recalled, and correlated model errors can limit independent oversight (Goel et al., 2025; Kandpal et al., 2023; Kim et al., 2025). An explicit atlas makes its documented contents, sources, coverage procedures, and revision history inspectable. When a missing option is discovered, it can be incorporated for all subsequent users. Undiscovered omissions remain possible, so literature access, independent coverage audits, and competing assessments remain essential. These requirements strengthen a shared atlas rather than undermine the case for building one.
Could deep research reconstruct the alternatives whenever an experiment is planned? Literature search is essential for finding unfamiliar or newly published methods. But a ranked list of documents does not establish the branching structure of methodological choices or identify options that the queries failed to retrieve. A routine offline RecSys experiment may run for a few hours on a workstation; reconstructing a field-wide inventory for each such experiment and again for its review would repeat substantial work. A maintained atlas makes that work reusable and gives both planner and reviewer a stable, versioned reference. Better search agents make it easier to discover and update entries, further strengthening the atlas’s value.
Might a comprehensive atlas encourage conformity? A map that favours common methods could entrench convention, and uncritical reliance on decision aids can introduce errors (Goddard et al., 2012). Scientists consult textbooks and surveys to expand their methodological knowledge without treating them as binding instructions. A research atlas extends this practice with broader coverage, including rare and disputed options and the limits of their evidence. Agents remain free to draw on other sources, challenge atlas entries, or devise methods absent from the map. They should explain their choices in light of relevant alternatives. The aim is more appropriate method selection, measured separately from conformity to established practice.
Who will maintain the atlas and protect it against manipulation? A resource consulted by many research agents will attract contributors seeking visibility for their methods, alongside legitimate experts correcting omissions. Public sources, attributed claims, contribution histories, independent expert adjudication, conflict-of-interest rules, and open correction procedures can reduce biased entries and promotional descriptions. Community stewardship must preserve the distinction between documenting that a method exists and judging how strong its supporting evidence is. The same incentive to appear in the atlas can support its growth if entries remain verifiable.
Might well-curated plain text suffice? A searchable text collection containing the same expert-checked information would be useful, with less initial structure to maintain. I nevertheless expect explicit decision points, typed alternatives, conditional relationships, and stable identifiers to make option discovery and independent comparison more reliable across repeated studies. This is a distinction worth testing: the research agenda compares the same curated information in plain text and atlas form, separately from a baseline using literature retrieval alone.
6 A Call to Action: Build and Test Research Atlases
I am asking research communities to build methodological atlases, and my claim is that the first draft of one is a weekend project that nobody has done. Offline RecSys evaluation is the field to start with: choices about splitting, candidate construction, baselines, and metrics are well documented and can change the conclusions drawn from experiments (Cañamares et al., 2020; Gusak et al., 2025; Krichene and Rendle, 2020; Zangerle and Bauer, 2023). A validated atlas is a long-term community undertaking, whereas a substantial, source-linked draft is a weekend’s work, and I expect a small team to produce one over a weekend with today’s research agents and an available full-text corpus. That draft is already useful for discovery and correction, and its absence after two decades of published RecSys evaluation is the part worth explaining. Three milestones turn the proposal into a practical research programme.
Milestone 1: build a source-linked atlas draft.
Start with a defined, accessible collection of full papers, such as a selected period of ACM RecSys proceedings. Use agents to identify documented methodological decision points, options, and variants; extract the passages that support them; reconcile terminology; and link applicability conditions and dependencies. Publish the resulting inventory with stable identifiers, source links, version information, and explicitly unresolved entries. My estimate is that a substantial initial draft could be produced in hours to a few days, potentially for a few hundred euros in model-inference costs once the full texts are available. This is a testable feasibility expectation, not a measured cost or a claim of audited completeness. Nothing in this step requires a curation committee, a standards process, or a new model. Independent experts are needed afterwards, to check entries and contested interpretations, search for overlooked long-tail variants, and revise the map. The Gene Ontology demonstrates that a versioned, evidence-linked knowledge resource can be maintained through sustained community stewardship (The Gene Ontology Consortium, 2023). Transparent contributions, conflict-of-interest rules, quality checks, and corrections matter for the atlas’s continued reliability, but do not have to precede a useful first draft.
Milestone 2: put the atlas inside a research agent.
AutoRecLab already turns a natural-language research request into explicit experimental requirements (Baumgart et al., 2026). Give its planner access to the atlas before the requirements are fixed, give a separate critic the same access, and let them discuss the decision: the planner proposes a protocol; the critic identifies a documented alternative; the planner adopts it, explains why it is inapplicable, or challenges the entry. Record the alternatives raised, the reasons exchanged, and any revisions. Compare the same research tasks under four conditions: model knowledge alone, strong literature retrieval, identical expert-curated information as searchable plain text, and structured atlas access. Match budgets and measure consequential alternatives discovered, methodological fit to the intended claim, and total cost across repeated studies. AutoRecLab is one convenient testbed, while the atlas remains independent of any particular agent architecture.
Milestone 3: test whether review and methodological scrutiny improve.
Give human and AI reviewers comparable studies with and without atlas access, and measure consequential omissions detected, false alarms, review time, and cost. Reproduction and robustness studies offer another application: an atlas can identify documented protocol variants and their applicability conditions, helping investigators select systematic sensitivity analyses and interpret changes in results. Whether an implementation correctly follows the chosen protocol still requires code inspection, execution checks, and empirical verification. In this way, the atlas extends the reusable methodological knowledge available for multiverse and robustness analyses (Steegen et al., 2016). If atlas-assisted planning, review, and scrutiny show no advantage over strong retrieval at comparable budgets, the proposed benefits are not supported in those settings. If identically curated plain text performs as well as the structured atlas, the formal representation has not earned its additional cost, even if the shared inventory remains valuable.
7 Conclusion
An AI scientist can design an experiment, run it, defend every step of it, and still answer a question nobody asked, because the method that would have answered the intended one was never on the table. Checking the execution catches nothing here, and neither does reproducing the result, since both take the design as given. A second agent trained on the same literature is liable to miss the gap for the same reason the first did. The omission leaves no trace in the plan, the code, or the log, which is why it has gone unaddressed while everything downstream of it has been automated.
Research atlases are my answer: shared, source-grounded, versioned maps of the documented methodological decision points and consequential variants a field has produced, queryable by the agents that design experiments and by the reviewers who judge them. The components already exist, since scientific mapping, design rationale, machine-readable experiments, and agent-assisted extraction have each built a part. What is missing is the object itself, for a reason that is plain enough: compiling such an inventory by hand was never worth anyone’s career, and consulting it by hand at every decision would have been worse. Agents change both halves of that calculation, because they can read a field’s published methodology at scale and traverse the result at every decision, which is precisely the combination that was unavailable when design-rationale systems were proposed and then abandoned (MacLean et al., 1991).
The ambition is not to make AI scientists as methodologically competent as a good researcher. It is to make them better than any researcher can be alone, because no individual has ever had access to everything their field knows about how to run an experiment. That access is now buildable, and every method identified, every disagreement recorded, and every correction filed improves the reference for the next study.
We have taught machines to run our experiments. Let us give them a map of how else those experiments could have been run.
Declaration on Generative AI
This paper is the result of several hundred prompts to ChatGPT and Claude and dozens of drafts that I read, checked, and sent back. The agents proposed; I kept disagreeing until I had what you are reading now.
References
Agarwal, S., Sahu, G., Puri, A., Laradji, I.H., Dvijotham, K.D., Stanley, J., Charlin, L., Pal, C., 2025. LitLLMs, LLMs for literature review: Are we there yet? Transactions on Machine Learning Research.
Arda, B., Yavuz, A., Gerry, P., Lobentanzer, S., Sarwar, N., Giner-Miguelez, J., Chen, K., Zhang, L., Sachan, M., Akhtar, M., 2026. CroissantMiner: Automated extraction and validation of croissant metadata for ML datasets. https://doi.org/10.48550/arXiv.2610.07132
Bakker, M., Veldkamp, C.L.S., Assen, M.A.L.M. van, Crompvoets, E.A.V., Ong, H.H., Nosek, B.A., Soderberg, C.K., Mellor, D., Wicherts, J.M., 2020. Ensuring the quality and specificity of preregistrations. PLOS Biology 18, e3000937. https://doi.org/10.1371/journal.pbio.3000937
Bandrowski, A., Brinkman, R., Brochhausen, M., Brush, M.H., Bug, B., Chibucos, M.C., Clancy, K., Courtot, M., Derom, D., Dumontier, M., Fan, L., Fostel, J., Fragoso, G., Gibson, F., Gonzalez-Beltran, A., Haendel, M.A., He, Y., Heiskanen, M., Hernandez-Boussard, T., Jensen, M., Lin, Y., Lister, A.L., Lord, P., Malone, J., Manduchi, E., McGee, M., Morrison, N., Overton, J.A., Parkinson, H., Peters, B., Rocca-Serra, P., Ruttenberg, A., Sansone, S.-A., Scheuermann, R.H., Schober, D., Smith, B., Soldatova, L.N., Stoeckert, C.J., Jr., Taylor, C.F., Torniai, C., Turner, J.A., Vita, R., Whetzel, P.L., Zheng, J., 2016. The ontology for biomedical investigations. PLOS ONE 11, e0154556. https://doi.org/10.1371/journal.pone.0154556
Baumgart, M., Meister, P., Krell, J., Schmidt, M., Gipp, B., Beel, J., 2026. AutoRecLab: An Autonomous Recommender Systems Lab, in: Proceedings of the 20th ACM Conference on Recommender Systems (RecSys ’26), RecSys ’26. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3773078.3841273
Benjelloun, O., Martins Bianco, L., Guyon, I., Khuong, T.G.H., Lebensold, J., Lobentanzer, S., Oala, L., Rachmat, B.K., Ullah, I., Vahidi, P., Vanschoren, J., 2026. Croissant tasks: A metadata format for reproducible machine learning evaluations. https://doi.org/10.48550/arXiv.2605.29786
Börner, K., 2010. Atlas of science: Visualizing what we know, 1st ed. MIT Press, Cambridge, MA.
Brack, A., Hoppe, A., Stocker, M., Auer, S., Ewerth, R., 2022. Analysing the requirements for an open research knowledge graph: Use cases, quality requirements, and construction strategies. International Journal on Digital Libraries 23, 33–55. https://doi.org/10.1007/s00799-021-00306-x
Cañamares, R., Castells, P., Moffat, A., 2020. Offline evaluation options for recommender systems. Information Retrieval Journal 23, 387–410. https://doi.org/10.1007/s10791-020-09371-3
Castelvecchi, D., 2026. Anthropic AI “formalizes” proof of fermat’s last theorem—a milestone for mathematics. Nature. https://doi.org/10.1038/d41586-026-02822-9
D’Amour, A., Heller, K., Moldovan, D., Adlam, B., Alipanahi, B., Beutel, A., Chen, C., Deaton, J., Eisenstein, J., Hoffman, M.D., Hormozdiari, F., Houlsby, N., Hou, S., Jerfel, G., Karthikesalingam, A., Lucic, M., Ma, Y., McLean, C., Mincu, D., Mitani, A., Montanari, A., Nado, Z., Natarajan, V., Nielson, C., Osborne, T.F., Raman, R., Ramasamy, K., Sayres, R., Schrouff, J., Seneviratne, M., Sequeira, S., Suresh, H., Veitch, V., Vladymyrov, M., Wang, X., Webster, K., Yadlowsky, S., Yun, T., Zhai, X., Sculley, D., 2022. Underspecification presents challenges for credibility in modern machine learning. Journal of Machine Learning Research 23, 1–61.
D’Souza, J., Sadruddin, S., Rula, A., Bossler, A., Fullana, A., Bas, E., Ather, S., Circi, D., Chen, A., Brinson, L.C., Columbus, A., Demetriou, G., Jeong, D., Kumar, T., Krüger, F., Genehr, S., Budde-Sagert, K., Leonescu, A., Lodola, F., Florindi, C., Murthy, G.B., Olagbile, S.O., Riasat, N., Sha, Y., Shen, K., Yang, S., 2026. SciSchema.org: A multidisciplinary collection of schemas for structured scientific process descriptions. https://doi.org/10.48550/arXiv.2607.27955
Garfield, E., 1981. Introducing the ISI Atlas of Science: Biochemistry and molecular biology, 1978/80. Current Contents 5–13.
Ghareeb, A.E., Chang, B., Mitchener, L., Yiu, A., Szostkiewicz, C.J., Shved, D., Gyimesi, G.J., Laurent, J.M., Wright, S.M., Razzak, M.T., White, A.D., Finnemann, S.C., Hinks, M.M., Rodriques, S.G., 2026. A multi-agent system for automating scientific discovery. Nature 655, 497–505. https://doi.org/10.1038/s41586-026-10652-y
Goddard, K., Roudsari, A., Wyatt, J.C., 2012. Automation bias: A systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association 19, 121–127. https://doi.org/10.1136/amiajnl-2011-000089
Goel, S., Strüber, J., Auzina, I.A., Chandra, K.K., Kumaraguru, P., Kiela, D., Prabhu, A., Bethge, M., Geiping, J., 2025. Great models think alike and this undermines AI oversight, in: Singh, A., Fazel, M., Hsu, D., Lacoste-Julien, S., Berkenkamp, F., Maharaj, T., Wagstaff, K., Zhu, J. (Eds.), Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research. PMLR, pp. 19621–19678.
Gottweis, J., Weng, W.-H., Daryin, A., Tu, T., Sirkovic, P., Myaskovsky, A., Glowaty, G., Weissenberger, F., Orlandi, A., Popovici, D., Palepu, A., Rong, K., Tanno, R., Saab, K., Zhang, F., Blum, J., Carroll, A., Kulkarni, K., Tomašev, N., Zverinski, D., Rendulic, I., Vedadi, E., Hasler, F., Rimanic, L., Boia, M., Budiselic, I., Feinstein, B., Bellaiche, M., Sheffer, T., Freyberg, J., Ratcliff, J., Bertolli, O., Chou, K., Hassidim, A., Gokturk, B., Vahdat, A., Guan, Y., Dhillon, V., Vaishnav, E.D., Lee, B., Costa, T.R.D., Penadés, J.R., Peltz, G., Matias, Y., Manyika, J., Hassabis, D., Xu, Y., Kohli, P., Pawlosky, A., Karthikesalingam, A., Natarajan, V., 2026. Accelerating scientific discovery with Co-Scientist. Nature 655, 487–496. https://doi.org/10.1038/s41586-026-10644-y
Gusak, D., Volodkevich, A., Klenitskiy, A., Vasilev, A., Frolov, E., 2025. Time to split: Exploring data splitting strategies for offline evaluation of sequential recommenders, in: Proceedings of the 19th ACM Conference on Recommender Systems. Association for Computing Machinery, pp. 874–883. https://doi.org/10.1145/3705328.3748164
Haddaway, N.R., Feierman, A., Grainger, M.J., Gray, C.T., Tanriver-Ayder, E., Dhaubanjar, S., Westgate, M.J., 2019. EviAtlas: A tool for visualising evidence synthesis databases. Environmental Evidence 8, 22. https://doi.org/10.1186/s13750-019-0167-1
Ifargan, T., Hafner, L., Kern, M., Alcalay, O., Kishony, R., 2025. Autonomous LLM-driven research—from data to human-verifiable research papers. NEJM AI 2. https://doi.org/10.1056/AIoa2400555
Kandpal, N., Deng, H., Roberts, A., Wallace, E., Raffel, C., 2023. Large language models struggle to learn long-tail knowledge, in: Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., Scarlett, J. (Eds.), Proceedings of the 40th International Conference on Machine Learning, Proceedings of Machine Learning Research. PMLR, pp. 15696–15707.
Kapoor, S., Narayanan, A., 2023. Leakage and the reproducibility crisis in machine-learning-based science. Patterns 4, 100804. https://doi.org/10.1016/j.patter.2023.100804
Kim, E.M., Garg, A., Peng, K., Garg, N., 2025. Correlated errors in large language models, in: Singh, A., Fazel, M., Hsu, D., Lacoste-Julien, S., Berkenkamp, F., Maharaj, T., Wagstaff, K., Zhu, J. (Eds.), Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research. PMLR, pp. 30038–30066.
King, R.D., Rowland, J., Oliver, S.G., Young, M., Aubrey, W., Byrne, E., Liakata, M., Markham, M., Pir, P., Soldatova, L.N., Sparkes, A., Whelan, K.E., Clare, A., 2009. The automation of science. Science 324, 85–89. https://doi.org/10.1126/science.1165620
Krichene, W., Rendle, S., 2020. On sampled metrics for item recommendation, in: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. Association for Computing Machinery, pp. 1748–1757. https://doi.org/10.1145/3394486.3403226
Lu, C., Lu, C., Lange, R.T., Foerster, J., Clune, J., Ha, D., 2024. The AI scientist: Towards fully automated open-ended scientific discovery. https://doi.org/10.48550/arXiv.2408.06292
MacLean, A., Young, R.M., Bellotti, V.M.E., Moran, T.P., 1991. Questions, options, and criteria: Elements of design space analysis. Human–Computer Interaction 6, 201–250. https://doi.org/10.1080/07370024.1991.9667168
Martín-Martín, A., Thelwall, M., Orduna-Malea, E., Delgado López-Cózar, E., 2021. Google Scholar, Microsoft Academic, Scopus, Dimensions, Web of Science, and OpenCitations’ COCI: A multidisciplinary comparison of coverage via citations. Scientometrics 126, 871–906. https://doi.org/10.1007/s11192-020-03690-4
Mishra, S., Sahu, G., Pedersoli, M., Charlin, L., Dolz, J., Pal, C., 2025. AInstein: Can LLMs Solve Research Problems From Parametric Memory Alone? https://doi.org/10.48550/arXiv.2510.05432
Publio, G.C., Esteves, D., Ławrynowicz, A., Panov, P., Soldatova, L., Soru, T., Vanschoren, J., Zafar, H., 2018. ML-Schema: Exposing the semantics of machine learning with schemas and ontologies. https://doi.org/10.48550/arXiv.1807.05351
Rocca-Serra, P., Brandizi, M., Maguire, E., Sklyar, N., Taylor, C., Begley, K., Field, D., Harris, S., Hide, W., Hofmann, O., Neumann, S., Sterk, P., Tong, W., Sansone, S.-A., 2010. ISA software suite: Supporting standards-compliant experimental annotation and enabling curation at the community level. Bioinformatics 26, 2354–2356. https://doi.org/10.1093/bioinformatics/btq415
Sadruddin, S., D’Souza, J., Poupaki, E., Watkins, A., Karasulu, B., Auer, S., Mackus, A., Kessels, E., 2026. schema-minerpro: Agentic AI for ontology grounding over LLM-discovered scientific schemas in a human-in-the-loop workflow. Semantic Web 17, 22104968261431521. https://doi.org/10.1177/22104968261431521
Sahu, G., Larochelle, H., Charlin, L., Pal, C., 2025. ReviewerToo: Should AI join the program committee? A look at the future of peer review. https://doi.org/10.48550/arXiv.2510.08867
Saran, A., White, H., 2018. Evidence and gap maps: A comparison of different approaches. Campbell Systematic Reviews 14, 1–38. https://doi.org/10.4073/cmdp.2018.2
Schmidgall, S., Su, Y., Wang, Z., Sun, X., Wu, J., Yu, X., Liu, J., Moor, M., Liu, Z., Barsoum, E., 2025. Agent laboratory: Using LLM agents as research assistants, in: Christodoulopoulos, C., Chakraborty, T., Rose, C., Peng, V. (Eds.), Findings of the Association for Computational Linguistics: EMNLP 2025. Association for Computational Linguistics, Suzhou, China, pp. 5977–6043. https://doi.org/10.18653/v1/2025.findings-emnlp.320
Shehzad, F., Jannach, D., 2023. Everyone’s a winner! On hyperparameter tuning of recommendation models, in: Proceedings of the 17th ACM Conference on Recommender Systems. Association for Computing Machinery, pp. 652–657. https://doi.org/10.1145/3604915.3609488
Silberzahn, R., Uhlmann, E.L., Martin, D.P., Anselmi, P., Aust, F., Awtrey, E., Bahník, Š., Bai, F., Bannard, C., Bonnier, E., Carlsson, R., Cheung, F., Christensen, G., Clay, R., Craig, M.A., Dalla Rosa, A., Dam, L., Evans, M.H., Flores Cervantes, I., Fong, N., Gamez-Djokic, M., Glenz, A., Gordon-McKeon, S., Heaton, T.J., Hederos, K., Heene, M., Hofelich Mohr, A.J., Högden, F., Hui, K., Johannesson, M., Kalodimos, J., Kaszubowski, E., Kennedy, D.M., Lei, R., Lindsay, T.A., Liverani, S., Madan, C.R., Molden, D., Molleman, E., Morey, R.D., Mulder, L.B., Nijstad, B.R., Pope, N.G., Pope, B., Prenoveau, J.M., Rink, F., Robusto, E., Roderique, H., Sandberg, A., Schlüter, E., Schönbrodt, F.D., Sherman, M.F., Sommer, S.A., Sotak, K., Spain, S., Spörlein, C., Stafford, T., Stefanutti, L., Tauber, S., Ullrich, J., Vianello, M., Wagenmakers, E.-J., Witkowiak, M., Yoon, S., Nosek, B.A., 2018. Many analysts, one data set: Making transparent how variations in analytic choices affect results. Advances in Methods and Practices in Psychological Science 1, 337–356. https://doi.org/10.1177/2515245917747646
Sinha, R., Gokturk, B., Parmar, M., Li, C.-L., Pfister, T., Dalvi Mishra, B., Song, Y., Chen, J., Ranganathan, P., Song, Y., Goyal, P., Meng, R., Yoon, J., 2026. Scientist-One: Verifiable autonomous research via chain-of-evidence.
Steegen, S., Tuerlinckx, F., Gelman, A., Vanpaemel, W., 2016. Increasing transparency through a multiverse analysis. Perspectives on Psychological Science 11, 702–712. https://doi.org/10.1177/1745691616658637
Tamm, Y.-M., Damdinov, R., Vasilev, A., 2021. Quality metrics in recommender systems: Do we calculate metrics consistently?, in: Proceedings of the 15th ACM Conference on Recommender Systems. Association for Computing Machinery, pp. 708–713. https://doi.org/10.1145/3460231.3478848
The Gene Ontology Consortium, 2023. The gene ontology knowledgebase in 2023. Genetics 224, iyad031. https://doi.org/10.1093/genetics/iyad031
Thornton, C., Hutter, F., Hoos, H.H., Leyton-Brown, K., 2013. Auto-WEKA: Combined selection and hyperparameter optimization of classification algorithms, in: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery, pp. 847–855. https://doi.org/10.1145/2487575.2487629
Trost, F., Zhang, B., Aring, I., Bauer, M., Glamann, L., Wessolly, M., Johnson, K., Göbel, H., Lerbs, T., Sangenne, T., Herrmann, P., Mairinger, F., Kopp, C., Michels, S., Rasokat, A., Heldwein, M., Wagner, S., Schömig-Markiefka, B., Wolf, J., Hartmann, S., Wickenhauser, C., Bychkov, A., Klussmann, J.P., Quaas, A., Buettner, R., Tolkach, Y., 2026. An agentic framework for autonomous scientific discovery in cancer pathology. Nature Medicine 32, 2254–2266. https://doi.org/10.1038/s41591-026-04357-y
Urbano, J., Lima, H., Hanjalic, A., 2019. Statistical significance testing in information retrieval: An empirical analysis of type i, type II and type III errors, in: Piwowarski, B., Chevalier, M., Gaussier, É., Maarek, Y., Nie, J.-Y., Scholer, F. (Eds.), Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, pp. 505–514. https://doi.org/10.1145/3331184.3331259
Vanschoren, J., Blockeel, H., Pfahringer, B., Holmes, G., 2012. Experiment databases: A new way to share, organize and learn from experiments. Machine Learning 87, 127–158. https://doi.org/10.1007/s10994-011-5277-0
Vanschoren, J., Rijn, J.N. van, Bischl, B., Torgo, L., 2014. OpenML: Networked science in machine learning. ACM SIGKDD Explorations Newsletter 15, 49–60. https://doi.org/10.1145/2641190.2641198
Waltemath, D., Adams, R., Bergmann, F.T., Hucka, M., Kolpakov, F., Miller, A.K., Moraru, I.I., Nickerson, D., Sahle, S., Snoep, J.L., Le Novère, N., 2011. Reproducible computational biology experiments with SED-ML—the simulation experiment description markup language. BMC Systems Biology 5, 198. https://doi.org/10.1186/1752-0509-5-198
Wicherts, J.M., Veldkamp, C.L.S., Augusteijn, H.E.M., Bakker, M., Aert, R.C.M. van, Assen, M.A.L.M. van, 2016. Degrees of freedom in planning, running, analyzing, and reporting psychological studies: A checklist to avoid p-hacking. Frontiers in Psychology 7, 1832. https://doi.org/10.3389/fpsyg.2016.01832
Yamada, Y., Lange, R.T., Lu, C., Hu, S., Lu, C., Foerster, J., Clune, J., Ha, D., 2025. The AI scientist-v2: Workshop-level automated scientific discovery via agentic tree search. https://doi.org/10.48550/arXiv.2504.08066
Zangerle, E., Bauer, C., 2023. Evaluating recommender systems: Survey and framework. ACM Computing Surveys 55, 170:1–170:38. https://doi.org/10.1145/3556536
0 Comments