AutoRecLab Alpha: Bringing the AI Scientist to Recommender Systems

How to cite this blog post:
Joeran Beel, Moritz Baumgart, and Philipp Meister. 2025. AutoRecLab Alpha: Bringing the AI Scientist to Recommender Systems. ISG Blog, 28 October 2025.

Today, we are releasing AutoRecLab Alpha, our first working implementation of an AI Scientist developed specifically for recommender-systems research. AutoRecLab starts with a research task described in natural language. It derives experimental requirements, generates and executes code, evaluates the implementation, and iteratively improves it. The current Alpha concentrates on one of the most time-consuming parts of empirical RecSys research: turning an experimental idea into working and inspectable experimental code.

This release follows a line of work that started during my research visit in Singapore, continued with our experiments using Sakana AI’s AI Scientist in February 2025, and led to a more specific goal: building an AI Scientist for the recommender-systems community. Last week, we published the position paper From AutoRecSys to AutoRecLab: A Call to Build, Evaluate, and Govern Autonomous Recommender-Systems Research Labs [32]. There, we argued that RecSys should expand research automation beyond algorithm selection and hyperparameter optimization toward the research process itself. AutoRecLab Alpha is a first concrete implementation of that direction.

The story started in Singapore

At the end of 2024 and beginning of 2025, I spent three months at the National University of Singapore visiting Min-Yen Kan and the WING group. A recurring topic during the visit was AI for Science and the automation of research. We discussed systems for literature analysis, idea generation, experimental design, implementation, scientific writing, reviewing, and reproducibility. I summarized some of these discussions in my January blog post about the research visit in Singapore.

Sakana AI’s AI Scientist was particularly interesting to us. Lu et al. had released the system in August 2024. It connected idea generation, novelty search, code modification, experiment execution, visualization, manuscript generation, and automated reviewing in one workflow [18]. I returned from Singapore convinced that this approach had considerable potential. I also wanted to know what happened when such a general-purpose AI Scientist was moved away from its original examples and asked to conduct recommender-systems research. So we tried it.

February 2025: our first RecSys experiments with an AI Scientist

Together with Min-Yen Kan and Moritz Baumgart, I evaluated Sakana’s AI Scientist in late 2024 / early 2025. Some of our experiments concerned recommender systems, including GreenRecSys research. We released the first version of our evaluation on 20 February 2025 [23] and discussed the results in more detail in a blog post the following day.

The results were mixed. Of twelve proposed experiments, five failed because of coding errors, a failure rate of 42%. Other experiments executed but contained problems in their design or interpretation. Across iterations, the generated experimental code changed relatively little: code size increased by only 8% in characters on average. The manuscripts contained a median of five citations, and only 5 of 34 cited papers were from 2020 or later. We also observed missing figures, repeated sections, placeholder text, and hallucinated numerical results [23]. At the same time, Sakana’s system could generate a complete manuscript for approximately $6–15 in API costs, with around 3.5 hours of human involvement in our experiments [23].

That combination was what convinced me to continue: substantial automation at low cost, combined with substantial reliability problems. The failures also pointed toward a possible solution. Recommender-systems research has its own methodological requirements. A general research agent may know how to write Python, but RecSys experiments depend on choices concerning data splitting, candidate construction, filtering, metrics, cutoffs, repeated runs, hyperparameter optimization, and leakage prevention. I therefore continued working on the idea with a more specific objective: bring AI Scientists to recommender-systems research.

To the best of our knowledge, our February 2025 study was the first explicit application and empirical evaluation of an AI Scientist in recommender-systems research. AutoRecLab is our next step: a RecSys AI Scientist.

AutoRecLab Alpha

The current AutoRecLab Alpha focuses on experimental planning, implementation, execution, and iterative refinement. A researcher starts with a natural-language request. AutoRecLab first acts as a planner and translates the request into an experimental plan and a set of explicit requirements. Depending on the task, these requirements can specify the dataset, recommender algorithm, train/test protocol, candidate construction, metric definitions, cutoffs, random seeds, and reproducibility constraints. A coder then produces Python code and executes it, while a reviewer assesses the implementation against the requirements.

Execution is part of the feedback loop. A failed program produces a stack trace that can inform another attempt. A program that runs produces outputs that can be checked against the requirements. AutoRecLab stores generated implementations as nodes in a search tree, allowing it to refine a promising implementation, debug a failed branch, or explore an alternative solution. The architecture is influenced by recent work on search-based scientific agents. AI Scientist-v2, for example, introduced progressive agentic tree search for experimental research [25]. AutoRecLab applies related ideas to RecSys experimentation.

An Alpha experiment: 20 cents, 7 iterations, 72 minutes

In one of our initial experiments, we asked AutoRecLab to investigate the impact of random seeds on an ImplicitMF recommender in LensKit using MovieLens 100K. The planner generated 16 requirements covering data loading, model initialization, consistent splitting and candidate generation, repeated training with controlled random seeds, and metric reporting. AutoRecLab then generated, executed, reviewed, and revised candidate implementations. After 7 iterations, it produced a working experimental implementation satisfying the generated requirements. The complete process took 72 minutes, and the API cost was approximately $0.20.

These numbers give a more useful picture of the current system than a general claim about autonomous experimentation. Twenty cents in API calls is inexpensive, while seven iterations and 72 minutes show that the current search process is far from instantaneous. A successful execution also still needs scientific inspection. Nevertheless, the experiment demonstrates the capability we want to develop: a researcher describes a RecSys experiment, and an agent translates that description into requirements and working experimental code through an iterative process.

AutoRecSys vs. AutoRecLab

We spent some time discussing the name of the project. AutoRecSys (or Auto-RecSys) would have been obvious candidates, but we decided against them because AutoRecSys already describes an established line of recommender-systems research, including several projects from our own group.

An early example is Mansoury and Burke’s librec-auto, which automated recommender-system experimentation and algorithm selection [1]. In 2020, Rohan Anand and I introduced Auto-Surprise: An Automated Recommender-System (AutoRecSys) Library [2], which searches across algorithms implemented in Surprise and optimizes their hyperparameters. Wang et al.’s AutoRec similarly explored automated model search and hyperparameter optimization [3]. Other work automated individual design decisions: AutoFIS searches feature interactions [4], AutoEmb and AutoDim search embedding dimensions [5,6], AutoField automates feature selection [7], and NASRec searches recommender architectures [10]. Zheng et al.’s survey organized much of this work under AutoML for recommender systems [8].

In 2023, Tobias Vente, Michael Ekstrand, and I introduced LensKit-Auto [9], which added automated algorithm selection, hyperparameter optimization, and ensembling to LensKit. One motivation came from our observation that 21 of 33 papers using LensKit algorithms, or 63.6%, did not perform algorithm selection or hyperparameter tuning [9]. More recently, in The Potential of AutoML for Recommender Systems, we compared 60 algorithms from 15 AutoML, AutoRecSys, machine-learning, and recommender-system libraries across 14 datasets [11]. AutoML systems obtained the best result on 6 of 14 datasets, while Auto-Surprise was the individual library with the most wins, achieving the best result on 5 of 14 datasets.

This existing body of work gives AutoRecSys a clear meaning: it automates model selection, hyperparameter optimization, architecture search, feature selection, and other decisions within a research problem defined by a researcher. AutoRecLab targets a different level of automation. A RecSys research agent should translate a research question into experimental requirements, implement and execute the experiment, identify failures, revise the implementation, explore alternatives, and eventually help decide what experiment should come next.

This distinction motivated the new name. The word Lab also reflects the larger direction. We expect an autonomous research environment eventually to contain multiple specialized agents for literature search, research ideas, experimental design, implementation, methodological checking, analysis, reproducibility, and scientific communication.

From AutoRecSys to AutoRecLab

On 20 October 2025, we made this broader agenda explicit in our preprint From AutoRecSys to AutoRecLab: A Call to Build, Evaluate, and Govern Autonomous Recommender-Systems Research Labs [32]. The paper argues for expanding the scope of RecSys automation from algorithm selection and hyperparameter tuning toward an autonomous research lab covering problem ideation, literature analysis, experimental design and execution, result interpretation, manuscript preparation, and provenance. It also argues that such systems require appropriate evaluation, reproducibility, attribution, and governance mechanisms.

The Alpha release described here is deliberately narrower than that vision. It starts with experimental implementation because experiments provide useful feedback: code can be executed, failures can be observed, outputs can be checked, and alternative implementations can be compared. This gives us a concrete place to start building the larger lab.

AI4Research and related work

AutoRecLab is part of a much larger development around AI for research. In May, Moritz started maintaining an overview of tools for conducting research autonomously, covering systems for literature research, idea generation, experimentation, analysis, autonomous research, writing, and reviewing.

The idea of autonomous science predates current LLMs. King et al.’s Robot Scientist and later Adam generated hypotheses and used experimental results to guide subsequent work [12,13]. More recent systems use language models as interfaces to scientific tools. ChemCrow connects an LLM with chemistry tools [14], while Coscientist combines LLM-based planning with search, code execution, documentation, and laboratory automation [15].

Since 2024, several systems have covered larger parts of the research workflow. ResearchAgent generates and refines research ideas grounded in literature [16], while data-to-paper connects data analysis, code execution, interpretation, and manuscript generation [17]. Sakana’s AI Scientist integrated idea generation, novelty search, implementation, experiments, visualization, writing, and reviewing [18], and CycleResearcher connected automated research with automated reviewing [19]. In 2025, Agent Laboratory [20], ToolMaker [21], Google’s AI Co-Scientist [22], CodeScientist [24], AI Scientist-v2 [25], Robin [26], AI-Researcher [27], and AlphaEvolve [28] explored different forms of automated ideation, experimentation, tool use, validation, and scientific discovery. In September, Google Research also presented an AI system for generating and optimizing empirical scientific software through iterative generation, execution, scoring, and search [29].

These systems differ substantially in scope, but they illustrate a common development: research automation is moving beyond isolated assistance toward agents that can make and revise decisions across several stages of the scientific process. Existing AutoRecSys work provides domain-specific automation for recommender systems, while AI Scientist systems automate larger parts of research. Our February evaluation brought a general AI Scientist into recommender-systems research [23], and our October position paper described the broader AutoRecLab agenda [32]. To the best of our knowledge, AutoRecLab is the first AI Scientist developed specifically for recommender-systems research.

What comes next

AutoRecLab Alpha can already turn some natural-language RecSys research requests into working experimental implementations. Our seed-sensitivity example required 16 generated requirements and 7 implementation iterations. It completed in 72 minutes at an API cost of about $0.20. Those numbers are encouraging, but a program can execute successfully while still answering the wrong scientific question. It can use an inappropriate split, construct the wrong candidate set, introduce leakage, calculate a metric incorrectly, or compare algorithms under incompatible conditions.

The next development work therefore needs to strengthen both generation and validation. We want better requirements engineering, stronger methodological checks, deeper integration with RecSys frameworks and their documentation, better provenance, and clearer records of why an agent made each experimental decision. The longer-term direction is the one we described in our 20 October position paper: move from individual AutoRecSys components toward an AutoRecLab in the literal sense, a lab of agents supporting increasingly large parts of recommender-systems research [32].

For now, we would like researchers to try AutoRecLab Alpha and find its limits. Give it real RecSys research tasks. Inspect its requirements and its generated code. Look for programs that fail, and especially for programs that run but implement the wrong experiment. Those cases tell us what a RecSys AI Scientist still needs to learn.

References

1. Mansoury, M., & Burke, R. (2019). Algorithm Selection with librec-auto. AMIR@ECIR 2019.
2. Anand, R., & Beel, J. (2020). Auto-Surprise: An Automated Recommender-System (AutoRecSys) Library with Tree of Parzens Estimator (TPE) Optimization. ACM RecSys 2020. DOI: 10.1145/3383313.3411467.
3. Wang, T.-H., Song, Q., Han, X., Liu, Z., Jin, H., & Hu, X. (2020). AutoRec: An Automated Recommender System. arXiv:2007.07224.
4. Liu, B., et al. (2020). AutoFIS: Automatic Feature Interaction Selection in Factorization Models for Click-Through Rate Prediction. KDD 2020. DOI: 10.1145/3394486.3403314.
5. Zhao, X., Wang, C., Chen, M., Zheng, X., Liu, X., & Tang, J. (2020). AutoEmb: Automated Embedding Dimensionality Search in Streaming Recommendations. arXiv:2002.11252.
6. Zhao, X., et al. (2021). AutoDim: Field-aware Embedding Dimension Search in Recommender Systems. The Web Conference 2021. DOI: 10.1145/3442381.3450124.
7. Wang, Y., Zhao, X., Xu, T., & Wu, X. (2022). AutoField: Automating Feature Selection in Deep Recommender Systems. The Web Conference 2022. DOI: 10.1145/3485447.3512071.
8. Zheng, R., Qu, L., Cui, B., Shi, Y., & Yin, H. (2022). AutoML for Deep Recommender Systems: A Survey. arXiv:2203.13922.
9. Vente, T., Ekstrand, M. D., & Beel, J. (2023). Introducing LensKit-Auto, an Experimental Automated Recommender System (AutoRecSys) Toolkit. ACM RecSys 2023, 1212–1216. DOI: 10.1145/3604915.3610656.
10. Zhang, T., et al. (2023). NASRec: Weight Sharing Neural Architecture Search for Recommender Systems. The Web Conference 2023. DOI: 10.1145/3543507.3583446.
11. Vente, T., Wegmeth, L., & Beel, J. (2025). The Potential of AutoML for Recommender Systems. UMAP Adjunct 2025, 371–378. DOI: 10.1145/3708319.3734173. Earlier preprint: arXiv:2402.04453.
12. King, R. D., et al. (2004). Functional Genomic Hypothesis Generation and Experimentation by a Robot Scientist. Nature, 427, 247–252. DOI: 10.1038/nature02236.
13. King, R. D., et al. (2009). The Automation of Science. Science, 324, 85–89. DOI: 10.1126/science.1165620.
14. Bran, A. M., Cox, S., Schilter, O., Baldassari, C., White, A. D., & Schwaller, P. (2023). ChemCrow: Augmenting Large-Language Models with Chemistry Tools. arXiv:2304.05376.
15. Boiko, D. A., MacKnight, R., Kline, B., & Gomes, G. (2023). Autonomous Chemical Research with Large Language Models. Nature, 624, 570–578. DOI: 10.1038/s41586-023-06792-0.
16. Baek, J., Jauhar, S. K., Cucerzan, S., & Hwang, S. J. (2024). ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models. arXiv:2404.07738.
17. Ifargan, T., Hafner, L., Kern, M., Alcalay, O., & Kishony, R. (2024). Autonomous LLM-driven Research from Data to Human-Verifiable Research Papers. arXiv:2404.17605.
18. Lu, C., Lu, C., Lange, R. T., Yamada, Y., Hu, S., Foerster, J., Ha, D., & Clune, J. (2024). The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery. arXiv:2408.06292.
19. Weng, Y., Zhu, M., Bao, G., Zhang, H., Wang, J., Zhang, Y., & Yang, L. (2024). CycleResearcher: Improving Automated Research via Automated Review. arXiv:2411.00816.
20. Schmidgall, S., et al. (2025). Agent Laboratory: Using LLM Agents as Research Assistants. arXiv:2501.04227.
21. Wölflein, G., Ferber, D., Truhn, D., Arandjelović, O., & Kather, J. N. (2025). LLM Agents Making Agent Tools. arXiv:2502.11705.
22. Gottweis, J., et al. (2025). Towards an AI Co-Scientist. arXiv:2502.18864.
23. Beel, J., Kan, M.-Y., & Baumgart, M. (2025). Evaluating Sakana’s AI Scientist: Bold Claims, Mixed Results, and a Promising Future? arXiv:2502.14297. First posted 20 February 2025.
24. Jansen, P., Tafjord, O., Radensky, M., et al. (2025). CodeScientist: End-to-End Semi-Automated Scientific Discovery with Code-based Experimentation. arXiv:2503.22708.
25. Yamada, Y., Lange, R. T., Lu, C., Hu, S., Lu, C., Foerster, J., Clune, J., & Ha, D. (2025). The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search. arXiv:2504.08066.
26. Ghareeb, A. E., Chang, B., Mitchener, L., et al. (2025). Robin: A Multi-Agent System for Automating Scientific Discovery. arXiv:2505.13400.
27. Tang, J., Xia, L., Li, Z., & Huang, C. (2025). AI-Researcher: Autonomous Scientific Innovation. arXiv:2505.18705.
28. Novikov, A., Vũ, N., Eisenberger, M., et al. (2025). AlphaEvolve: A Coding Agent for Scientific and Algorithmic Discovery. arXiv:2506.13131.
29. Dorfman, L., & Brenner, M. (2025). Accelerating Scientific Discovery with AI-Powered Empirical Software. Google Research Blog, 9 September 2025.
30. Beel, J. (2025). Prof. Beel’s 3-months visit of Prof. Min-Yen Kan in Singapore comes to an end. ISG Blog, 20 January 2025.
31. Baumgart, M. (2025). AI4Research: An overview of tools for conducting research autonomously. ISG Blog, 26 May 2025.
32. Beel, J., Gipp, B., Vente, T., Baumgart, M., & Meister, P. (2025). From AutoRecSys to AutoRecLab: A Call to Build, Evaluate, and Govern Autonomous Recommender-Systems Research Labs. arXiv:2510.18104. First posted 20 October 2025.