AutoRecLab: An Autonomous Recommender Systems Lab

Abstract. Recommender systems (RecSys) research depends on extensive empirical evaluation, yet translating experimental designs into executable code remains a manual, error-prone process. This paper presents AutoRecLab, a Python-based autonomous RecSys lab that automates RecSys experimentation from natural-language prompts. Starting from a research idea, AutoRecLab derives explicit experiment requirements, develops and validates a prototype, and iteratively refines it into the requested full experiment by combining retrieval-augmented generation (RAG) documentation lookup, static type verification, and execution-steered tree search. As a demonstration, AutoRecLab autonomously implements an explicit-to-implicit feedback conversion study and, in a baseline comparison across six algorithms and three datasets, achieves 8 out of 9 successful runs at an average cost of approximately $1 per run using GPT-5.4-mini.

Moritz Baumgart, Philipp Meister, Justus Krell, Michael Schmidt, Bela Gipp, and Joeran Beel. 2026. AutoRecLab: An Autonomous Recommender Systems Lab. In 20th ACM Conference on Recommender Systems (RecSys '26), September 27-October 02, 2026, Minneapolis, MN, USA. ACM, New York, NY, USA, 3 pages. https://doi.org/10.1145/3773078.3841273

ACM: https://dl.acm.org/doi/10.1145/3773078.3841273
DOI: https://doi.org/10.1145/3773078.3841273
Source Code: https://code.isg.beel.org/AutoRecLab

@inproceedings{10.1145/3773078.3841273, author = {Baumgart, Moritz and Meister, Philipp and Krell, Justus and Schmidt, Michael and Gipp, Bela and Beel, Joeran}, title = {AutoRecLab: An Autonomous Recommender Systems Lab}, year = {2026}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, isbn = {979-8-4007-2284-4}, url = {https://doi.org/10.1145/3773078.3841273}, doi = {10.1145/3773078.3841273}, booktitle = {20th ACM Conference on Recommender Systems (RecSys ’26)}, numpages = {3}, location = {Minneapolis, MN, USA}, series = {RecSys ’26} }

Keywords: Recommender Systems, Autonomous Agents, Code Generation, RAG, LLM

1 Introduction

Autonomous science agents based on Large Language Models (LLMs) are becoming increasingly popular for automating science and engineering tasks from natural language prompts [8, 14]. Recommender systems (RecSys) research is a promising domain for this automation, given that setting up and implementing experiments is inherently time- and labor-intensive. In particular, this includes writing custom data preprocessing, configuring evaluation loops, and adapting code to different libraries, such as LensKit [6], RecBole [15], or meta-frameworks like OmniRec [12].

Learning these libraries helps researchers understand experimental choices but can also create substantial setup overhead for newcomers. Experienced researchers likewise spend considerable time implementing and debugging experiments, while manual implementations remain prone to evaluation and reproducibility errors across libraries [4, 7]. These challenges motivate systems that translate high-level research ideas into explicit experimental requirements, reduce routine implementation work, and validate generated code against these requirements.

Standard LLMs can support researchers but frequently hallucinate API calls for these domain-specific libraries, and existing general science agents lack RecSys-specific configuration. We investigated multiple LLM-based science agents, including Sakana’s AI Scientist [8, 14], Agent Laboratory [10], the AI-Researcher [11], and Zochi [16], to determine their capabilities in the domain of recommender systems research. However, due to their focus on machine learning tasks, the science agents we were able to test needed additional context. In particular, our detailed evaluation showed that, e.g., the AI Scientist’s performance on RecSys tasks fell short of expectations [5].

In our prior work [2, 3], we call for developing and evaluating automated RecSys research systems to address these limitations. Here, we present AutoRecLab as one such proof of concept. Starting from a single natural-language prompt, AutoRecLab derives explicit experiment requirements, iteratively develops and validates a prototype, and finally scales it into the requested full experiment. This staged workflow aims to automate this labor-intensive process while validating that the generated implementation meets experiment-specific requirements.

2 AutoRecLab

AutoRecLab is an open-source Python CLI tool that translates a natural-language research prompt into valid, executable code. It structures the experimental process into three key phases: requirements engineering, prototyping, and refinement of the prototype to the complete experiment. Throughout these phases, AutoRecLab automatically checkpoints intermediate states and collects all artifacts, such as Python code, generated plots, and execution logs, in a dedicated workspace. To execute these experiments, the tool leverages the OmniRec [12] meta-framework, which standardizes data loading and training across over 230 datasets and multiple RecSys Python libraries, namely RecPack [9], RecBole [15], LensKit [6], and Elliot [1].

2.1 Demonstration

Since AutoRecLab is not intended as a web service, we do not provide a link to a live system. Users can download the source code from our Git repository and run AutoRecLab locally.

We evaluated AutoRecLab on four representative empirical RecSys scenarios, focusing on an experiment to test explicit-to-implicit feedback conversion on the MovieLens 1M dataset. The experiment was initiated with the following prompt:

Test the influence of […] feedback conversion strategies on recommendation accuracy by comparing multiple binarization thresholds […]. Evaluate […] on the MovieLens1M dataset. Report metrics […], and compare ranking quality […].

From this input, AutoRecLab generated in total 22 requirements covering data loading, conversion thresholds, train-test splitting, and evaluation. AutoRecLab successfully generated 169 lines of executable Python code and the requested comparative plots (see Figure 2) at a total API cost of USD 0.76 using GPT-5.4-mini.

To evaluate reproducibility across multiple standard algorithms, we tasked AutoRecLab with establishing a performance baseline of 6 algorithms for model comparison. Eight of the nine executed runs (≈ 89%) successfully generated bug-free code and plots at the cost of around $1 per run. Detailed statistics for these runs are reported in Table 1. The overall runtime is high for some runs because it is dominated by the execution time of the generated code.

Additionally, we evaluated AutoRecLab on a dataset filtering task, where it analyzed the performance impact of pruning users with few interactions. In a final scenario, AutoRecLab analyzed how evaluation metrics are affected by the random seeds used during user splitting. Notably, the results reproduced qualitative trends reported in existing human-conducted research [13], indicating that AutoRecLab can support empirical RecSys experimentation.

2.2 System Architecture

AutoRecLab coordinates experiment implementation and execution in the three stages illustrated in Figure 1.

2.2.1 Requirements Engineering

AutoRecLab converts the user’s natural-language prompt into a research plan and two sets of requirements: (1) Prototype Requirements: These define a small, fast-running experiment, typically restricted to one dataset, baseline algorithm, and metric cutoff. (2) Full Requirements: These retain the complete request, including all specified algorithms, datasets, metrics, and visualizations.

This separation allows AutoRecLab to test the implementation on a small experiment before scaling it to the full setup.

2.2.2 Prototyping Stage

AutoRecLab currently uses the OmniRec library and is therefore limited to its available libraries and datasets. Compared with using OmniRec directly, AutoRecLab reduces manual setup, while direct use provides more immediate control over configuration and implementation.

Code Generation & Verification: AutoRecLab generates code and statically checks it for type mismatches, triggering an automated correction loop when errors are detected.

RAG Documentation Server (MCP): To reduce hallucinated API calls, AutoRecLab retrieves indexed documentation and code for OmniRec, LensKit, and RecBole through the Model Context Protocol (MCP).

Evaluation & Tree Search: Each candidate is executed in an independent workspace. AutoRecLab uses an LLM to evaluate the generated code and console output, determining for each prototype requirement whether it is fulfilled and whether the candidate is buggy or bug-free. The proportion of fulfilled requirements defines the node score S ∈ [0,1]. The score measures requirement coverage rather than providing a general confidence estimate or a guarantee of correctness. AutoRecLab organizes candidate implementations as a search tree. During node selection, it first decides whether to sample from the buggy or bug-free node set. Within the selected set, an ε-greedy strategy chooses either the highest-scoring candidate or a random alternative for improvement or debugging. The process continues until a candidate reaches S = 1 or AutoRecLab reaches the configured iteration limit.

2.2.3 Refinement Stage

AutoRecLab incrementally extends the executable prototype to satisfy the full requirements instead of generating the complete experiment from scratch. Each revision is executed and evaluated until all requirements are satisfied. Finally, AutoRecLab returns the Python code, generated plots, execution logs, and a Markdown summary for inspection and modification.

3 Conclusion

Our evaluation focuses on a limited set of comparatively simple offline RecSys tasks. Across nine runs, AutoRecLab generated code that was classified as bug-free in eight cases, although not all resulting analyses were scientifically meaningful. These results suggest the technical feasibility of the approach while indicating that requirement coverage and successful execution alone may not fully capture experiment quality.

AutoRecLab should therefore be understood as an early proof of concept toward the broader vision of autonomous RecSys research labs [2], rather than as a realization of the full vision. Its current capabilities are bounded by OmniRec, the underlying LLM, coverage of indexed documentation, and long sequential runtimes. AutoRecLab can reduce implementation effort for supported tasks and provides inspectable artifacts, while researchers remain responsible for experimental design, code inspection, result interpretation, and deciding when user studies are needed. Potential directions for future work include evaluations on more complex RecSys tasks and across different LLMs, explicit assessments of generated code quality, indexing additional recommendation libraries, and parallelizing the tree search. Further extensions could investigate support for literature search and manuscript preparation.

Figure 1: AutoRecLab workflow. The research task is translated into requirements. The prototyping stage uses tree search and MCP-based documentation lookup to produce a working prototype, which is then scaled in the final refinement loop to generate the experimental summary, code, and plots.

Figure 2: Plot generated by AutoRecLab in the explicit-to-implicit conversion experiment. Shows NDCG@10 (top) and Precision@10 (bottom) on the MovieLens 1M dataset for different algorithms and rating thresholds (greater than or equal to (ge) 1, greater than 3, and greater than 4).

Table 1: Statistics of AutoRecLab baseline experiment across nine runs (P: Prototype, F: Final Refinement)

Metric	1.1	1.2	1.3	2.1	2.2	2.3	3.1	3.2	3.3
Dataset	MU	MU	MU	ML1M	ML1M	ML1M	VI	VI	VI
Algos	6	6	6	6	6	6	6	6	6
Cost ($)	1.09	1.08	1.10	0.95	0.98	0.91	1.04	1.01	0.90
Runtime P	4.37h	12.23h	40m	23m	58m	1.52h	17.32h	7m	19m
Runtime F	30.2h	33.65h	16.23h	32.33h	32.08h	22.43h	5m	24.48h	16.58h
Best Score P	0.89	0.84	0.375	0.6	0.875	0.857	0.56	0.45	0.4
Avg. LoC	110	111	139	100	134	118	100	141	109
Nodes P	8	8	8	8	8	8	8	8	8
Nodes F	4	4	4	4	4	4	4	4	4
Buggy P?	No	No	Yes	No	No	No	Yes	Yes	Yes
Buggy F?	No	No	No	No	No	No	Yes	No	No

Datasets: ML1M: MovieLens 1M, MU: Amazon2018MusicalInstruments, VI: Amazon2018VideoGames.

References

1. Vito Walter Anelli, Alejandro Bellogín, Antonio Ferrara, Daniele Malitesta, Felice Antonio Merra, Claudio Pomo, Francesco Maria Donini, and Tommaso Di Noia. 2021. Elliot: A comprehensive and rigorous framework for reproducible recommender systems evaluation. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2405–2414.
2. Joeran Beel, Bela Gipp, Tobias Vente, Moritz Baumgart, and Philipp Meister. 2025. From AutoRecSys to AutoRecLab: A Call to Build, Evaluate, and Govern Autonomous Recommender-Systems Research Labs. arXiv preprint arXiv:2510.18104 (2025).
3. Joeran Beel, Bela Gipp, Tobias Vente, Moritz Baumgart, Philipp Meister, and Sinan Pourazari. 2026. A RecSys Paper for $20: Why We Must Build, Evaluate, and Govern Autonomous Recommender Systems Research Labs (AutoRecLabs). In Methodology First – Rethinking Research Assessment in RecSys (FRAME 2026). Minneapolis, Minnesota, USA. Accepted for publication.
4. Joeran Beel, Dietmar Jannach, Alan Said, Guy Shani, Tobias Vente, and Lukas Wegmeth. 2024. Best-Practices for Offline Evaluations of Recommender Systems. In Report from Dagstuhl Seminar 24211 – Evaluation Perspectives of Recommender Systems: Driving Research and Education (2024-01-01), Christine Bauer, Alan Said, and Eva Zangerle (Eds.).
5. Joeran Beel, Min-Yen Kan, and Moritz Baumgart. 2025. Evaluating Sakana’s AI Scientist: Bold Claims, Mixed Results, and a Promising Future? ACM SIGIR Forum 59, 1 (June 2025), 1–20. https://doi.org/10.1145/3769733.3769747
6. Michael D Ekstrand. 2020. Lenskit for python: Next-generation software for recommender systems experiments. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management. 2999–3006.
7. Maurizio Ferrari Dacrema, Paolo Cremonesi, and Dietmar Jannach. 2019. Are we really making much progress? A worrying analysis of recent neural recommendation approaches. In Proceedings of the 13th ACM Conference on Recommender Systems (RecSys ’19). ACM, 101–109. https://doi.org/10.1145/3298689.3347058
8. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. 2024. The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery. https://doi.org/10.48550/ARXIV.2408.06292
9. Lien Michiels, Robin Verachtert, and Bart Goethals. 2022. Recpack: An (other) experimentation toolkit for top-n recommendation using implicit feedback data. In Proceedings of the 16th ACM Conference on Recommender Systems. 648–651.
10. Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Michael Moor, Zicheng Liu, and Emad Barsoum. 2025. Agent Laboratory: Using LLM Agents as Research Assistants. https://doi.org/10.48550/ARXIV.2501.04227
11. Jiabin Tang, Lianghao Xia, Zhonghang Li, and Chao Huang. 2025. AI-Researcher: Autonomous Scientific Innovation. https://doi.org/10.48550/ARXIV.2505.18705
12. Lukas Wegmeth, Moritz Baumgart, Philipp Meister, Bela Gipp, and Joeran Beel. 2026. OmniRec: The All-In-One Solution for Reproducible and Interoperable Recommender Systems Experimentation. In European Conference on Information Retrieval. Springer, 129–135.
13. Lukas Wegmeth, Tobias Vente, Lennart Purucker, and Joeran Beel. 2023. The Effect of Random Seeds for Data Splitting on Recommendation Accuracy. In Perspectives@RecSys.
14. Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. 2025. The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search. https://doi.org/10.48550/ARXIV.2504.08066
15. Wayne Xin Zhao, Shanlei Mu, Yupeng Hou, Zihan Lin, Yushuo Chen, Xingyu Pan, Kaiyuan Li, Yujie Lu, Hui Wang, Changxin Tian, et al. 2021. Recbole: Towards a unified, comprehensive and efficient framework for recommendation algorithms. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management. 4653–4664.
16. Andy Zhou, Ron Arel, Soren Dunn, and Nikhil Khandekar. 2025. Zochi Technical Report. https://www.intology.ai/blog/zochi-tech-report. Accessed: 2025-10-20.