FINALLY: A Dataset Recommender for RecSys Experiments

Abstract. FINALLY is a web-based recommender system for dataset selection in RecSys experiments. It supports a decision often made manually or by convention: choosing datasets to evaluate a recommendation algorithm offline. With FINALLY, users choose seed datasets, filters, a target set size, and a recommendation strategy. FINALLY returns dataset recommendations that can be inspected through metadata and, where available, an Algorithm Performance Space (APS), and exported for experiments and publications. FINALLY supports APS-based diverse and non-diverse strategies as well as random selection, with diversity as the default. FINALLY is publicly accessible, operates on more than 90 recommender-system datasets, and builds on the APS Explorer code base with a recommendation-centered workflow. To the best of our knowledge, FINALLY is the first operational system that recommends configurable sets of datasets specifically for recommender-systems experiments.

Louis Owie, Tobias Vente, and Joeran Beel. 2026. FINALLY: A Dataset Recommender for RecSys Experiments. In 20th ACM Conference on Recommender Systems (RecSys '26), September 27-October 02, 2026, Minneapolis, MN, USA. ACM, New York, NY, USA, 3 pages. https://doi.org/10.1145/3773078.3841276

ACM: https://dl.acm.org/doi/10.1145/3773078.3841276
Live System: https://finally.recommender-systems.com/
Source Code: https://code.isg.beel.org/FINALLY
APS Explorer: https://datasets.recommender-systems.com/

@inproceedings{10.1145/3773078.3841276,
author = {Owie, Louis and Vente, Tobias and Beel, Joeran},
title = {FINALLY: A Dataset Recommender for RecSys Experiments},
year = {2026},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
isbn = {979-8-4007-2284-4},
url = {https://doi.org/10.1145/3773078.3841276},
doi = {10.1145/3773078.3841276},
booktitle = {20th ACM Conference on Recommender Systems (RecSys '26)},
numpages = {3},
location = {Minneapolis, MN, USA},
series = {RecSys '26}
}

CCS Concepts: Information systems → Recommender systems; Evaluation of retrieval results; Human-centered computing → Interactive systems and tools.

Keywords: dataset selection, algorithm performance spaces, datasets

1 Introduction

Dataset selection is a central design decision in offline recommender-systems experiments [1, 6]. The datasets used in an offline experiment determine the empirical setting and can influence observed performance trends, runtime behavior, and the interpretation of results [1, 4]. Consequently, selecting datasets that fit the experiment objective is essential for meaningful and generalizable findings.

However, dataset selection in recommender-systems experiments is often insufficiently justified [16]. In an analysis of RecSys 2024 full papers, only 14% explicitly justified their dataset selection [16]. The same analysis reports concentrated dataset usage, with Amazon (38%), MovieLens (34%), Yelp (15%), and Gowalla (12%) among the most frequently used datasets [16]. Similar concentration had been observed in earlier RecSys research, with MovieLens emerging as a de facto standard dataset [3]. Not justifying dataset selection does not make the selected datasets inappropriate. However, it makes it harder to assess whether they match the research question and whether empirical conclusions generalize beyond frequently used datasets [2, 7, 14].

FINALLY addresses this lack of systematic support by constructing configurable dataset selections under user-specified constraints. It draws on more than 90 recommender-system datasets and lets users define an initial selection, apply dataset-level filters, choose a recommendation strategy, and inspect and export the resulting datasets. To the best of our knowledge, FINALLY is the first operational dataset recommender specifically designed for recommender-systems experiments.

2 Related Work

Related work supports dataset selection through both dataset surveys and discovery tools. Polatidis et al. provide a taxonomy and comparative overview of recommender-system datasets to support researchers in identifying datasets suitable for their experiments [11]. Existing discovery tools include Google Dataset Search, which provides general-purpose dataset search [5], DS4RS, which offers semantic search for recommender-system datasets [13], and scientific dataset recommenders that suggest datasets based on textual problem descriptions [8]. These approaches help researchers identify and assess candidate datasets rather than constructing a complete dataset selection under experiment-specific constraints.

The closest related system is the APS Explorer, which was developed in previous work by members of our group and supports interactive visualization and comparison of recommender-system datasets [16]. Algorithm Performance Spaces (APS) were initially introduced for instance-level algorithm selection and meta-learning [15] and later adapted to informed dataset selection [4]. APS provide the performance-based representation used in the APS Explorer [4]. In the APS Explorer, researchers inspect datasets and their positions in the APS and manually decide which datasets to select. FINALLY changes this interaction model from exploration to recommendation: researchers specify optional seed datasets, candidate constraints, a target set size, and a recommendation strategy, and the system constructs a complete dataset selection that can subsequently be inspected and exported.

Related work has additionally examined the influence of dataset characteristics on algorithm performance [1, 6] and the effects of evaluation practices and dataset dependence [2, 7, 14]. Open benchmarking efforts such as BARS [17] and ORBIT [9] provide infrastructure for systematic and reproducible recommender-system evaluation, but do not address the problem of constructing dataset selections for a specific experimental setup.

3 The FINALLY Dataset Recommender

FINALLY is an open-source web-based dataset recommender for recommender-systems research. It operates on a corpus of more than 90 recommender-system datasets and is based on the APS Explorer code base, while organizing the user workflow around dataset recommendations rather than exploration. A user configuration defines the initial selection, candidate constraints, target set size, and recommendation strategy. FINALLY filters the candidate pool accordingly, applies the selected strategy to construct the dataset selection, and presents the result through dataset metadata and APS-based visualization before it can be exported for downstream use. The recommendation workflow is browser-centered: the frontend obtains the required dataset and performance data through the existing API layer, while filtering and dataset selection are executed client-side.

3.1 Use Cases

FINALLY supports three main use cases. First, researchers without a predefined dataset selection can generate an initial selection according to a chosen strategy. Second, researchers can extend an existing selection with additional datasets, for example to increase APS-based diversity or sample randomly from the available candidates. Third, after developing and tuning an algorithm, researchers can request additional datasets not used during development for post-development validation. APS-diverse recommendations can expose the algorithm to different empirical performance conditions and help assess whether conclusions extend beyond the development datasets.

3.2 Recommendation Workflow

Users define the starting point for the recommendation by optionally selecting one or more datasets to include in the final selection. They then specify the desired total number of datasets and, optionally, restrict the candidate pool through dataset-level filters. These filters include, among others, feedback type, number of interactions, users, items, user-item ratio, density, and rating statistics. FINALLY first determines the eligible candidate datasets from the user-defined constraints and then selects from this candidate pool according to the chosen recommendation strategy.

The default recommendation objective is APS-based diversity. An Algorithm Performance Space represents each dataset as a point whose coordinates correspond to the performance of multiple recommendation algorithms. Datasets are farther apart in this space when the algorithms exhibit more distinct performance patterns on them [4]. For this objective, FINALLY implements the two diversity measures proposed by Reising [12]. The convex-hull-based method combines the mean nearest-neighbor distance with the intrinsic convex-hull volume of the selected datasets, while EffCov combines the same distance term with a structure measure derived from the effective covariance eigenvalues.

For both measures, users can choose between diverse selections, which maximize the respective diversity score, and non-diverse selections, which minimize it and therefore favor more similar datasets. FINALLY also supports random selection. Holding the seed datasets, filters, and target set size fixed therefore allows users to vary the recommendation strategy while keeping the remaining experimental constraints unchanged. In a direct comparison with otherwise identical settings, the Diverse EffCov selection was visibly more spread out in the APS than the Non-Diverse EffCov selection.

Figure 1 shows the recommendation configuration, resulting dataset selection, and APS visualization. FINALLY returns the selected datasets with relevant characteristics and supports export as image, Markdown, HTML, LaTeX, and BibTeX for reuse in experiments and publications.

Figure 1: The FINALLY recommendation configuration, APS visualization, and resulting dataset selection.

4 Discussion

Dataset selection remains an experimental decision that cannot be fully automated. The suitability of a dataset depends on the research question and evaluation objective, and different datasets can expose different algorithm-performance patterns [1, 4, 6]. FINALLY therefore provides decision support through filters, recommendation strategies, metadata, and APS-based visualizations rather than replacing researcher judgment.

APS-based recommendations are limited by the available dataset corpus and the underlying performance representation. Each APS is defined by a particular algorithm portfolio, evaluation metric, and cutoff, and requires corresponding performance data for the represented datasets. FINALLY currently provides APSs based on HR, nDCG, and Recall. Consequently, APS distance captures differences in algorithm-performance patterns under these settings, but does not establish dataset suitability for other objectives such as fairness or cold-start evaluation. Researchers must therefore still consider study-specific requirements such as feedback type, dataset scale, preprocessing compatibility, licensing, and the research question itself.

FINALLY currently operates on a pre-integrated dataset corpus. The public interface does not support uploading new or private datasets; for APS-based recommendation, additional datasets require the necessary metadata and corresponding performance data to be integrated into the underlying data source. Likewise, additional recommendation strategies currently require implementation changes rather than being supplied through a user-facing plug-in interface. The system also does not expose aggregated community dataset selections as a recommendation signal. Finally, we did not conduct a user study or empirically evaluate whether researchers prefer FINALLY over manual dataset selection or whether FINALLY-based selections improve downstream scientific outcomes. Future work therefore includes improving extensibility, incorporating additional recommendation signals such as dataset meta-features, and evaluating the system with researchers.

Acknowledgments

This work is based on the first author’s bachelor’s thesis [10]. Generative AI tools were used for drafting assistance and language refinement. The authors reviewed the generated content and take full responsibility for the final manuscript.

References

1. Gediminas Adomavicius and Jingjing Zhang. 2012. Impact of Data Characteristics on Recommender Systems Performance. ACM Transactions on Management Information Systems 3, 1, Article 3 (2012), 17 pages. https://doi.org/10.1145/2151163.2151166
2. Christine Bauer, Eva Zangerle, and Alan Said. 2024. Exploring the Landscape of Recommender Systems Evaluation: Practices and Perspectives. ACM Transactions on Recommender Systems 2, 1, Article 11 (2024), 31 pages. https://doi.org/10.1145/3629170
3. Joeran Beel and Victor Brunel. 2019. Data Pruning in Recommender Systems Research: Best-Practice or Malpractice?. In Proceedings of ACM RecSys 2019 Late-Breaking Results (CEUR Workshop Proceedings, Vol. 2431). CEUR-WS.org, 26–30.
4. Joeran Beel, Lukas Wegmeth, Lien Michiels, and Steffen Schulz. 2024. Informed Dataset Selection with ‘Algorithm Performance Spaces’. In Proceedings of the 18th ACM Conference on Recommender Systems (RecSys ’24). 1085–1090. https://doi.org/10.1145/3640457.3691704
5. Dan Brickley, Matthew Burgess, and Natasha Noy. 2019. Google Dataset Search: Building a Search Engine for Datasets in an Open Web Ecosystem. In Proceedings of the World Wide Web Conference (WWW ’19). 1365–1375. https://doi.org/10.1145/3308558.3313685
6. Jin Yao Chin, Yile Chen, and Gao Cong. 2022. The Datasets Dilemma: How Much Do We Really Know About Recommendation Datasets?. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining (WSDM ’22). 141–149. https://doi.org/10.1145/3488560.3498519
7. Paolo Cremonesi and Dietmar Jannach. 2021. Progress in Recommender Systems Research: Crisis? What Crisis? AI Magazine 42, 3 (2021), 43–54. https://doi.org/10.1609/aimag.v42i3.18145
8. Michael Färber and Ann-Kathrin Leisinger. 2021. Recommending Datasets for Scientific Problem Descriptions. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management (CIKM ’21). 3014–3018. https://doi.org/10.1145/3459637.3482166
9. Jingyuan He, Jiongnan Liu, Vishan Vishesh Oberoi, Bolin Wu, Mahima Jagadeesh Patel, Kangrui Mao, Chuning Shi, I-Ta Lee, Arnold Overwijk, and Chenyan Xiong. 2025. ORBIT: Open Recommendation Benchmark for Reproducible Research with Hidden Tests. In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track.
10. Louis Owie. 2026. FINALLY: A Dataset Recommender System for Recommender-Systems Research. Bachelor’s thesis. University of Siegen. https://doi.org/10.13140/RG.2.2.11866.50883
11. Nikolaos Polatidis, Almas Baimagambetov, Dionysios Kehagias, Marcello Trovati, Panagiotis Sarigiannidis, and Yannis Manolopoulos. 2026. Datasets for Recommender Systems: A Survey and Comparative Overview. Journal of Intelligent Information Systems (July 2026). https://doi.org/10.1007/s10844-026-01068-1
12. Jonas Reising. 2026. Towards Systematic Dataset Selection in Algorithm Performance Spaces. Bachelor’s thesis. University of Siegen. https://doi.org/10.13140/RG.2.2.23085.65764
13. Xinyang Shao and Tri Kurniawan Wijaya. 2025. DS4RS: Community-Driven and Explainable Dataset Search Engine for Recommender System Research. arXiv preprint arXiv:2508.10238 (2025). https://doi.org/10.48550/arXiv.2508.10238
14. Zhu Sun, Di Yu, Hui Fang, Jie Yang, Xinghua Qu, Jie Zhang, and Cong Geng. 2020. Are We Evaluating Rigorously? Benchmarking Recommendation for Reproducible Evaluation and Fair Comparison. In Proceedings of the 14th ACM Conference on Recommender Systems (RecSys ’20). 23–32. https://doi.org/10.1145/3383313.3412489
15. Bryan Tyrrell, Edward Bergman, Gareth Jones, and Joeran Beel. 2020. Algorithm-Performance Personas for Siamese Meta-Learning and Automated Algorithm Selection. In 7th ICML Workshop on Automated Machine Learning, Vol. 1. 16.
16. Tobias Vente, Michael Heep, Abdullah Abbas, Theodor Sperle, Joeran Beel, and Bart Goethals. 2025. APS Explorer: Navigating Algorithm Performance Spaces for Informed Dataset Selection. In Proceedings of the Nineteenth ACM Conference on Recommender Systems (RecSys ’25). 1322–1324. https://doi.org/10.1145/3705328.3759342
17. Jieming Zhu, Quanyu Dai, Liangcai Su, Rong Ma, Jinyang Liu, Guohao Cai, Xi Xiao, and Rui Zhang. 2022. BARS: Towards Open Benchmarking for Recommender Systems. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’22). 2912–2923. https://doi.org/10.1145/3477495.3531723