Tobias Vente
Ph.D. Student
Phone: +49 271 740-4235
Email: tobias.<last-name>@uni-siegen.de
Office: H-C 8318
Address: Office and postal address
Bio
Tobias Vente is a Ph.D. Student of the Intelligent Systems Group at the University of Siegen. Before joining the ISG he completed his bachelor’s – and master’s degree at the University of Siegen. In addition, Tobias took part in the University of Siegen’s exchange program and spent one year of his masters’ degree at the University of Tulsa in OK, USA.
Tobias will focus his Ph.D. on Recommender Systems (RecSys) and Automated machine learning (AutoML) research field.
Publications
2025
Beel, Joeran; Gipp, Bela; Jannach, Dietmar; Said, Alan; Wegmeth, Lukas; Vente, Tobias
Checky, the Paper-Submission Checklist Generator for Authors, Reviewers and LLMs Proceedings Article
In: 47th European Conference on Information Retrieval (ECIR), pp. 10–15, Springer, 2025.
@inproceedings{Beel2025,
title = {Checky, the Paper-Submission Checklist Generator for Authors, Reviewers and LLMs},
author = {Joeran Beel and Bela Gipp and Dietmar Jannach and Alan Said and Lukas Wegmeth and Tobias Vente},
url = {http://isg.beel.org/pubs/2025-Checky-Checklist-Generator-ECIR-Beel-et-al.pdf},
year = {2025},
date = {2025-01-01},
booktitle = {47th European Conference on Information Retrieval (ECIR)},
pages = {10–15},
publisher = {Springer},
series = {LNCS},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
2024
Beel, Joeran; Said, Alan; Vente, Tobias; Wegmeth, Lukas
Green Recommender Systems: A Call for Attention Journal Article
In: ACM SIGIR Forum, vol. 58, no. 2, pp. 1–5, 2024.
@article{Beel2024e,
title = {Green Recommender Systems: A Call for Attention},
author = {Joeran Beel and Alan Said and Tobias Vente and Lukas Wegmeth},
url = {https://isg.beel.org/pubs/2024_Green_Recommender_Systems-A_Call_for_Attention.pdf},
doi = {10.31219/osf.io/5ru2g},
year = {2024},
date = {2024-12-01},
journal = {ACM SIGIR Forum},
volume = {58},
number = {2},
pages = {1–5},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Beel, Joeran; Jannach, Dietmar; Said, Alan; Shani, Guy; Vente, Tobias; Wegmeth, Lukas
Best-Practices for Offline Evaluations of Recommender Systems Proceedings Article
In: Bauer, Christine; Said, Alan; Zangerle, Eva (Ed.): Report from Dagstuhl Seminar 24211 – Evaluation Perspectives of Recommender Systems: Driving Research and Education, 2024.
@inproceedings{Beel2024,
title = {Best-Practices for Offline Evaluations of Recommender Systems},
author = {Joeran Beel and Dietmar Jannach and Alan Said and Guy Shani and Tobias Vente and Lukas Wegmeth},
editor = {Christine Bauer and Alan Said and Eva Zangerle},
year = {2024},
date = {2024-01-01},
booktitle = {Report from Dagstuhl Seminar 24211 – Evaluation Perspectives of Recommender Systems: Driving Research and Education},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Baumgart, Moritz; Wegmeth, Lukas; Vente, Tobias; Beel, Joeran
e-Fold Cross-Validation for Recommender-System Evaluation Proceedings Article
In: International Workshop on Recommender Systems for Sustainability and Social Good (RecSoGood) at the 18th ACM Conference on Recommender Systems (ACM RecSys), 2024.
@inproceedings{Baumgart2024,
title = {e-Fold Cross-Validation for Recommender-System Evaluation},
author = {Moritz Baumgart and Lukas Wegmeth and Tobias Vente and Joeran Beel},
url = {https://isg.beel.org/pubs/2024-e-folds-recsys-baumgart.pdf},
year = {2024},
date = {2024-01-01},
booktitle = {International Workshop on Recommender Systems for Sustainability and Social Good (RecSoGood) at the 18th ACM Conference on Recommender Systems (ACM RecSys)},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Beel, Joeran; Wegmeth, Lukas; Vente, Tobias
E-fold Cross-validation: A Computing and Energy-efficient Alternative to K-fold Cross-validation with Adaptive Folds [Proposal] Journal Article
In: OSF Preprints, 2024.
@article{Beel2024c,
title = {E-fold Cross-validation: A Computing and Energy-efficient Alternative to K-fold Cross-validation with Adaptive Folds [Proposal]},
author = {Joeran Beel and Lukas Wegmeth and Tobias Vente},
url = {https://osf.io/preprints/osf/exw3j
https://isg.beel.org/blog/2024/04/07/e-fold-cross-validation-a-computing-and-energy-efficient-alternative-to-k-fold-cross-validation-with-adaptive-folds-proposal/},
doi = {10.31219/osf.io/exw3j},
year = {2024},
date = {2024-01-01},
journal = {OSF Preprints},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Wegmeth, Lukas; Vente, Tobias; Said, Alan; Beel, Joeran
EMERS: Energy Meter for Recommender Systems Proceedings Article
In: International Workshop on Recommender Systems for Sustainability and Social Good (RecSoGood) at the 18th ACM Conference on Recommender Systems (ACM RecSys), 2024.
@inproceedings{Wegmeth2024a,
title = {EMERS: Energy Meter for Recommender Systems},
author = {Lukas Wegmeth and Tobias Vente and Alan Said and Joeran Beel},
url = {https://arxiv.org/pdf/2409.15060},
year = {2024},
date = {2024-01-01},
booktitle = {International Workshop on Recommender Systems for Sustainability and Social Good (RecSoGood) at the 18th ACM Conference on Recommender Systems (ACM RecSys)},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Vente, Tobias; Wegmeth, Lukas; Said, Alan; Beel, Joeran
From Clicks to Carbon: The Environmental Toll of Recommender Systems Proceedings Article
In: Proceedings of the 18th ACM Conference on Recommender Systems, pp. 580–590, Association for Computing Machinery, Bari, Italy, 2024, ISBN: 9798400705052.
@inproceedings{Vente2024a,
title = {From Clicks to Carbon: The Environmental Toll of Recommender Systems},
author = {Tobias Vente and Lukas Wegmeth and Alan Said and Joeran Beel},
url = {https://arxiv.org/abs/2408.08203},
doi = {10.1145/3640457.3688074},
isbn = {9798400705052},
year = {2024},
date = {2024-01-01},
booktitle = {Proceedings of the 18th ACM Conference on Recommender Systems},
pages = {580–590},
publisher = {Association for Computing Machinery},
address = {Bari, Italy},
series = {RecSys '24},
abstract = {As global warming soars, the need to assess the environmental impact of research is becoming increasingly urgent. Despite this, few recommender systems research papers address their environmental impact. In this study, we estimate the environmental impact of recommender systems research by reproducing typical experimental pipelines. Our analysis spans 79 full papers from the 2013 and 2023 ACM RecSys conferences, comparing traditional “good old-fashioned AI’’ algorithms with modern deep learning algorithms. We designed and reproduced representative experimental pipelines for both years, measuring energy consumption with a hardware energy meter and converting it to CO2 equivalents. Our results show that papers using deep learning algorithms emit approximately 42 times more CO2 equivalents than papers using traditional methods. On average, a single deep learning-based paper generates 3,297 kilograms of CO2 equivalents—more than the carbon emissions of one person flying from New York City to Melbourne or the amount of CO2 one tree sequesters over 300 years.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
As global warming soars, the need to assess the environmental impact of research is becoming increasingly urgent. Despite this, few recommender systems research papers address their environmental impact. In this study, we estimate the environmental impact of recommender systems research by reproducing typical experimental pipelines. Our analysis spans 79 full papers from the 2013 and 2023 ACM RecSys conferences, comparing traditional “good old-fashioned AI’’ algorithms with modern deep learning algorithms. We designed and reproduced representative experimental pipelines for both years, measuring energy consumption with a hardware energy meter and converting it to CO2 equivalents. Our results show that papers using deep learning algorithms emit approximately 42 times more CO2 equivalents than papers using traditional methods. On average, a single deep learning-based paper generates 3,297 kilograms of CO2 equivalents—more than the carbon emissions of one person flying from New York City to Melbourne or the amount of CO2 one tree sequesters over 300 years.
Mahlich, Christopher; Vente, Tobias; Beel, Joeran
From Theory to Practice: Implementing and Evaluating e-Fold Cross-Validation Proceedings Article
In: International Conference on Artificial Intelligence and Machine Learning Research (CAIMLR), 2024.
@inproceedings{Mahlich2024,
title = {From Theory to Practice: Implementing and Evaluating e-Fold Cross-Validation},
author = {Christopher Mahlich and Tobias Vente and Joeran Beel},
url = {https://isg.beel.org/blog/2024/09/16/e-fold-cross-validation/},
year = {2024},
date = {2024-01-01},
booktitle = {International Conference on Artificial Intelligence and Machine Learning Research (CAIMLR)},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Vente, Tobias; Mehta, Zainil; Wegmeth, Lukas; Beel, Joeran
Greedy Ensemble Selection for Top-N Recommendations Proceedings Article
In: RobustRecSys Workshop at the 18th ACM Conference on Recommender Systems (ACM RecSys), 2024.
@inproceedings{Vente2024b,
title = {Greedy Ensemble Selection for Top-N Recommendations},
author = {Tobias Vente and Zainil Mehta and Lukas Wegmeth and Joeran Beel},
year = {2024},
date = {2024-01-01},
booktitle = {RobustRecSys Workshop at the 18th ACM Conference on Recommender Systems (ACM RecSys)},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Arabzadeh, Ardalan; Vente, Tobias; Beel, Joeran
Green Recommender Systems: Optimizing Dataset Size for Energy-Efficient Algorithm Performance Proceedings Article
In: International Workshop on Recommender Systems for Sustainability and Social Good (RecSoGood) at the 18th ACM Conference on Recommender Systems (ACM RecSys), 2024.
@inproceedings{Arabzadeh2024,
title = {Green Recommender Systems: Optimizing Dataset Size for Energy-Efficient Algorithm Performance},
author = {Ardalan Arabzadeh and Tobias Vente and Joeran Beel},
url = {https://isg.beel.org/pubs/2024-Green-RecSys-Dataset-Sampling-Ardalan.pdf},
year = {2024},
date = {2024-01-01},
booktitle = {International Workshop on Recommender Systems for Sustainability and Social Good (RecSoGood) at the 18th ACM Conference on Recommender Systems (ACM RecSys)},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Wegmeth, Lukas; Vente, Tobias; Beel, Joeran
Recommender Systems Algorithm Selection for Ranking Prediction on Implicit Feedback Datasets Proceedings Article
In: 18th ACM Conference on Recommender Systems (ACM RecSys), 2024.
@inproceedings{Wegmeth2024,
title = {Recommender Systems Algorithm Selection for Ranking Prediction on Implicit Feedback Datasets},
author = {Lukas Wegmeth and Tobias Vente and Joeran Beel},
url = {https://arxiv.org/pdf/2409.05461},
year = {2024},
date = {2024-01-01},
booktitle = {18th ACM Conference on Recommender Systems (ACM RecSys)},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Meister, Philipp; Wegmeth, Lukas; Vente, Tobias; Beel, Joeran
Removing Bad Influence: Identifying and Pruning Detrimental Users in Collaborative Filtering Recommender Systems Proceedings Article
In: RobustRecSys Workshop at the 18th ACM Conference on Recommender Systems (ACM RecSys), 2024.
@inproceedings{Meister2024,
title = {Removing Bad Influence: Identifying and Pruning Detrimental Users in Collaborative Filtering Recommender Systems},
author = {Philipp Meister and Lukas Wegmeth and Tobias Vente and Joeran Beel},
year = {2024},
date = {2024-01-01},
booktitle = {RobustRecSys Workshop at the 18th ACM Conference on Recommender Systems (ACM RecSys)},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Wegmeth, Lukas; Vente, Tobias; Purucker, Lennart
Revealing the Hidden Impact of Top-N Metrics on Optimization in Recommender Systems Proceedings Article
In: Goharian, Nazli; Tonellotto, Nicola; He, Yulan; Lipani, Aldo; McDonald, Graham; Macdonald, Craig; Ounis, Iadh (Ed.): 46th European Conference on Information Retrieval (ECIR), pp. 140–156, Springer Nature Switzerland, Cham, 2024, ISBN: 978-3-031-56027-9.
@inproceedings{Wegmeth2024b,
title = {Revealing the Hidden Impact of Top-N Metrics on Optimization in Recommender Systems},
author = {Lukas Wegmeth and Tobias Vente and Lennart Purucker},
editor = {Nazli Goharian and Nicola Tonellotto and Yulan He and Aldo Lipani and Graham McDonald and Craig Macdonald and Iadh Ounis},
url = {https://link.springer.com/chapter/10.1007/978-3-031-56027-9_9
https://arxiv.org/pdf/2401.08444},
doi = {10.1007/978-3-031-56027-9_9},
isbn = {978-3-031-56027-9},
year = {2024},
date = {2024-01-01},
booktitle = {46th European Conference on Information Retrieval (ECIR)},
pages = {140–156},
publisher = {Springer Nature Switzerland},
address = {Cham},
abstract = {The hyperparameters of recommender systems for top-n predictions are typically optimized to enhance the predictive performance of algorithms. Thereby, the optimization algorithm, e.g., grid search or random search, searches for the best hyperparameter configuration according to an optimization-target metric, like nDCG or Precision. In contrast, the optimized algorithm, e.g., Alternating Least Squares Matrix Factorization or Bayesian Personalized Ranking, internally optimizes a different loss function during training, like squared error or cross-entropy. To tackle this discrepancy, recent work focused on generating loss functions better suited for recommender systems. Yet, when evaluating an algorithm using a top-n metric during optimization, another discrepancy between the optimization-target metric and the training loss has so far been ignored. During optimization, the top-n items are selected for computing a top-n metric; ignoring that the top-n items are selected from the recommendations of a model trained with an entirely different loss function. Item recommendations suitable for optimization-target metrics could be outside the top-n recommended items; hiddenly impacting the optimization performance. Therefore, we were motivated to analyze whether the top-n items are optimal for optimization-target top-n metrics. In pursuit of an answer, we exhaustively evaluate the predictive performance of 250 selection strategies besides selecting the top-n. We extensively evaluate each selection strategy over twelve implicit feedback and eight explicit feedback data sets with eleven recommender systems algorithms. Our results show that there exist selection strategies other than top-n that increase predictive performance for various algorithms and recommendation domains. However, the performance of the top $$backslashsim 43backslash%$$∼43%of selection strategies is not significantly different. We discuss the impact of our findings on optimization and re-ranking in recommender systems and feasible solutions. The implementation of our study is publicly available.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
The hyperparameters of recommender systems for top-n predictions are typically optimized to enhance the predictive performance of algorithms. Thereby, the optimization algorithm, e.g., grid search or random search, searches for the best hyperparameter configuration according to an optimization-target metric, like nDCG or Precision. In contrast, the optimized algorithm, e.g., Alternating Least Squares Matrix Factorization or Bayesian Personalized Ranking, internally optimizes a different loss function during training, like squared error or cross-entropy. To tackle this discrepancy, recent work focused on generating loss functions better suited for recommender systems. Yet, when evaluating an algorithm using a top-n metric during optimization, another discrepancy between the optimization-target metric and the training loss has so far been ignored. During optimization, the top-n items are selected for computing a top-n metric; ignoring that the top-n items are selected from the recommendations of a model trained with an entirely different loss function. Item recommendations suitable for optimization-target metrics could be outside the top-n recommended items; hiddenly impacting the optimization performance. Therefore, we were motivated to analyze whether the top-n items are optimal for optimization-target top-n metrics. In pursuit of an answer, we exhaustively evaluate the predictive performance of 250 selection strategies besides selecting the top-n. We extensively evaluate each selection strategy over twelve implicit feedback and eight explicit feedback data sets with eleven recommender systems algorithms. Our results show that there exist selection strategies other than top-n that increase predictive performance for various algorithms and recommendation domains. However, the performance of the top $$backslashsim 43backslash%$$∼43%of selection strategies is not significantly different. We discuss the impact of our findings on optimization and re-ranking in recommender systems and feasible solutions. The implementation of our study is publicly available.
Vente, Tobias; Beel, Joeran
The Potential of AutoML for Recommender Systems Journal Article
In: arXiv, pp. 18, 2024.
@article{Vente2024,
title = {The Potential of AutoML for Recommender Systems},
author = {Tobias Vente and Joeran Beel},
url = {https://arxiv.org/abs/2402.04453},
doi = {10.48550/arXiv.2402.04453},
year = {2024},
date = {2024-01-01},
journal = {arXiv},
pages = {18},
abstract = {Automated Machine Learning (AutoML) has greatly advanced applications of Machine Learning (ML) including model compression, machine translation, and computer vision. Recommender Systems (RecSys) can be seen as an application of ML. Yet, AutoML has found little attention in the RecSys community; nor has RecSys found notable attention in the AutoML community. Only few and relatively simple Automated Recommender Systems (AutoRecSys) libraries exist that adopt AutoML techniques. However, these libraries are based on student projects and do not offer the features and thorough development of AutoML libraries. We set out to determine how AutoML libraries perform in the scenario of an inexperienced user who wants to implement a recommender system. We compared the predictive performance of 60 AutoML, AutoRecSys, ML, and RecSys algorithms from 15 libraries, including a mean predictor baseline, on 14 explicit feedback RecSys datasets. To simulate the perspective of an inexperienced user, the algorithms were evaluated with default hyperparameters. We found that AutoML and AutoRecSys libraries performed best. AutoML libraries performed best for six of the 14 datasets (43%), but it was not always the same AutoML library performing best. The single-best library was the AutoRecSys library Auto-Surprise, which performed best on five datasets (36%). On three datasets (21%), AutoML libraries performed poorly, and RecSys libraries with default parameters performed best. Although, while obtaining 50% of all placements in the top five per dataset, RecSys algorithms fall behind AutoML on average. ML algorithms generally performed the worst.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Automated Machine Learning (AutoML) has greatly advanced applications of Machine Learning (ML) including model compression, machine translation, and computer vision. Recommender Systems (RecSys) can be seen as an application of ML. Yet, AutoML has found little attention in the RecSys community; nor has RecSys found notable attention in the AutoML community. Only few and relatively simple Automated Recommender Systems (AutoRecSys) libraries exist that adopt AutoML techniques. However, these libraries are based on student projects and do not offer the features and thorough development of AutoML libraries. We set out to determine how AutoML libraries perform in the scenario of an inexperienced user who wants to implement a recommender system. We compared the predictive performance of 60 AutoML, AutoRecSys, ML, and RecSys algorithms from 15 libraries, including a mean predictor baseline, on 14 explicit feedback RecSys datasets. To simulate the perspective of an inexperienced user, the algorithms were evaluated with default hyperparameters. We found that AutoML and AutoRecSys libraries performed best. AutoML libraries performed best for six of the 14 datasets (43%), but it was not always the same AutoML library performing best. The single-best library was the AutoRecSys library Auto-Surprise, which performed best on five datasets (36%). On three datasets (21%), AutoML libraries performed poorly, and RecSys libraries with default parameters performed best. Although, while obtaining 50% of all placements in the top five per dataset, RecSys algorithms fall behind AutoML on average. ML algorithms generally performed the worst.
2023
Vente, Tobias
Advancing Automation of Design Decisions in Recommender System Pipelines Proceedings Article
In: Proceedings of the 17th ACM Conference on Recommender Systems, pp. 1355-1360, 2023.
@inproceedings{Vente2023,
title = {Advancing Automation of Design Decisions in Recommender System Pipelines},
author = {Tobias Vente},
doi = {https://dl.acm.org/doi/10.1145/3604915.3608886},
year = {2023},
date = {2023-01-01},
booktitle = {Proceedings of the 17th ACM Conference on Recommender Systems},
pages = {1355-1360},
abstract = {Recommender systems have become essential in domains like streaming services, social media platforms, and e-commerce websites. However, the development of a recommender system involves a complex pipeline with preprocessing, data splitting, algorithm and model selection, and postprocessing stages. Every stage of the recommender systems pipeline requires design decisions that influence the performance of the recommender system. To ease design decisions, automated machine learning (AutoML) techniques have been adapted to the field of recommender systems, resulting in various AutoRecSys libraries. Nevertheless, these libraries limit flexibility in integrating automation techniques. In response, our research aims to enhance the usability of AutoML techniques for design decisions in recommender system pipelines. We focus on developing flexible and library-independent automation techniques for algorithm selection, model selection, and postprocessing steps. By enabling developers to make informed choices and ease the recommender system development process, we decrease the developer’s effort while improving the performance of the recommender systems. Moreover, we want to analyze the cost-to-benefit ratio of automation techniques in recommender systems, evaluating the computational overhead and the resulting improvements in predictive performance. Our objective is to leverage AutoML concepts to automate design decisions in recommender system pipelines, reduce manual effort, and enhance the overall performance and usability of recommender systems.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Recommender systems have become essential in domains like streaming services, social media platforms, and e-commerce websites. However, the development of a recommender system involves a complex pipeline with preprocessing, data splitting, algorithm and model selection, and postprocessing stages. Every stage of the recommender systems pipeline requires design decisions that influence the performance of the recommender system. To ease design decisions, automated machine learning (AutoML) techniques have been adapted to the field of recommender systems, resulting in various AutoRecSys libraries. Nevertheless, these libraries limit flexibility in integrating automation techniques. In response, our research aims to enhance the usability of AutoML techniques for design decisions in recommender system pipelines. We focus on developing flexible and library-independent automation techniques for algorithm selection, model selection, and postprocessing steps. By enabling developers to make informed choices and ease the recommender system development process, we decrease the developer’s effort while improving the performance of the recommender systems. Moreover, we want to analyze the cost-to-benefit ratio of automation techniques in recommender systems, evaluating the computational overhead and the resulting improvements in predictive performance. Our objective is to leverage AutoML concepts to automate design decisions in recommender system pipelines, reduce manual effort, and enhance the overall performance and usability of recommender systems.
Vente, Tobias; Ekstrand, Michael; Beel, Joeran
Introducing LensKit-Auto, an Experimental Automated Recommender System (AutoRecSys) Toolkit Proceedings Article
In: Proceedings of the 17th ACM Conference on Recommender Systems, pp. 1212-1216, 2023.
@inproceedings{Vente2023a,
title = {Introducing LensKit-Auto, an Experimental Automated Recommender System (AutoRecSys) Toolkit},
author = {Tobias Vente and Michael Ekstrand and Joeran Beel},
url = {https://dl.acm.org/doi/10.1145/3604915.3610656},
year = {2023},
date = {2023-01-01},
booktitle = {Proceedings of the 17th ACM Conference on Recommender Systems},
pages = {1212-1216},
abstract = {LensKit is one of the first and most popular Recommender System libraries. While LensKit offers a wide variety of features, it does not include any optimization strategies or guidelines on how to select and tune LensKit algorithms. LensKit developers have to manually include third-party libraries into their experimental setup or implement optimization strategies by hand to optimize hyperparameters. We found that 63.6% (21 out of 33) of papers using LensKit algorithms for their experiments did not select algorithms or tune hyperparameters. Non-optimized models represent poor baselines and produce less meaningful research results. This demo introduces LensKit-Auto. LensKit-Auto automates the entire Recommender System pipeline and enables LensKit developers to automatically select, optimize, and ensemble LensKit algorithms.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
LensKit is one of the first and most popular Recommender System libraries. While LensKit offers a wide variety of features, it does not include any optimization strategies or guidelines on how to select and tune LensKit algorithms. LensKit developers have to manually include third-party libraries into their experimental setup or implement optimization strategies by hand to optimize hyperparameters. We found that 63.6% (21 out of 33) of papers using LensKit algorithms for their experiments did not select algorithms or tune hyperparameters. Non-optimized models represent poor baselines and produce less meaningful research results. This demo introduces LensKit-Auto. LensKit-Auto automates the entire Recommender System pipeline and enables LensKit developers to automatically select, optimize, and ensemble LensKit algorithms.
Wegmeth, Lukas; Vente, Tobias; Beel, Joeran
The Challenges of Algorithm Selection and Hyperparameter Optimization for Recommender Systems Journal Article
In: COSEAL Workshop 2023, 2023.
@article{Wegmeth2023b,
title = {The Challenges of Algorithm Selection and Hyperparameter Optimization for Recommender Systems},
author = {Lukas Wegmeth and Tobias Vente and Joeran Beel},
url = {http://dx.doi.org/10.13140/RG.2.2.24089.19049},
year = {2023},
date = {2023-01-01},
journal = {COSEAL Workshop 2023},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Wegmeth, Lukas; Vente, Tobias; Purucker, Lennart; Beel, Joeran
The Effect of Random Seeds for Data Splitting on Recommendation Accuracy Proceedings Article
In: Proceedings of the 3rd Perspectives on the Evaluation of Recommender Systems Workshop, 2023.
@inproceedings{Wegmeth2023,
title = {The Effect of Random Seeds for Data Splitting on Recommendation Accuracy},
author = {Lukas Wegmeth and Tobias Vente and Lennart Purucker and Joeran Beel},
url = {https://ceur-ws.org/Vol-3476/paper4.pdf},
year = {2023},
date = {2023-01-01},
booktitle = {Proceedings of the 3rd Perspectives on the Evaluation of Recommender Systems Workshop},
abstract = {The evaluation of recommender system algorithms depends on randomness, e.g., during randomly splitting data into training and testing data. We suspect that failing to account for randomness in this scenario may lead to misrepresenting the predictive accuracy of recommendation algorithms. To understand the community’s view of the importance of randomness, we conducted a paper study on 39 full papers published at the ACM RecSys 2022 conference. We found that the authors of 26 papers used some variation of a holdout split that requires a random seed. However, only five papers explicitly repeated experiments and averaged their results over different random seeds. This potentially problematic research practice motivated us to analyze the effect of data split random seeds on recommendation accuracy. Therefore, we train three common algorithms on nine public data sets with 20 data split random seeds, evaluate them on two ranking metrics with three different ranking cutoff values k, and compare the results. In the extreme case with k = 1, we show that depending on the data split random seed, the accuracy with traditional recommendation algorithms deviates by up to ∼6.3% from the mean accuracy achieved on the data set. Hence, we show that an algorithm may significantly over- or under-perform when maliciously or negligently selecting a random seed for splitting the data. To showcase a mitigation strategy and better research practice, we compare holdout to cross-validation and show that, again, for k = 1, the accuracy of algorithms evaluated with cross-validation deviates only up to ∼2.3% from the mean accuracy achieved on the data set. Furthermore, we found that the deviation becomes smaller the higher the value of k for both holdout and cross-validation.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
The evaluation of recommender system algorithms depends on randomness, e.g., during randomly splitting data into training and testing data. We suspect that failing to account for randomness in this scenario may lead to misrepresenting the predictive accuracy of recommendation algorithms. To understand the community’s view of the importance of randomness, we conducted a paper study on 39 full papers published at the ACM RecSys 2022 conference. We found that the authors of 26 papers used some variation of a holdout split that requires a random seed. However, only five papers explicitly repeated experiments and averaged their results over different random seeds. This potentially problematic research practice motivated us to analyze the effect of data split random seeds on recommendation accuracy. Therefore, we train three common algorithms on nine public data sets with 20 data split random seeds, evaluate them on two ranking metrics with three different ranking cutoff values k, and compare the results. In the extreme case with k = 1, we show that depending on the data split random seed, the accuracy with traditional recommendation algorithms deviates by up to ∼6.3% from the mean accuracy achieved on the data set. Hence, we show that an algorithm may significantly over- or under-perform when maliciously or negligently selecting a random seed for splitting the data. To showcase a mitigation strategy and better research practice, we compare holdout to cross-validation and show that, again, for k = 1, the accuracy of algorithms evaluated with cross-validation deviates only up to ∼2.3% from the mean accuracy achieved on the data set. Furthermore, we found that the deviation becomes smaller the higher the value of k for both holdout and cross-validation.
2022
Vente, Tobias; Purucker, Lennart; Beel, Joeran
The Feasibility of Greedy Ensemble Selection for Automated Recommender Systems Journal Article
In: COSEAL Workshop 2022, 2022.
@article{Vente2022,
title = {The Feasibility of Greedy Ensemble Selection for Automated Recommender Systems},
author = {Tobias Vente and Lennart Purucker and Joeran Beel},
url = {http://dx.doi.org/10.13140/RG.2.2.16277.29921},
year = {2022},
date = {2022-01-01},
journal = {COSEAL Workshop 2022},
keywords = {},
pubstate = {published},
tppubtype = {article}
}