The International Journal on Digital Libraries (IJDL) published our manuscript “Choice Overload and Recommendation Effectiveness in Related-Article Recommendations: Analyzing the Sowiport Digital Library”. The paper is freely available as open access via Springer. The paper is an extended version of a previous paper published at the 5th International Workshop on Bibliometric-enhanced Information Retrieval (BIR). In this extension, we analyze a dataset 12.1 times larger than the dataset from the previous analysis. Furthermore, in addition to the click-through rate (CTR), we use three further metrics to assess choice overload and recommendation effectiveness: the percentage of clicked sets (clicked set rate, CSR), the average clicks per clicked recommendation set (ACCS), and the time between recommendation delivery and first click (time to first click, TTFC). Additionally, we investigate the effectiveness of different recommender algorithms.
Abstract
Choice overload describes a situation in which a person has difficulty in making decisions due to too many options. We
examine choice overload when displaying related-article recommendations in digital libraries, and examine the effectiveness of recommendation algorithms in this domain. We first analyzed existing digital libraries and found that only 30% of digital libraries show related-article recommendations to their users. Of these libraries, the majority (74%) displays 3–5 related
articles; 28% of them display 6–10 related articles; and no digital library displayed more than ten related-article recommendations. We then conducted our experimental evaluation through GESIS’ digital library Sowiport, with recommendations delivered by recommendations-as-a-service provider Mr. DLib. We use four metrics to analyze 41.3 million delivered recommendations: click-through rate (CTR), percentage of clicked recommendation sets (clicked set rate, CSR), average clicks per clicked recommendation set (ACCS), and time to first click (TTFC), which is the time between delivery of a set of recommendations to the first click. These metrics help us to analyze choice overload and can yield evidence for finding the ideal number of recommendations to display. We found that with increasing recommendation set size, i.e., the numbers of displayed recommendations, CTR decreases from 0.41% for one recommendation to 0.09% for 15 recommendations. Most recommendation sets only receive one click. ACCS increases with set size but increases more slowly for six recommendations and more. When displaying 15 recommendations, the average clicks per set is at a maximum (1.15). Similarly, TTFC increases with larger recommendation set size but increases more slowly for sets of more than five recommendations. While CTR and CSR do not indicate choice overload, ACCS and TTFC point toward 5–6 recommendations as being optimal for Sowiport. Content-based filtering yields the highest CTR with 0.118%, while stereotype recommendations yield the highest ACCS (1.28). Stereotype recommendations also yield the highest TTFC. This means that users take more time before clicking stereotype recommendations when compared to recommendations based on other algorithms.
Reference
Felix Beierle, Akiko Aizawa, Andrew Collins, and Joeran Beel. 2019. Choice overload and recommendation effectiveness in related-article recommendations. International Journal of Digital Libraries (IJDL) (2019). DOI:https://doi.org/10.1007/s00799-019-00270-7
BibTeX
@Article{BeierleAizawaCollinsBeel2019,
author = {Beierle, Felix and Aizawa, Akiko and Collins, Andrew and Beel, Joeran},
title = {Choice overload and recommendation effectiveness in related-article recommendations},
journal = {International Journal of Digital Libraries (IJDL)},
year = {2019},
doi = {10.1007/s00799-019-00270-7},
}
0 Comments