New publication: Position Bias in Recommender Systems for Digital Libraries

We recently had a paper accepted to iConference. We used click data from Mr DLib, our recommender-as-a-service, to see if users of recommender systems in digital libraries are affected by position bias. We found that users are affected by position bias on average. Some users do seem to examine recommendations more carefully than other users.

@inproceedings{collins2018position,
  title={Position bias in recommender systems for digital libraries},
  author={Collins, Andrew and Tkaczyk, Dominika and Aizawa, Akiko and Beel, Joeran},
  booktitle={International Conference on Information},
  pages={335--344},
  year={2018},
  organization={Springer}
}

As far as we know, this is the only investigation of position bias in digital libraries, and in recommender systems for digital libraries. We have made the data available. It contains 10 million recommendations delivered by us to users, with 12,543 clicks on recommendations logged. We hope that other researchers will find this useful. For example, perhaps the click behavior of a digital library user could be modeled using deep learning.

The paper can be read here, and the data is available here.

Abstract. “Position bias” describes the tendency of users to interact with items on top of a list with higher probability than with items at a lower position in the list, regardless of the items’ actual relevance. In the domain of recommender systems, particularly recommender systems in digital libraries, position bias has received little attention. We conduct a study in a real-world recommender system that delivered ten million related-article recommendations to the users of the digital library Sowiport, and the reference manager JabRef. Recommendations were randomly chosen to be shuffled or non-shuffled, and we compared click-through rate (CTR) for each rank of the recommendations. According to our analysis, the CTR for the highest rank in the case of Sowiport is 53% higher than expected in a hypothetical non-biased situation (0.189% vs. 0.123%). Similarly, in the case of Jabref the highest rank received a CTR of 1.276%, which is 87% higher than expected (0.683%). A chi-squared test confirms the strong relationship between the rank of the recommendation shown to the user and whether the user decided to click it (p < 0.01 for both Jabref and Sowiport). Our study confirms the findings from other domains, that recommendations in the top positions are more often clicked, regardless of their actual relevance.