We are pleased to announce that Colin Daly is joining our core group of researchers. Colin is a part-time researcher and will work towards a Ph.D. over a six-year period. Colin has a BSc from University College Cork, an MBA, and MSc from the Open University, a Higher Diploma from Read more…
AMIR 2019 — The 1st Interdisciplinary Workshop on Algorithm Selection and Meta-Learning in Information Retrieval — is over, and it was a full success. We had many interesting presentations and around 20 attendees. The following photos give some impressions.
Our proposal for the “1st Interdisciplinary Workshop on Algorithm Selection and Meta-Learning in Information Retrieval (AMIR)“, to be held at the 41st European Conference on Information Retrieval (ECIR), was accepted. AMIR will take place on the 14th of April 2019 in Cologne, Germany. The algorithm selection problem describes the challenge Read more…
Lars Kotthoff and I have applied to organize the 1st Interdisciplinary Workshop on Algorithm Selection and Meta-Learning in Information Retrieval (AMIR) at the 41st European Conference on Information Retrieval (ECIR). Let’s cross fingers and hope it will get accepted. In the following, you find the proposal (also available on ResearchGate as PDF). Read more…
We recently submitted a paper to UMAP (The Conference on User Modelling, Adaptation, and Personalization). The paper was about how mind-maps could be utilized by information retrieval applications such as recommender systems. The paper got accepted, which means we will be in Aalborg, Denmark from July 7 until July 11 to present the Read more…
As you may know, Docear offers literature recommendations and as you may know further, it’s part of my PhD to find out how to make these recommendations as good as possible. To accomplish this I need to know what a ‘good’ recommendation is. So far we have been using Click Through Rates (CTR) to evaluate different recommendation algorithms. CTR is a common performance measure in online advertisement. For instance, if a recommendation is shown 1000 times and clicked 12 times, then the CTR is 1,2% (12/1000). That means if an algorithm A has a CTR of 1% and algorithm B has a CTR of 2%, B is better.
Recently, we submitted a paper to a conference. The paper summarized the results of some evaluations we did with different recommendation algorithms. The paper was rejected. Among others, a reviewer criticized the CTR as a too simple evaluation metric. We should rather use metrics that are common in information retrieval such as Mean Absolute Error (MAE), Mean Squared Error (MSE), Root Mean Squared Error (RMSE), or Precision (i.e. Mean Average Precision, MAE).
The funny thing is, CTR, MAE, MSE, RMSE and Precision are basically all the same, at least in a binary classification problem (recommendation relevant / clicked vs. recommendation irrelevant / not clicked). The table shows an example. Assume, you show ten recommendations to users (Rec1…Rec10). Then is the ‘Estimate’ for each recommendation ‘1’, i.e. it’s clicked by a user. The ‘Actual‘ value describes if a user actually clicked on a recommendation (‘1) or not (‘0’). The ‘Error’ is either 0 (if the recommendation actually was clicked) or 1 (if it was not clicked). The mean absolute error (MAE) is simply the sum of all errors (6 in the example) devided by the number of total recommendations (10 in the example). Since we have only zeros and ones, it makes no difference if they are squared or not. Consequently, the mean squared error (MSE) is identical to MAE. In addition, precision and mean average precision (MAP) is identical to CTR; precision (and CTR) is exactly 1-MAE (or 1-MSE), and also RMSE perfectly correlates with the other values because it’s simply the root square of MSE (or MAE).
In a binary evaluation (relevant / not relevant) in information retrieval, there is no difference in the significance between Click Through Rate (CTR), Mean Absolute Error (MAE), Mean Squared Error (MSE), Root Mean Squared Error (RMSE), and Precision.
The third Beta of Docear was released today (download here). Besides several bug fixes, the major changes are:
An automatic update-check which informs you when a new version of Docear was released. We also have a proper licence now you have to accept when starting Docear for the first time. Of course, the main licences still remains GPL2+ but we now have additional terms of service for using our online backup etc. We also implemented an optional function to transmit your mind maps to our servers so we can use them for our research (read here for more details). We would kindly ask you to activate this function because it allows us to perform our research on mind maps and the more and better research we do, the more likely we will get additional funding from the German Government to further develop Docear. As stated in our terms of use and data processing terms, we are bound to the very strict German data privacy law and try everything to protect your data. However, if you feel uncomfortable having your mind maps analyzed by us, please just deactivate the options that are shown on first start. You can then use Docear without any data being submitted to us. Finally, you can directly register a Docear account when starting Docear the first time. This account is required if you want to backup your files to our servers. We would highly recommend this since you never know what happens to your computer and you can also access your mind maps through our web interface if you are not at home. Read also here for more information about the backup and alternatives.
Here are all the changes: