Measures are tuned for Interactive UIThis implementation is targeted for interactive use in search engines. A search UI usually has the first few results shown in ranked order, with the option to go to the next few results. This UI is intended to show the first three ranked sentences at the top of an entry with the theme words highlighted. Users are not forgiving of mistakes in these situations. The first result is much more important than the second, and so forth. People rarely click through to the second page.
The measures of effectiveness are formulated with this in mind. We used three:
- A variant of Mean Reciprocal Rank (MRR).
- "Rating" is a measure we created to model the user's behavior in a summarization UI. Our MMR variant and Rating are defined in the next Post.
- Non-zero counts whether the algorithm placed any recommendations in the top three. "Did we even hit the dartboard?"
Overall Comparison Chart
- Key to algorithm names: "binary_normal" means that "binary" was used to create each cell, while "normal" multiplied each term vector with the mean normalized term vector. If there is no second key, the global value was 1. See post #1 for the full list of algorithms.