Computing & AIPreprintData analysis4 min read

YOU KNOW WHAT YOU WILL LIKE BETTER THAN THE ALGORITHM

Recommender systems decide much of what people watch, read and buy; at Netflix, according to a 2015 paper the authors cite, the recommender “influences choice for about 80% of hours streamed”. To work, a platform has to guess what each person will like. It can learn that from behaviour — the ratings, the clicks, the titles finished by you and by millions of others — or it can simply ask.

Asking is costly. On MovieLens, a film-rating site, 98.8% of 3.0 million questions went unanswered. And theory gives conflicting advice. One tradition, which the authors call the crowd view, says statistics built from many people beat individual judgment, and that people are poor at predicting their own enjoyment. Another, the view of dispersed knowledge, says each of us holds information no database records: you loved the director’s earlier work but never rated it, the trailer left you cold, a friend with trusted taste will not stop talking about it.

A question on the homepage

Zihan Chen and Di Zhu (Stevens Institute of Technology) and Lei Nico Zheng (University of Massachusetts Boston) used a dataset that makes the test possible. MovieLens, a non-commercial recommender run by the GroupLens lab at the University of Minnesota, placed a question on its homepage from March 2023 to May 2024. It showed a film title and asked whether the member had seen it. If not: how many stars do you think you would give it, from half a star to five?

Of 26,020 such guesses, 551 — from 277 members — were followed by a real rating at least a day later. On exactly these pairs, the team compared the guess with four machines: the prediction of MovieLens’s own recommender, two classic recommendation algorithms trained on 33 million earlier ratings and updated with everything the member had rated up to the moment of the guess, and a “stack” combining every machine forecast with what the platform knew about the film and the member.

People win

The member’s guess correlated 0.69 with the later rating, on a scale where 1 means the two always move together and 0 means no link at all. The trained algorithms reached 0.54 to 0.59; MovieLens’s own prediction, 0.34. On average, the guess missed the later rating by 0.61 stars, against at least 0.72 for any machine. It was also nearly unbiased, while MovieLens overestimated the later rating by half a star.

The advantage is not just knowing which films are good or how generously one rates. Remove the film’s average appeal and the member’s usual generosity, and the guess still predicts what is left — the personal match between this person and this film — far better than the machines (0.63 against at most 0.47).

Added to everything the machines knew, the guess cut the prediction error by 26%. The reverse was not true: adding all the machine information to the guess brought no detectable gain.

Where the machines catch up

The picture is more balanced for ordering. Given two films a member later rated differently, the guess put them in the right order 72% of the time. MovieLens managed 56% — a coin flip scores 50% — but the trained algorithms reached 64 to 67%, which the authors find statistically indistinguishable from the person.

And on familiar ground, the machines do as well or better. For films that members declared unseen but had in fact rated years earlier — older, heavily rated films, rated by the site’s heaviest users — the algorithms matched or beat the person. The person’s edge is largest where data are thin: for little-rated films (0.65 against 0.41 for one algorithm and 0.13 for MovieLens) and for genres the member rarely rates.

What does not help

Two shortcuts fail. A member’s earlier guesses about other films, even weighted by how similar those films are, do not replace a fresh guess about the film at hand. And asking how sure people are adds almost nothing: certainty mostly flags extreme opinions, and the surest members turned out somewhat overoptimistic.

The authors also checked that people were not simply copying the site’s displayed prediction: their guesses matched it about as often as chance would.

Four rules for platforms

From these results the team draws four design principles: ask for the expected rating of a specific item; ask about the items being decided on now; judge recommenders on the top candidates they actually re-rank, where MovieLens ordered only 56% of pairs correctly; and give any algorithm everything the platform knows before comparing it with people.

The limits are clear in the paper. The 551 pairs are 2.1% of all guesses, from enthusiasts on a single site, and people rate what they chose to watch. A randomised experiment, the authors suggest, would tell whether asking actually improves the recommendations people follow. Until then the conclusion stands as their title puts it: you know something about the film in front of you that no rating history has recorded.