ASK PLAYERS FOR REVIEWS, AND THE SCORE GOES QUIET

Online ratings steer what we buy, which apps we install and which games we play. But they are built from the few customers who bother to write: most people never review anything. The standard fix is to ask. On 30 October 2019, Valve released a new library for its Steam game store, in which the software asked players whether they wanted to review a game they had recently played. Across the store, reviews rose from about 17,000 a day to about 70,000.

Did asking make ratings better or worse? Theories disagree. More reviewers should mean less noise. But people who write unprompted often do so because something surprised them — and a prompted review may carry less of that news. And when scores crowd toward the top, they may simply say less.

What a rating knows, and what it shows

Zihan Chen and Di Zhu, at the Stevens Institute of Technology, and Lei Nico Zheng, at the University of Massachusetts Boston, borrowed a tool long used to study human judgment and diagnostic tests: signal detection theory. It separates two things. Sensitivity is how well a system can tell a real change from ordinary noise — what a rating knows. The criterion is how much evidence it needs before it reacts — what it shows. A score can move less because it knows less, or because its scale has squeezed toward the ceiling while the display still waits for the same jump.

The ceiling matters because a rating is the share of positive reviews among all reviews: the same shift in the balance between praise and complaints moves a 95% score far less than a 60% one.

A three-hour trigger

Steam is an ideal test. Every public review shows how long its author had played when writing it, and Valve’s prompt is triggered by playing time. Valve does not publish the trigger, so the authors found it in the data: 180 minutes. After the launch, 11,163 reviews were written at exactly 180 minutes of play against 2,316 at 179; before, the two minutes had 3,390 and 3,255.

Using 21.6 million reviews of 315 popular games, gathered by an independent collector and posted online, they compared reviews just above and just below three hours, before and after the launch. Just past three hours, reviews became 4.25 times as numerous as just below, and the jump in the share of positive reviews grew by 6.6 points. The added reviews are about 96% positive, against 86% just below the threshold, and short: 71% of them are under 50 characters.

Ranking holds, alarms fade

For choosing between games, little changed at first. Holding the number of reviews per game equal, the scores’ agreement with critics stayed put: a rank correlation with the critic scores compiled by Metacritic — a measure of how closely two rankings agree, where 1 means the same order — of 0.42 before and 0.41 after, across more than a thousand games.

Noticing that a game has changed is another story. The authors use the reviews written in the first two hours of play as a reference signal, and measure how much of its monthly movement shows up in other groups of reviews. Just above the threshold, that share fell from 0.73 to 0.18; just below it, only from 0.81 to 0.64. The break appears at three hours and at the launch date, and nowhere else.

The labels Steam shows went quiet too. Steam sums up each score with a word label whose tiers start at fixed shares of positive reviews (95%, 80%, 70%, 40%, 20%). Computed only from the reviews written just past three hours, such a tier dropped in 42% of bad months — months when the reference reviews from the first two hours took a bad turn — before the launch, and in only 12% after; just below the threshold, it went from 47% to 41%. On a reconstruction of the public 30-day score, the label fell in 38% of bad months before and 9% after — with false alarms dropping as well.

Fewer alarms, not less knowledge

Here the two notions part ways. Each added review responds to a bad month about half as much as a voluntary one. But there are so many of them that, at the volumes actually observed, the stream above the threshold lost no detectable information compared with the stream below: its agreement with the reference reviews from the first two hours changed by only −0.03 relative to the stream below, a gap that could just as well be zero (95% interval from −0.13 to 0.06). The rating knows nearly as much; it shows it less, because the scale has compressed and the labels have not moved.

Some bad news still gets through. Failures that developers acknowledged in their own announcements — outages, rollbacks, lost progress — mostly bring complaints, and complaints were inflated less than praise. At those failures, the overall score fell by 2.9 points after the launch, against 1.0 before.

The authors also checked their model at Steam’s later prompts, at 10 and 30 hours, which add far fewer reviews: there it predicted almost no break, and none appeared.

Read the score differently

The study covers one platform, popular games and data that stop in January 2021, and the prompt and the added reviews are inferred rather than flagged. Its advice is about reading, not about asking less: platforms that solicit reviews should expect fixed labels to fall silent, judge a display by how often it catches real problems against how often it cries wolf, and recalibrate. Old rules of thumb do not transfer — a five-point drop that once signalled trouble has become rare in any month.