Valve Reviewing Review Reviews

A new post on the Steam Blog has further thoughts from Valve on the process of refining the user review process on Steam to make them more useful to other users. They offer an extensive discussion of what makes for helpful reviews, and how they evaluate reviews for this quality. They note that one of the problems they must deal with is that the system for reviewing reviews itself is as subject to abuse as the review system itself. Here's word on what they are doing about this:
What we are changing
Up to this point, we've tried to maintain just showing the simple math behind how we calculate whether a review is helpful or not--the percentage and number of people who indicated a review as helpful. We like systems that are transparent and easy to understand, as they are also easier to believe and trust. Unfortunately, this has resulted in a system that allows a small group to manipulate reviews to a degree that is clearly decreasing the value of Steam for many other players. So we're making two main changes.

  1. Firstly, our system will use a new method of calculating the helpfulness of each review, taking into account the users that are trying to manipulate the system. One way we're doing that is by counting the helpful ratings on reviews differently for users that are far outside the norm. Ratings from users that follow normal patterns of rating will continue to be counted the same way that they have, whereas accounts that rate an excessive number of reviews on an individual game will see the weight of each individual rating count for less and less.
  2. Secondly, store pages will now show the default helpful positive and negative reviews in a similar proportion to that of the overall review score for the game. For example, if the game is reviewed positively by 80% of reviewers, then the ten reviews shown by default on the store page will be 80% positive, showing eight positive and two negative. This should keep the reviews shown on a game's page from being so easily manipulated by a few determined players and should more accurately represent the overall sentiment of the people playing the game.

We're rolling out both of these changes as a beta today. You can turn the new method on and off to see how it impacts the default display of reviews on any given store page. Note that these changes only impact the default listing of reviews (Called "Summary") and the "Most Helpful" display option.