What we are changing
Up to this point, we've tried to maintain just showing the simple math behind how we calculate whether a review is helpful or not--the percentage and number of people who indicated a review as helpful. We like systems that are transparent and easy to understand, as they are also easier to believe and trust. Unfortunately, this has resulted in a system that allows a small group to manipulate reviews to a degree that is clearly decreasing the value of Steam for many other players. So we're making two main changes.
- Firstly, our system will use a new method of calculating the helpfulness of each review, taking into account the users that are trying to manipulate the system. One way we're doing that is by counting the helpful ratings on reviews differently for users that are far outside the norm. Ratings from users that follow normal patterns of rating will continue to be counted the same way that they have, whereas accounts that rate an excessive number of reviews on an individual game will see the weight of each individual rating count for less and less.
- Secondly, store pages will now show the default helpful positive and negative reviews in a similar proportion to that of the overall review score for the game. For example, if the game is reviewed positively by 80% of reviewers, then the ten reviews shown by default on the store page will be 80% positive, showing eight positive and two negative. This should keep the reviews shown on a game's page from being so easily manipulated by a few determined players and should more accurately represent the overall sentiment of the people playing the game.
We're rolling out both of these changes as a beta today. You can turn the new method on and off to see how it impacts the default display of reviews on any given store page. Note that these changes only impact the default listing of reviews (Called "Summary") and the "Most Helpful" display option.
ItBurn wrote on Nov 22, 2017, 03:00:^these +1jdreyer wrote on Nov 22, 2017, 02:04:
As for Ragnarok, it's great. I don't know if it's my favorite Marvel film. I love the first Cap movie, the first Iron Man, the first Avengers, and the first GotG, too.
For me it's Ragnarock, Deadpool and GotG1. The others are so far away that I won't even mention them.
ItBurn wrote on Nov 22, 2017, 03:00:jdreyer wrote on Nov 22, 2017, 02:04:
As for Ragnarok, it's great. I don't know if it's my favorite Marvel film. I love the first Cap movie, the first Iron Man, the first Avengers, and the first GotG, too.
For me it's Ragnarock, Deadpool and GotG1. The others are so far away that I won't even mention them.
jdreyer wrote on Nov 22, 2017, 02:04:
As for Ragnarok, it's great. I don't know if it's my favorite Marvel film. I love the first Cap movie, the first Iron Man, the first Avengers, and the first GotG, too.
ItBurn wrote on Nov 21, 2017, 16:57:jdreyer wrote on Nov 21, 2017, 16:33:ItBurn wrote on Nov 21, 2017, 15:55:jdreyer wrote on Nov 21, 2017, 15:16:
Rotten Tomatoes offers both: a fresh/rotten meta calculation of like/didn't like as well as a 10 point scale average. It includes these for both reviewers and audience members.
That's a system that pleases everyone it would seem.
I was pretty sure that the rotten tomatoes point system was based on thumbs up and not a 10 point system, just like Steam. Anyone can confirm?
Although Dacote pointed you to the RT website, it's better to look at a specific movie like Thor Ragnarok to see the scale. 92% of reviewers liked the movie, giving it an average 10 point rating of 7.5. 88% of audience members liked it, with a 4.3 out of 5 rating.
I don't understand your numbers, so I don't know if you're with me or against me :p But Thor Ragnarock isn't a good barometer of the system because it was too good of a movie. You need to get a really mediocre one. There's tons of examples in the Marvel library...
Side note, Thor Ragnarock is my favorite Marvel movie ever. The others are so... mediocre. Some are better than others, but they place too much emphasis on the hero carrying the movie. Everything else is meh, and since I'm not really a superhero guy, they rarely do anything for me. Ragnarock is just a good movie overall.
Cutter wrote on Nov 21, 2017, 15:28:ItBurn wrote on Nov 21, 2017, 13:04:
Statistically, the thumbs up would rate a 10/10 game and a 6/10 game the same, this is bad. A 10/10 system is all over the place and people have bias, but it doesn't matter. The aggregate of tons of reviews will be closer to the real rating than a thumbs up system would. I mean, have you seen critic movie reviews on rotten tomatoes? The most mediocre films have 80s and 90s. If you look at games on metacritic, where most of critic reviews are on a base 10 system, you get MUCH more accurate results.
My point exactly. There are a ton of games I would rate a 4/5/6 out of 10 which would give you a much better idea of whether to take a chance on buying it or not. Whereas the current system says - to me at any rate - either buy or don't buy, no middle ground.
jdreyer wrote on Nov 21, 2017, 15:16:Similar to Metacritic, just with less vegetable (or is a tomato a fruit?).
Rotten Tomatoes offers both: a fresh/rotten meta calculation of like/didn't like as well as a 10 point scale average. It includes these for both reviewers and audience members.
That's a system that pleases everyone it would seem.
jdreyer wrote on Nov 21, 2017, 16:33:ItBurn wrote on Nov 21, 2017, 15:55:jdreyer wrote on Nov 21, 2017, 15:16:
Rotten Tomatoes offers both: a fresh/rotten meta calculation of like/didn't like as well as a 10 point scale average. It includes these for both reviewers and audience members.
That's a system that pleases everyone it would seem.
I was pretty sure that the rotten tomatoes point system was based on thumbs up and not a 10 point system, just like Steam. Anyone can confirm?
Although Dacote pointed you to the RT website, it's better to look at a specific movie like Thor Ragnarok to see the scale. 92% of reviewers liked the movie, giving it an average 10 point rating of 7.5. 88% of audience members liked it, with a 4.3 out of 5 rating.
ItBurn wrote on Nov 21, 2017, 15:55:jdreyer wrote on Nov 21, 2017, 15:16:
Rotten Tomatoes offers both: a fresh/rotten meta calculation of like/didn't like as well as a 10 point scale average. It includes these for both reviewers and audience members.
That's a system that pleases everyone it would seem.
I was pretty sure that the rotten tomatoes point system was based on thumbs up and not a 10 point system, just like Steam. Anyone can confirm?
ItBurn wrote on Nov 21, 2017, 15:55:https://www.rottentomatoes.comjdreyer wrote on Nov 21, 2017, 15:16:
Rotten Tomatoes offers both: a fresh/rotten meta calculation of like/didn't like as well as a 10 point scale average. It includes these for both reviewers and audience members.
That's a system that pleases everyone it would seem.
I was pretty sure that the rotten tomatoes point system was based on thumbs up and not a 10 point system, just like Steam. Anyone can confirm?
jdreyer wrote on Nov 21, 2017, 15:16:
Rotten Tomatoes offers both: a fresh/rotten meta calculation of like/didn't like as well as a 10 point scale average. It includes these for both reviewers and audience members.
That's a system that pleases everyone it would seem.
ItBurn wrote on Nov 21, 2017, 13:04:
Statistically, the thumbs up would rate a 10/10 game and a 6/10 game the same, this is bad. A 10/10 system is all over the place and people have bias, but it doesn't matter. The aggregate of tons of reviews will be closer to the real rating than a thumbs up system would. I mean, have you seen critic movie reviews on rotten tomatoes? The most mediocre films have 80s and 90s. If you look at games on metacritic, where most of critic reviews are on a base 10 system, you get MUCH more accurate results.
BIGtrouble77 wrote on Nov 21, 2017, 11:48:The add a feature puleeze along with the troll-brigade upset about a developer tweet are just 2 of the reason why Steam reviews are a bitch.
The thing that drives me crazy about Steam reviews in general is that people tend to use them to shit on the developer to get a feature added or bug fixed. I can't count how many times I read "I'll thumb up if the dev adds {fill in stupid feature here}".
NKD wrote on Nov 21, 2017, 13:36:ItBurn wrote on Nov 21, 2017, 13:04:
Statistically, the thumbs up would rate a 10/10 game and a 6/10 game the same, this is bad.
No, because the 10/10 game is going to have a higher proportion of "thumbs up" than the 6/10 game. This system works extremely well in a system where you have sample sizes in the thousands, as opposed to your example of movie reviews. The only "accuracy" you gain is a finer grain view of each individual score for the game. But when dealing with so many reviews, the individual scores don't matter, just the overall proportions. You can still view the text of the reviews if you need more detail.
ItBurn wrote on Nov 21, 2017, 13:04:
Statistically, the thumbs up would rate a 10/10 game and a 6/10 game the same, this is bad.
saluk wrote on Nov 21, 2017, 12:59:ItBurn wrote on Nov 21, 2017, 12:41:Quboid wrote on Nov 21, 2017, 12:24:Cutter wrote on Nov 21, 2017, 11:21:
I wish they'd get rid of the stupid thumbs up/dowm BS and use a normal out of 10 sytem. We live in a base 10 world after all. And it's still the best way to get an overall feel for reception of the game.
Reviewers can use whatever rating system they want in their reviews. Using out-of-10 ratings for the actual score would mess up the aggregate score, which is by far the most important aspect of the entire review feature. With such a system, any score other than 0/10 or 10/10 is just telling the algorithm that your opinion is less important than the one-eyed fanboy who can only see the extremes.
Here's 3 hypothetical reviews, with numeric/thumb ratings:
1) Decent game, no masterpiece but fans of the genre should check it out. 6/10, thumbs up.
2) Very good game, they really listened to complaints about the previous title. 8/10, thumbs up.
3) LAME! MEDAL OF BATTLEDUTY IS FARR BETTErrr! 0/10, thumbs down.
Scores:
Numeric: 4.67/10, poor.
Thumbs: 66.67%, good.
The aggregate thumb rating is much more representative of the actual opinions expressed.
I absolutely disagree. I hate the thumbs up system and I can't stand the fact that the best game ever and a barely passable, generic, lowest common denominator one both get a positive rating.
Sure, there's abuse in the base 10 system, but it doesn't matter, it still gives better results.
edit: with that said, I'm not saying that the thumbs up system doesn't work, just that I prefer base 10.
You failed to comment on the point Quboid made. Thumb up/down gives much better statistical rating when there are many reviews, and since for the most part you don't know the bias of the people writing the reviews, the statistical value is more relevant to users than each individual review.
However, what they could do, is let reviewers give an x out of 10, and then statistically count things up anyway they choose, instead of JUST averaging the ratings.
ItBurn wrote on Nov 21, 2017, 12:41:Quboid wrote on Nov 21, 2017, 12:24:Cutter wrote on Nov 21, 2017, 11:21:
I wish they'd get rid of the stupid thumbs up/dowm BS and use a normal out of 10 sytem. We live in a base 10 world after all. And it's still the best way to get an overall feel for reception of the game.
Reviewers can use whatever rating system they want in their reviews. Using out-of-10 ratings for the actual score would mess up the aggregate score, which is by far the most important aspect of the entire review feature. With such a system, any score other than 0/10 or 10/10 is just telling the algorithm that your opinion is less important than the one-eyed fanboy who can only see the extremes.
Here's 3 hypothetical reviews, with numeric/thumb ratings:
1) Decent game, no masterpiece but fans of the genre should check it out. 6/10, thumbs up.
2) Very good game, they really listened to complaints about the previous title. 8/10, thumbs up.
3) LAME! MEDAL OF BATTLEDUTY IS FARR BETTErrr! 0/10, thumbs down.
Scores:
Numeric: 4.67/10, poor.
Thumbs: 66.67%, good.
The aggregate thumb rating is much more representative of the actual opinions expressed.
I absolutely disagree. I hate the thumbs up system and I can't stand the fact that the best game ever and a barely passable, generic, lowest common denominator one both get a positive rating.
Sure, there's abuse in the base 10 system, but it doesn't matter, it still gives better results.
edit: with that said, I'm not saying that the thumbs up system doesn't work, just that I prefer base 10.
Quboid wrote on Nov 21, 2017, 12:24:Cutter wrote on Nov 21, 2017, 11:21:
I wish they'd get rid of the stupid thumbs up/dowm BS and use a normal out of 10 sytem. We live in a base 10 world after all. And it's still the best way to get an overall feel for reception of the game.
Reviewers can use whatever rating system they want in their reviews. Using out-of-10 ratings for the actual score would mess up the aggregate score, which is by far the most important aspect of the entire review feature. With such a system, any score other than 0/10 or 10/10 is just telling the algorithm that your opinion is less important than the one-eyed fanboy who can only see the extremes.
Here's 3 hypothetical reviews, with numeric/thumb ratings:
1) Decent game, no masterpiece but fans of the genre should check it out. 6/10, thumbs up.
2) Very good game, they really listened to complaints about the previous title. 8/10, thumbs up.
3) LAME! MEDAL OF BATTLEDUTY IS FARR BETTErrr! 0/10, thumbs down.
Scores:
Numeric: 4.67/10, poor.
Thumbs: 66.67%, good.
The aggregate thumb rating is much more representative of the actual opinions expressed.
Cutter wrote on Nov 21, 2017, 11:21:
I wish they'd get rid of the stupid thumbs up/dowm BS and use a normal out of 10 sytem. We live in a base 10 world after all. And it's still the best way to get an overall feel for reception of the game.