Valve Reviewing Review Reviews

A new post on the Steam Blog has further thoughts from Valve on the process of refining the user review process on Steam to make them more useful to other users. They offer an extensive discussion of what makes for helpful reviews, and how they evaluate reviews for this quality. They note that one of the problems they must deal with is that the system for reviewing reviews itself is as subject to abuse as the review system itself. Here's word on what they are doing about this:
What we are changing
Up to this point, we've tried to maintain just showing the simple math behind how we calculate whether a review is helpful or not--the percentage and number of people who indicated a review as helpful. We like systems that are transparent and easy to understand, as they are also easier to believe and trust. Unfortunately, this has resulted in a system that allows a small group to manipulate reviews to a degree that is clearly decreasing the value of Steam for many other players. So we're making two main changes.

  1. Firstly, our system will use a new method of calculating the helpfulness of each review, taking into account the users that are trying to manipulate the system. One way we're doing that is by counting the helpful ratings on reviews differently for users that are far outside the norm. Ratings from users that follow normal patterns of rating will continue to be counted the same way that they have, whereas accounts that rate an excessive number of reviews on an individual game will see the weight of each individual rating count for less and less.
  2. Secondly, store pages will now show the default helpful positive and negative reviews in a similar proportion to that of the overall review score for the game. For example, if the game is reviewed positively by 80% of reviewers, then the ten reviews shown by default on the store page will be 80% positive, showing eight positive and two negative. This should keep the reviews shown on a game's page from being so easily manipulated by a few determined players and should more accurately represent the overall sentiment of the people playing the game.

We're rolling out both of these changes as a beta today. You can turn the new method on and off to see how it impacts the default display of reviews on any given store page. Note that these changes only impact the default listing of reviews (Called "Summary") and the "Most Helpful" display option.

View : : :
23 Replies. 2 pages. Viewing page 1.
Newer [  1  2  ] Older
23.
 
Re: Valve Reviewing Review Reviews
Nov 22, 2017, 07:36
Bub
23.
Re: Valve Reviewing Review Reviews Nov 22, 2017, 07:36
Nov 22, 2017, 07:36
Bub
 
Ha Ha Ha, As If those Steam reviews were a good indicator of game quality and fun or how so many Steam titles won't install, or run, or run in various screen resolutions.

To get the real skinny on any Steam game on sale:

- always filter the Steam reviews for the negative ones first. Look for repeated mention of technical issues, and core gameplay criticisms, then go check the positive reviews.

- check the community discussion board, look for any of these bad signs: game won't run/install in Win10, game won't run in certain resolutions, major in-game bugs.

- go to Youtube, search the game title followed by "gameplay". Actual game play is always better to judge a game by than reviews which tend to be dominated by people with other agendas.
==================================================
Bubb Stubbley
... I miss BBS..
"There is a sucker born every minute." - PT Barnum
==================================================
Avatar 58208
22.
 
Re: Valve Reviewing Review Reviews
Nov 22, 2017, 07:34
22.
Re: Valve Reviewing Review Reviews Nov 22, 2017, 07:34
Nov 22, 2017, 07:34
 
ItBurn wrote on Nov 22, 2017, 03:00:
jdreyer wrote on Nov 22, 2017, 02:04:
As for Ragnarok, it's great. I don't know if it's my favorite Marvel film. I love the first Cap movie, the first Iron Man, the first Avengers, and the first GotG, too.

For me it's Ragnarock, Deadpool and GotG1. The others are so far away that I won't even mention them.
^these +1
GotG2 was a huge disappoint, here's to hoping Deadpool 2 can keep some of the magic that made the first movie so special.
Avatar 58135
21.
 
Re: Valve Reviewing Review Reviews
Nov 22, 2017, 03:12
21.
Re: Valve Reviewing Review Reviews Nov 22, 2017, 03:12
Nov 22, 2017, 03:12
 
ItBurn wrote on Nov 22, 2017, 03:00:
jdreyer wrote on Nov 22, 2017, 02:04:
As for Ragnarok, it's great. I don't know if it's my favorite Marvel film. I love the first Cap movie, the first Iron Man, the first Avengers, and the first GotG, too.

For me it's Ragnarock, Deadpool and GotG1. The others are so far away that I won't even mention them.

That's right, Deadpool is frickin' awesome. Forgot that's a Marvel movie.
If Russia stops fighting, the war ends. If Ukraine stops fighting, Ukraine ends. Slava Ukraini!
Avatar 22024
20.
 
Re: Valve Reviewing Review Reviews
Nov 22, 2017, 03:00
20.
Re: Valve Reviewing Review Reviews Nov 22, 2017, 03:00
Nov 22, 2017, 03:00
 
jdreyer wrote on Nov 22, 2017, 02:04:
As for Ragnarok, it's great. I don't know if it's my favorite Marvel film. I love the first Cap movie, the first Iron Man, the first Avengers, and the first GotG, too.

For me it's Ragnarock, Deadpool and GotG1. The others are so far away that I won't even mention them.
Avatar 56211
19.
 
Re: Valve Reviewing Review Reviews
Nov 22, 2017, 02:04
19.
Re: Valve Reviewing Review Reviews Nov 22, 2017, 02:04
Nov 22, 2017, 02:04
 
ItBurn wrote on Nov 21, 2017, 16:57:
jdreyer wrote on Nov 21, 2017, 16:33:
ItBurn wrote on Nov 21, 2017, 15:55:
jdreyer wrote on Nov 21, 2017, 15:16:
Rotten Tomatoes offers both: a fresh/rotten meta calculation of like/didn't like as well as a 10 point scale average. It includes these for both reviewers and audience members.

That's a system that pleases everyone it would seem.

I was pretty sure that the rotten tomatoes point system was based on thumbs up and not a 10 point system, just like Steam. Anyone can confirm?

Although Dacote pointed you to the RT website, it's better to look at a specific movie like Thor Ragnarok to see the scale. 92% of reviewers liked the movie, giving it an average 10 point rating of 7.5. 88% of audience members liked it, with a 4.3 out of 5 rating.

I don't understand your numbers, so I don't know if you're with me or against me :p But Thor Ragnarock isn't a good barometer of the system because it was too good of a movie. You need to get a really mediocre one. There's tons of examples in the Marvel library...

Side note, Thor Ragnarock is my favorite Marvel movie ever. The others are so... mediocre. Some are better than others, but they place too much emphasis on the hero carrying the movie. Everything else is meh, and since I'm not really a superhero guy, they rarely do anything for me. Ragnarock is just a good movie overall.

I think that both systems have merit and measure different things, so including both is a good policy. I'll often look at a movies fresh/rotten score, and if a number is higher or lower than I thought, I'll look at the numeric score for perspective. Then maybe I'll go read some reviews. Even for Ragnarok, the numeric scores are lower than the meta score, lending credence to Cutter's theory.

As for Ragnarok, it's great. I don't know if it's my favorite Marvel film. I love the first Cap movie, the first Iron Man, the first Avengers, and the first GotG, too.
If Russia stops fighting, the war ends. If Ukraine stops fighting, Ukraine ends. Slava Ukraini!
Avatar 22024
18.
 
Re: Valve Reviewing Review Reviews
Nov 21, 2017, 18:42
Quboid
 
18.
Re: Valve Reviewing Review Reviews Nov 21, 2017, 18:42
Nov 21, 2017, 18:42
 Quboid
 
Cutter wrote on Nov 21, 2017, 15:28:
ItBurn wrote on Nov 21, 2017, 13:04:
Statistically, the thumbs up would rate a 10/10 game and a 6/10 game the same, this is bad. A 10/10 system is all over the place and people have bias, but it doesn't matter. The aggregate of tons of reviews will be closer to the real rating than a thumbs up system would. I mean, have you seen critic movie reviews on rotten tomatoes? The most mediocre films have 80s and 90s. If you look at games on metacritic, where most of critic reviews are on a base 10 system, you get MUCH more accurate results.

My point exactly. There are a ton of games I would rate a 4/5/6 out of 10 which would give you a much better idea of whether to take a chance on buying it or not. Whereas the current system says - to me at any rate - either buy or don't buy, no middle ground.

The present system gives a score out of 100. If you want more detail from a single review, read the review. As in, the actual text. Also, reviewers can put whatever sort of score they want there.
Avatar 10439
17.
 
Re: Valve Reviewing Review Reviews
Nov 21, 2017, 18:01
17.
Re: Valve Reviewing Review Reviews Nov 21, 2017, 18:01
Nov 21, 2017, 18:01
 
jdreyer wrote on Nov 21, 2017, 15:16:
Rotten Tomatoes offers both: a fresh/rotten meta calculation of like/didn't like as well as a 10 point scale average. It includes these for both reviewers and audience members.

That's a system that pleases everyone it would seem.
Similar to Metacritic, just with less vegetable (or is a tomato a fruit?).
Take Justice League for example;
Metascore 46
User Score 7.1
http://www.metacritic.com/about-metascores

And there is always opencritic for games, it looks like it may be less splooged by the troll-brigade.


Avatar 58135
16.
 
Re: Valve Reviewing Review Reviews
Nov 21, 2017, 16:57
16.
Re: Valve Reviewing Review Reviews Nov 21, 2017, 16:57
Nov 21, 2017, 16:57
 
jdreyer wrote on Nov 21, 2017, 16:33:
ItBurn wrote on Nov 21, 2017, 15:55:
jdreyer wrote on Nov 21, 2017, 15:16:
Rotten Tomatoes offers both: a fresh/rotten meta calculation of like/didn't like as well as a 10 point scale average. It includes these for both reviewers and audience members.

That's a system that pleases everyone it would seem.

I was pretty sure that the rotten tomatoes point system was based on thumbs up and not a 10 point system, just like Steam. Anyone can confirm?

Although Dacote pointed you to the RT website, it's better to look at a specific movie like Thor Ragnarok to see the scale. 92% of reviewers liked the movie, giving it an average 10 point rating of 7.5. 88% of audience members liked it, with a 4.3 out of 5 rating.

I don't understand your numbers, so I don't know if you're with me or against me :p But Thor Ragnarock isn't a good barometer of the system because it was too good of a movie. You need to get a really mediocre one. There's tons of examples in the Marvel library...

Side note, Thor Ragnarock is my favorite Marvel movie ever. The others are so... mediocre. Some are better than others, but they place too much emphasis on the hero carrying the movie. Everything else is meh, and since I'm not really a superhero guy, they rarely do anything for me. Ragnarock is just a good movie overall.
Avatar 56211
15.
 
Re: Valve Reviewing Review Reviews
Nov 21, 2017, 16:33
15.
Re: Valve Reviewing Review Reviews Nov 21, 2017, 16:33
Nov 21, 2017, 16:33
 
ItBurn wrote on Nov 21, 2017, 15:55:
jdreyer wrote on Nov 21, 2017, 15:16:
Rotten Tomatoes offers both: a fresh/rotten meta calculation of like/didn't like as well as a 10 point scale average. It includes these for both reviewers and audience members.

That's a system that pleases everyone it would seem.

I was pretty sure that the rotten tomatoes point system was based on thumbs up and not a 10 point system, just like Steam. Anyone can confirm?

Although Dacote pointed you to the RT website, it's better to look at a specific movie like Thor Ragnarok to see the scale. 92% of reviewers liked the movie, giving it an average 10 point rating of 7.5. 88% of audience members liked it, with a 4.3 out of 5 rating.
If Russia stops fighting, the war ends. If Ukraine stops fighting, Ukraine ends. Slava Ukraini!
Avatar 22024
14.
 
Re: Valve Reviewing Review Reviews
Nov 21, 2017, 16:08
14.
Re: Valve Reviewing Review Reviews Nov 21, 2017, 16:08
Nov 21, 2017, 16:08
 
ItBurn wrote on Nov 21, 2017, 15:55:
jdreyer wrote on Nov 21, 2017, 15:16:
Rotten Tomatoes offers both: a fresh/rotten meta calculation of like/didn't like as well as a 10 point scale average. It includes these for both reviewers and audience members.

That's a system that pleases everyone it would seem.

I was pretty sure that the rotten tomatoes point system was based on thumbs up and not a 10 point system, just like Steam. Anyone can confirm?
https://www.rottentomatoes.com
13.
 
Re: Valve Reviewing Review Reviews
Nov 21, 2017, 15:55
13.
Re: Valve Reviewing Review Reviews Nov 21, 2017, 15:55
Nov 21, 2017, 15:55
 
jdreyer wrote on Nov 21, 2017, 15:16:
Rotten Tomatoes offers both: a fresh/rotten meta calculation of like/didn't like as well as a 10 point scale average. It includes these for both reviewers and audience members.

That's a system that pleases everyone it would seem.

I was pretty sure that the rotten tomatoes point system was based on thumbs up and not a 10 point system, just like Steam. Anyone can confirm?
Avatar 56211
12.
 
Re: Valve Reviewing Review Reviews
Nov 21, 2017, 15:28
12.
Re: Valve Reviewing Review Reviews Nov 21, 2017, 15:28
Nov 21, 2017, 15:28
 
ItBurn wrote on Nov 21, 2017, 13:04:
Statistically, the thumbs up would rate a 10/10 game and a 6/10 game the same, this is bad. A 10/10 system is all over the place and people have bias, but it doesn't matter. The aggregate of tons of reviews will be closer to the real rating than a thumbs up system would. I mean, have you seen critic movie reviews on rotten tomatoes? The most mediocre films have 80s and 90s. If you look at games on metacritic, where most of critic reviews are on a base 10 system, you get MUCH more accurate results.

My point exactly. There are a ton of games I would rate a 4/5/6 out of 10 which would give you a much better idea of whether to take a chance on buying it or not. Whereas the current system says - to me at any rate - either buy or don't buy, no middle ground.
"Van Gogh painted alone and in despair and in madness and sold one picture in his entire life. Millions struggled alone, unrecognized, and struggled as heroically as any famous hero. Was it worthless? I knew it wasn't."
11.
 
Re: Valve Reviewing Review Reviews
Nov 21, 2017, 15:16
11.
Re: Valve Reviewing Review Reviews Nov 21, 2017, 15:16
Nov 21, 2017, 15:16
 
Rotten Tomatoes offers both: a fresh/rotten meta calculation of like/didn't like as well as a 10 point scale average. It includes these for both reviewers and audience members.

That's a system that pleases everyone it would seem.
If Russia stops fighting, the war ends. If Ukraine stops fighting, Ukraine ends. Slava Ukraini!
Avatar 22024
10.
 
Re: Valve Reviewing Review Reviews
Nov 21, 2017, 13:59
10.
Re: Valve Reviewing Review Reviews Nov 21, 2017, 13:59
Nov 21, 2017, 13:59
 
BIGtrouble77 wrote on Nov 21, 2017, 11:48:
The thing that drives me crazy about Steam reviews in general is that people tend to use them to shit on the developer to get a feature added or bug fixed. I can't count how many times I read "I'll thumb up if the dev adds {fill in stupid feature here}".
The add a feature puleeze along with the troll-brigade upset about a developer tweet are just 2 of the reason why Steam reviews are a bitch.

Not to mention the ridiculous thumbs system. Give me a point scale any-day.
Avatar 58135
9.
 
Re: Valve Reviewing Review Reviews
Nov 21, 2017, 13:46
9.
Re: Valve Reviewing Review Reviews Nov 21, 2017, 13:46
Nov 21, 2017, 13:46
 
NKD wrote on Nov 21, 2017, 13:36:
ItBurn wrote on Nov 21, 2017, 13:04:

Statistically, the thumbs up would rate a 10/10 game and a 6/10 game the same, this is bad.

No, because the 10/10 game is going to have a higher proportion of "thumbs up" than the 6/10 game. This system works extremely well in a system where you have sample sizes in the thousands, as opposed to your example of movie reviews. The only "accuracy" you gain is a finer grain view of each individual score for the game. But when dealing with so many reviews, the individual scores don't matter, just the overall proportions. You can still view the text of the reviews if you need more detail.

I don't know man, in all my years of Steam use, overwhelmingly, when buying a ~90% game, I found out it was more of a ~75% game. I figured out how to know if the game is actually a ~90% one though. First, I check if the game is at least very positive. If so, then it MIGHT be a ~90% game, but there's no way to know, because of the thumbs up system. So, next step is to read reviews, mostly the negative ones, and that will tell me if it's a good game. So yeah, I think that a base 10 system would give me way more precise information upfront. Also, with the review usefulness rating, Steam could improve the base 10 system dramatically.
Avatar 56211
8.
 
Re: Valve Reviewing Review Reviews
Nov 21, 2017, 13:36
NKD
8.
Re: Valve Reviewing Review Reviews Nov 21, 2017, 13:36
Nov 21, 2017, 13:36
NKD
 
ItBurn wrote on Nov 21, 2017, 13:04:

Statistically, the thumbs up would rate a 10/10 game and a 6/10 game the same, this is bad.

No, because the 10/10 game is going to have a higher proportion of "thumbs up" than the 6/10 game. This system works extremely well in a system where you have sample sizes in the thousands, as opposed to your example of movie reviews. The only "accuracy" you gain is a finer grain view of each individual score for the game. But when dealing with so many reviews, the individual scores don't matter, just the overall proportions. You can still view the text of the reviews if you need more detail.
Do you have a single fact to back that up?
Avatar 43041
7.
 
Re: Valve Reviewing Review Reviews
Nov 21, 2017, 13:04
7.
Re: Valve Reviewing Review Reviews Nov 21, 2017, 13:04
Nov 21, 2017, 13:04
 
saluk wrote on Nov 21, 2017, 12:59:
ItBurn wrote on Nov 21, 2017, 12:41:
Quboid wrote on Nov 21, 2017, 12:24:
Cutter wrote on Nov 21, 2017, 11:21:
I wish they'd get rid of the stupid thumbs up/dowm BS and use a normal out of 10 sytem. We live in a base 10 world after all. And it's still the best way to get an overall feel for reception of the game.

Reviewers can use whatever rating system they want in their reviews. Using out-of-10 ratings for the actual score would mess up the aggregate score, which is by far the most important aspect of the entire review feature. With such a system, any score other than 0/10 or 10/10 is just telling the algorithm that your opinion is less important than the one-eyed fanboy who can only see the extremes.

Here's 3 hypothetical reviews, with numeric/thumb ratings:

1) Decent game, no masterpiece but fans of the genre should check it out. 6/10, thumbs up.
2) Very good game, they really listened to complaints about the previous title. 8/10, thumbs up.
3) LAME! MEDAL OF BATTLEDUTY IS FARR BETTErrr! 0/10, thumbs down.

Scores:
Numeric: 4.67/10, poor.
Thumbs: 66.67%, good.

The aggregate thumb rating is much more representative of the actual opinions expressed.

I absolutely disagree. I hate the thumbs up system and I can't stand the fact that the best game ever and a barely passable, generic, lowest common denominator one both get a positive rating.
Sure, there's abuse in the base 10 system, but it doesn't matter, it still gives better results.

edit: with that said, I'm not saying that the thumbs up system doesn't work, just that I prefer base 10.

You failed to comment on the point Quboid made. Thumb up/down gives much better statistical rating when there are many reviews, and since for the most part you don't know the bias of the people writing the reviews, the statistical value is more relevant to users than each individual review.

However, what they could do, is let reviewers give an x out of 10, and then statistically count things up anyway they choose, instead of JUST averaging the ratings.

Statistically, the thumbs up would rate a 10/10 game and a 6/10 game the same, this is bad. A 10/10 system is all over the place and people have bias, but it doesn't matter. The aggregate of tons of reviews will be closer to the real rating than a thumbs up system would. I mean, have you seen critic movie reviews on rotten tomatoes? The most mediocre films have 80s and 90s. If you look at games on metacritic, where most of critic reviews are on a base 10 system, you get MUCH more accurate results.
Avatar 56211
6.
 
Re: Valve Reviewing Review Reviews
Nov 21, 2017, 12:59
6.
Re: Valve Reviewing Review Reviews Nov 21, 2017, 12:59
Nov 21, 2017, 12:59
 
ItBurn wrote on Nov 21, 2017, 12:41:
Quboid wrote on Nov 21, 2017, 12:24:
Cutter wrote on Nov 21, 2017, 11:21:
I wish they'd get rid of the stupid thumbs up/dowm BS and use a normal out of 10 sytem. We live in a base 10 world after all. And it's still the best way to get an overall feel for reception of the game.

Reviewers can use whatever rating system they want in their reviews. Using out-of-10 ratings for the actual score would mess up the aggregate score, which is by far the most important aspect of the entire review feature. With such a system, any score other than 0/10 or 10/10 is just telling the algorithm that your opinion is less important than the one-eyed fanboy who can only see the extremes.

Here's 3 hypothetical reviews, with numeric/thumb ratings:

1) Decent game, no masterpiece but fans of the genre should check it out. 6/10, thumbs up.
2) Very good game, they really listened to complaints about the previous title. 8/10, thumbs up.
3) LAME! MEDAL OF BATTLEDUTY IS FARR BETTErrr! 0/10, thumbs down.

Scores:
Numeric: 4.67/10, poor.
Thumbs: 66.67%, good.

The aggregate thumb rating is much more representative of the actual opinions expressed.

I absolutely disagree. I hate the thumbs up system and I can't stand the fact that the best game ever and a barely passable, generic, lowest common denominator one both get a positive rating.
Sure, there's abuse in the base 10 system, but it doesn't matter, it still gives better results.

edit: with that said, I'm not saying that the thumbs up system doesn't work, just that I prefer base 10.

You failed to comment on the point Quboid made. Thumb up/down gives much better statistical rating when there are many reviews, and since for the most part you don't know the bias of the people writing the reviews, the statistical value is more relevant to users than each individual review.

However, what they could do, is let reviewers give an x out of 10, and then statistically count things up anyway they choose, instead of JUST averaging the ratings.
5.
 
Re: Valve Reviewing Review Reviews
Nov 21, 2017, 12:41
5.
Re: Valve Reviewing Review Reviews Nov 21, 2017, 12:41
Nov 21, 2017, 12:41
 
Quboid wrote on Nov 21, 2017, 12:24:
Cutter wrote on Nov 21, 2017, 11:21:
I wish they'd get rid of the stupid thumbs up/dowm BS and use a normal out of 10 sytem. We live in a base 10 world after all. And it's still the best way to get an overall feel for reception of the game.

Reviewers can use whatever rating system they want in their reviews. Using out-of-10 ratings for the actual score would mess up the aggregate score, which is by far the most important aspect of the entire review feature. With such a system, any score other than 0/10 or 10/10 is just telling the algorithm that your opinion is less important than the one-eyed fanboy who can only see the extremes.

Here's 3 hypothetical reviews, with numeric/thumb ratings:

1) Decent game, no masterpiece but fans of the genre should check it out. 6/10, thumbs up.
2) Very good game, they really listened to complaints about the previous title. 8/10, thumbs up.
3) LAME! MEDAL OF BATTLEDUTY IS FARR BETTErrr! 0/10, thumbs down.

Scores:
Numeric: 4.67/10, poor.
Thumbs: 66.67%, good.

The aggregate thumb rating is much more representative of the actual opinions expressed.

I absolutely disagree. I hate the thumbs up system and I can't stand the fact that the best game ever and a barely passable, generic, lowest common denominator one both get a positive rating.
Sure, there's abuse in the base 10 system, but it doesn't matter, it still gives better results.

edit: with that said, I'm not saying that the thumbs up system doesn't work, just that I prefer base 10.
Avatar 56211
4.
 
Re: Valve Reviewing Review Reviews
Nov 21, 2017, 12:24
Quboid
 
4.
Re: Valve Reviewing Review Reviews Nov 21, 2017, 12:24
Nov 21, 2017, 12:24
 Quboid
 
Cutter wrote on Nov 21, 2017, 11:21:
I wish they'd get rid of the stupid thumbs up/dowm BS and use a normal out of 10 sytem. We live in a base 10 world after all. And it's still the best way to get an overall feel for reception of the game.

Reviewers can use whatever rating system they want in their reviews. Using out-of-10 ratings for the actual score would mess up the aggregate score, which is by far the most important aspect of the entire review feature. With such a system, any score other than 0/10 or 10/10 is just telling the algorithm that your opinion is less important than the one-eyed fanboy who can only see the extremes.

Here's 3 hypothetical reviews, with numeric/thumb ratings:

1) Decent game, no masterpiece but fans of the genre should check it out. 6/10, thumbs up.
2) Very good game, they really listened to complaints about the previous title. 8/10, thumbs up.
3) LAME! MEDAL OF BATTLEDUTY IS FARR BETTErrr! 0/10, thumbs down.

Scores:
Numeric: 4.67/10, poor.
Thumbs: 66.67%, good.

The aggregate thumb rating is much more representative of the actual opinions expressed.
Avatar 10439
23 Replies. 2 pages. Viewing page 1.
Newer [  1  2  ] Older