Tuesday, August 23, 2011

How much can you trust average star ratings?

I recently read an article in the Christian Science Monitor about fabricated reviews/ratings on websites "from Amazon to Trip Advisor to Yelp". Of course, the one I'm most concerned with is Amazon.

I tend to give average star ratings more weight than I probably should when I'm looking at books, CDs, or other media. When I'm using a travel site to plan a trip, I'm far more likely to read individual reviews to determine whether they're reliable or written by an employee or someone with an axe to grind. After all, almost any aspect of a trip--whether it's plane tickets or a couple nights in a hotel--is far more expensive than a book. Also, I usually don't want to read reviews of books until after I've read them because I don't want to be spoiled, so the average star rating is useful from that perspective.

However, I am fully aware that the average star rating system, especially on Amazon, is flawed. How many times have you seen someone giving a book a one-star review because the Amazon Marketplace seller they bought it from didn't ship it quickly enough? Or because the Kindle edition is too expensive? Or for any other reason that has nothing to do with the book's content?

On the other hand, how many times have you seen a review that's suspiciously glowing--either written by the author, one of the author's friends, or possibly someone who's being paid to write positive reviews?

I think average star ratings on sites where you can't purchase a book--or at least where purchasing is a secondary concern--are more reliable. That's why, in the months since I started using Goodreads to track my reading, I've started going there instead of Amazon when I'm deciding whether a book is worth buying.

I've noticed that average star ratings on Goodreads tend to be slightly lower than those on Amazon, so I decided to do some comparisons between Amazon and Goodreads for books I've read recently (though Goodreads provides numeric averages, Amazon does not so I calculated themselves. Therefore, if there's an error in the Amazon numbers, it's my fault):
Alice Bliss (Laura Harrington)
Amazon average star rating: 4.8 (22 ratings)
Goodreads average star rating: 3.97 (207 ratings) 
Faceless Killers (Henning Mankell)
Amazon: 3.85 (90 ratings)
Goodreads: 3.61 (3,990 ratings) 
Georgia Bottoms (Mark Childress)
Amazon: 4.00 (40 ratings)
Goodreads: 3.22 (669 ratings) 
Await Your Reply (Dan Chaon)
Amazon: 3.86 (189 ratings)
Goodreads: 3.58 (4,451 ratings)
As you can see, not only do these books have somewhat lower ratings on Goodreads than on Amazon, there are also far more ratings for each book on Goodreads, which also makes the Goodreads averages more accurate. This could be because Goodreads doesn't require users to write reviews; they can simply choose the star rating for a book without having to put the thought into writing why they did or didn't like a book. It could also be because Goodreads has a broader base of users--people who read, not just people who buy from Amazon (I realize that you can leave reviews for products not purchased on Amazon, but I doubt that happens often). So I'm going to keep using Goodreads when I want a general idea about whether I should buy a book, try to get it from the library, or skip it altogether.

In addition to the discrepancy in average star ratings, I have also noticed that Goodreads reviews tend to be a little more thoughtful and well written than those on Amazon. This could also be because of the broader user base at Goodreads--Await Your Reply, for example, has 1,314 written reviews compared to 189 on Amazon. But I also think (snobbish as this probably sounds) that the average Goodread user probably tends to be more bookish and to write more in general than the average Amazon user. Regardless of the reason, I much prefer reading through Goodreads reviews after I've finished a book to reading through Amazon reviews.

I'll definitely keep the Christian Science Monitor article in mind from now on when I'm looking at online reviews, no matter what I'm buying.

No comments:

Post a Comment