: Any good research studies on effectiveness of web rating systems? I'm going to be creating a website that allows users to upload and vote on each others' content (specific kinds of images),

@Shanna517

Define "effectiveness". If by "effective" you mean effective at capturing the community consensus, then they're all effective. If by "effectiveness" you mean the richness of the data returned, then using a point scale is going to be more "effective" than using a binary system. If by "effectiveness" you mean ease implementing a collaborative filtering system, then a binary system is preferable.

This question is too vague and it also depends on the type of rating (data) you are looking for and what is being rated. They each have their place.

The article you cited is concerned with poorly weighted point scale systems. Just because most people who rate a movie tend to be fans of the movie/genre doesn't mean the system is useless. That just means you need to scale/calibrate the rating system appropriately. If the median rating of all movies is 3.5 instead of 2.5, then you need to calibrate the rating system so that a 2.5 star is actually a 3.5 average. Similarly, you need to take the standard deviation into account and scale the star rating appropriately. If you do both of these things, then the precision of the 5 star rating system is very real and very valuable.

Another benefit of the point scale system is that it's not just a community consensus. You can actually see how much an individual likes or dislike a film and the rating distribution for different demographics. With a simple binary system, you lose all of that. You only know that X% of girls like this film, not the degree to which they like it. And even that may not be accurate since people who don't strongly like/dislike a film probably won't vote with such limited options.

With a unary system the data quality is even worse—there's no distinction between a movie that has very few votes because it's not well known versus a movie that has very few votes because it sucks and everyone hates it. A movie which is loved by 50% of the viewers could have the exact same rating as one which is loved by 100% of the viewers, and there'd be no way to tell them apart.

OTOH, there are all kinds of cool things you can only do with a point scale system, such as throwing away votes that are way outside the standard deviation or diluting the votes of users who consistently vote at one extreme or another. You can also make better recommendations based on more detailed individual voting data. E.g. not just group users based on which items they vote on, but also the relative ratings they give to movies: a user who gives the Boondock Saints a 5/10 and Basketball Diaries a 9/10 is going to have very different tastes from a user who gives Boondock Saints a 9/10 and Basketball Diaries 5/10.

So do you want to be able to see the degree to which an individual likes/dislikes a photo, or do you only want to collect data for collaborative filtering? Is the item being rated of enough qualitative nuance and diversity that users will even attempt to rate it properly on a 5-point scale (e.g. movies versus youtube videos). Even the unary system is useful if your primary goal is to maximize user participation or you're just measuring popularity (e.g. holding a vote).

On a side note, another alternative is a relative rating system in which users are shown 2 items simultaneously and pick which they prefer. There's some slightly more complicated math involved to rank items in this way, but it offers the simplicity (for users) of the regular binary system along with quality of data of a point system.

Lastly, there are all kinds of tweaks you can make to voting/rating systems to account for the psychology of voting and encourage/discourage certain behaviors. For example, research has shown that users vote very differently depending on whether a vote is "for keeps" or inconsequential. Similarly, if users have a limited number of votes (which they need to earn), they will vote more wisely. And you can force users to expend the same number of positive votes as negative votes to compensate for the tendency of users not voting at all for items they dislike. Similarly, if giving a movie 5 stars costs 5 voting points, you'll probably see less polarized rating patterns from users. And would users vote more diligently if their votes were broadcasted (e.g. on Facebook)?

Vote Up Vote Down