How Much Can You Trust Beer Ratings?

Ryan Langrill, August 01, 2013

Not all reviews is created equal. This is true for single reviews and aggregate ratings. Nathan Gismot touched on a couple sources of bias last week, and this week I am going to discuss other forms of bias and how they should influence your use of beer reviews.

Individual reviews face two types of bias: Cognitive and motivational, while review aggregators, including BAR, face a third bias: selection bias.

Motivational Biases

Motivational biases enter into beer reviews when the reviewer rates the beer based on something other than the quality of the beer, such as in Nathan’s example of the recently dumped Pliny drinker who writes angsty beer reviews instead of poetry. When people talk about a biased news program, for instance, they’re usually talking about motivational biases. If your friend just returned from an amazing trip to Munich, he’ll probably report that every Munich beer tasted like the nectar of the gods, just like <insert news organization> will likely describe any policy put forth by <insert party> as if it was inspired by the gods. In neither case can you take the opinion at face value. 

One benefit of aggregation is that it strips motivational biases from rankings, as long as biases are evenly distributed across beers; that is, if people are no more likely to drink one beer in a bad mood than any other beer.  This might not always hold, though: if reviewers of a certain beer are systematically positively or negatively emotionally influenced, the aggregate will be biased. When people quest after a beer, the mere attainment of which is an accomplishment to be celebrated, people will tend to overrate the beer. A beer reviewed in a bad mood—say, a beer commonly served at the stadium of a disappointing baseball team—will be underrated, better than its reviews seem. People might drink expensive beers more likely as a celebration, and cheap beers only if money is tight, which will distort the picture by making expensive beers seem like they are better than they are—the rating difference will represent some of the effect of the mood instead of just the difference in beer quality. 

Cognitive Biases 

Even when emotions do not sway our evaluations of a beer, we are not perfectly objective beer review machines. When we think, we do not take into account all available information. Instead, we use heuristics, rules of thumb that allow us to solve complex problems with much less computational power and information.

An example, often used by psychologist Daniel Kahnemann, concerns the way that people catch baseballs. The quickest route between two points is a straight line; thus, a rational baseball player ought to calculate the trajectory of the ball and run straight to where he calculates that the ball is going to land. Baseball players do not have infinitely capable minds and rely instead on a much simpler way of judging the location of the ball: he fixes his gaze on the ball, runs forward if the ball drops in his gaze, and backwards if it rises. Using this heuristic introduces some biases from the hypothetical optimum. Players run in slight arcs, not straight lines and they often catch balls while running instead of finding the point of landing and stopping there. 

The list of cognitive biases is enormous, but a few in particular matter to beer reviewers. Confirmation bias happens because we look for information we expect to be there. If we have a negative impression of a beer, we are likely to find aspects of the beer we do not like, and vice versa. More subtle than that is the expectations that we bring to a beer: if we associate a creamed corn taste with American Adjunct Lagers, we’re more likely to find that taste even if it isn’t all that strong than we would in a blind taste test. A study at the University of Bordeaux in 2001 showed that wine experts, given white wines that were dyed red, described the wines as red wines—confirmation bias trapped even trained reviewers, not one realized he was drinking white wine; they tasted what they expected to taste. Beer ratings, too, will tend to reinforce the existing impression of a beer.

With BAR in particular, the status quo bias presents a problem. With the status quo bias, we believe that the existing state of the world is better than it actually is. If people get into a habit drinking and reviewing certain beers, the status quo bias will push the rating of any ‘novel’ beers down. Newly released beers do not have anyone who has acclimated to them, yet, and will be systematically rated less often, and enjoyed less, than the exact same beer would be in a familiar bottle. 

There are all sorts of explanations for the source of these biases, evolutionary or otherwise. For the most part, psychologists agree that the heuristics that lead to these biases are quite useful. But knowing about our biases can help us come to more objective conclusions (as long as we don't succumb to bias blind spot, where we believe that we are less biased than other people!), and knowing the way that reviewers are biased can help us learn more from people's reviews.

Selection Biases

Selection biases can cause even more problems for review aggregators. While I may be stating the obvious, it is important to note that there are two ways for a beer to get better ratings: 1) being reviewed more highly, or 2) having a biased sample of people rate the beer. Ideally, a beer's rating would change only with the quality of beer. In reality, there are several factors that will affect the type of person who rates the beer.

One factor is price. No one goes to buy beer knowing exactly how he or she will like each beer; the lower the price, the lower the risk involved trying a new brew. People will be more willing to try, and dislike, a cheap beer than a pricy one--the pricy beers will likely sell to people who really love the style or are otherwise predisposed to liking it. On average, a cheaper beer will invite more people to try it who end up disliking it and giving it a poor rating. 

Imagine a local brewery decides to offer two new selections, which happen to be identical in every way but price: a double IPA for $6/6-pack and a double IPA for $10/6-pack. Quite a few people would be willing to try the $6/6-pack beer, even if those people don't particularly like IPAs, but bypass the $10/6-pack for beers that are less risky to them. Perhaps the beer is a great example of its style, but it gets middling reviews because it just does not match the palette of its drinkers very well. The $10/6-pack double IPA is going to attract people who are more certain that they like double IPAs -- and though the beer is identical in taste to the cheaper beer, it gets rave reviews (and this ignores any psychological biases that price points may induce). 

In economics this is called the Alchian-Allen effect. The Alchian-Allen effect states that if there is a fixed cost attached to a class of goods, people will select, on average, higher quality goods. It means that apples that Texans get from Washington are going to be of a higher quality on average than the apples that Washingtonians eat from Washington. The higher the ‘shipping’ cost, the greater the quality variation. When trips from Calcutta to London took the better part of a year, the goods shipped were assured to be quite valuable. These days the cost and time of shipping is small so it can pay to ship lower-quality goods. 

When we look at beer reviews, the ‘shipping cost’ of a review is a person actually drinking and reviewing the beer. The higher the price of the beer, the higher the quality of people who drink it—higher quality meaning the expectation of liking the beer. To try to gauge the selection bias that price induces, we have to compare apples to apples, which means we can’t just ‘adjust’ for price (e.g., run a regression that holds cancels out price). If we did that, we would cancel out quite a bit of signal in our attempt to get rid of some noise—it might just be that more expensive beers are more likely to use higher quality ingredients (in larger quantities). As we accumulate more information on price changes, we might be able to tease the effect out better. At the moment, just remember that an $8 beer with good reviews might not really be all that much better than a similar beer that is only $1 a bottle.


Everything gets even more complicated once the selection biases start affecting people psychologically. A pricy or hard-to-find beer attracts only those who really want to try it, and as a result the beer gets great reviews, which in turn positively biases future drinkers. The perception of a beer, and of a brand, depends a great deal on historical contingency (and good marketing). 

As a beer drinker, you can use these biases to improve your beer drinking experience. Avoid negative information, and be skeptical of expensive beers with reviews only marginally better than their cheaper cousins. This holds not just for beer, but anything reviewed by the general public: expect books in hardcover to have better ratings than their paperback, and expensive restaurants to seem better than cheap restaurants. 

For those seeking capital-T Truth in the science of beer, these biases are some of the hurdles that stand in the way of uncovering the nature of good beer.