We've done some investigating around here on the influence ABV has on ratings, and the consensus seems to be that within styles, the higher ABV offerings receive better ratings, at least up to the point at which the ABV becomes inappropriate to the style. Expectations matter. However, some styles seem to be influenced by more than just the ABV.
The World Beer Cup lists four styles of barrel-aged beers. Here on BeerGraphs, via Untappd, we have none, instead including barrel aged versions next to their standard counterparts. It would appear that this practice is leading to some truly excellent beers getting shafted (staved?) by our metrics. Essentially, an increasingly common adjunct is defining the style. Imagine if every above average amber ale contained pumpkin, or all of the most sought after hefeweizens were also fruit beers.
Sorted by Style+ (easily my favorite metric on the leaderboards), American Imperial/Double Stouts exhibit a clear bias towards bourbon (or other booze) barrel aged variants. The top 25 are all barrel aged. Twenty-six through fifty feature a single non-barrel-aged beer, Speedway Stout Vietnamese Coffee, at #49. Continuing down that leaderboard, we have to get to pages 9 and 10 before we start to see a majority of non-barrel aged beers, and #250 is sitting at exactly 100 (perfectly average for the style) by Style+. It would appear that in order to make an above average American Imperial Stout, brewers simply must age it in oak previously used to condition hard liquor.
What about Russian Imperial Stouts? First observation: a crapload of overlap between Russian and American Imperial stouts. Most of the beers on page one of the Style+ leaderboard for each style are the same beers. Second observation: the bias in favor of barrel-aged variants is not quite as drastic for RIS as it is for AIS, but it clearly exists here as well. Of the three imperial stout styles available for sorting here, only the Imperial Oatmeal seems relatively immune to the effects of barrel aging on aggregate ratings.
Let's have a look at barleywines. Much like the RIS, the preference isn't as drastic as it is for American Imperial Stouts, but there is a clear preference at the top for barrel aged versions. Also, how in the nether hell does Old Guardian 2004 have enough check-ins to hit 5 BAR and 112 Style+? That's a goddamned decade-old beer. Actually, looking at the RIS and Barleywine leaderboards, it appears that some of the barrel bias is offset by bias in favor of ancient vintages of non-barrelled versions.
But that's the top of the leaderboards. Maybe there are a bunch of below-average for their style barrel aged versions as well and this is all in my head. For American Imperial Stouts, there are several, but the ratio seems to be reversed at the bottom -- instead of barrels dominating as they do at the top, there are occasional barrels floating in a sea of "normal" imperial stouts. The same appears to be true of Russian Imperials. Barleywines? Same story.
So, why is this happening? A few guesses, based on my own misanthropic observations as a former professional participant in craft beer culture. First, barrel-aged variants tend to be rare, single day releases, and anything with that sort of cachet is going to see a rating bump. Miles Liebtag has eloquently discussed the impact of the hoarders and Ahabs on this, our beer, and I strongly suspect that those folks are among the most active in the Untappd database from which all of our lovely numbers flow. Second, barrel aging increases ABV, and within these styles, higher ABV tends to result in higher ratings. Actually, that's only two guesses, and they pretty much cover all the bases. Relative scarcity + high ABV + beer nerd one-upsmanship = top of the Style+ leaderboards.
Now the question becomes how we deal with this, assuming my own preliminary and not quite numerically-sound observations hold weight. I'd originally intended to separate the barrel-aged and standard versions of the various styles and calculate the actual effect on our style ratings, but that query, if possible at all, would be hideously complex, and my database skills are as yet virtually nonexistent.
But I'm reasonably confident that the replacement level of these styles, especially American Imperial Stout, is skewed quite high by the barrel aged variants. Maybe we just accept it and move on, maybe better mathematical minds than mine can use this to adjust the metrics accordingly, maybe we figure out a way to bin barrel aged variants of high gravity styles together into a single style. Hell, maybe figure out a way to filter for barrel aging. I don't know, I'm just the asshole pissing in everyone's Cheerios. What does everyone else think? Is hard wood messing with our perception of beer?
Ben wrote this under the influence of Gogol Bordello, Rush, and Stone IPA. Not necessarily in that order. He says stuff and drinks beer. And is still uncomfortable talking about himself in the third person.