What's Going On With Solid?

Ben Sammis, April 08, 2014

I was goofing off with the leaderboards while eating lunch today, and something very strange caught my eye. Less than two months ago, when the nifty new leaderboard features were introduced, there were nine breweries with a 100% Solid ranking. Today, there are two hundred twenty-one. That seems somewhat unsustainable, in addition to crazier than a toddler on bath salt flavored ice cream.

I promise I've never given my toddler bath salts.

So, what the hell is going on? First, once the filter is set for a BAR > 10, the number of perfect breweries drops back down to five,  which seems much more reasonable. Being a generally paranoid and conspiracy minded sort of fellow, my initial thought was that several someones had found a way to game the ratings. But 212 breweries all over the world simultaneously being a) aware of the BG metrics and b) arriving at the idea of tilting the ratings in their favor is even crazier than the (purely hypothetical, I swear) bath salt infused toddler.

Next, let's take a look at a few of these breweries Untappd pages. First up, Fiddlehead. Two beers on the leaderboards, average BAR of 3.57. But they have 54 unique beers checked in by 3,648 people. Next up, Otley. Four leaderboard beers, 1.24 average BAR, but 43 unique beers checked in by 1,228 individuals.  I'm sensing a pattern here. One more, Heater Allen, with basically the same pattern. A small percentage of their total offerings with check in volume high enough to hit the leaderboards, over a thousand unique individuals checking in, and at the moment a very high Solid% rank.

There are anomalies here, like Small Town. They're apparently getting a big bump by being classified as root beer, and I'm sure there are other similar cases, but I'm not parsing through 221 individual brewery pages just to identify more oddities. I am reasonably confident in concluding that what we're seeing here is an inevitable result of the increase in the overall size of the Untappd user base. So how do we deal with it? We could increase the minimum check in quantity for a beer to appear on the leaderboards, but that's just going to recreate the same problem further down the line as total volume increases. Another solution would be to automatically include all offerings from a brewery once they hit a certain threshold of number of beers appearing on the leaderboards and total number of brewery check ins. Each of these approaches will probably lead to further skewing of the numbers, though.  

Ultimately, the most useful solution is probably to use multiple filters when searching the leaderboards. Simply relying on Solid% is no longer the guarantee of quality it once was, and some combination of Solid%, Style +, and BAR or BAR25, along with the relevant region will likely yield more satisfying results.