A different approach to the Pitchfork score

Iyas Daghlas

Figures and analyses generated using either the R statistical programming language or Excel

Summary

1. The 2010-2014 distribution of Pitchfork’s review scores is left-skewed
2. The average (median) Pitchfork score from 2010-2014 is a 7.3
3. A table of percentiles allows for a more meaningful interpretation of Pitchfork album reviews

Pitchfork recently gave the new Magical Cloudz EP Wait & See a score of 7.5. This number means different things to different people- I would be intrigued to give the album a listen, but a more time-strapped reader might eschew it in favor of albums scoring higher than an 8.0. Looking at the rating in this sense means you are using the “absolute score” in reference to the rating scale; a 5.3 is two points lower than 7.3, a 7.0 is three tenths of a point lower than 9.0, etc. Giving the score meaning is difficult. What is the meaning of a 7.5? What is the meaning of a 7.6, and how does that differ from the meaning of a 7.5?

I think a clearer way of thinking about album scores is in terms of percentiles. A 50th percentile score tells you that the album scored better than 50% of all other albums reviewed by Pitchfork. That last bit was key- this scale is only in reference to Pitchfork’s taste and rating tendencies. The number would differ depending upon the website you are using; Stereogum, Rolling Stone, etc. would each have different percentiles for the same absolute score. It is important to visualize Pitchfork’s album review scores to get a sense for their tendencies. I had access to data from 2000-mid 2015, but I chose to focus on the five year period encompassing 2010-2014 because it would be more generalizable to the present than data generated much earlier. Below is a histogram I generated from a database containing all Pitchfork reviews from 2010-2014 that plots the frequency of an album score against the album score. It is interesting to note that the “middle 50%” (interquartile range) of the data is between the scores of 6.5 and 7.8. The 50th percentile is 7.3. Unlike the textbook normal distribution, the pitchfork review distribution has a left-skew, or a long tail. The short right tail probably reflects the fact that the interquartile range is close to the rating scale maximum, and distant from the rating scale minimum. In other words, there’s a lot less room to the right of the average than to the left of the average. I watched a talk recently that showed the distribution of IMDB TV show ratings, and was really interested to see that the graph looks very similar to the one below. Perhaps this left skewness is a characteristic common to review scales.

The percentile of an album score can be calculated by comparing it to all other scores. The percentile approach allows for the key insight that “not all intervals are created equally.” In other words, the percentile difference between a 0.0 and a 2.0 is different than the difference between a 7.0 and 9.0. This is reflected by the table below, which gives you an idea of where an album’s score lies in relation to other albums. I personally don’t think a team of reviewers can distinguish between a 7.1 and 7.2, and so to use the table I would round the score up or down to the nearest 0.5 interval (7.2->7.0; 7.3->7.5).

Album Review Score	Percentile
0.5	0
1.0	0.1
1.5	0.1
2.0	0.12
2.5	0.3
3.0	0.4
3.5	0.6
4.0	1.4
4.5	2.3
5.0	4.1
5.5	7.0
6.0	14.8
6.5	26.3
7.0	43.6
7.2	50
7.5	66.2
8.0	88.2
8.5	97.4
9.0	99.4
9.5	~100
10.0	100

Check out that nice sigmoid shape

The reader might get the wrong impression that I’m putting too much credence or weight in the score of a pitchfork review. This is not my goal. In fact, my enjoyment of an album is basically independent of its Pitchfork score. I love The Mars Volta and At The Drive In, who have both been skewered by Pitchfork reviews. The presidential debate-style review of ATDI’s Relationship of Command is in fact one of the notorious views in Pitchfork history. Note this was written in 2004, which is during the time of Pitchfork’s “wild west” style of reviewing. I don’t think this type of review would make its way into a review today, hence my inclusion of reviews from 2010-2014.

Rather, my goal is to make Pitchfork review scores more useful by offering a concrete way to interpret them in the context of Pitchfork’s other reviews. The Magical Cloudz EP is no longer a 7.5, but rather a score in the 66th percentile. Does that new information make you re-evaluate your decision to listen or not to listen to the album? I stated earlier that I would be intrigued to give the album a listen based on the absolute score, but that percentile is not nearly as appealing as the absolute score.