[Today’s post is brought to you by Neil Paine, my comrade at Pro-Football-Reference.com and expert on all things Sports-Reference related. You can follow Neil on twitter, @Neil_Paine.]
Most fans like to think of the NFL’s playoff system as being the final word on each team’s season — run the table and you’re the champs, the “best team in football”; lose, and your season means nothing. But what if I told you that the NFL playoffs are getting a lot more random in recent seasons? Would it change your attitude if you knew we were getting closer to the point where every playoff outcome might as well be determined by a coin flip?
David Tyree and Rodney Harrison use their bodies to attempt to depict the normal distribution.
To research this phenomenon, I want to explore two models of predicting playoff games: one powered by as much information as possible, the other completely ruled by randomness. I then want to simulate the last 34 postseasons, and see how much of a predictive edge that information actually gives you. If it’s giving you less of an edge, it means the playoffs are being ruled more by randomness.
First, I grabbed every playoff game since 1978 and looked at the Vegas lines. To convert from a pointspread to a win probability, you have to use Wayne Winston’s assumption that “the probability […] of victory for an NFL team can be well approximated by a normal random variable margin with a mean of the Vegas line and a standard deviation of 13.86.” If the Patriots are favored by 7 over the Ravens, this means you can calculate their odds of winning in Excel via:
p(W) = (1-NORMDIST(0.5,7,13.86,TRUE))+0.5*(NORMDIST(0.5,7,13.86,TRUE)-NORMDIST(-0.5,7,13.86,TRUE)) = 69.3%
This gives us a good prediction — in fact, perhaps the best possible prediction — of the outcome going into the game. So for each playoff, I’m going to say a “Smart” fan picks winners based on these numbers; 69.3% of the time he’ll pick the Patriots, and 30.7% of the time he’ll pick the Ravens. Of course, we also need a control, a fan who picks completely at random, so I’m also going to track a “Dumb” fan who thinks every single game is a coin flip.
I’m going to simulate these decision-making processes for the Smart and Dumb fans in every playoff since 1978, running through each year 1,000 times. How much better at picking do you think the Smart fan will be than the Dumb one?
To be clear, it was Neil who called you the dumb fan. It was Neil!
Well, over the course of the whole sample, the Smart fan averaged a little more than 204 correct picks in 356 games, which is good for a 56.6% rate. The Dumb fan had 178 correct picks, a 50% success rate. In other words, being “Smart” gave you an edge of 6.6% over the fan who picked Aaron Eckhart-style.
But what I really want to know is whether this number has changed over time. The logical comparison I wanted to make was pre- and post-free agency, but it turns out there is practically no difference. From 1978 through 1993, the Smart fan would pick winners at a 56.6% rate (6.8% better than his Dumb counterpart), and from 1995-2011, he picks at a 56.3% clip (6.2% better than the Dumb fan). That observed difference, less than a half a percentage point, can be chalked up completely to random variation, so there’s no evidence that the playoffs have been more or less random in the salary cap era.
However, if you compare pre-2005 to post-2005, you see a major difference that cannot be explained away by chance alone. From 2005-2011, the Smart fan would have picked only 53.2% of playoff games correctly; that’s a difference of 3.2 percent from 2005-11, vs. 6.6 percent over the course of the full sample!
Let me restate this finding: the difference between an intelligent prediction of NFL playoff games and a pure coinflip has been sliced in half in the last seven postseasons. In other words, the playoffs are more random now than they’ve ever been in the last 35 years, something we’ve all seen anecdotally with the 2005 Steelers, both Giants championships (especially last year, when they were actually outscored during the regular season), and the 2008 Cardinals’ unexpected SB run, among others.
So does this change how you feel about the playoffs? Do you still think the “best team” is synonymous with the Super Bowl Champion, or do you think it’s more of a crapshoot than ever before?