Regular readers surely recall my “What are the Odds of That” post from this summer. In that article, I referenced an obscured Jacoby Jones stat: in 2011, he gained three times as many receiving yards against teams at the back end of the alphabet as he did against the teams he faced in the front of the alphabet. Then I asked, “what are the odds of that?”

This is a very good reason why it’s often inappropriate to apply standard significance tests to football statistics. Surely Jones’ splits would pass any standard significance test, signaling that his wild split was in fact “real” even though we know it wasn’t. With a large enough sample, you would expect to have false positives, which isn’t a knock on standard significant testing. If something is statistically significant at the 1% level, that doesn’t mean you shouldn’t expect to see a false positive if you have 100 different samples…
…
Some in the statistical community refer to this as the Wyatt Earp Effect. You’ve undoubtedly heard of Wyatt Earp, who is famous precisely because he survived a large number of duels. What are the odds of that? Well, it depends on your perspective. The odds that one person would survive a large number of duels? Given enough time, it becomes a statistical certainty that someone would do just that. Think back to the famous Warren Buffett debate on the efficient market hypothesis. Suppose that 225 million Americans partake in a single elimination national coin-flipping contest, with one coin flip per day. After 20 days, we would expect 215 people to successfully call their coin flips 20 times out of 20. But that doesn’t mean those 215 people are any better at calling coins than you or I am. The Wyatt Earp Effect, the National Coin Flipping Example, and my Splits Happen post all illustrate the same principle. Asking “what are the odds of that?” is often meaningless in retrospect. If you look at enough things, enough players’ splits, enough 4th quarter comeback opportunities, enough coin flips, or enough roulette wheel spins, you will see some things that seem absurdly unlikely.

In December, I highlighted Matt Schaub’s struggles in night games compared to day games as yet another example. Well now, Ray Rice is the latest protagonist in What are the Odds of That? In case you missed it, Rice fumbled twice in Baltimore’s playoff win over Indianapolis, with the Colts recovering both times. Rice has struggled with fumbles in the playoffs in the past, but he’s always been outstanding during the regular season at holding on to the ball. In 2012, he lost just one fumble — which went harmlessly out of bounds — giving him a clean record for the season. So what’s going on? Here’s what Bill Barnwell wrote earlier this week:

In 1,527 regular-season touches during his career, Rice has just seven fumbles. That’s an average of just over 218 touches per fumble for Rice during the first 16 games of the year. Rice’s two fumbles against the Colts on Sunday brought him to five fumbles amid just 152 [post-season] touches. That’s one fumble every 30.4 touches. Rice has been more than seven times more likely to fumble in the playoffs than he has been during the regular season.

The odds of this happening are astronomical. If we assume that Rice’s “true” fumble rate is his rate during the regular season, the odds that Rice would have five or more fumbles in 152 touches are an incredible 1,364 to 1. (The odds that he would have exactly five are 1,534 to 1.) So is there something innately wrong with Rice that causes him to fumble in the playoffs far more frequently than he does during the regular season?
…
If anything, I think the flip side of the argument is possible: Rice has been lucky to make it through so many regular-season touches with such a low fumble rate. Over the past five years, the average runner with 1,000 carries or more has fumbled once every 94.6 touches. Nobody’s come close to matching Rice’s fumble rate. In all likelihood, Rice’s fumble rate will drop during the playoffs over the coming seasons, but his fumble rate during the regular season will rise. (Emphasis added)

Barnwell’s conclusion is correct, but the key takeaway for me is that Rice is simply Wyatt Earp (or Matt Schaub). Applying standard significance tests — or calculating the odds that someone with his fumble rate would fumble 5 times in 152 post-season touches — is not helpful in retrospect (to be fair, Barnwell also implied in the article that Rice may be a Wyatt Earp). We’re calculating those odds precisely because we know what happened.

Here’s the better question to ask. At any point in the last 10 years, what are the odds that some good running back would fumble five times in 152 touches in the playoffs despite having a great history at holding onto the ball? Because if LaDainian Tomlinson or Curtis Martin or Jerome Bettis did this, you can bet we’d be hearing the same concerns about their failure to hold onto the ball in the playoffs.

With really small sample sizes, these “random” events are bound to pop up. In 32 games in 2003 and 2004, Rodney Harrison had 5 interceptions in 32 games. That was one for every 231 opposing pass attempts faced by the Patriots, roughly analogous to Rice’s one fumble for every 218 touches during the regular season. In six playoff games those seasons, Harrison had six interceptions on just 223 opposing pass attempts. What are the odds that Harrison would record six or more interceptions on those passes? Just one in 2,127. What does that mean? From a predictive standpoint, not much: in his remaining three playoff games, Harrison had just one interception. The point is if you want to look for random events in small sample sizes (hey, Brooks Reed has 4.5 sacks in 3 career playoff games but only 8.5 sacks in 28 regular season games!), you will find them.

What does it mean for Rice? One thing that’s important to remember is that each playoff game is not necessarily an independent event. If — and I have no idea if this is true — being labeled a fumbler (or fumbling in a high profile game) makes you more likely to fumble because you are so focused on not fumbling that you fumble, then Rice’s odds of fumbling against Denver are probably higher than his base rate. This would be the ‘fumbling is in his head’ theory.

If — and I have no idea if this is true, but I would guess this to be more likely — being labeled a fumbler (or fumbling in a high profile game) makes you less likely to fumble because you will be so concerned about ball security (perhaps at the expense of production — see this cool article by Doug on Adrian Peterson) then Rice is less likely to fumble against Denver.

Either way, asking ‘what are the odds of that’ is not the right question when it comes to Rice. It’s asking “how likely is it that we’ll see some crazy statistical outliers over the course of three games?” The answer to that, as always, is, “very.”