Last week, I sat down with Brian Burke and discussed the work he’s done with NFL teams. Aaron Schatz, founder of Football Outsiders, an indispensable resource for fans of advanced football statistics, has been consulting with NFL teams for years. Schatz is also the lead writer, editor, and statistician on the book series Football Outsiders Almanac and writes for ESPN.com and ESPN The Magazine. Below is my interview with Aaron.
Q: Aaron, can you go into specifics on the type of work you do for NFL teams? Do you envision ultimately working for a team?
As far as consulting with teams, I’ve done two different sorts of things. First, I’ve done some in-game decision analysis, some fourth down stuff as well as some analysis on when to accept or decline penalties. Second, I’ve done reports for teams in February that gave analysis of the season with our stats, looking at what issues were likely to statistically regress and what issues really needed to be addressed, along with suggestions for possible free agent signings. Actually, it’s more accurate to say “we’ve done” rather than “I’ve done.” Some consulting I’ve done alone, and sometimes two or three guys on the FO staff work together.
Consulting for teams is great, but as advanced analysis people gradually move into front offices I don’t think I will be one of them. I don’t know about the various other folks who have followed in FO’s footsteps, but my heart has always been with the media, going back to my days running my high school paper, through my time as a radio disc jockey, doing the Lycos 50, and now Football Outsiders. I set out to revolutionize the way people analyzed the NFL, not the way they managed teams. If I end up improving the way people manage teams a little bit too, that’s just extra coolness.
Q: You publish your DVOA rankings every week, one of the most popular football articles on the web. Have you ever gotten flak from a team for them (i.e., how come we’re hiring you, we have a winning record, and you have us 24th!)?
No flak, no. A couple times I’ve had teams that I’ve worked with or that I’m otherwise in contact with ask me why their rating is particularly low in one area. However, unlike a lot of fans, people who work for teams understand that our stats are objective based on a general formula and don’t get tweaked to favor one team over another depending on how we feel each week. I think when people ask me why their team is low in one area, they often ask so that they can improve that area. And when a team hits rock bottom, I mean, they know it. The Jacksonville people don’t need to ask me why the Jaguars are ranked 30th in DVOA, or whatever it is this week. They don’t care as much about their DVOA right now as they do about their DVOA (and record) next year or two years from now. Jim Schwartz has told me he would rather have his defense ranked highly in DVOA than in yards per game. Of course, he’d rather have more wins than either. (In case it’s not clear otherwise, I should point out there are more teams where I’ve got contacts among various coaches and front office people than there are teams that I have actually worked for and received a check from.)
Q: Fumble luck is a drum FO has beaten for awhile. Is this something that you’ve discussed with NFL teams?
A little bit. I think they basically buy the concept, at least the people I talk to. People I don’t talk to? I mean, I would guess Lovie Smith does believe that his coaching leads to a higher fumble recovery rate, not just a higher fumble forced rate, but I don’t know, I’ve never talked to him.
A lot of it gets back to something I’ve said over and over again: when it comes to dinosaur thinking in the NFL, the media is a much bigger problem than front offices are. There certainly are things I would like to see teams be smarter about — fourth downs, understanding aging curves when they sign free agents, understanding where their scheme requires them to spend money and where their scheme lets them be more frugal — but there certainly are teams that aren’t living in the old world of “establish the run” anymore. More coaches are old school than front office people. But a good number of front office people, and a few coaches, do understand concepts of risk and randomness.
The media, on the other hand… ugh. Beat reporters are better now than they were 10 years ago — there are some really smart beat reporters now, who read FO and PFF and Advanced NFL Stats and even your site and they understand where we are all coming from. There are people like Greg Bedard who will do their own tape analysis articles, and writers like Mike Sando who will do their own stat analysis. However, you still have some guys trapped in 1975. Most color commentators still stand behind coaches when they make ridiculously conservative game decisions. I don’t know if Mike Mayock does, he’s so smart about everything else that maybe he’s smart about this too, I can’t remember a situation where I’ve noted it one way or another. And all the former players and coaches on the various studio shows know a lot about motivation and what it’s like to prepare for games, and sometimes they know about X’s and O’s, but they don’t understand advanced stats or the concept of probability at all, and they constantly grab onto completely absurd storylines just so they can fill airtime.
Mike Smith's team either just converted a 4th-and-1 or won another one-point game.
Yeah, it has been frustrating. You see a couple of famous decisions like some of Mike Smith’s get all this attention, and it makes you think maybe coaches are being more aggressive, and then you look at overall numbers and actually they seem to be getting less aggressive over the last couple years, even guys like Belichick seem to be less aggressive than they were a couple years ago. However, I do think (and hope) this is a situation where fans of advanced analysis need to just wait things out. As people move into front offices, eventually they will get the ears of the coaches, and I do think eventually you will have some coaches who are more aggressive. Maybe not as aggressive as Brian Burke’s win probability calculator thinks they should be, but certainly more than now.
Chip Kelly coming to the NFL might help also, especially if he has success quickly. I’ve said this before, but what we really need is for an owner to come out in public and say that he wants his coach to be more aggressive on fourth down. You need an owner who gives public backing for smarter decisions to try to get younger coaches over that hump of being worried about their job security if results don’t work out in the short term.
Q: Do you think 20 years from now, advanced analytics, especially of the type FO provides, will become widespread? On a scale of 1 to Baseball, if the NFL is at a 4 right now, how long do you think until we get to an 8 or 9?
Hmmmm… well, I guess there are two different issues here. Advanced analytics spread in two ways. The first is the ideas — color commentators understanding that fielding percentage is pretty useless, general managers understanding the value of walks or Ultimate Zone Rating, managers using more bullpen flexibility. Then you have the stats — OPS showing up everywhere, or WAR, or in the case of basketball, Houston putting Dean Oliver’s Four Factors up on their scoreboard.
In football, I would expect that the ideas will spread much easier than the stats. The fact is that many of the stats are proprietary. I explain what’s basically in DVOA, but I don’t give out the specific baselines. ESPN doesn’t give the specific formula for Total QBR. We both have business reasons behind these decisions. The “charting” type stats are also proprietary. You have all these groups doing charting and for the most part we don’t share data so the numbers aren’t out there for everyone to use and they don’t even agree with each other, ESPN may have one count of drops and PFF another and Stats Inc. a third.
I would hope that the “ideas” part of things can get to an 8 or 9 in 20 years. I still think the “stats” part will be something that’s there for the teams to use, and for a particularly hardcore subset of fans. They aren’t going to show up much on broadcasts.
Q: Thanks for taking the time, Aaron. Last question: We know that records in close games has little predictive value, and we’re seeing the Falcons again ride a string of close wins to a great record. Is that something you’ve talked about with teams? I picture this being difficult to bring up with a head coach, who only sees his team’s grit and a never-say-die attitude.
I’ve never talked about this with a team specifically, no. But I’ll say this. We know that in the long run, records in close games have little predictive value. In the short run, however, obviously teams often do specific things to win specific close games. They make the right coaching calls and they make plays when it counts. Not every close win is based entirely on one lucky bounce. So I don’t want players thinking that records in close games are random. They better think that it matters that they do things right because in each specific game, considered on its own, it often does. And I don’t think it’s a problem for a coach to think his team has grit and a never-say-die attitude. What is important is that when you go into the offseason, the personnel people don’t misread how good the team truly is because that team happened to have a 6-2 record in close games thanks to its grit and never-say-die attitude, because that same team is likely to show a sometimes-say-die attitude the following year.
Like a lot of issues with regression towards the mean, the problem isn’t patting yourself on the back for something you have accomplished, it is being unprepared for the future because you were deluded about the long-term meaning of those accomplishments.
My bigger question about this stuff is not close games, but things like red-zone performance. We know that red-zone performance regresses towards the mean significantly from season to season, where your red-zone performance tends to move back towards your overall performance if it is really high or really low. What does that say about the importance of designing specific plays for the red zone? I honestly don’t know.