The preseason is meaningless, right? Well, as it turns out, it might give us a window into quarterback development, despite what you might think. The threshold for whether the preseason is useful is whether including that information tells us anything about a quarterback’s potential that we don’t already know from his draft position (or perhaps certain analytics). I have been putting together data from preseason box scores going back to 1997. The data show that, for some quarterbacks, the preseason is not quite meaningless.

Neil Paine showed some interesting evidence relating to this idea on Friday. Looking at team performance since 2009 for teams with new quarterbacks, Neil showed that preseason passing efficiency helps predict regular season passing efficiency. It’s important to note that part of this result may have been pretty predictable even before we watched those preseason games. The 2012 Redskins replaced Rex Grossman and John Beck with the #2 pick in the draft who would have been #1 in an average year. So we would expect a big improvement to come just by way of moving from Grossman to a healthy RGIII.

Still, maybe the preseason does offer hints as to whether a quarterback is likely to succeed or fail even controlling for what we would have expected. Were there hints in the preseason that someone like Russell Wilson would outperform his draft status? ^[1]Mike Tanier also looked at this topic last week, and lauded Wilson’s 2012 preseason performance. Does Blake Bortles’ impressive preseason performance so far tell us something about his chances of succeeding in the NFL even for someone drafted so highly? ^[2]His numbers on Thursday were inflated by a meaningless pass at the end of the first half, but he also had some very pretty passes that I wasn’t expecting. Notable were the pass right before the 1st … Continue reading

The answers to these questions are subtle. I think the answers look like yes and no, respectively. Let’s take this to the data.

Historical Performance of Early-Round QBs in their First Preseason

By going back to 1997, we can get enough data to look at how new quarterbacks perform during the preseason according to draft position. I collected these data from John Troan’s database. ^[3]I’m grateful that John made these data available. It’s wonderful that these data are even available, and they are mostly in good shape. I have checked them and fixed some coding errors, but it is important to emphasize that there may still be a few mistakes. ^[4]For example, I thought JaMarcus Russell had an incredible 2008 preseason, until I saw that the box score gave him 751 yards for the second game. I don’t think there are many.

John’s database reports box scores that include completions, attempts, yards, and touchdowns going back to 1997. Sacks are not broken down by quarterback and interceptions are only available since 2007, so I’m going to look at just Y/A. ^[5]Looking at Y/A misses some of the fairly hilarious Leaf-to-Manning comparisons that Chase referenced in his post. While Leaf was thought to have had the better preseason in 1998, Manning actually had … Continue reading

The table below lists the quarerbacks selected in the draft who threw at least 25 passes in the preseason the year they were drafted. The table is sorted as yards over average, with the average Y/A of these quarterbacks checking in at 6.12. ^[6]Note that we are not endorsing this as the preferred method to rank the quarterbacks; it just makes more sense than using pure Y/A given the varying number of attempts. The last column in the table is value above expectation (VAE), or how much more Approximate Value (AV) the player produced in his first five years relative to the expectation for other quarterbacks with his draft position. ^[7]To calculate this, I regressed the AV in the first five years on a cubic in draft position for all QBs. The difference between actual first-five AV and the predicted value from that regression is VAE. For example, Kyle Orton checks in as the rookie quarterback with the most impressive preseason since 1997. He averaged 10.2 Y/A on his 46 attempts, and wound up producing 23 points of AV during his first five seasons. Since Orton was the 106th pick in the draft, 23 points of AV represents 17.4 points more than expected. That means Orton represents one piece of evidence that yes, preseason statistics can provide some insight into a quarterback’s career.

Looking at the table, most of the top preseason performers did pretty well in their first five regular seasons. Wilson had the 7th-highest preseason Y/A since 1997 and he doesn’t factor into any of the calculations for AV because he hasn’t played his fifth season yet. Peyton Manning and Tom Brady are both in the top 20 for rookie preseason Y/A. Jay Cutler, Nick Foles, Andrew Luck, Carson Palmer, and Ben Roethlisberger also are in the top 20.

But so are Mark Sanchez and Tim Tebow. And look for Aaron Rodgers, way down at 4.65 Y/A. Donovan McNabb is at 3.52, two spots from the bottom. So preseason performance clearly doesn’t tell us everything about how a quarterback will perform. Does it tell us something, though?

For first round passers, I think the answer is not so much. The picture below has the first round quarterbacks from ’97 to ’10 along with a best-fit line to predict their VAE as a function of preseason Y/A.

On the other hand, for quarterbacks drafted in later rounds, preseason performance — at least in this data set — does tell us something useful about how they will do in the regular season. The graph below looks just at QBs drafted in rounds 2-4 from 1997 to 2010. ^[8]The line looks similar if we go all the way up to round 7, but the image gets very cluttered. In addition, we have to exclude quarterbacks like Wilson who have not played for five seasons yet.

For every additional one yard per attempt a quarterback produced, that adds about 2.4 points of AV in the first five years for a quarterback drafted in rounds two through four. The estimated effect is 2.0 points of AV added if we look from rounds 2-7.

Overall, it looks like preseason performance is useful for unearthing diamonds in the rough. It can help identify middle-round signal callers who the draft process undervalued, such as Wilson and Orton. What the preseason can’t do is separate QBs who the draft process already highly valued. So I’m not sure it’s going to tell us much about someone like Blake Bortles and perhaps not Teddy Bridgewater, either.

And maybe even the information about middle-round quarterbacks is something we could have found in the player’s college performance. For example, Marques Tuiasosopo may have been drafted in the second round, but his college numbers did not project well. Russell Wilson was a famous favorite of both Matt Waldman and  Football Outsiders leading up to the 2012 Draft. So maybe preseason performance is picking up stuff that we could have already seen in college stats.

I do think college stats can tell a big part of the story. College stats pointed towards players such as Kyle Orton and Matt Schaub outperforming their draft position, so that maybe we didn’t need the preseason to know they were probably going to be pretty good for where they were picked. Still, there are some players with high college completion percentages (era-adjusted) whose poor preseason stats portended future doom. Examples include Brian Brohm and Kellen Clemens. This issue is an open question, but I lean towards the preseason probably mattering a little for rookie passers even controlling for college stats. I could easily be wrong, though, and it would be great to check this out.

At this point, where I think the preseason might predict future success is not so much for 1st round QBs such as Bortles and Teddy Bridgewater, but for players like Jimmy Garoppolo, who got less attention leading up to the draft. And it’s not just that Garoppolo is a second-round passers. We have less information to go on with him based on his college performances coming against inferior competition. Based on the results above, I think it probably does tell us a little something that Garoppolo’s averaging 9.2 Y/A so far. ^[9]The Football Outsiders LCF forecast was bullish on Garoppolo, too. Of course, full disclosure that I am invested in the success of the Pats latest GQ-worthy quarterback.