This week at the New York Times, I take my annual look at ESPN’s Total QBR:

In 2011, ESPN introduced Total QBR, or Total Quarterback Rating, a proprietary statistic intended to capture several of the hidden aspects of quarterback play. The next year, the Indianapolis Colts drafted quarterback Andrew Luck. And while he has helped revitalize the franchise, he has also served as one of the shining examples of how Total QBR credits players for positive plays that are otherwise ignored.

As a rookie, Luck ranked 26th in passer rating but ninth in Total QBR; last year he finished 18th in passer rating but eighth in Total QBR. So what was traditional passer rating missing when it came to Luck? In both years, he ranked in the top three in value added via penalties and on the ground. He did the little things — drawing a significant number of penalties (including value pass interference flags), making key contributions with his legs — that traditional passer rating ignored.

This year, though, Luck ranks slightly higher in passer rating (seventh) than Total QBR (eighth). Alok Pattani and Sharon Katz of ESPN Stats & Information, via email, shed some light on Luck’s season, along with other quarterbacks who have exhibited key differences between their Total QBR and passer rating numbers.

While Luck is having a breakout year via traditional metrics — he leads the N.F.L. in passing yards and is on a pace to set career highs in completion percentage, passer rating and touchdown percentage — he has taken a step back in some of the areas in which he used to excel. Luck picked up a first down on 78 percent of his rushes last year, but that number has dropped to 40 percent in 2014. Scrambles represent the majority of his rushes, and he gained a first down on 75 percent of them last year, compared with 29 percent of them this year. He also ranked third among quarterbacks in Expected Points Added via penalties in 2012 (+12.6), in part because he drew 13 defensive pass interference plays for 238 yards. This year Luck ranks 14th in penalty E.P.A. (+3.5), with five pass interference calls for 104 yards

You can read the full article here.

Finally, a Brian Hoyer note or two that made its way to the cutting room floor. Hoyer ranks 10th in ANY/A but 23rd in Total QBR. I was curious about that, and here is what Alok and Sharon were able to tell me:

• Hoyer has really struggled on third downs. He ranks 2nd-to-last among qualifiers in completion percentage (49%), Y/A (5.7), and first down pct (32%) on 3rd downs.  Not coincidentally, the Browns rank 2nd-to-last in NFL in 3rd down conversion rate.

• Hoyer has also been really bad at running, with 4 yards on 22 carries (only 4 1st downs).  Total rush EPA of -5.6 is lowest in NFL this season.

• Hoyer’s also getting bad grades for the context of his interceptions: five of his interceptions cost his team 4+ EPA, including two of his picks against Falcons.  Only Cutler, Bortles, and Dalton have more “really bad” interceptions.

Ryan Fitzpatrick averaged 9.61 Adjusted Net Yards per Attempt in week 1, good enough for the 4th best grade of the week. But the Houston signal caller — who went 14/22 for 206 yards with 1 touchdown, no interceptions, and 1 sack — was not a very good fantasy quarterback. Using the Footballguys.com standard scoring system of 1 point per 20 yards passing, 1 point per 10 yards rushing, 4 points per touchdown pass, and -1 point per interception, Fitzpatrick had just 15.3 fantasy points (he rushed for 10 yards). That tied him for only the 25th best performance by a quarterback in week one.

Obviously there’s a big difference between ANY/A and fantasy points.  But while we use ANY/A as our main metric for lots of reasons, it’s always helpful to compare it to other statistics.  For example, RG3 ranked 17th in ANY/A in week 1, but only 27th in ESPN’s Total QBR. Why is that? Well, Griffin fumbled twice (losing one), and he completed a lot of very short throws (he had the third lowest air yards per throw and air yards per completion).  But another factor is that his third down performance was a bit misleading using conventional metrics, which is something Total QBR is good at identifying.

Griffin gained 75 net yards on 10 third down dropbacks in the game: that’s pretty good, but he only picked up first downs on 3 of 10 opportunities.   He had a 48-yard completion on a 3rd-and-7, which is great, but it also inflates his average gain; he also had a pair of 9 yard completions on third and very long that added little value.

We can also look at Football Outsiders’ main efficiency metric, DVOA, and compare that to other statistics.  Matt Cassel is an interesting player to analyze.  In DVOA, he ranked 5th.  In ANY/A, he ranked 10th.  In Total QBR, he was 15th, and in fantasy points, he was 21st!   So what gives?

As noted by Vince Verhei, Cassel’s “average pass traveled just 4.8 yards past the line of scrimmage, nearly a full yard shorter than the next shortest quarterback (Derek Carr, 5.6).” That would explain why QBR would be less high on Cassel than other statistics.  And since Cassel threw just 25 passes for only 170 yards, his fantasy value won’t be very high. Football Outsiders, on the other hand, gives Cassel credit for things like his a 9-yard pass on third-and-10 that created better field goal range.  Overall, comparing what Cassel did to the baseline, he looks really good according to FO, and just pretty good according to QBR.  As for ANY/A, it’s impressed by his 2 TD/0 INT ratio, but it’s hard to get a great ANY/A grade when you are averaging just 10.0 yards per completion.

The table below shows each quarterback’s stats in each metric.  For example, Matthew Stafford averaged 11.55 ANY/A in week 1, scored 31.5 fantasy points, had a Total QBR of 97.5, and a DVOA of 90.3%.  Those ratings, among the 33 quarterbacks in week 1 (curses, Rams!), ranked him 1st in ANY/A, 3rd in fantasy points, 1st in QBR, and 1st in DVOA, for an average rank of 1.5. [continue reading…]

A few weeks ago, I put ESPN’s Total QBR under the microscope. Today, I want to look at the quarterbacks whose passing statistics most differ from their QBR grades.

Total QBR grades go back to 2006, so to start, I ran a regression using Adjusted Net Yards per Attempt to predict Total QBR. The best-fit formula was:

Total QBR = -13.5 + 11.23 * ANY/A

For those curious, the R^2 was 0.80, indicating a very strong relationship between ANY/A and Total QBR. What this formula tells us is that a passer needs to average 5.65 ANY/A to be “projected” to have a QBR of 50; from there, every additional adjusted net yard per attempt is worth 11.2 points of QBR. Last year, Peyton Manning averaged 8.87 ANY/A, which projects to a QBR of 86.2. In reality, Manning had a QBR of “only” 82.9; this means Manning’s QBR says he wasn’t quite as amazing as his excellent efficiency numbers would indicate (to say nothing of his otherworldly gross numbers). One likely reason for this result is that Manning ranked 29th in average pass length in the air (according to NFLGSIS) and 6th in yards after the catch per completion; this matters because ESPN gives more credit to quarterbacks on the yards they accumulate through the air. (Throughout this post, we will be forced to deal with educated guesses, because Total QBR is a proprietary formula.)

As it turns out, Manning rating higher in actual QBR than projected QBR is a stark departure from prior years. In 2012, he finished 7.2 points higher in actual QBR than projected QBR, but that’s nothing compared to his time with the Colts. In five years in Indianapolis during the Total QBR era, Manning finished at least 10 points higher in actual QBR each season.

Along with Manning, Matt Ryan and Andrew Luck are the two quarterbacks who are most likely to over-perform relative to their “projected” ratings. Let’s be careful about exactly what this means: whatever the ingredients that go into the QBR formula that don’t go into the ANY/A formula, Manning, Ryan, and Luck seem to have a lot of them.

Luck is a fascinating case. In 2012, he ranked just 20th in ANY/A, but 11th in QBR. I wrote several articles during Luck’s rookie season about how his QBR ratings surpassed his standard stats. ^[1]Although now I can’t recall if his 2012 ratings were inflated because of his 4th quarter comebacks.  And I can’t check, because once ESPN decided to cap the clutch weight associated with … Continue reading Last year, he ranked 16th in ANY/A and 9th in QBR. Does this make Luck the quarterback most underrated (if you buy into QBR) by his traditional passing numbers (if you buy into ANY/A)? [continue reading…]

One of the very first posts at Football Perspective measured how various passing stats were correlated with wins.  One of the main conclusions from that post was that passer rating, because of its heavy emphasis on completion percentage and interception rate, was not the ideal way to measure quarterback play. But what about ESPN’s Total QBR, a statistic invented specifically to improve on — and supersede — traditional passer rating?

As a reminder, we can’t simply correlate a statistic with wins to determine the utility of that metric. The simplest way to remember this is that 4th quarter kneeldowns are highly correlated with wins. Just because you notice it’s raining when the ground is wet doesn’t mean a wet ground causes rain; i.e., just because two variables are correlated doesn’t mean variable A leads to variable B (alternatively, variable B could lead to variable A, variable C could lead to both variable A and B, or the sample size could be too small to determine any legitimate causal relationship). That said, it at least makes sense to begin with a look at how various statistics have correlate with wins.

The Sample Set

Throughout this post, I will be looking at a set of quarterback data consisting of the 152 quarterback seasons from 2006 to 2013 where the player had at least 14 games with 20+ action plays. Games where the quarterback had fewer than 20 plays were excluded, but the quarterback was still included if he otherwise had 14 such games.

The next step was to sum the weekly quarterback data on various metrics, including wins, and create season data. ^[1]For ESPN’s QBR, I took a weighted average of the weekly QBR data. I should note that this is not the way ESPN calculates QBR. As explained to me via email, the scaling function that gives the … Continue reading This allowed me to measure the correlation between a quarterback’s statistics over those 14+ games with that player’s winning percentage in those games.

As it turns out, ESPN’s Total QBR is very highly correlated with wins, with a 0.68 correlation coefficient. ^[2]As a reminder, the correlation coefficient is a measure of the linear relationship between two variables on a scale from -1 to 1. If two variables move in the same direction, their correlation … Continue reading This is to be expected; after all, Total QBR is based off Expected Points Added on the team level, which generally tracks wins and losses. The second most correlated statistic with wins was Adjusted Net Yards per Attempt, my favorite non-proprietary quarterback metric. After ANY/A, both traditional passer rating and touchdowns per attempt were the next most correlated statistics with wins (after all, this is only a step or two away from saying scoring points is correlated with wins). In another unsurprising result, passing yards had almost no correlation with wins, while pass attempts had a slight negative correlation (as any Game Scripts observer would know).  Take a look:

When ESPN first introduced QBR, I wrote that I was intrigued by the possibility of this metric, but frustrated that the specific details of the formula remained confidential. At the time, a clutch weight feature was included in the calculations, which made the metric more of a retrodictive statistic than a predictive one. Since then, ESPN has tweaked the formula several times, and the clutch weight has been capped. ^[3]When Dean Oliver was on the Advanced NFL Stats podcast, he noted that the formula was tweaked in 2013 so that the “clutch index” part of the formula was essentially capped. He added … Continue reading ESPN is not engaged in academia, so I understand why they have not published all the fine print; as a researcher, I’m still frustrated by that decision. Still, with 8 years of QBR data now publicly available, we can answer two questions: does Total QBR predict wins and how sticky is Total QBR?

We know that a high Total QBR is correlated with winning games, but we also know that there’s limited value to such a statement. If having a high Total QBR was one of the driving factor behind winning games, than such a variable would manifest itself in all games, not just the current one. So with my sample of 152 quarterbacks, I used a random number generator to divide each quarterback season into two half-seasons. Then I calculated each quarterback’s average in several different categories and measured the correlation between a quarterback’s average in such category in each half-season with his winning percentage in the other half-season. ^[4]Then I did the entire process again, using a new set of random numbers, and averaged the results. The results:

As you would expect, all of our correlations are now smaller. But ESPN’s quarterback rating metric remains the best measure to predict wins. Perhaps even more impressively, Total QBR is more correlated with future wins than past wins. That’s pretty interesting. Another interesting result is that passer rating fares pretty well here, although much of the same issues as before remain with using correlation to derive causal direction. ^[5]For example, because passer rating is biased towards high completion percentage and low interception rates, quarterbacks who play with the lead tend to produce strong passer ratings; well, playing … Continue reading

One other concept to remember is that our sample of quarterbacks consists of players who were heavily involved in at least 14 games. That makes sure Peyton Manning, Tom Brady, and Drew Brees are involved, while filtering out some Christian Ponder, Blaine Gabbert, and Brandon Weeden seasons. In other words, the data set contains more above-average quarterbacks than a random sample would, so we may not be able to justify certain conclusions from this study.

The other important question is whether Total QBR is predictive of itself; i.e., how “sticky” is this metric over different time periods. We know that interceptions are very random, and knowing a quarterback’s prior interception rate is not all that helpful in predicting his future interception rate. Where does Total QBR fall along those lines?

The most “sticky” stats were passing yards and pass attempts, which in retrospect isn’t too surprising. These reflect the style of the offense, the talent of the quarterback, and the quality of the defense, so they should be easier to predict. The second-least sticky metric was wins, which also makes sense. After that, ESPN’s Total QBR fits in a narrow tier with most of our other metrics as being somewhat predictable.

Conclusion

The numbers here indicate that Total QBR is worth examining.  It may be a proprietary measure of quarterback play, but it’s not a subjective one with no basis in reality.  It does seem to be the “best” measure of quarterback play, although whether the tradeoff in accuracy for transparency is worth it remains up to each individual reader. One of the drawbacks I see in Total QBR is the failure to incorporate strength of schedule. And while no other traditional passer metric does, either, it’s also easy enough to make those adjustments. Hopefully, an SOS-adjusted Total QBR measure will be released soon (I’ll note that the college football version does include a strength-of-schedule adjustment).  My sense is that Total QBR is underutilized because (1) ESPN haters hate it because it’s an ESPN statistic, (2) it’s proprietary, and (3) analytics types disliked it because of the (now-eliminated) clutch rating.  While I would not suggest making it the only tool at your disposal, it does appear to deserve a prominent place in your toolbox.