A few weeks ago, I put ESPN’s Total QBR under the microscope. Today, I want to look at the quarterbacks whose passing statistics most differ from their QBR grades.

Total QBR grades go back to 2006, so to start, I ran a regression using Adjusted Net Yards per Attempt to predict Total QBR. The best-fit formula was:

Total QBR = -13.5 + 11.23 * ANY/A

For those curious, the R^2 was 0.80, indicating a very strong relationship between ANY/A and Total QBR. What this formula tells us is that a passer needs to average 5.65 ANY/A to be “projected” to have a QBR of 50; from there, every additional adjusted net yard per attempt is worth 11.2 points of QBR. Last year, Peyton Manning averaged 8.87 ANY/A, which projects to a QBR of 86.2. In reality, Manning had a QBR of “only” 82.9; this means Manning’s QBR says he wasn’t quite as amazing as his excellent efficiency numbers would indicate (to say nothing of his otherworldly gross numbers). One likely reason for this result is that Manning ranked 29th in average pass length in the air (according to NFLGSIS) and 6th in yards after the catch per completion; this matters because ESPN gives more credit to quarterbacks on the yards they accumulate through the air. (Throughout this post, we will be forced to deal with educated guesses, because Total QBR is a proprietary formula.)

As it turns out, Manning rating higher in actual QBR than projected QBR is a stark departure from prior years. In 2012, he finished 7.2 points higher in actual QBR than projected QBR, but that’s nothing compared to his time with the Colts. In five years in Indianapolis during the Total QBR era, Manning finished at least 10 points higher in actual QBR each season.

Along with Manning, Matt Ryan and Andrew Luck are the two quarterbacks who are most likely to over-perform relative to their “projected” ratings. Let’s be careful about exactly what this means: whatever the ingredients that go into the QBR formula that don’t go into the ANY/A formula, Manning, Ryan, and Luck seem to have a lot of them.

Luck is a fascinating case. In 2012, he ranked just 20th in ANY/A, but 11th in QBR. I wrote several articles during Luck’s rookie season about how his QBR ratings surpassed his standard stats. ^[1]Although now I can’t recall if his 2012 ratings were inflated because of his 4th quarter comebacks.  And I can’t check, because once ESPN decided to cap the clutch weight associated with … Continue reading Last year, he ranked 16th in ANY/A and 9th in QBR. Does this make Luck the quarterback most underrated (if you buy into QBR) by his traditional passing numbers (if you buy into ANY/A)?

Yes and no. He’s the most underrated among active quarterbacks, but one other quarterback (minimum 700 action plays since 2006) has him beat. Here’s what I did:

1) Calculate a career projected QBR grade, based on the projected QBR grade in each season (using the regression formula above) and weighting the rating in each season by number of action plays.

2) Calculate a career actual QBR grade, based on a weighted average (for number of action plays) of each season’s QBR grade. This is not the way ESPN comes up with their career QBR grades (assuming they even come up with them), but I am not too concerned about about the trivial distinction. ^[2]As explained in the footnotes to the earlier post, the scaling function that gives the “final” QBR on a 0-100 scale is nonlinear; as a result, you can’t just calculate a weighted average of the … Continue reading

3) Subtract the result from step 1 from the result in step 2.

As it turns out, the quarterback most harmed by using traditional stats is… Vince Young. It’s easy to snicker and say this is a drawback of QBR if we’re saying Young is underrated, but I don’t think that’s fair. We have to remember what the baseline is: out of the 55 quarterbacks with 700 action plays since 2006, Young ranks 45th in projected QBR, which puts him behind end-of-career Marc Bulger and Trent Edwards. He’s just slightly ahead of Tarvaris Jackson and Chad Henne.

Now if you’re speaking to a Longhorns homer who thinks Young was an above-average quarterback because of his career 30-17 record, well, then he’s not underrated. We already know that Young’s record is way out of whack with his passing stats. But seeing him atop the list here indicates that Young deserves slightly more credit than you would normally assign him, and there’s a pretty obvious reason why: ANY/A ignores rushing contributions, and Young was an outstanding runner.

Here’s how to read the table below: Young had three qualifying seasons since 2006 ^[3]Note that because Young posted an excellent ANY/A average in 2010 but on only 156 attempts, Young’s actual career ANY/A rating is better than you would think based on his projected QBR. When … Continue reading on 1,360 action plays in those seasons. He has a projected career QBR grade of 42.3 but an actual grade of 53.8, which means he’s outperformed the projection by 11.5 points.

With Young, it’s easy to identify why QBR loved him. What about the three active passers atop the list?

• In 2012, Luck received a lot of credit for two areas ignored by ANY/A: rushing production and benefits gained via penalty. Luck ranked 3rd in rushing EPA among quarterbacks, behind only Cam Newton and Russell Wilson. And he ranked 1st in expected points added due to penalties. In 2012, the Colts offense led the league by a large margin in both penalties drawn and penalty yards gained (who knew?). In other words, ESPN QBR gives Luck credit for a 40 yard gain that comes via pass interference, or for drawing a player offsides, while ANY/A ignores both plays. Last year, Luck led all quarterbacks in rushing EPA, and he ranked third in penalty EPA. Luck ranks 4th among all players and (a very distant) 2nd among quarterbacks in rushing first downs picked up on 3rd/4th down over the past two seasons. While Luck’s traditional passing numbers are not overwhelming, I’m very inclined to side with Total QBR when it comes to the Colts passer. He seems to be an appropriate outlier.

• Ryan also excels at adding value via penalty yards. And Ryan, like Luck, has done very well in the fourth quarter of games. Even though ESPN has removed the clutch bonus associated with QBR, Ryan might still be getting some extra credit for all the comebacks and game-winning drives he’s led (without knowing the formula, one can’t quite say how Ryan’s QBR grade was derived). And, for whatever reason, Ryan’s passing EPA has outstripped his ANY/A, particularly in the 2010 season (perhaps related to the fact that the Falcons ranked 4th in passing first downs that season). Before running the numbers, I thought his great supporting cast of Roddy White, Tony Gonzalez, and Julio Jones (albeit not in 2010) meant Ryan would look better in ANY/A than QBR, because Total QBR tries to only reward a quarterback for his individual contributions (so a great catch by Jones won’t be worth as much to Ryan as it would in ANY/A). Ultimately, I can’t quite explain why Ryan stars here (the simple answer could just be the Ryan is better than his stats indicate, although I’m not even sure what that means in this context.)

• With Manning, it’s never surprising to see him “overachieve” in some metric, although it is kind of odd to see an argument that ANY/A underrates him. Manning tends to have prolific offenses and is excellent on third downs, so presumably that would help him here. And since he passes the eye test with flying colors, my guess is that all the little things that Manning does so well gets rewarded in QBR, even if they may not all get picked up in ANY/A.

What about the reverse?

• Brandon Weeden is the anti-Luck (in more ways than one). Weeden doesn’t have very good standard numbers, but he’s downright dreadful in QBR. There are lots of potential explanations — Weeden struggles on third down, has issues with fumbling, and it’s easy to envision QBR’s analysts assigning Josh Gordon a good chunk of the production on the few good plays Weeden makes [Please see the update below]. I will note that Weeden actually led the NFL last year in air yards per completion (Norv!), so I wouldn’t have guessed that Weeden would fare so badly (relative to his already bad statistics) in this analysis.

• Nick Foles led the NFL in ANY/A but ranked just fifth in Total QBR. In ESPN’s season in review of QBR, no justification was given to explain this difference. Eagles running backs easily led the league in yards per reception: it’s possible (and appropriate) that Foles’ QBR grade is lower because ANY/A overvalues him when LeSean McCoy takes a short pass for 30 yards. But otherwise, I can’t quite think of why Foles’ QBR was so low: perhaps more credit was given to the Eagles offensive line than other quarterbacks? I do know that Foles threw a number of potential interceptions that were dropped, but as I understand it, QBR wouldn’t penalize him for that. While I like Total QBR, this is a good reason why it’s heavily criticized: it’s silly that we don’t know what Foles ranks only 5th in this metric.

• Alex Smith is another player who does not excel in Total QBR. It’s easy to assume that screen passes to Jamaal Charles and a general aversion to deep throws are the reasons, but Smith “only” undershot his QBR projection by 3.8 points in 2013. It was actually his 2006 and 2010 seasons that gave him such a low place on the list.

Update: Alok Pattani of ESPN Stats and Information was kind enough to email me this morning to discuss the post. He noted, among other things, that there is no subjective adjustment on “Weeden to Gordon” type of passes. He noted that ESPN’s video trackers do not have the ability to “decide” how much credit the quarterback or receiver should get on each play.

I asked him how QBR treats the following three plays, which are obviously treated the exact same way using conventional metrics:

1) QB throws a perfect pass to WR, who makes an easy catch for 25 yards.

2) QB throws a terrible pass, but Elite WR makes an incredible diving catch for 25 yards.

3) QB throws a terrible pass, which hits a DB in the hands, but bounces into the WR’s hands, who records a 25 yard catch.

Alok noted that currently, all three plays will likely be looked at the same way from a division of credit standpoint based on the data ESPN records (i.e. if the air yards, pass direction, etc. are all the same). ESPN does not look at the “quality” of a completed pass, which would venture into more subjective territory (on the other hand, overthrow/underthrow/drop data is recorded for incomplete passes). ^[4]However, it’s possible such concepts, if ESPN could get comfortable with the data and history, could be made in the future. While subjective, I personally think it makes sense to treat these … Continue reading In addition, he noted:

All the trackers can do is enter in “video-based” data – whether the QB was under duress or not, which direction the pass went, how many air yards/YAC there was, whether an incompletion was a drop or an overthrow, etc. The QBR algorithm looks at that data and then applies certain rules (based on analysis of past data) to divide credit on the play – e.g. if there were more air yards, relatively more credit to the QB.

So this is not really subjective – the division of credit is based on data relevant to what Weeden and Gordon did on the play. The trackers can’t say something like “Josh Gordon deserves 60% credit for the EPA on that play” and enter that – they just input the data they see on the video.

It looks like several of the guys you mention as QBR-ANY/A outliers are those with higher/lower sack rates. I know ANY/A takes into account sacks and sack yardage, but my thought is that QBR gives a higher penalty on sacks, so that’s another source of explanation for some of those QBs with discrepancies. Peyton Manning, Matt Ryan, Andrew Luck and even Vince Young have relatively low sack rates. Nick Foles, Alex Smith, and JaMarcus Russell have relatively high ones. Makes a big difference in QBR.

As for Weeden in particular, Alok noted that three possible explanations (for anyone who really wants to know more about Brandon Weeden and has made it this far, I salute you):

• He’s worse on 3rd downs than other downs (where he is still not good). This is going to be emphasized in a system based on EPA more than any stats that don’t look at the context of each play.
• While he didn’t rely on a lot of YAC in 2013, he was one of the league leaders in YAC/completion in 2012. So he got less credit for his completions than the average passer in his rookie year.
• In 2012, his interceptions were really penalizing. So his interceptions may have been worse, on average, than others (usually pick-sixes or other shorter throws where returns are more attributable to the QB).

I’d like to thank Alok and ESPN reaching out to help inform Football Perspective and its readers.