Earlier this week, ESPN announced three key changes to the way its Total QBR metric is calculated. Let’s review them:

1) Interception returns

The base statistic used throughout QBR is EPA, which stands for Expected Points Added per play. So if an interception was returned for a touchdown, that play would obviously have a large negative EPA. For example, when the Chargers had 3rd-and-8 at the St. Louis 8-yard line in the 2nd quarter of a game in week 12 of last season, PFR calculated the Expected Points for that situation as +3.58 for San Diego. When Rivers threw a pick-6 on that play, that situation turned into a -7, which is a swing of 10.58 points. Presumably ESPN’s formula came to a pretty similar result.  And that leaves Rivers with an enormous penalty.

So now, instead of penalizing the quarterback for the actual EPA swing, ESPN will penalize the quarterback for the expected swing based on the type and location of the interception.  This means much smaller penalties on pick sixes, and (one would assume) slightly larger ones on all other interceptions.

This makes sense to me, although it highlights the question of what is QBR actually supposed to measure.  This change, while eliminating some of the randomness involved in a play, moves away from the way QBR has been tied to EPA. On some (though not all) interceptions, whether a player returns it 90 yards or 10 yards is completely random, so penalizing a quarterback a fixed amount (that varies by type and location) is likely going to improve the predictability of the model. What I mean by that is that QBR will become “stickier” from time period to time period, which is a good thing (if you like predictive models).

On the other hand, this change will decrease the relationship between QBR and say, winning percentage. That seems unavoidable if you are decreasing the penalty on pick sixes. Whether that’s a good thing or not is up to the individual reader, but I find it interesting in light of how QBR was originally marketed: ^[1]I acknowledge that those who created the metric likely had little influence in how it was marketed. One of the selling points has always been the correlation between QBR and winning percentage (e.g., from ’08 to ’10, quarterbacks with the higher QBR won 85.7% of games).    And, in fact, in the very article announcing the changes, there’s this tidbit:

The changes outlined above do not affect QBR’s status as a key statistic — a game-level QBR of 80 is the level of QB performance that generally wins the game 80 percent of the time.

I will note that (1) being correlated with winning does not make QBR a key statistic any more than 2nd half kneels or 4th quarter interceptions is a key statistic, and (2) it’s can’t be true that this change to interceptions doesn’t decrease the correlation with winning.  It probably has an immaterial impact, given how rare pick sixes are, but regardless, I’m in the camp approves of changes that makes QBR more of a predictive stat and less of a retrodictive stat.

Let’s move on to the second change.

2) Pressure

The exact changes aren’t clear, but we have been told that “steps” were taken to give quarterbacks more credit if they succeed in spite of pressure, and less credit if they succeed because the offensive line gave them a bunch of time. In addition, if a quarterback is sacked because his receivers didn’t get open, he’ll be penalized less than in the past.

This introduces an element of subjectivity into the process, which will rankle some feathers: ESPN is going to use live video tracking to determine whether there was pressure, and that basically means relying on game charters to determine the level of pressure. Subjective is not the same thing as random, but this adds a bit of a human element to things (perhaps even more so than calling something a drop or not).

Putting aside that note, ^[2]While some people get really bothered by the idea of subjectivity in their stats, I’m not one of them. I don’t know how I feel about this one: for example, let’s say a quarterback misses a read, which leads to him being under pressure.  If he still completes the pass, would he get more credit than if he made the original read correctly? Is that appropriate?  If a quarterback changes the protection call before a play, or identifies a hot read, that’s something he should get credit for, but it’s possible that a quarterback who fails to do those things but succeeds under pressure will get more credit.  The article cites Russell Wilson as an example of a quarterback who has succeeded under pressure, and that’s undoubtedly true: but I also don’t think it’s a coincidence that Russell was under pressure on a league-high 41% of his passes. He’s good under pressure, but he also is responsible for some of that pressure. ^[3]One Wilson-neutral reason: given how good the Seahawks defense is, Wilson is likely coached to be conservative, which could lead to him holding on to the ball longer than he would if the defense … Continue reading

At a very high level, this is probably a good change: if a quarterback does well under pressure, that’s a good thing. And I’m suspect that on average, the quarterbacks most under pressure will have been the victims of poor blocking (and the quarterbacks least under pressure will have been the beneficiaries of good blocking), but who knows how will this will work with any individual quarterback.  And given the proprietary nature of the statistic, this is only going to add to the murkiness.  I don’t mean to throw cold water on what’s probably a good idea — hey, making good throws under pressure should be rewarded — but one is left having to have a lot of faith in ESPN.  I think the guys there (at least those working on QBR) are very smart, so that may be warranted, but each person will have to come to their own conclusion.  But these sorts of changes make it very hard to really understand what a QBR is, since such a small percentage of the formula is actually made public.

3) Credit for yards after the catch:

As with interception returns, the quarterback has little control over whether a receiver takes a bubble screen 5 yards or 50 yards. Did the defender fall down or miss a tackle? Did the receiver juke the cornerback, resulting in a long play? Neither of those situations was a result of the quarterback’s throw. Like with interceptions, there is an expected yards after the catch based on the type of throw. Credit for YAC is now based on what is expected, given the throw.

I was under the impression that this was always the case.  Back in 2011, ESPN noted that “yards after the catch is more about what the receiver does. Statistical analysis was able to show this, and we divided credit based on those things.”  In that same article, it was explained that their analysis of the data showed “that a pass that is in the air for 40 yards is more reflective of a quarterback than a pass that is in the air for 5 yards and the receiver has 35 yards of run after the catch.”  My understanding of QBR was that yards gained after the catch were already heavily discounted.

I suppose there is a slight difference here: maybe ^[4]Yes, it’s frustrating that we are left having to guess. a quarterback used to receive a small percentage of the YAC actually gained, and it was therefore subject to some variation. Now, he gets a constant amount.  That’s probably an improvement as for as the predictive elements of QBR, but it would also seem to make the metric less correlated to winning (a quarterback whose receivers get a lot of YAC is, all else equal, going to win more games).

This game by Ben Roethlisberger is used as an example of the change: his QBR drops by 20.4 points under the new formula.  In that game, Roethlisberger averaged an excellent 8.21 ANY/A, and the Steelers won, but under the new methodology, more of the credit is given to the offensive line and the wide receivers. But since good offensive line play and good wide receivers are correlated with winning, discounting those factors will make QBR less correlated with winning (but a better predictive stat for individual quarterbacks).  ESPN appears to be serving two masters: tying QBR more to a quarterback’s individual skill and performance level, and tying QBR to winning.  That’s simply not possible to do, because quarterbacks are not the only players that matter.

As another example example, Aaron Rodgers is cited as a player who now drops in QBR, and that makes some sense given how good Jordy Nelson and Randall Cobb are after the catch. On the other hand, that should make it clear how this will make QBR less correlated with winning, as does the pass pressure change, and as would any metric that tries to isolate a quarterback from his performance.  Green Bay wins in part because it has very good receivers; if we are removing that when analyzing the quarterback’s play, that’s a good thing to analyze the quarterback and a bad thing to create a statistic that’s correlated with winning.

Like the interception change, this is probably a good thing, but the black box element of QBR feels even stronger here (at least, to me) given how I thought YAC was always treated.

Conclusion

The three changes here seem to make QBR more of an individual stat than a team stat, and to reduce some of the randomness associated with the game of football.  It also is likely to make QBR more of a sticky metric, which is a good thing if you like predictive stats.  And while the changes are probably small in the grand scheme of things, they also are going to make QBR less correlated with winning.  We’ll see if that leads to more changes to the system next year.