Two years ago, I set a baseline for what a pre-season projection system should hope to accomplish. The simplest baseline of all would be to project each team to go 8-8. That would require no thought at all: a person could wake up 49 years and eight months from now and project each then-existing team in the NFL (or its successor league) to go .500 in the 2067 season. So any projection system has to beat that, at a minimum. As I wrote two years ago:
If you did that in every season from 1989 to 2014, your model would have been off by, on average, 2.48 wins per team. This is calculated by taking the absolute value of the difference between 0.500 and each team’s actual winning percentage, and multiplying that result by 16. So that should be the absolute floor for any projection model: you have to come closer than that.
Now, let’s flash back to July of this year. The USA Today published its preseason predictions in a rather provocative fashion, particularly with respect to two teams. I see a ton of preseason projections every year and forget them minutes later, so please forgive me if you feel like I am picking on the USA Today here. That is not the intent, and other publications have made more egregious errors but are not at my fingertips. But this publication picked the Patriots to go 16-0 and the Jets to go 1-15, which was rather extreme.
Upon further view of their predictions, many of them are pretty ugly. So I decided to compare those predictions to the “every team in the same” test, or the “8-8” system.
There were six teams the USA Today got really wrong, where an 8-8 projection would be at least 3.0 games closer to being accurate:
The Buffalo Bills are 8-7 (off of 0.500 by 0.5 wins); the USA Today predicted them to go 4-12 (off by 4.5 wins).
The Green Bay Packers are 7-8 (0.500 projection is off by 0.5); the USA Today predicted them to go 12-4 (off by 4.5).
The Los Angeles Rams are 11-4; an 8-8 projection would be off by 3.5 games, but the USA Today had the Rams are 4-12, off by 7.5 games.
The Oakland Raiders are 6-9 (off by 1.5); the USA Today had them at 11-5 (off by 4.5).
The Detroit Lions are 8-7 (off by 0.5); the USA Today had them at 5-11 (off by 3.5).
The Tennessee Titans are 8-7 (off by 0.5); the USA Today had them at 12-4 (off by 3.5).
You might say the loss of Aaron Rodgers shouldn’t be held against them, and that the Rams success caught everyone off guard. Both of those things are true, but we also know that superstars get hurt and surprise teams happen every year. Both of those facts should urge predictors to be more conservative in the aggregate.
The USA Today thought the Titans would be great and the Bills terrible; both are 8-7. And the USA Today had the Raiders at 11-5 and the Lions at 5-11; instead, Detroit has two more wins than Oakland. These, again, are signs that we shouldn’t be too overconfident in our preseason projections (a look at the Giants and the Rams projected wins totals would also work to that effect).
Now, there were also three teams where the USA Today beat the 8-8 system by at least 3 games.
The Indianapolis Colts were projected to go 5-11 by the USA Today; they are 3-12 (USAT off by 1.5, .500 prediction off by 4.5).
The Cleveland Browns were projected to go 4-12; they are 0-15 (USAT off by 3.5, .500 prediction off by 7.5).
The Pittsburgh Steelers were projected to go 12-4; they are 12-3 (USAT off by 0.5, .500 prediction off by 4.5).
In the case of the Colts, Browns, and Bears, predicting them to be bad — but not terrible — was the wise move. (That would have worked with the Jets, too: the 1-15 projection is now off by 4 or 5 wins, giving another win to the 8-8 system). The Steelers were projected to be very good, and that prediction was nailed. The USA Today even wins the Patriots bet as we stand right now (12-3 is closer to 16-0 than 8-8), but a more conservative approach would have been better.
Overall, the USA Today fared worse than the blind 8-8 system. The USA Today projections were closer than the 8-8 system for 13 teams. Meanwhile, the 8-8 system was closer on 15 teams, with four teams (Houston, Kansas City, Jacksonville, New Orleans) currently graded as ties. Take a look:
| Team | W | L | Win% | USAT W% | Diff | Abs Val Diff (USAT) | Abs Val Diff (8-8) | Closer? |
|---|---|---|---|---|---|---|---|---|
| New York Giants | 2 | 13 | 0.133 | 0.625 | 0.492 | 0.492 | 0.367 | 8-8 |
| Tampa Bay Buccaneers | 4 | 11 | 0.267 | 0.625 | 0.358 | 0.358 | 0.233 | 8-8 |
| Oakland Raiders | 6 | 9 | 0.400 | 0.688 | 0.288 | 0.288 | 0.100 | 8-8 |
| Green Bay Packers | 7 | 8 | 0.467 | 0.750 | 0.283 | 0.283 | 0.033 | 8-8 |
| Cleveland Browns | 0 | 15 | 0.000 | 0.250 | 0.250 | 0.250 | 0.500 | USAT |
| Houston Texans | 4 | 11 | 0.267 | 0.500 | 0.233 | 0.233 | 0.233 | tie |
| Tennessee Titans | 8 | 7 | 0.533 | 0.750 | 0.217 | 0.217 | 0.033 | 8-8 |
| New England Patriots | 12 | 3 | 0.800 | 1.000 | 0.200 | 0.200 | 0.300 | USAT |
| Arizona Cardinals | 7 | 8 | 0.467 | 0.625 | 0.158 | 0.158 | 0.033 | 8-8 |
| Dallas Cowboys | 8 | 7 | 0.533 | 0.688 | 0.154 | 0.154 | 0.033 | 8-8 |
| Atlanta Falcons | 9 | 6 | 0.600 | 0.750 | 0.150 | 0.150 | 0.100 | 8-8 |
| Indianapolis Colts | 3 | 12 | 0.200 | 0.313 | 0.113 | 0.113 | 0.300 | USAT |
| Denver Broncos | 5 | 10 | 0.333 | 0.438 | 0.104 | 0.104 | 0.167 | USAT |
| Seattle Seahawks | 9 | 6 | 0.600 | 0.688 | 0.088 | 0.088 | 0.100 | USAT |
| Miami Dolphins | 6 | 9 | 0.400 | 0.438 | 0.038 | 0.038 | 0.100 | USAT |
| Los Angeles Chargers | 8 | 7 | 0.533 | 0.563 | 0.029 | 0.029 | 0.033 | USAT |
| Chicago Bears | 5 | 10 | 0.333 | 0.313 | -0.021 | 0.021 | 0.167 | USAT |
| Baltimore Ravens | 9 | 6 | 0.600 | 0.563 | -0.038 | 0.038 | 0.100 | USAT |
| Pittsburgh Steelers | 12 | 3 | 0.800 | 0.750 | -0.050 | 0.050 | 0.300 | USAT |
| Cincinnati Bengals | 6 | 9 | 0.400 | 0.313 | -0.088 | 0.088 | 0.100 | USAT |
| Kansas City Chiefs | 9 | 6 | 0.600 | 0.500 | -0.100 | 0.100 | 0.100 | tie |
| Washington Redskins | 7 | 8 | 0.467 | 0.313 | -0.154 | 0.154 | 0.033 | 8-8 |
| Jacksonville Jaguars | 10 | 5 | 0.667 | 0.500 | -0.167 | 0.167 | 0.167 | tie |
| San Francisco 49ers | 5 | 10 | 0.333 | 0.125 | -0.208 | 0.208 | 0.167 | 8-8 |
| Detroit Lions | 8 | 7 | 0.533 | 0.313 | -0.221 | 0.221 | 0.033 | 8-8 |
| New Orleans Saints | 11 | 4 | 0.733 | 0.500 | -0.233 | 0.233 | 0.233 | tie |
| Minnesota Vikings | 12 | 3 | 0.800 | 0.563 | -0.238 | 0.238 | 0.300 | USAT |
| New York Jets | 5 | 10 | 0.333 | 0.063 | -0.271 | 0.271 | 0.167 | 8-8 |
| Buffalo Bills | 8 | 7 | 0.533 | 0.250 | -0.283 | 0.283 | 0.033 | 8-8 |
| Carolina Panthers | 11 | 4 | 0.733 | 0.438 | -0.296 | 0.296 | 0.233 | 8-8 |
| Philadelphia Eagles | 13 | 2 | 0.867 | 0.563 | -0.304 | 0.304 | 0.367 | USAT |
| Los Angeles Rams | 11 | 4 | 0.733 | 0.250 | -0.483 | 0.483 | 0.233 | 8-8 |
So how bad were the USA Today’s predictions? Well, they have been off by an average winning percentage of 0.197 per team. How bad is that? The 8-8 system is currently off by an average of 0.169 per team. On a per-16 game basis, the 8-8 system is off by 2.70 wins per team, while the USA Today’s projections are off by 3.16 wins per team. That is… a lot.
Now recall, in an average year, the 8-8 system will be off by 2.48 wins per team, so this is not a year where teams are huddled around .500 more than usual. If you want to throw the USA Today a bone, you might argue that the standings are more unpredictable than usual (although some data to support that claim would be wise), but they are not more tightly bunched.
But again, the intent here is not to criticize one publication, whose intent may not have been accuracy, and certainly wasn’t accuracy vs. a robot system to be judged by a crazy blogger five months later. The true lesson is to understand that there is a lot of uncertainty in the world, and more concretely, in the NFL. And if your projections are less accurate than a system that says every team will finish with the same record, you should probably rethink your projection system.