A.J. Smith knows a perfect throwing motion when he sees one.
Think back to April 2007. A.J. Smith and Bill Polian were widely considered the two best draft minds in the NFL. At the time, Smith’s last three draft classes had been outstanding. He added Philip Rivers (via the Eli Manning trade), Igor Olshansky, Nick Hardwick, Shaun Phillips, Michael Turner, and Nate Kaeding in 2004, and followed that up with Shawne Merriman, Luis Castillo, Vincent Jackson, and Darren Sproles in 2005 and another strong class (Antonio Cromartie, Marcus McNeill, and Jeromey Clary) in 2006.
The defending Super Bowl champions were the Indianapolis Colts, a team that Polian had built from scratch. From 1996 to 2003, Polian’s eight first round picks were spent on Marvin Harrison, Tarik Glenn, Peyton Manning, Edgerrin James, Rob Morris, Reggie Wayne, Dwight Freeney, and Dallas Clark. His most recent first round pick was Joseph Addai, who had 1400 yards from scrimmage as a rookie and 143 yards in the Super Bowl victory over Chicago.
Fast forward six years later, and both Smith and Polian have been fired. From 2007 to 2011, Polian’s first round picks brought Anthony Gonzalez, Donald Brown, Jerry Hughes, and Anthony Castonzo to town. The Colts were without a first round pick in 2008, because Polian packaged his 2007 fourth rounder (Dashon Goldson) and his 2008 first rounder to the 49ers for the the 42nd pick in the draft, which was spent on …. Tony Ugoh. Indianapolis’ second- and third- round picks were even worse over that five year stretch: in addition to Ugoh, the Colts drafted Mike Pollak, Pat Angerer, Fili Moala, Ben Ijalana, Dante Hughes, Philip Wheeler, Jerraud Powers, Drake Nevis, Kevin Thomas, and Quinn Pitcock in those rounds. After Peyton Manning missed the entire 2011 season, and Polian’s combination of Kerry Collins, Curtis Painter, and Dan Orlovsky predictably failed, Polian was fired. He went from scouting genius to draft failure overnight.
Meanwhile, A.J. Smith went from masterful talent evaluator to the most hated man in San Diego in roughly the same amount of time. The Chargers’ first round picks starting in 2007: Craig Davis, Antoine Cason, Larry English, Ryan Mathews, Corey Liuget, and Melvin Ingram. After fielding perhaps the most talented roster in the NFL in 2006, San Diego failed to restock as veterans moved on, and Smith left Rivers with an anemic supporting cast. The Chargers drafted 40 players from 2007 to 2012, and only one — Eric Weddle — has made a Pro Bowl.
There are teams that are good at drafting just like there are players who are clutch and captains who can correctly call the coin toss. The problem is, we recognize that someone who correctly calls the coin toss is just lucky while we label “good drafters” as oracles capable of separating the draft wheat from the chaff. Being a great drafter is simply another example of What Are the Odds of That:
You’ve undoubtedly heard of Wyatt Earp, who is famous precisely because he survived a large number of duels. What are the odds of that? Well, it depends on your perspective. The odds that one person would survive a large number of duels? Given enough time, it becomes a statistical certainty that someone would do just that. Think back to the famous Warren Buffett debate on the efficient market hypothesis. Suppose that 225 million Americans partake in a single elimination national coin-flipping contest, with one coin flip per day. After 20 days, we would expect 215 people to successfully call their coin flips 20 times out of 20. But that doesn’t mean those 215 people are any better at calling coins than you or I am. The Wyatt Earp Effect, the National Coin Flipping Example, and my Splits Happen post all illustrate the same principle. Asking “what are the odds of that?” is often meaningless in retrospect.
It would have been meaningless six years ago to ask what the odds were that A.J. Smith was actually an average (or worse) drafter who just happened to have three excellent drafts in a row. The odds would be very low, but that doesn’t answer the question we really want to answer. Smith was excellent in the draft right up until the moment he stopped being excellent and instead became terrible. If you think of the draft as an efficient market — and I believe it is — then there is no reason to think any one team/GM/scouting department is going to be better than another.
All of that is theory, of course. What about the data? That’s pretty easy. In this post, I graded every draft class from 2000 to 2007. I credited each team with the marginal Approximate Value generated by each player selected subject to two caveats: one, I limited the production of each player to only his AV in his first five years, and two, only production that came with the team that drafted him was included. Finally, I then assigned each team credit for the amount of marginal Approximate Value generated relative to what the expectation was (based solely on draft pick). This way, teams with higher picks did not have an advantage.
Using those same methods, I went back and graded every draft since 1970 (Friend-of-the-Program Danny Tuccitto just did something similar.) That enabled me to compare over 1000 pairs of team seasons and determine how likely teams were to sustain their good or bad draft fortune in consecutive years. As it turns out, the correlation coefficient between a team’s draft grade in one year and draft grade in the next year was just 0.07. This means that there is essentially no relationship between how well a team does relative to expectation in the draft this year to how they did the prior year.
The 1975 Bears had one of the most amazing draft classes in history. With the 4th pick in the draft, they selected Walter Payton, but the depth of the class is the real story. The Bears used a 12th round pick on Doug Plank, who was a starter for seven years. With a 17th round pick, Chicago added Roland Harper, Payton’s starting fullback for five years. The second round pick was used on Mike Hartenstine, a defensive end who played for 12 years with the Bears, most of them as a starter (and he started 8 games on the ’85 team). The fourth round pick was Virgil Livers, a starting corner and returner for half a decade. Two sixth round picks yielded Bob Avellini (started 50 games at quarterback) and Tom Hicks (four-year starter at middle linebacker), while fifth-round guard Revie Sorey started 77 games for Chicago. On a pick-for-pick basis, I don’t know if a team has ever had a higher batting average in a draft (Chicago did not have a third round pick that year), and the Bears received tremendous value from their late round picks.
But guess what? In 1976 and 1977, the Bears were below average. They were the proverbial coin that went “heads” on 9 out of 10 tosses but then came up heads on just 9 of the next 20 flips.
What about the famous Steelers draft class of 1974? With Jack Lambert, Lynn Swann, Mike Webster and John Stallworth, that was obviously an excellent draft. But the season before, their second round pick, Ken Phares, never played a game (to be fair, this was injury-related), while their third and fourth round picks Roger Bernhardt and Gail Clark provided almost no value. Their first round pick wasn’t a bust (J.T. Thomas), but the class as a whole came in far below expectation. And in 1975, it was even worse: not one of their 21 picks ever started a game for the Steelers. Pittsburgh had picks in each of their first six rounds: those players combined to play in just 24 games for Pittsburgh. The Steelers surrounded the greatest draft of all time with two duds.
Some of the low value provided by the ’73 and ’75 classes were no doubt in part because the Steelers were so talented that roster spots weren’t plentiful. Admittedly that’s a small problem, and I don’t know what you can do about that. In general, though, there appears to be almost no relationship between draft years. I looked at the top 50 draft classes from 1970 to 2007. On average, those classes produced 133 points of marginal AV against 56 points of marginal AV, meaning those teams outproduced expectation by 77 points. In the following year, those 50 teams, on average, outproduced expectations by only 4 points of AV. In the prior year, they outproduced by just 3 points of AV.
In retrospect, there are good and bad drafting teams. But in retrospect, there are people who make lots of money picking the stock market and flipping houses, and there are people who lose just as much money on the same endeavors. The true question of whether something is skill or luck is if it is repeatable. I’m not saying the door is closed on the issue, but there appears to be no real evidence that picking winners in the draft is a repeatable skill. If you have any other suggestions for how to measure whether “picking winners” in the draft is a skill, I have the data, so leave a note in the comments. (I’ll also note that I’ve conflated the issues of “drafting well” and “developing players” here, although I don’t know if there’s any way to untangle them.)
That’s not a knock on NFL GMs. In fact, it’s a pat on their collective backs. It’s not that all GMs are stupid or blindly lucky, it’s that scouting is so good that there are no “steals” left to find. If you threw 31 random fans and a GM into a draft, I’m sure that one GM would do very well most years. But graded against 31 other GMs who are focused on the exact same goal, consistently beating the pack is an unrealistic expectation.
If you want to take a look at my draft class grades from 1970 to 2007, they are all available here.
[Update: Some good comments from Brian Burke this morning on the same issue.]