[Chase note: Take a look at the name at the top of this post. Our good friend Andrew continues to desire to post here, and we thank him for that.]

In 1906, Sir Francis Galton probably wasn’t thinking about the NFL draft when he asked almost 800 fair goers to guess the weight of an ox. No one person accurately guessed its weight, and the guesses were all over the map, but the mean of all the guesses (1197 lbs) was within one pound of the actual weight of the ox. As I looked through endless mock drafts leading up to last Thursday night, I wondered if there was anything to be gained by looking at the wisdom of the crowds. Could we do a better job of predicting the NFL draft if we took all the knowledge and tried to put it together?

And the answer appears to be yes… to an extent. The NFL draft is not exactly a place where we’d expect the wisdom of crowds to be particularly strong. The power of the wisdom of crowds comes from lots of people bringing their own independent information to the table. For example, prediction markets appear to do a great job of predicting events like a president’s chances of being reelected. Sports prediction markets (a.k.a sportsbooks) similarly succeed in predicting game outcomes. And the stock market often reveals companies’ true values. In each case, every individual transaction represents a piece of information which gets reflected in the price.

Of course, the crowd is not always so wise. Stock markets can go haywire. Betting lines can be affected by people’s biases. The wisdom of crowds can break down when groupthink occurs and people stop having independent opinions. The NFL draft certainly looks like such a case. All the mock drafts are out there and the experts have the implicit pressure to not be too different. ^[1]In some cases, there may be incentives to stand out from the crowd with an original prediction, too. Overall, there are incentives that can make predictions depend on those made by others. In those circumstances, we could lose in a haze of groupthink much of the original information that people have.

And it looks like there may be some of that going on, too. Looking across a bunch of mock drafts, it simultaneously looks like the crowd both does better than all but a couple predictions and that there is some important information in those draft boards that look different from the rest.

The Data

I collected 27 mock first round drafts from the internet on May 8. These were the final draft predictions for each individual I collected. I took many of the top ranked forecasters from the last five years on The Huddle Report. I also included a few more of the particularly well-known football media people such as Peter King, Mike Mayock, and Mike Florio. I included some of these guys in part to try to get some heterogeneous opinions.

To construct the Wisdom of Crowds (WOC) draft, I assign each draft pick a total point score. A player gets 32 points for someone predicting them first overall, 31 points for second, and 1 point if someone was predicted to get last in the first round. The idea here is basically to take the mean prediction across all the predictors. So Jadeveon Clowney is the predicted #1 pick because he collected 855 points for his 23 first place votes, 3 third place votes, and 1 fourth place vote.

The Wisdom of Crowds Draft

Here’s the WOC draft. Each player is listed with their total number of points. In parentheses is the player’s actual draft position.

One way to assess the accuracy of the WOC draft is to compute the distance of each of the actual players drafted from their predicted position. For example, Johnny Manziel has a prediction error of 15 because the WOC prediction was for him to go at #7, when he actually went at #22. Ja’Wuan James has a prediction error of 12 because the WOC had him at #31, while he went at #19.

To assign a distance score for the players who were not predicted to go in the first round and for all the first round mock drafts, I use the difference between 33 and where the player was selected, adding a 3 point penalty for not picking the player in the first round. It doesn’t really matter what penalty is picked when we compare the WOC to the individual mocks.

The total distance between the WOC draft and the actual players picked in the first round is 162. Looking at all the other mocks, the WOC finishes tied for first out of the 28 entrants.

So the WOC draft does better than most drafts at estimating where players will end up. There is some important value to be gained in adding up all the information contained across the different mock drafts. But the WOC draft, like the mocks on which it was based, also had some notable prediction failures. The data offer some hints as to why those failures might have happened.

Groupthink and Prediction Failures

When Galton was estimating the weight of the ox, he actually cared about the median guess first. That also turned out to be close to the actual weight, but not quite as close as the mean. There are some reasons to think the median might do better. If somebody guessed 100,000 lbs, that would cause the mean to be off, but the median would still be OK. At the same time, if someone has some valuable information that’s different from the crowd, the mean will do a better job of picking that up.

For the NFL draft, is there any evidence that people who made unusual predictions may have been on to something that the rest of the crowd missed? There are a few examples that suggest this might be the case. Blake Bortles is maybe one example. His WOC prediction was 17 and the only reason it’s that low is because a few people picked him to go pretty early. Three predictions had him going at #8, one at #4, and one #1. These outliers wound up being closer to the truth than the majority of the forecasters. ^[2]Chase’s mock draft also wisely put Bortles towards the top at #6. Of course, he also had Bridgewater at #3, which was slightly less accurate.

Another case is Ja’Wuan James. Four people actually picked him exactly right at #19, while 21 had him out of the first round altogether. It seems likely that the four who had him at the right spot had some valuable information that the rest of the crowd missed. Most likely, those mock drafts recognized that (a) Miami really would want to draft an offensive tackle, and (b) the top tier tackles would all be off the board, and the Dolphins would go with need over best player available.

Finally, consider the graph below for Darqueze Dennard, who went 24th overall to the Bengals. Twenty of the 27 predictions had him going in the top 15. But there was a small cluster of other people who had him pegged almost correctly. Three people pegged him at #25, just one spot off where he landed. It is possible that the crowd herded on earlier spots in the round, while this smaller group had some valuable information about where Dennard would actually go.

Conclusion

The Wisdom of Crowds (WOC) draft performed much better than most mock drafts in terms of putting players near where they were eventually selected. Since the experts are putting those mock drafts out there semi-constantly, those opinions are far from independent. Still, the wisdom of crowds mostly works, subject to some clear limitations. ^[3]It seems almost certain that people shade their mock drafts to match the crowd. This kind of shading of results can happen even with polling data, where it’s just a little bit more nefarious and … Continue reading

Of course, predicting the draft could mean something different. For example, the Huddle Report’s prediction contest gives points for predicting people to the correct team and getting them in the first round. We could think about creating a smarter wisdom of crowds prediction machine that would attempt to achieve the objective of assigning players to teams, too. Here, the prediction machine is pretty simple, just picking the highest remaining player regardless of position. That will still tend to account for much of the variation, but it can put someone at 17 who is actually equally likely to be selected at 14 and 20, for example, with the teams in between interested in other positions.

One of the reasons the WOC draft works somewhat better than the individual drafts themselves is that it incorporates outliers who may have some independent information. It does better than most mocks with players like Ja’Wuan James and Darqueze Dennard where most people have herded on a certain option, but a minority has a very different opinion. Note, too, that the cases of James and Dominique Easley illustrate that we may particularly want to pay attention to people with localized knowledge. Local mock drafts from Miami and New England got those two surprise picks exactly right.