Love the Bowl Championship Series or (more likely) hate it, tonight marks the end of college football’s 16-year BCS experiment. Designed to bring some measure of order to the chaotic state college football had been in under the Bowl Alliance/Coalition, the BCS did streamline the process of determining a national champion — though it was obviously not without its share of controversies either.
If various opinion polls conducted over the years are any indication, the public is ready to move on from the BCS to next season’s “plus-one”-style playoff system. But before it bids farewell forever, how does the BCS grade out relative to other playoff systems in terms of selecting the best team as a champion?
Back in 2008, I concluded that it didn’t really do much worse of a job than a plus-one system would have. But that was more of an unscientific survey of the 1992-2007 seasons than a truly rigorous study. Today, I plan to take a page from Doug’s book and use the power of Monte Carlo simulation to determine which playoff system sees the true best team win the national title most often.
(Note: If you just want the results and don’t want to get bogged down in the details, feel free to skip the next section.)
The Details
The basic framework of this post is essentially a college football version of Doug’s fabled Ten Thousand Seasons NFL post. For each of the 125 FBS schools, I generated a random Simple Rating System value with a mean of the school’s average SRS since 1998 and a standard deviation of 6.44 (the typical FBS school’s yearly variance about its overall average in the BCS era). For non-FBS schools, I assigned a constant SRS of -23.4, which is the implied SRS of all non-major schools since ’98, based on the scores of their games and their opponents’ ratings.
I then plugged all of those ratings into the 2013 season’s schedule, simulating wins/losses for each team based on Wayne Winston’s normal distribution win probability method (if you’re curious, the standard deviation of scoring margin around pregame predictions in college football is 13.89, which is remarkably close to what Hal Stern found for NFL games in the early 1980s).
After simulating an outcome for every regular-season game, I tallied up conference records (with SRS “talent” as the tie-breaker) to determine conference-championship game participants in the ACC, Big Ten, MAC, and SEC. Following those games, I computed a makeshift BCS computer ranking using the same structure as college basketball’s RPI, but with the following weights instead of 25%-50%-25%:
RPI = 75% * Winning Percentage + 5% * Opponent’s Winning Percentage + 20% * Opponents’ Opponents’ Winning Percentage
(These weights were chosen in order to most closely correlate with Jeff Sagarin’s Pure Elo Chess rating, which typifies the kind of computer rankings the BCS uses.)
With that, it was time to program various playoff systems for the universe I laid out above. The following formats were considered:
- Standard BCS – The current BCS system in place for a final time in the 2013 season… #1 plays #2 for all of the marbles.
- 4-Team Playoff – The system that will be in place next season. #1 plays #4 and #2 plays #3 in the semis, then the winners face off in the championship game.
- 6-Team Playoff – In this setup, #1 & #2 have 1st-round byes. #3 plays #6 and #4 plays #5 in the opening round; the worst-seeded remaining team after round 1 plays #1 in the semis, with the other 1st-round winner playing #2.
- 8-Team Playoff – A straightforward bracket that looks like this.
- 10-Team Playoff – In the first round, #7 plays #10 and #8 goes against #9. The worst seed among those winners gets to face #1 with #4 facing #5 in the same half of the bracket; the other winner from round 1 faces #2 in the same half of the bracket with the #3-vs-#6 matchup.
- 12-Team Playoff – A bracket that looks like this.
- 16-Team Playoff – Your standard NCAA Basketball Tournament regional bracket; looks like this.
The Results
For each of the playoff systems detailed above, I ran 5,000 simulations apiece and tracked:
- The average BCS ranking of the “true” most talented team — aka the team with the highest SRS in the simulated universe (note that this is the same in every playoff variant)
- The average talent ranking of the BCS/playoff champion
- How often the BCS/playoff champion is the nation’s “true” most talented team
- How often the BCS/playoff field contains the nation’s “true” most talented team
Here were the results:
| Standard BCS | 4-Team Playoff | 6-Team Playoff | 8-Team Playoff | 10-Team Playoff | 12-Team Playoff | 16-Team Playoff | |
|---|---|---|---|---|---|---|---|
| True Best Tm's BCS Rk | 6.30 | ||||||
| Avg Champ's Talent Rk | 5.69 | 5.06 | 4.74 | 4.67 | 4.56 | 4.72 | 4.94 |
| Champ is True Best Tm | 29.4% | 31.4% | 32.4% | 31.6% | 32.7% | 32.9% | 30.5% |
| True Best Tm in Field | 40.6% | 58.7% | 70.0% | 77.6% | 82.6% | 85.7% | 90.9% |
The purpose of this exercise is to estimate the sweet spot where two factors intersect for a playoff system: the probability of the field actually including the true best team, and the ease of the true best team in winning once they are in the bracket.
The current BCS fares worst among the proposed systems above because it fails in the former category; the best team has a very good chance of winning if they do rank among the top 2 (needing to win just a single game), but the likelihood of them being selected as one of the top 2 teams is relatively low. Even the new 4-team bracket includes the true best team less than 60% of the time.
Meanwhile, the more teams that are included in the bracket, the more likely it is that the true best team will be included — but also the more likely it is that they’ll be upset before winning the championship. (This is the problem faced by the 16-team bracket, which offers the best team as champion at a rate barely above the current BCS despite including the best team in the field more than twice as frequently.)
Balance seems to be achieved somewhere in between. In terms of most frequently seeing the true best team win, the 6-, 10-, and 12-team brackets came out ahead of the rest of the pack, with the 10-team playoff’s champ ranking the highest in true talent on average (coincidentally, Chase’s preferred college football playoff is the 10-team structure). Why? One major reason is that all 3 setups give superior teams a boost via 1st-round byes, something not offered in the 8- and 16-team bracket varieties. As a favored team, it’s always better to not have to play a game than to introduce the chance of an upset, however slim.
The best team won’t always win (or even usually win) no matter which setup is used, but this simulation suggests that moving to a Final Four next season won’t mean college football finally has the most optimal system possible. The powers that be always speak of “bracket creep” as though adding more teams is a bad thing, but here’s hoping they eventually give in to the temptation of a bigger tournament and settle at 6, 10, or 12 teams.