Playoff Selection Committee Faces Real Ambiguity Problems

This week's College Football Playoff spotlight is shining on the fact that college football can be just plain weird at times. Even if you expect the unexpected, you'll still get some surprises.

It's probably for that reason why the College Football Playoff organizers aren't tying the selection committee down to any specific criteria. Stewart Mandel's column today highlights the sort of things the committee will need to take into consideration in regards to strength of schedule:

"They’ve been very clear to us," said Stephen Prather, whose company, SportSource Analytics, is providing the committee with all of its official data. "They do not want an actual number or metric on strength of schedule." ...

Members will be able to look at two teams’ schedules and see not only the records of their opponents and where the games were played but also which teams their opponents did or did not face. ...

"… We could build a fancy algorithm, but kind of how the Supreme Court said you know pornography when you see it, you just know a hard schedule when you see it."

As someone who believes in the ability of stats to overcome fallible human biases, that last quote bothers me a lot. But even when you're composing formulas to determine stuff like strength of schedule, you will run into philosophical questions before the supposedly impartial numbers take over.

Consider Florida's cancelation of its game against Idaho from last week. I know the Gators won't be making the playoff this year, but just imagine that it happened to a real contender. Does dropping a pretty bad Idaho team from the slate make a team's strength of schedule better or worse?

If you measure SOS by the average quality of opponent, the schedule rates tougher without Idaho in it. The Vandals' poor metrics won't be included, so Florida's average opponent ends up tougher without them. The NCAA's schedule strength formula (which is poor) looks at the total wins and losses of all of a team's opponents. By a method like that, losing Idaho will again make the schedule look stronger.

In another way, Florida's schedule strength is slightly weaker without Idaho in it. Brian Fremeau, whose FEI rating is half of the F/+ metric, calculates SOS based on how likely a team is to go undefeated against its set of opponents. Idaho is bad enough that UF's schedule rating is basically unchanged, but every time a team takes the field, it has an opportunity to lose. It is tougher, even if only by a smidgeon, for a team to face a collection of 11 opponents plus Idaho instead of just those 11 opponents. The fact that Idaho is the opponent in question for the cancelled game makes this angle pretty insignificant for this specific scenario, but imagine it had been someone better like a top mid-major or even a mediocre-to-decent Power 5 team—someone below average for the schedule but much closer to it. SOS would show as harder by the average opponent method, but it would show as noticeably easier by the likelihood of going undefeated method. Which one is right?

Now, here's something even more controversial. What happens if referees blow a call and that determines the outcome of a game? This is not some wild hypothetical. It happened in 2006 with the Oregon-Oklahoma game.

Oklahoma had a 33-27 lead following an Oregon touchdown. There was 1:12 on the clock, and the Ducks performed an onside kick as you'd expect. Replays clearly showed both an Oregon player touching the ball before it went 10 yards and OU being the first team to gain possession of the ball. The refs on the field called the reverse of both of those, awarding Oregon with the ball. The latter aspect about possession wasn't reviewable according to replay rules of the season, but the touching-before-10-yards part was. The replay official got that wrong, and Oregon got the ball.

Oregon had only one timeout left, so even with the old 25-second play clock that was in effect at the time, OU could have run most of the remaining 1:06 off the clock and almost certainly come away with the victory. The Ducks were able to go down and score a go-ahead touchdown that was aided by another bad call on defensive pass interference. The ball was tipped at the line, which makes pass interference impossible, but the replay official blew that one too.

Then-commissioner of the Pac-10 Tom Hansen apologized to Oklahoma, saying the referees' mistakes changed the outcome of the game, and he punished the refs involved. That was a nice gesture and all, but it didn't change the fact that the official records logged the game as an Oregon win.

Had the playoff been in effect at the time, this exchange would have mattered a lot. OU won the Big 12 and finished the regular season 11-2. If the refs hadn't bungled that game, the Sooners would have been 12-1 and right near the top of the rankings. Looking at the final BCS standings for that year, we see a pair of two-loss teams in LSU and USC in the No. 4 and No. 5 spots, respectively. A one-loss Big 12 champ almost certainly would have taken that fourth spot away from a two-loss team that didn't even win its division.

Suppose something similar happens during the playoff era. Imagine that refs blow a late critical call that affects the outcome of a game so badly that everyone knows it and the refs get punished. How will the committee deal with that game? In 2006 terms, would they still count it as an Oregon win and an Oklahoma loss? Would they reverse the bad call to the proper one and give the win to Oklahoma for their own purposes despite what the records say? Could they even mentally give both teams credit for a win, acknowledging the bad call but also not penalizing Oregon for taking advantage of the circumstances?

I don't know, and neither does anyone else. The playoff doesn't officially have a policy on scenarios like that. It comes down to the judgment of the individuals on the committee. That's why the credibility of committee members is so important. At some point, they're going to ask us to just trust their judgment.

It's an unenviable position to be in, and the craziness that happens in college football only makes the job even harder.