ESPN’s Double-Secret QBR Still Shrouded in Mystery

I wasn't the best athlete as a kid, but I was still pretty good in sports. Sports statistics, that is. By third grade, I could calculate free throw percentages, and in middle school, I understood how a hitter's batting average could exceed his OBP. Then one autumn during high school, I went after the Goliath of all sports stats: the NFL's passer rating.

I waited for the first week's numbers to come out — which, back then, took until Wednesday — and went to work convinced the esoteric code could be cracked by using just about every permutation in the limited data population existing after only one game until I could duplicate a QB's rating. Even still, I never did figure it out. It wasn't until the advent of the Internet, which brought arcane information like this to my desktop, that the inner workings of this rating were exposed.

Now it is 2011. I've had an Internet connection for two decades. The passer rating system has been in place for 38 years and under attack for just as long. There's a rallying cry for a new quarterback measurement, and ESPN has embraced the mandate. In a one-hour special that aired last Friday night, the network lifted the shroud surrounding its much-anticipated Total Quarterback Rating, or QBR. Nevertheless, I'm no better off than that high school kid trying to reverse-engineer the old formula while his friends were out picking up girls. That's because ESPN won't disclose its recipe.

Of course, they don't have to. The Soup Nazi stashed away his recipes in an armoire, and Colonel Sanders had his 11 herbs and spices mixed all over the country to proect his formula. Both enjoyed secrecy and widespread acceptance. But that doesn't work everywhere. The Supreme Court can't simply announce that the video gaming industry just beat California by a score of 7-2 and expect us to rewrite laws without a basis for doing so. Sports also need to explain themselves. Imagine watching a football game where one team fights its way across the goal line and celebrates the touchdown while a ref matter-of-factly picks up the ball, spots it on the 20, and brings out the opponent's offense to take over. No signal, no miked zebras explaining the call, just first-and-ten going the other way. That's what ESPN has done.

The network willing to devote a full season to the importance of the quarterback now wants to rewrite the rules on how to measure one, but they've learned a valuable lesson from the rating they hope to supplant: keep it secret. No one can criticize a methodology they can't analyze. Just move the sticks, swap balls, and huddle up.

The website trailer for Friday's "Year of the Quarterback" SportsCenter special shows Tom Brady completing a 3-yard pass on 3rd-and-3 to pick up the first down but losing a point of passer rating in the process. The reason the incumbent system can be mocked is because the three-man committee led by the late Don Smith that devised the rating laid bare its machinations. You know what you get and can independently calculate the impact of every play. On the contrary, it's not possible to exploit QBR's flaws, and that may well be its most endearing quality.

ESPN has defended their position. In an e-mail, Senior Director of Communications Bill Hofheimer told Sports Central, “the formula will not be available because it's based on a complex algorithm and can't be boiled down to a basic formula where you plug in a few numbers.”

The mouthpiece of QBR is the Monday Night Football lineup of Mike Tirico, Ron Jaworski, and Jon Gruden, the special's anchors, who fawned over their employer's new toy to the brink of embarrassment. The MNF cast was also joined by QBR consultant and former NFL QB Trent Dilfer, who was nearly brought to tears by the pure white light of a rating that could reduce all facets of a quarterback's game – passing, running, thinking, clutch play — into one magical quantitative value.

In Dilfer, ESPN sought an outside ambassador to lend credibility to its system. He is sent into the mix as our eyes and ears, which is fine for those of us who can accept that the 14-year NFL veteran, whose 70.2 rating belies his Super Bowl championship, isn't grinding axes against a system that rated him as close to a pedestrian passer as ever the NFL could produce.

So how does Dilfer's passer rating translate to QBR? ESPN can't say, and for good reason. The underlying database of expected results that is a vital component of QBR was built by using NFL play-by-play data of some 200,000 snaps over the last ten years, but specific values can only be assigned by using video evidence, and they've only collected film going back to 2008. Dilfer retired in 2007. For all we know, his QBR may be on par with Sammy Baugh or Joe Montana.

The rating system is the brainchild and effort of the Production Analytics unit of ESPN's Stats & Information Group, a collection of geeks destined for actuarial school had their payday from Bristol, Connecticut not come in. They've allegedly logged dozens of attributes from those films, many involving subjective analysis. Was the pass in the breadbasket, or did the receiver make a circus catch? Did he drop a catchable ball, or tip one that resulted in an interception? How far was the ball in the air? The actuaries like it when the ball goes far, and are loathe to give the QB credit for any yards after the catch.

Dilfer is also cited as a contributing architect, which undoubtedly had special appeal to him. Pennington may have had to entice Mr. Loo into saving humanity by promising him the opportunity to kill 50, maybe 60, people in The Kentucky Fried Movie, but ESPN had Dilfer at one: arch-nemesis Jay Cutler. In one 60-second segment on Negative Non-Throwing Plays, Gruden explains the devastating effects of sacks, which he says happen when QBs hold the ball too long — cut to Cutler being sacked by the Eagles — or when they don't understand the protection scheme — cut to Cutler being sacked by the Redskins — and are exacerbated when they result in turnovers — cut to Cutler fumbling against the Raiders.

It remains to be seen how logistics factor into the deliverability of this product. While passer ratings can be determined real-time and flashed on our televisions along with receptions and rushing yards, video feeds will have to be analyzed and adjudicated by the actuaries back in Bristol. Who will decide if a pass was intentionally thrown away, or if a sack was the result of a blown blitzing assignment, and how fast will that get to our living rooms? How do we know some on-call techie didn't just pick a number from 1 to 100 and feed it to us?

If ESPN wants acclaim, wants to have a part in NFL rules-writing, wants to stock the material that feeds our sports radio the day after a game, they have to become accountable by providing the recipe to their system. That way, we can determine for ourselves how any isolated play affects QBR. The NFL did it for ESPN, and ESPN has exploited that visibility.

It might be amusing to observe a quarterback hitting his tailback on the dead run behind the line of scrimmage knowing he can slip through the outside linebacker for a long gain, only to see his QBR drop in the process. Forget yards after catch; actuaries dig the long ball.