How Great Is Mariano Rivera?

New analytic tools have led us to understand baseball better and in new ways. They have magnified the under-appreciated and shined light on the game's nuances. They have helped clubs work more efficiently and provided their early adapters with an edge on the competition. 

But they have largely failed one area of the game. And that area's name is Mariano Rivera.

There doesn't seem to be any debate today that Rivera is the greatest reliever of all time. He's the all-time saves leader and possesses the lowest ERA relative to the league ever. He's allowed the fewest baserunners per inning in history. He's been well-nigh unhittable in 96 post-season games. He's spun one pitch into 19 dominant seasons. His worst season, at age 37, was merely excellent.

And yet, the analytic tools don't believe he's a Hall of Famer. 

When applied to closers, new analysis asks the following question: where ya been? Appearing for just three outs every other day limits the value of a closer, according to Wins Against Replacement and its ilk, even when his work comes in high-leverage situations. Despite pitching into his 40s, Rivera has just 1200 frames under his belt -- about seven good seasons for a starter. The result is that he has accumulated just 54 wins against replacement, well below the standard for a Hall of Fame pitcher. WAR (or WARP) matters because it's the statistic that comes closest to describing a player's overall value in a single number. 

That 54 is from Baseball-Reference, which is the most credible of the three main Sabermetric sites with regard to relievers. Baseball Prospectus pegs it at 31.4 while Fangraphs is at the median of 38.7. The discrepancy has much to do with whether the measurements use underlying data as the true level of performance; i.e., whether Rivera is given the credit (as he should be) for a career-long outlier BABIP or whether that's dismissed as luck.

By comparison, Fangraphs credits Mark Buehrle with 45.7 wins against replacement. 
 
The problem, of course, is that comparing a closer to a starter because they're both pitchers is like comparing the ACLU to the Teamsters because they're both unions.
Misused and under-utilized though they are, closers can still be fairly analyzed by comparing them to each other. That limits the sample more or less to Rivera's contemporaries, but that should be enough, given his dominance.

Rivera's lifetime ERA+ of 205 is about half-again beyond the next best closer in history, Trevor Hoffman. He has the lowest walk rate, by far the highest strikeout rate, by far the fewest blown saves and nearly double the Win Probability Added of any short reliever in history.

ERA+ measures a pitcher's ERA relative to the league at the time. Rivera has limited opponents to 2.22 earned runs per nine innings at a time when the American League is scoring 4.55 against non-Rivera pitchers.

The bottom line is that this is a SABR problem, not a Sandman problem. Clearly, WAR (or WARP) is not designed to handle closers, or at least to compare them to starters. Whatever number of wins Rivera allegedly added to the Yankees with his bat-breaking contributions, they are not being reflected in the sabermetric record.