In his response to last week’s news that the Nationals had fired manager Matt Williams, The Washington Post’s Adam Kilgore described a scene from the fourth and final game of the 2014 National League Division Series, when Williams went to rookie reliever Aaron Barrett with one out and two on in a seventh-inning tie. Citing the proverbial “person familiar with the situation,” Kilgore wrote that “Giants Manager Bruce Bochy turned giddily incredulous that Williams had not instead chosen all-star Tyler Clippard, whom Bochy feared because of his experience and ability to face both left-handed and right-handed hitters. As Barrett warmed up, the person said, Bochy expressed to one of his coaches that Williams’s decision had just given the Giants the series.” We’ll never know what would have happened had Williams signaled for someone else, but Bochy wasn’t wrong: Barrett lost the game, walking Hunter Pence to load the bases and throwing a wild pitch to Pablo Sandoval that scored the winning run. The Nats slipped into the offseason without turning to Clippard, Drew Storen, or Stephen Strasburg (who was available out of the pen) for salvation in the decisive game.

We’ve all been Bochy, either giddy or aghast about a major league manager’s decision to commit suicide by bullpen. We all know what it’s like for a soul to be scarred by the memory of a game that might have ended differently had a manager not summoned the last guy in the world everyone watching wanted to see. And we know that these emotions are particularly acute in October, when the importance of every managerial move is magnified.¹ Some managers are known for making mind-blowing bullpen mistakes, but their bad bullpen reps are supported less by numbers than by an amorphous anxiety among fans as the inning gets later and the starter’s pitch count climbs.

There’s no well-known statistic we can use to quantify bad bullpenning. But as the Grinder would say, “What if there was?” After the 2009 season, former Baseball Prospectus author and quantitative consultant Tim Kniker developed a managerial reliever-use rating system, which he dubbed Bullpen Management Above Random. BMAR’s purpose is simple: It’s a way of assessing how closely each manager lined up his best relievers with his team’s highest-leverage relief opportunities. Essentially, it’s a stat that tells us how often a manager’s bullpen moves made fans lie awake and wonder, “Why him?”

Some of the managers who ranked most favorably in the BMAR standings in 2009 (Joe Girardi, Mike Scioscia, and, yes, Bruce Bochy) are still with us, while others (Ron Gardenhire, Ken Macha, Tony La Russa) are no longer active. Several seasons have passed since the original research, so I asked Kniker to bring BMAR up to date by generating ratings for 2012 to 2015.

BMAR rates managers based on their relievers’ Weighted On-Base Average allowed to righties and lefties over the previous 12 calendar months.² BMAR’s baseline — the standard by which all managers are judged — is a team’s theoretical distribution of bullpen outings had the manager assigned relievers using eeny, meeny, miny, moe. To produce the ratings, Tim sums each bullpen’s actual totals of left- and right-handed batters faced and calculates how much more effectively the manager could have distributed the same workload, given the relievers on his roster at the time. You can think of each manager’s BMAR as an answer to this question: In light of the bullpen he had, how much better (in wOBA points allowed) were the relievers he did choose than the relievers he could’ve chosen at random?

As an example of a high-leverage matchup that BMAR deems misguided, consider the eighth inning of an August 18 game between the Dodgers and A’s. With the Dodgers up 4-1, right-handed Dodgers reliever Pedro Baez replaced Clayton Kershaw to begin the bottom of the eighth and allowed three consecutive hits, trimming Los Angeles’s lead to one run. After Baez got Billy Butler to ground out, Dodgers manager Don Mattingly made a sensible choice, calling on lefty reliever J.P Howell to face Oakland left-handed hitter Josh Reddick. But Reddick managed an infield single and then Mattingly made a mistake: He kept in the lefty to face right-handed hitter Marcus Semien, who lined a single to left, scoring one run and tying the game; Oakland would go on to win 5-4 in the 10th inning. It was the first time the Dodgers had blown a three-run lead in the eighth inning or later since 2011.

Commenters in the game thread at Dodgers Digest split their anger between the bullpen and the bullpen management. “Our pen is a random mess,” wrote one commenter, who noted that “JP is horrible with righties” and that leaving him in was “[expletive] dumb.” As Chad Moriyama wrote in his recap at the same site, “Kenley Jansen threw zero pitches, by the way.” Howell’s wOBA allowed to right-handed hitters over the previous 12 calendar months was .356. Compare that to the wOBAs allowed to right-handed hitters over the same span by the other options in the Dodgers’ pen: Jansen (.200), Yimi Garcia (.279), Luis Avilan (.288), Chris Hatcher (.315), and Jim Johnson (.322), who relieved Howell to secure the last out of the inning. From BMAR’s perspective, the optimal choice was Jansen, and the “random,” baseline choice was .291 — the weighted-average wOBA allowed to righties of the bullpen as a whole. BMAR holds the difference between the baseline and Howell against Mattingly, whose error was compounded by the situation’s high stakes. The more make-or-break the spot, the more BMAR makes it matter, and Howell’s ill-fated faceoff with Semien had a Leverage Index of 4.9 (where the average is 1.0).

A couple of caveats: First, while we know which pitchers were on the active roster on a given day, we don’t know which pitchers were actually unavailable (or less effective than usual) because of overuse, illness, nagging injury, or a stubborn latch on the bullpen-bathroom door. And we’re assessing only one aspect of bullpen management: which reliever the manager chose once he decided to make a move. If some managers are better than others at judging when to take out their starters, or at keeping relievers fresh by limiting how often they warm up or enter games — and some of them undoubtedly are — then this method will miss that extra value. And since we’re simply reassigning each team’s actual matchups against left-handed and right-handed batters to the most logically leveraged scenarios based on ability, we’re also overlooking the ability of some managers to gain the platoon advantage more often. But we can judge managers independent of bullpen quality: To adjust for the fact that some managers have better late-game relievers to work with, and can therefore make more of a difference by choosing their best available reliever at a crucial moment instead of their mop-up man, we’ll compare each manager to a personalized standard of how well he could have done with his own array of relievers.

The table below shows 2015 BMAR ratings for every manager, excluding Dave Roberts, who guided the Padres for one game between Bud Black and Pat Murphy. The second column, “Reliever BF,” tells us how many batters the bullpen faced during each manager’s tenure. The third, “wOBA Points Saved,” shows how much better the arms each manager chose were expected to be, relative to random selections. And the fourth, “% of Optimal,” tells us how close each manager came to the maximum difference between the baseline and the best option available on his team — in other words, how close he was to making the perfect move at every opportunity.

Manager	Reliever BF	wOBA Points Saved	% of Optimal
Bud Black	446	.021	38.7
Joe Girardi	2222	.022	37.2
Mike Scioscia	2013	.015	36.5
Ned Yost	2160	.017	35.0
Kevin Cash	2214	.012	34.0
Pat Murphy	1187	.012	32.6
Pete Mackanin	1287	.014	26.8
Robin Ventura	1848	.010	26.5
Clint Hurdle	2102	.010	26.0
Bryan Price	2321	.010	25.8
Buck Showalter	2137	.012	24.7
Dan Jennings	1637	.013	23.7
A.J. Hinch	1830	.007	23.4
Ryne Sandberg	1067	.010	22.6
Paul Molitor	2130	.009	21.1
Torey Lovullo	304	.011	19.5
Bob Melvin	2141	.009	19.0
AVERAGE	—	.008	18.2
Joe Maddon	2123	.008	17.9
John Farrell	1835	.008	17.1
Jeff Banister	2124	.007	16.9
Don Mattingly	1947	.008	16.8
Lloyd McClendon	2200	.009	16.1
Matt Williams	1939	.008	16.1
Brad Ausmus	2161	.008	15.2
Mike Redmond	516	.008	14.2
Bruce Bochy	2031	.006	13.4
Terry Collins	1884	.005	12.7
Fredi Gonzalez	2083	.005	9.8
Walt Weiss	2460	.004	8.1
Terry Francona	1875	.002	6.4
Craig Counsell	1850	.003	6.2
Ron Roenicke	316	.002	4.5
Chip Hale	2385	.001	2.3
John Gibbons	1907	.000	1.4
Mike Matheny	2011	-.005	-18.3

Among all managers, there isn’t much of a year-to-year correlation in BMAR, which suggests that a single season’s ranking isn’t all skill. Still, certain names tend to rise to the top or sink to the bottom year after year. The following table shows 2012 to 2015 weighted averages for every manager who was active in at least three of those four years. The first thing to notice is that even the best managers didn’t come close to what BMAR considers optimal bullpen usage, which would include unheard-of ideas like bringing in the closer in non-save situations, even — brace yourself for some serious heresy — tie games on the road. This might explain why fans of almost every team seem so convinced that their manager has no idea what he’s doing. The second thing to notice is that 2009 standouts Girardi and Scioscia are still at the top. The third thing to notice is Mike Matheny, whom BMAR says was actually worse at running his bullpen than a random number generator wearing a warm-up jacket. You might remember Matheny from his odd last act of the 2014 season, which was using struggling starter Michael Wacha, for the first time in weeks, in the ninth inning of a tied elimination game, instead of a rested closer whom he ruled out because of bullpen orthodoxy. If we believe BMAR, the Cardinals would have had better bullpen matchups this season had Matheny made his moves by closing his eyes, spinning around 10 times, and gesturing in the pen’s general direction. (Maybe Matheny’s new iPad will help.) All snark aside, though, these rankings are another reminder that the manager’s impact pales in comparison to the quality of his players. Matheny and the Blue Jays’ John Gibbons may have been among the worst bullpen managers this season, but they didn’t prevent their teams from taking division titles.

Manager	wOBA Points Saved	% of Optimal
Joe Girardi	.014	28.8
Mike Redmond	.010	26.0
Bud Black	.013	25.7
Bruce Bochy	.012	25.5
Ned Yost	.011	24.1
Mike Scioscia	.011	24.0
Buck Showalter	.010	23.2
Clint Hurdle	.008	22.4
Bob Melvin	.010	22.3
Fredi Gonzalez	.011	21.9
Joe Maddon	.009	21.3
Don Mattingly	.009	19.6
AVERAGE	.009	18.9
Robin Ventura	.008	18.4
Ron Roenicke	.008	18.0
Ryne Sandberg	.007	17.7
John Farrell	.009	16.7
John Gibbons	.006	14.8
Terry Francona	.005	13.1
Walt Weiss	.005	11.3
Terry Collins	.005	11.1
Mike Matheny	.003	7.5

Matheny — who told USA Today’s Ted Berg in April that he uses three-year statistical stats as a “tiebreaker” when making bullpen moves but believes that managers “first and foremost have to trust our gut” — ranks at the bottom over a multiyear sample as well as in 2015. Mets manager Terry Collins and Toronto’s Gibbons, who made the perplexing decision to use David Price in low-leverage relief in Monday’s Game 4, aren’t far behind. But Girardi is BMAR’s golden boy.

Girardi is already regarded as a bullpen whisperer; BMAR just clinches the case. In April 2014, Fox Sports reporter Ken Rosenthal passed on the praise of a “rival AL exec” who’d told him that Girardi was “the best [he’d] seen at managing a bullpen.” The BBWAA is onboard: Girardi (who won the NL Manager of the Year Award in 2006) hasn’t finished lower than sixth in AL MotY voting since 2008, even though his high-payroll teams have missed the playoffs twice. After eight seasons in the country’s largest media market, Girardi’s bullpen use still gets a seal of approval from both bloggers and tabloid beat writers, two tough-to-please constituencies. It’s even more impressive that Girardi leads in BMAR given his reluctance to use relievers three days in a row, which limits his options in some high-leverage situations: This season, the Yankees ranked 29th in reliever appearances on zero days’ rest, but they also ranked first in multi-inning outings. For what it’s worth, the Yankees have exceeded their Pythagorean win-loss record by an average of two games per season during Girardi’s years at the helm, as have the Angels during Scioscia’s 16 years in Anaheim. Bochy’s teams have exceeded their Pythagorean records by an average of one win during his 21 seasons.

With Girardi’s Yankees eliminated from the postseason, the best remaining bullpen manager by BMAR belongs to the Royals. (So much for the notion that stats have nothing nice to say about Ned Yost.) Black, a former pitcher and bullpen coach, appears to have been something of a bullpen savant,³ and Mike Redmond, another manager who lost his job in 2015, also seemed to excel in this area. Buck Showalter, who distinguished himself by being unusually flexible with his closer and setup men in the 2014 ALDS, rounds out the top seven on the multiyear list.

According to Kniker’s wOBA-to-wins conversion, the difference between the best and worst manager by BMAR in a given season is on the order of four wins: not enough to turn a good team into a bad one (or vice versa), but more than enough to make the difference in a tight race (or a must-win game), even though there are elements of bullpen management that this method still can’t capture. So if you believe that Girardi deserves partial credit for leading the Yankees to winning records with negative run differentials from 2013 to 2014, or for winning a wild-card spot this season with Andrew Miller and Dellin Betances, BMAR has your back. And if you’re a Cardinals fan feeling a little extra Matheny-related agita about today’s must-win Game 4 against the Cubs, BMAR won’t bring you back from the ledge.

Thanks to Tim Kniker, and R.J. Anderson and Rob McQuown of Baseball Prospectus for research assistance.

1. Jeff Weaver, anyone?

2. In its original incarnation, BMAR used only single-season stats. This is an improvement, although 12-month splits are still subject to small-sample alerts.

3. And maybe a master of mitigating the effects of fatigue.