If you watch Tuesday’s 30 for 30 Elway to Marino, which I strongly advise that you do, you will hear the story about how Dan Marino fell to the 27th pick of the first round of the 1983 NFL draft. NFL teams had the opportunity to draft one of the greatest quarterbacks in league history 26 times, and passed. Five quarterbacks came off the board before Marino, and while you can forgive the Colts and Bills for drafting John Elway and Jim Kelly, respectively, the other three guys don’t quite match up. Given the chance to draft a franchise-defining Hall of Fame quarterback, the evaluation processes of three other teams somehow led to the selections of Todd Blackledge, Tony Eason, and Ken O’Brien ahead of Marino. Taking a player like Blackledge — who couldn’t complete 50 percent of his passes, threw more interceptions than touchdowns, and started just 29 pro games — over Marino is just about the biggest draft mistake a team can make. It implies they weren’t able to tell the difference between a legendary quarterback and a middling backup before the two started their professional careers.
The documentary features interviews with teams who passed on Marino in an attempt to understand why Mr. Isotoner fell so far, and while I won’t spoil what they have to say, their logic isn’t always sound. Many cover their asses or suggest they were smarter than the rest of the league and rated Marino highly, but I have my doubts. There’s enough evidence from the video footage and newsreels included from 1983 to make it clear that Marino was seen at the time as a bottom-of-the-first-round quarterback. And of course, we don’t need to go back to 1983 to find a situation in which teams drastically misevaluated a crop of quarterbacks in the draft; just last year, Russell Wilson fell to the 75th pick, with every single organization in the NFL passing on him at one point. Five other teams took quarterbacks before him. How could the draft process somehow suggest that Brandon Weeden was a better prospect — 53 picks better, to be exact — than Russell Wilson?1
Misestimations that dramatic raise a question that’s worth discussing: Do NFL teams actually have any idea what they’re doing on draft day?
I think it’s clear the league as a whole can do a decent job of sifting through the draft pool and separating guys into relative tiers; in other words, that they can differentiate between a guy worthy of a first-round pick, one better suited for the fourth round, and a future undrafted free agent who isn’t worth the time. There will always be exceptions, but as Kevin Meers’s research on the NFL draft shows, the average value produced by a player from a given round drops with each passing set of selections. You’ll get the occasional lunatic who insists on taking a punter over a franchise quarterback because he would rather take a starter than a backup, but as a collective entity, the league does an acceptable job of putting guys in buckets.
When it comes to valuing draftees at the same position who are clumped together in a similar tier, I’m skeptical. I wonder whether the margin of error in analyzing those players is larger than any advantage a particular franchise has in scouting talent. No organization leaves draft day thinking it’s come away with a crop of useless busts, of course, but there are teams that have terrible drafts each year. It isn’t always the dumb teams, either. Baltimore might have the best general manager in football, but Ozzie Newsome still used a first-round pick on Kyle Boller in 2003. And in 2007, as the much-maligned Matt Millen used the second overall pick to draft Calvin Johnson,2 two of the league’s most talent-rich organizations were using picks at the bottom of the first round to draft wideouts of their own. Both Craig “Buster” Davis and Anthony Gonzalez failed to launch in the NFL, and the genius general managers who drafted them — A.J. Smith in San Diego and Bill Polian in Indianapolis, respectively — are no longer in charge of their teams. Even our perceptions of what a “smart” organization looks like can change pretty quickly.
Smart folks have looked into this topic and raised their own doubts. Wharton professor Cade Massey published a study suggesting that there’s no difference in the drafting ability of teams after you take a long-term view and account for the expectations of the picks they have. Football Perspective’s Chase Stuart wrote a piece on Monday that found virtually no relationship from year to year in a team’s ability to produce drafts that would exceed expectations. Stuart notably pointed out how legendary drafts from given teams would tend to be surrounded by disappointing ones. Take Pittsburgh’s famous 1974 draft, one that produced four Hall of Famers with its first five picks. That’s a truly amazing performance by the Pittsburgh front office, but its 1973 draft was middling and its 1975 performance was downright poor, as its 21 picks didn’t produce even a single start in a Steelers uniform. If Pittsburgh was so great at scouting in 1974, why was it awful in 1975?
The limitations to these studies come with the fact that it’s extremely difficult to define “value” and “return” for NFL players across different positions. Because the responsibilities and output of a quarterback are very different from that of a defensive tackle, judging those two positions using one number is a frustrating task. You could use games started, but that artificially makes draft picks from bad teams look better while hurting selections from good teams. (That helps explain the 1975 Steelers draft; there weren’t many starting spots left on those teams.) It also flattens the value of positions and you end up with somebody like David Diehl — a passable offensive lineman who started every game during his first seven seasons after being taken in the fifth round — as the greatest draft pick of the aughts. The studies conducted by Meers and Stuart use the Pro-Football-Reference.com statistic Approximate Value, which guesstimates its way to providing each player on a team with a measure of his likely contributions. A statistic like “guaranteed money in second professional contract” would be a better metric, but even that can go screwy. So the numbers are a good starting point here, but just as Massey and Stuart get to the causes of why there doesn’t appear to be much difference between each team’s drafting ability, it’s more important to understand the theoretical reasons why each team would know so little about the prospects upon which they stake their respective futures.
Why Wouldn’t They Know What’s Going On?
Scouting is tough.
Well, there’s more to it than that. But it starts with the issue of imperfect information. If you gave teams an infinite amount of time to scout an infinite amount of information about each player, they would probably do a nearly perfect job of drafting. That’s obviously not the case. Scouts have to sleep. General managers balance the obligations of the draft with free agency. Meddlesome owners occasionally insist on stumping for a player.
More disconcerting is that teams really don’t see very much of a player in college. If a player starts for four years and doesn’t get hurt, he might end up playing 50 games across his college career. Assuming he plays for a decent school in a big conference, 10 of those games will come against significantly inferior competition, teams from I-AA or the worst independent schools. Another 15 will be against teams in the bottom of the conference, producing meaningless statistics against players who will be selling insurance in a year. So that leaves 25 games of possibly meaningful film against competent competition. With liberal roster sizes and more frequent substitutions, maybe a guy gets 40 snaps per game. That’s 1,000 snaps over four years, and that’s assuming his freshman or sophomore film looks anything like the guy who finished up at school. Maybe you knock that down to 600 snaps. If you want to scout a specific skill — say, a linebacker’s awareness in zone coverage — you’re ignoring run plays or pass plays in which he was in man coverage. Now you’re down to 150 snaps, which might even be in a 3-4 when you play a 4-3. And remember: That’s the best-case scenario. Imagine what you look at for a pass rusher who had one big year and left school early.
The innate nature of the draft process also ensures that players will come out at the peak of their draft stock far more frequently than they will at the nadir. You get the occasional example of a player coming out early because of financial or medical hardship with his stock dipping (like Rob Gronkowski), but that’s a rare exception. In most cases, players will leave when they look their absolute best. You may only see their peak without seeing any possible regression. As an example, take this made-up pass rusher and his sack totals from year to year:
In the first scenario, he works his way into the starting lineup, shows some promise, and has a breakout season as a junior. The sky is the limit for this kid; he keeps getting better, and he could register double-digit sacks and be a monster pass rusher at the pro level. Maybe he goes in the middle of the first round. Player 1A has the same career path, but he doesn’t turn pro after his junior year; instead, he goes back to school to try to win a national title and he has a middling season. Now his big junior season seems like a fluke. Your area scout hears whispers that he let the attention get to his head and he took a few plays off against Bethune-Cookman.3 That first-round pick looks a lot more like a third-rounder after his senior year. When a guy comes out early, not only do you lose an opportunity to see more film, you miss out on more information that helps define what sort of talent he has. This is an abstract example, but there are plenty of examples of players who went through this same wringer — just ask Matt Leinart or Brian Brohm. On the flip side, guys like Devin Thomas were able to parlay a lone year of above-average play into a relatively high draft position without ever revealing that they weren’t more than a one-hit wonder.
Even that process of wanting to know as much about each player as possible lends itself to bad logic. As human beings, we naturally want to be proven right. The evaluation process lends itself to trying to find some little piece of information that creates a stronger opinion than the crowd about a given player. Maybe you find some way that a receiver runs his routes that you think you can fix in two days at camp, turning him into a superstar in the process. The scouting system is set up to fall in love with a particular player.
Often, there is value in the asset that nobody ever argues for: the unknown. That’s why those 2-for-1 trades end up looking horrific for the teams who gave up a future first-round pick for a second-rounder. It’s easy to make the case for a tangible asset now, even if it’s a second-round pick. It’s far more difficult to look into the future, where a far superior player might wait if your team is just brave enough to hold onto its pick. When Denver dealt a future first-round pick to draft Alphonso Smith in the second round of the 2009 draft, it was able to see an undersize cornerback with speed and attitude who could possibly work out of the slot for them as a rookie. By the time the 2010 draft rolled around, the Broncos had decided to change defensive schemes and move to a 3-4; Smith would be dealt by the end of training camp for a third-string tight end. That asset they had given up became a far superior player, Pro Bowl safety Earl Thomas. The Broncos didn’t know they could get a Pro Bowl safety by waiting a year, of course, but if they really trusted their scouting sense enough to justify dealing a future first-round pick for a second-rounder, why wouldn’t they trust those same scouts to find a guy who would be significantly better in the first round of the following year’s draft?
That relates to the moral hazard of sports, something I brought up with regard to Marty Hurney’s tenure in Carolina early last season. In short, the best interests of your organization and the best interests of your general manager aren’t always aligned; in fact, they can be dramatically opposed. Let’s say that you’re a GM who presided over a veteran team that came close to winning a Super Bowl without ever getting there. After a couple of conference championship trips, you go 9-7 and get dumped out in the first round, and then follow that with a 7-9 campaign. The best thing for your organization might be to rebuild, to dump salaries and start fresh with a new crop of talent. Unless you have a particularly keen owner, you’re going to lose your job as part of that rebuilding process. Instead, to try to get your team over the hump and keep your job long enough to earn a contract extension, you’ll likely go out and spend heavily in free agency and add a big contract or two, even if it ensures that your team will be in cap hell three years down the line. Who among us would be strong enough to resist that temptation? Great organizations like the Patriots and Ravens don’t operate this way, which is one of the reasons why those organizations succeed.
Finally, it’s close to impossible to extricate the gap between good scouting and good development in determining how “well” a team drafts. Do the Steelers consistently find great talent at linebacker because they draft well, or is it because they play a stable defensive style, have a great coordinator, and do an ideal job of integrating their young talent into the lineup? It’s difficult to tease out because we don’t get to see a player come into the league for more than one team. We’ll never know what LaMarr Woodley would have turned into if he had been drafted by the Saints as opposed to the Steelers. That’s the reality of sports, and it colors how we view players in hindsight. Tuesday night’s documentary notes how Don Shula was the perfect coach for Dan Marino. You might argue that Don Shula was the perfect coach for a lot of guys. What would have happened if the hometown Steelers had taken Marino with the 21st pick of that draft? Would Marino have struggled under the pressure of playing at home and replacing Terry Bradshaw? What if he went to some team in the top 10 with little track record of developing quarterbacks or winning, while the Dolphins ended up taking Tony Eason at the end of the first round? Would he have had a better career than Marino? We assume not, but how can we be sure? After all, this documentary makes it very clear that people were not sure about how these quarterbacks ranked below Elway before they were drafted.
What If One Team Really Does Draft Better?
If it really is true that one team doesn’t draft significantly better than another, the course of action is actually pretty simple. Massey lays it out in his study: Give yourself as many chances as possible to get it right. In other words, amass as many high draft picks as possible. Massey would be close friends with Jimmy Johnson, then, since Johnson followed that strategy to a T in building the Cowboys dynasty of the early ’90s.
That strategy suggests you’ll do a better job of evaluating drafted talent when they’re in your building and practicing with your coaches in your schemes of choice every day, which makes sense. On the other hand, of course, it’s even possible to screw up when you have the talent right in front of your eyes. The Falcons traded away Brett Favre. The Chargers cut Wes Welker. More recently, the Cowboys buried Miles Austin on the back of the roster for years, but even though he was wildly productive when given a chance, it took an injury in front of him to get him significant playing time.
Wait, Is That Possible?
I’m taking a position in between the numbers and the scouts. I can’t imagine that Matt Millen and Ozzie Newsome would be equally effective if you gave each of them an expansion-caliber roster and told them to draft for 100 years. On the other hand, there’s plenty of evidence telling us — at least on an anecdotal level and on some quantitative levels — that teams routinely do a subpar job of judging college talent as it enters the league. I think there is a difference between the relative scouting abilities of teams, but that difference in ability is probably one the media, fans, and those respective organizations overstate.
