A few years ago, the economist David Romer got interested in the fourth down decisions of NFL coaches. He saw the sport as a novel way to test a basic assumption of macroeconomic models, which is that business firms in a competitive environment will maximize expected profits. While executives aren’t perfect, they should, over time, converge on an ideal set of decisions. (After all, that’s why CEO’s and coaches get paid millions of dollars.) Milton Friedman said it best: “Unless the behavior of businessmen in some way or other approximated behavior consistent with the maximization of returns, it seems unlikely that they would remain in business for long. The process of ‘natural selection’ thus helps to validate the hypothesis [of return-maximization].”

What does this have to do with football? One of the most obvious ways a football coach can influence the game comes on fourth down. There are three options: they can go for it, punt or kick a field goal. Romer wanted to figure out the ideal conditions for each option. Then, he wanted to see if football coaches had converged on this ideal. Did the intensely competitive environment of the NFL – a sport in which each win is worth millions of dollars – lead to a maximization of profits?

The first thing Romer did was analyze every fourth down during the first quarter of every NFL game between 1998 and 2000. (He had help from a computer program.) Then, he figured out the fluctuating value of a first down at each point on the football field. After all, a first down was more valuable for a team if it occurred on an opponents two yard line than on their own twenty yard line. The next thing Romer calculated was the statistical likelihood of going for it on fourth down under various circumstances and actually getting a first down. He also calculated the probability of kicking a successful field goal from various spots on the field.

So let’s say you are NFL coach, and you have a fourth and three on your opponent’s 30 yard line. Romer could tell you that 1) you have a 60 percent chance of getting a first down, and that teams with 1st downs inside the thirty yard line score a touchdown 40 percent of the time, for an expected point value of 1.7 and 2) that field goal attempts from the 32 yard line failed almost 65 percent of the time, which meant that going for a field goal only had an expected point value of 1.05. In other words, it’s almost twice as effective to go for it than to attempt a field goal.

So what do most coaches do? They consistently make the wrong decision. According to Romer’s analysis, teams would have been better off going for it on fourth down during the 1st quarter on 1100 different drives. Instead, coaches decided to kick the ball 992 times. This meant that NFL coaches made the wrong decision over 90 percent of the time. Romer summarized his counterintuitive results: “This analysis implies that teams should be quite aggressive. A team facing fourth and goal is better off on average trying for a touchdown as long as it is within 5 yards of the endzone. At midfield, being within 5 yards of a first down makes going for it on average desirable. Even on its own 10 yard line – 90 yards from a score – a team within three yards of a first down is better off on average going for it.” Romer conservatively estimates that a more aggressive approach on fourth downs would make a team 5 percent more likely to win the game. This is a significant advantage: a coach willing to endure the risks would win one more game in three seasons out of every four.

But if kicking a field goal or punting on fourth down is such a bad idea, then why do coaches always do it? To explain the consistently bad decisions of NFL coaches, Romer offered two different answers. The first is risk aversion. If coaches followed Romer’s strategy, they would fail about half the time they were within ten yards of the endzone. This means that instead of kicking an easy field goal and settling for three points, they would come away empty handed. Although that’s a winning strategy in the long-run, it’s hard to stomach. (As Daniel Kahneman notes, “Worst case scenarios overwhelm our probabilistic assessment, as the mere prospect of the worst case has so much more emotional oomph behind it.”) After a long drive down the field, fans expect some points. A coach that routinely disappointed the crowd would quickly get fired.

The second reason coaches stink at making decisions on fourth down is that they stink at statistics. As Romer politely writes, “Many skills are more important to running a successful football team than a command of mathematical and statistical tools…It may be that individuals involved want to make the decisions to maximize their teams’ chance of winning, but that they rely on experience and intuition rather than formal analysis.”

So firms don’t always maximize profits, if only because coaches aren’t rational agents. But I’m most interested in what happens next. Can coaches learn from their mistakes? Can a rigorous analysis of flawed behavior help us correct that flaw? This is the optimistic hope behind much of behavioral economics, which assumes that identifying the irrational quirks of humans allows us to escape those quirks. Thanks to Romer’s analysis, it’s now easy to make the right decision on fourth down.

So how have coaches reacted to this data? In 2001, before Romer published his findings, the average team went for it on fourth down 15.1 times per season. During the 2010 season, the average NFL team went for it on fourth down…15.125 times. Perhaps 2011 will be the year coaches start to maximize profits. But I’m doubtful.

There are a few sad lessons here. For one thing, it appears that NFL teams don’t closely follow the behavioral economics literature, even when it directly involves the sport. But the lack of change in fourth down decision-making is also a depressing reminder that human biases are exceedingly hard to fix. When the game is on the line we default to our lazy hunches and instincts, even when the rational choice couldn’t be more clear.

Image: New York Jets coach Rex Ryan, center, reacts near the end of the Jets’ 28-21 win over the New England Patriots in NFL divisional playoff football game in Foxborough, Mass., Sunday, Jan. 16, 2011. (AP Photo/Winslow Townson)