My opinion has long been that football teams are too scared to take an action that is smart but opens the coach to criticism. So instead of attempting to make it on 4th down (if you don’t understand American football, just skip this post) they punt because that is the decision that is accepted as reasonable.
So instead of doing what is wise they do what avoids criticism. Fear drives them to take the less advantageous action. Now I have never looked hard at the numbers, but my impression is that it is well worth the risk to go for it on 4th down often. In a quick search I don’t see a paper by a Harvard professor (this article refers to it also – Fourth down: To punt or to go?) on going for it on 4th down but I found on by a University of California, Berkeley economist (David Romer wrote called “Do Firms Maximize? Evidence from Professional Football.”).
My guess is that the advantages to going for it on 4th down are greater for high school than college which is greater than the advantage for the pros (but I may be wrong). My guess is this difference is greater the more yardage is needed. Basically my feeling is the variation in high school is very high in high school and decreases with greater skill, experience and preparation. Also the kicking ability (punting and field goals) impacts the choices of going for it on 4th down and that dramatically increases in college. So if I am correct, I think pro coaches should be more aggressive on 4th down, but likely less aggressive than high school coaches should be.
But in any event the data should be explored and strategies should be tested.
I am not at all surprised to see very competitive and well studied situations not being optimized by very smart individuals. We often just keep doing what is expected even when the evidence is not there to support the wisdom of such actions. There are lots of things managers should have stopped doing decades ago that they still do. There are lots of things managers should have started doing decades ago that they still don’t do. Evidence based management is still rare.
It does seem a few coaches over the last decade or so have been more willing to go for it on 4th down. They tend to be very successful coaches, probably less fearful of being criticized – including Les Miles (LSU), Nick Saban (Alabama), Bill Belichick (New England Patriots).
The psychology is more complex than just fear. It is also a common trait to view errors of commission as worse than errors of omission. Even though the decision to punt is a decision (one of commission) since the default accepted action is to punt the decision to go for it is seen as commission while not going for it is just seen as omission. This adds to the reluctance to go for it on 4th down.
I believe many organizations fail to take risks due to fear as well as people seeing failure due to commission as worse than failure due to omission. And that most organizations are far too slow to continual improve – too tied to old ways of doing things.
Related: Randomization in Sports, my father’s suggestion – Why Don’t Football Players Just Thrown the Ball Out of Bounds to Stop the Clock – My Father Had Suggestions for Football and Basketball Coaches (but didn’t find willing coaches) (Dad also liked the idea of going for it more on 4th down) – The 4th Down Study by Brian Burke – posts on sports on the Curious Cat Science blog – University pay rates (football coaches at the top) – Accept Taking Risks, Don’t Blithely Accept Failure Though