6 ms·

How Coaches and the NYT 4th Down Bot Compare

50 points by trevin 13y ago

Uhhrrr 13y ago
Hm, so the data points for likelihood of getting another first down is based on data from the fourth downs that coaches actually attempted? It may be that the fourth downs they didn't attempt were not as propitious for some reason - wind (which the authors mention), injuries on key players, lateral positioning which doesn't favor one formation or another, etc.
- jhermsmeyer 13y ago
  Agreed. The sample is very likely tainted by factors that made success much more likely than average if we believe that NFL coaches are both conservative (they forgo punts only when the odds are much in their favor) and rational (they are win maximizing). Still, there is probably a case to be made that coaches should be a bit more aggressive, but that edge could be small enough that other factors (like perceived incompetence) dominate it.
- cheald 13y ago
  The choice to go for a 4th down conversion is more likely than not primarily decided by the current score and time left in the game. However, the success of a 4th down conversion probably isn't heavily influenced by those factors (other than perhaps player fatigue), which would suggest that coaches are indeed too conservative.
  - 001sky 13y ago
    is more likely than not primarily This seems pretty loose. I'm not trying to be pedantic, but the theoretic flaws in the data are such that you would want a pretty tight logic in your model to have any level of comfort with it. The proportion of the state-space that is unobserved seems ~large and critically relevant.
- zeroonetwothree 13y ago
  No, it's based on third down data. The likelihood of converting 3rd and 1 is probably similar to that of converting 4th and 1.
  - Uhhrrr 13y ago
    Thanks! I read poorly the first time.
timfrietas 13y ago
This does not take into account an important factor that is somewhat intangible, but real to the psychology of the game: point swings. If you go for it on 4th and 1 at your own 1 and lose you are definitely down 3 points and probably 7. This puts pressure on your offense--which just failed on a 4th and 1--to somehow make it all the way down the field next time. That can be demoralizing for that offense, paralyzing in fact. That is much harder to model, but an important part of a correct model, I believe.
- aero142 13y ago
  This actually seems measurable. Do teams that are behind perform worse on average? Do teams perform worse than normal after failing a 4th down attempt?
  - twoodfin 13y ago
    The same statistics wonk has done a lot of work on "momentum" in games. His conclusion is that there is no such thing: http://www.advancednflstats.com/2013/11/momentum-1.html http://www.advancednflstats.com/2013/11/momentum-1.html That jibes with my own anecdotal experience watching a lot of football, despite how often announcers like to cite it as a factor.
Zimahl 13y ago
Why would you ever go for it on 4th and 1 inside your own 10 yard line? You would put yourself at a severe disadvantage if you failed. Since the data from this is from previous plays in the last 10 or so seasons it is flawed. There probably isn't very little data of a team going for 4th and 1 and that deep in their own territory. The only time this has probably happened is in extraordinary circumstances - specifically in a game-winning drive scenario with time expiring, where the defense is in 'prevent'. In these cases it is highly successful in that situation but you don't have the time to get the score.
- thejteam 13y ago
  In place of 4th and 1 data AdvancedNFlStats would substitute 3rd and 1 stats. And I would agree that this substitution is suspect. Just a bit shy of bogus in fact.
  - Zimahl 13y ago
    Well, getting a yard in the NFL isn't typically a big problem and 3rd and 1 vs 4th and 1 are feasibly no different. But turning the ball over inside 30 yards is at least 3 points for the other team with only rare exceptions.
    - omegaham 13y ago
      I think the biggest difference is that on 3rd and 1, the defense has already assumed that they're going to get a 1st down. The main job of the defense is to prevent a big gain and then get ready for stopping the offense on the next four downs. I wouldn't say that they're cheesedicking it, but they're not going all out. Meanwhile, on fourth down, the defense is amped up and ready to roll. You'll usually see a strong blitz against a sputtering offense, and the percentage of plays that actually get a yard is going to be lower. Not only that, going three-and-out means that the offense is unlikely to do any better going from the 10 to the 20 yard line. Better to punt the ball, trust your defense, and have your offensive coordinator fix whatever is going wrong.
- vacri 13y ago
thejteam 13y ago
This was discussed some in https://news.ycombinator.com/item?id=6728821 https://news.ycombinator.com/item?id=6728821 I've thought about it some since then and I'm also not convinced that expected points is a good metric. That will maximize the expected point differential per season but I am not convinced (although I could be wrong... I haven't put pencil to paper to make my thoughts rigorous) that there are enough possessions in the game on average to make this metric useful even in the first half.
- gerad 13y ago
  The best defense is keeping the other team's offense off the field. Going for it on fourth down has a much higher probability of doing that than a punt does.
kosei 13y ago
Tough to take everything into account, but I find it troubling that all non-success/failure outcomes are not part of the calculation (ex: 8-yard gain on a 4th and 10; a fumble on a 4th and 2 run, etc). It simplifies the formula to the layman, but I have to imagine that it adds enough additional variance to matter, especailly considering we're talking in variances of less than half a point in their stated example. Also (and to their point), there is a ton of variation between offenses, defenses, punters, running backs, etc. Using this as a rubric is a nice idea for a rule of thumb for an armchair quarterback, but I'd strongly disagree with using it to accurately criticize any decision within a couple points of expected value.
- altoz 13y ago
  This is especially true of 4th down plays. A surprise play-action pass on 4th and 1 can go for an especially long gain and thus should have higher expected value. There's also the fake-punt scenario, which I suspect tends to be more successful than a regular 4th down.
  - kosei 13y ago
    Absolutely. And I'm sure if coaches started trying to run more 4th down plays, that would dramatically change the algorithm as well.
slg 13y ago
These type of articles never take into consideration the reason why coaches don't go for it as often as a computer might. The coaches face a reverse moral hazard when taking risks because the negative outcomes are multiplied for them. Even if the statistics support their decisions, a coach will be fired for taking too many risks and failing. Is gaining an extra 2% (or whatever) in expected returns worth the risk of losing everything if the low(er) percentage outcome does actually happen? It is the same idea behind the whole "no one gets fired for buying IBM/Microsoft". Until you get a coach's motivations more in line with the motivations of the team, you won't get a coach that is as aggressive as these computer simulations suggest they should be.
- symbaton 13y ago
  People always say this, but it is actually true? Has any coach ever been fired for missing too many fourth downs?
  - slg 13y ago
    I can't think of one off the top of my head. Mike Smith was under a lot of pressure a little while back for his aggressive decisions on 4th down (that were statistically correct IIRC) but wasn't actually fired. However, the obvious counter to your question is how many coaches really go for it frequently on fourth downs? Most coaches just follow the conventional wisdom unless they are already in a desperate situation like Ron Riviera or they have built up a lot of good will like Bill Belichick.
    - hkmurakami 13y ago
      So it sounds like fans and the media need to grasp EV first before coaches will be allowed to take the appropriate risks.
  - judk 13y ago
    Not even missing too many. Just missing more visibly, compared to coaches with equivalent records.
  - 13y ago
vacri 13y ago
There's an interesting hole in the data at 4 & 10 on the opponent's 36. In a sea of punting and kicking, this one data point has actual coaches going for it. Also, looking at the overall pattern of the data, my gut tells me that the optimal rubric is somewhere in-between the 'bot and the meatcoach. There are some strange holes as well - at 4th&3, you should go for if you're on your own 8 or 9, but punt if you're on your own 7 or 10?
- judk 13y ago
  Without confidence levels, the differences are meaningless. Maybe that's the flip from 49% to 51%
- thejteam 13y ago
  That makes sense. At the 36, it is a 53 yard or so field goal. At the outer range of most kickers and pretty low percentage, so you give the ball to the opponent at the 40 something yardline with high probability. On the other hand, its also close enough to be very difficult for a punter to pin in inside the 5 or 10, so even if you fail you only give the opponent 16 yards over a punt. But the holes at the your own end you mentioned, that makes no sense at all.
reddog 13y ago
Here is a high school coach from Little Rock that never punts, and always onside kicks. His reasoning is sound and it seems to work out for him. http://www.youtube.com/watch?v=AGDaOJAYHfo http://www.youtube.com/watch?v=AGDaOJAYHfo
dfan 13y ago
It is interesting what an emotional reaction people instinctively have to 4th down. Going for it on 4th down and failing feels like a catastrophe. But punting would have given the ball away anyway, so all you've done is give the opponent an extra 40 yards or so. Which is bad, sure, but it's a hell of a lot less bad than, say, throwing an interception on 1st and 10. But somehow giving up the ball on downs has come to be seen as a massive failure, on par with an regular turnover.
- snowwrestler 13y ago
  Turning it over on downs is a turnover, as sure as fumbling is. It's not accounted that way in the box score, but in terms of impact on the game it is the same. An interception is better or worse than turning it over on downs by the field position delta. If it is returned past the line of scrimmage, then it's worse. If it is not returned, it's better. A deep interception with no return is like a punt in terms of impact on the game.
snowwrestler 13y ago
The fundamental flaw with this analysis is that statistics are descriptive, not predictive. You can't reliably predict the future with statistics alone. If you could, then technical analysis of the stock market would be the most successful investing strategy. It is not. Likewise, a lot of folks seem to think that taking an historical statistical trend and extending it into the future is a scientific approach to football coaching. It's not. Scientists use statistics to understand evidence, but they formulate theories in terms of root causes. In other words it's not enough to know that dropping a hammer has a high correlation with acceleration of 10 meters per second squared. If you want to predict how fast other objects in the future will fall, you need a theory that describes why and how anything falls. It doesn't look to me like the 4th down bot embodies or implements any cohesive theory of football success. It just finds trends and dots the line into the future.
nova22033 13y ago
It's the same thing with penalty kicks in soccer. Statistically, you are more likely to score if you kick the ball down the middle. The goalie is going to go one way or another and there's a 50% chance he'll guess right. So why don't players kick it down the middle..because if they kick it down the middle and the goalie manages to block the kick, they'll come off looking like idiots and it would look real bad on the highlight shows.

50 points by trevin 13y ago

Hm, so the data points for likelihood of getting another first down is based on data from the fourth downs that coaches actually attempted? It may be that the fourth downs they didn't attempt were not as propitious for some reason - wind (which the authors mention), injuries on key players, lateral positioning which doesn't favor one formation or another, etc.

jhermsmeyer 13y ago
Agreed. The sample is very likely tainted by factors that made success much more likely than average if we believe that NFL coaches are both conservative (they forgo punts only when the odds are much in their favor) and rational (they are win maximizing). Still, there is probably a case to be made that coaches should be a bit more aggressive, but that edge could be small enough that other factors (like perceived incompetence) dominate it.
cheald 13y ago
The choice to go for a 4th down conversion is more likely than not primarily decided by the current score and time left in the game. However, the success of a 4th down conversion probably isn't heavily influenced by those factors (other than perhaps player fatigue), which would suggest that coaches are indeed too conservative.
- 001sky 13y ago
  is more likely than not primarily This seems pretty loose. I'm not trying to be pedantic, but the theoretic flaws in the data are such that you would want a pretty tight logic in your model to have any level of comfort with it. The proportion of the state-space that is unobserved seems ~large and critically relevant.
zeroonetwothree 13y ago
No, it's based on third down data. The likelihood of converting 3rd and 1 is probably similar to that of converting 4th and 1.
- Uhhrrr 13y ago
  Thanks! I read poorly the first time.

timfrietas 13y ago

This does not take into account an important factor that is somewhat intangible, but real to the psychology of the game: point swings. If you go for it on 4th and 1 at your own 1 and lose you are definitely down 3 points and probably 7. This puts pressure on your offense--which just failed on a 4th and 1--to somehow make it all the way down the field next time. That can be demoralizing for that offense, paralyzing in fact. That is much harder to model, but an important part of a correct model, I believe.

aero142 13y ago
This actually seems measurable. Do teams that are behind perform worse on average? Do teams perform worse than normal after failing a 4th down attempt?
- twoodfin 13y ago
  The same statistics wonk has done a lot of work on "momentum" in games. His conclusion is that there is no such thing: http://www.advancednflstats.com/2013/11/momentum-1.html http://www.advancednflstats.com/2013/11/momentum-1.html That jibes with my own anecdotal experience watching a lot of football, despite how often announcers like to cite it as a factor.

Zimahl 13y ago

Why would you ever go for it on 4th and 1 inside your own 10 yard line? You would put yourself at a severe disadvantage if you failed. Since the data from this is from previous plays in the last 10 or so seasons it is flawed. There probably isn't very little data of a team going for 4th and 1 and that deep in their own territory. The only time this has probably happened is in extraordinary circumstances - specifically in a game-winning drive scenario with time expiring, where the defense is in 'prevent'. In these cases it is highly successful in that situation but you don't have the time to get the score.

thejteam 13y ago
In place of 4th and 1 data AdvancedNFlStats would substitute 3rd and 1 stats. And I would agree that this substitution is suspect. Just a bit shy of bogus in fact.
- Zimahl 13y ago
  Well, getting a yard in the NFL isn't typically a big problem and 3rd and 1 vs 4th and 1 are feasibly no different. But turning the ball over inside 30 yards is at least 3 points for the other team with only rare exceptions.
  - omegaham 13y ago
    I think the biggest difference is that on 3rd and 1, the defense has already assumed that they're going to get a 1st down. The main job of the defense is to prevent a big gain and then get ready for stopping the offense on the next four downs. I wouldn't say that they're cheesedicking it, but they're not going all out. Meanwhile, on fourth down, the defense is amped up and ready to roll. You'll usually see a strong blitz against a sputtering offense, and the percentage of plays that actually get a yard is going to be lower. Not only that, going three-and-out means that the offense is unlikely to do any better going from the 10 to the 20 yard line. Better to punt the ball, trust your defense, and have your offensive coordinator fix whatever is going wrong.
vacri 13y ago

thejteam 13y ago

This was discussed some in https://news.ycombinator.com/item?id=6728821 https://news.ycombinator.com/item?id=6728821 I've thought about it some since then and I'm also not convinced that expected points is a good metric. That will maximize the expected point differential per season but I am not convinced (although I could be wrong... I haven't put pencil to paper to make my thoughts rigorous) that there are enough possessions in the game on average to make this metric useful even in the first half.

gerad 13y ago
The best defense is keeping the other team's offense off the field. Going for it on fourth down has a much higher probability of doing that than a punt does.

kosei 13y ago

Tough to take everything into account, but I find it troubling that all non-success/failure outcomes are not part of the calculation (ex: 8-yard gain on a 4th and 10; a fumble on a 4th and 2 run, etc). It simplifies the formula to the layman, but I have to imagine that it adds enough additional variance to matter, especailly considering we're talking in variances of less than half a point in their stated example. Also (and to their point), there is a ton of variation between offenses, defenses, punters, running backs, etc. Using this as a rubric is a nice idea for a rule of thumb for an armchair quarterback, but I'd strongly disagree with using it to accurately criticize any decision within a couple points of expected value.

altoz 13y ago
This is especially true of 4th down plays. A surprise play-action pass on 4th and 1 can go for an especially long gain and thus should have higher expected value. There's also the fake-punt scenario, which I suspect tends to be more successful than a regular 4th down.
- kosei 13y ago
  Absolutely. And I'm sure if coaches started trying to run more 4th down plays, that would dramatically change the algorithm as well.

slg 13y ago

These type of articles never take into consideration the reason why coaches don't go for it as often as a computer might. The coaches face a reverse moral hazard when taking risks because the negative outcomes are multiplied for them. Even if the statistics support their decisions, a coach will be fired for taking too many risks and failing. Is gaining an extra 2% (or whatever) in expected returns worth the risk of losing everything if the low(er) percentage outcome does actually happen? It is the same idea behind the whole "no one gets fired for buying IBM/Microsoft". Until you get a coach's motivations more in line with the motivations of the team, you won't get a coach that is as aggressive as these computer simulations suggest they should be.

symbaton 13y ago
People always say this, but it is actually true? Has any coach ever been fired for missing too many fourth downs?
- slg 13y ago
  I can't think of one off the top of my head. Mike Smith was under a lot of pressure a little while back for his aggressive decisions on 4th down (that were statistically correct IIRC) but wasn't actually fired. However, the obvious counter to your question is how many coaches really go for it frequently on fourth downs? Most coaches just follow the conventional wisdom unless they are already in a desperate situation like Ron Riviera or they have built up a lot of good will like Bill Belichick.
  - hkmurakami 13y ago
    So it sounds like fans and the media need to grasp EV first before coaches will be allowed to take the appropriate risks.
- judk 13y ago
  Not even missing too many. Just missing more visibly, compared to coaches with equivalent records.
- 13y ago

vacri 13y ago

There's an interesting hole in the data at 4 & 10 on the opponent's 36. In a sea of punting and kicking, this one data point has actual coaches going for it. Also, looking at the overall pattern of the data, my gut tells me that the optimal rubric is somewhere in-between the 'bot and the meatcoach. There are some strange holes as well - at 4th&3, you should go for if you're on your own 8 or 9, but punt if you're on your own 7 or 10?

judk 13y ago
Without confidence levels, the differences are meaningless. Maybe that's the flip from 49% to 51%
thejteam 13y ago
That makes sense. At the 36, it is a 53 yard or so field goal. At the outer range of most kickers and pretty low percentage, so you give the ball to the opponent at the 40 something yardline with high probability. On the other hand, its also close enough to be very difficult for a punter to pin in inside the 5 or 10, so even if you fail you only give the opponent 16 yards over a punt. But the holes at the your own end you mentioned, that makes no sense at all.

reddog 13y ago

Here is a high school coach from Little Rock that never punts, and always onside kicks. His reasoning is sound and it seems to work out for him. http://www.youtube.com/watch?v=AGDaOJAYHfo http://www.youtube.com/watch?v=AGDaOJAYHfo

dfan 13y ago

It is interesting what an emotional reaction people instinctively have to 4th down. Going for it on 4th down and failing feels like a catastrophe. But punting would have given the ball away anyway, so all you've done is give the opponent an extra 40 yards or so. Which is bad, sure, but it's a hell of a lot less bad than, say, throwing an interception on 1st and 10. But somehow giving up the ball on downs has come to be seen as a massive failure, on par with an regular turnover.

snowwrestler 13y ago
Turning it over on downs is a turnover, as sure as fumbling is. It's not accounted that way in the box score, but in terms of impact on the game it is the same. An interception is better or worse than turning it over on downs by the field position delta. If it is returned past the line of scrimmage, then it's worse. If it is not returned, it's better. A deep interception with no return is like a punt in terms of impact on the game.

snowwrestler 13y ago

The fundamental flaw with this analysis is that statistics are descriptive, not predictive. You can't reliably predict the future with statistics alone. If you could, then technical analysis of the stock market would be the most successful investing strategy. It is not. Likewise, a lot of folks seem to think that taking an historical statistical trend and extending it into the future is a scientific approach to football coaching. It's not. Scientists use statistics to understand evidence, but they formulate theories in terms of root causes. In other words it's not enough to know that dropping a hammer has a high correlation with acceleration of 10 meters per second squared. If you want to predict how fast other objects in the future will fall, you need a theory that describes why and how anything falls. It doesn't look to me like the 4th down bot embodies or implements any cohesive theory of football success. It just finds trends and dots the line into the future.

nova22033 13y ago

It's the same thing with penalty kicks in soccer. Statistically, you are more likely to score if you kick the ball down the middle. The goalie is going to go one way or another and there's a 50% chance he'll guess right. So why don't players kick it down the middle..because if they kick it down the middle and the goalie manages to block the kick, they'll come off looking like idiots and it would look real bad on the highlight shows.