15 ms·

Every NFL play for the past 10 years in CSV format

421 points by edw519 14y ago

ScottWhigham 14y ago
The comments on that are awesome too - great advice for parsing, categorizing, and such. I couldn't download 2010 though - "Sorry, we are unable to generate a view of the document at this time. Please try again later."
- ScottWhigham 14y ago
  If you click the little down arrow (top left), it will download the file. Just a heads-up in case others see this message as well.
euroclydon 14y ago
How many of you are thinking right now: I'm going to generate an HTML page for every game and throw ads on it? Be Honest!
- pinchyfingers 14y ago
  hush :)
- DanBC 14y ago
  Is anyone going to try a 'moneyball' style Fix_Your_Fantasy_League_LineUp site with ads?
  - 404error 14y ago
    I might try to use the data to create mock drafts.
arscan 14y ago
I'm frankly surprised that this information is allowed to be distributed. I spent awhile in the financial services industry, and while it was really easy to obtain "public" information like stock quote data, I recall that we weren't allowed to simply scrape data from public sites... we had to pay a license fee to get a feed of the data if we were planning on repackaging & distributing it. It seems to me that the NFL would want to have exclusive rights to distribute this data and charge people a fee for access to it. Clearly I'm no expert in these legal affairs though.
- aidenn0 14y ago
  IANAL, but I asked one about this while ago; let's see if I can remember: It's complicated. The NFL broadcasts are copyrighted, and come with a statement that (among other things) distributing descriptions of the game is not allowed. That could be considered a derivative work. On the other hand, a live performance is generally not protected from copyright, so if you attend a live game to collect the data, you may be in the clear. The data isn't owned by the NFL, but all recordings of the games are, and so any data obtained by watching recordings of the games could potentially be controlled by the NFL.
  - sethist 14y ago
    IANAL too, but the NFL lost a relatively recent court case regarding fantasy football that, as far as I am aware, made the leagues statistics public knowledge as long as you compile the information yourself. Therefore the main legal issue with this data would be the source.
  - _delirium 14y ago
    It might not even violate the NFL's copyright if extracted from tapes. For one thing, something is only a "derived work" for copyright purposes if it's a "creative work" subject to copyright at all, and in the U.S., data sets comprising factual information aren't typically considered "creative". For another, it's not clear whether data about a recording is derived from the recording for copyright purposes. For example, a re-edit or mash-up of a film is clearly a derived work, but is a count of how many minutes each character speaks a derived work? Or is a Spotify-style algorithmic analysis of a song's musical style a derived work of the song? I wouldn't want to put a large bet on where exactly those lines are drawn, though.
- kodablah 14y ago
  The NFL is not near as strict as MLB (note how you never see an MLB highlight on youtube?) yet MLB allows http://retrosheet.org/ http://retrosheet.org/ to exist. I don't believe it's technically "dissemination". Edit: I should note that even http://gdx.mlb.com/components/game/mlb/ http://gdx.mlb.com/components/game/mlb/ contents are governed by http://gdx.mlb.com/components/copyright.txt http://gdx.mlb.com/components/copyright.txt which doesn't allow commercial use.
edw519 14y ago
From Line 42536 of the 2008 CSV file: 20090201_PIT@ARI,2,30,18,ARI,PIT,1,1,1,(:18) (Shotgun) K.Warner pass short middle intended for A.Boldin INTERCEPTED by J.Harrison at PIT 0. J.Harrison for 100 yards TOUCHDOWN. Super Bowl Record longest interception return yards. Penalty on ARZ-E.Brown Face Mask (15 Yards) declined. The Replay Assistant challenged the runner broke the plane ruling and the play was Upheld.,7,10,2008 They forgot: for(i=0;i<92;i++){yell('edw519','GO!')} Seriously, I had plans for the next 4 days, but I just scrapped them. Funny how jazzed I get when it's data that I can really relate to... I've already structured my data warehouse and started the loads. (I'll probably need a whole day just to parse the text in Field 10.) Then I'm going to build a Business Intelligence system on top of it. I will finally have the proof I need that I, not the offensive coordinator, should be texting each play to Coach Tomlin. See you guys on Monday. EDIT: OK, I'm back, but not for long. I'm having way too much fun with this... fleaflicker: Cool website & domain name. Thanks for the tips. I expect shortcomings in the data, but it looks like it's in a lot better shape than the usual free form enterprise quality/vendor/customer comments I usually have to parse. We'll see... MattSayer & sjs382: I don't plan to do any analysis. I prefer to build an app that enables others to do their own analyses, answering questions that nobody else is asking. Like "Which Steeler makes the most tackles on opposing runs or more than 5 yards when it's 3rd down and longer than 8 yards to go, the temperature is below 38, and edw519 is twirling his Terrible Towel clockwise?" jerf: Nice thought. I've spent years trying to earn enough money to buy the Pittsburgh Steelers just to fire the slackers and fumblers and win the Super Bowl every year. Maybe I should just take an easier route and solve that problem like any self-respecting hacker should: with data & logic. No Steeler game this weekend; I may have found my destiny </sarcasm>
- MattSayar 14y ago
  Please post your results, I'd love to read your analysis.
  - sjs382 14y ago
    Agreed. I'd love to learn about the process, too. Not just the results/findings.
- 14y ago
tghw 14y ago
Looking through the 2002 season, there's an oddity around touchdowns and extra points. It seems that the 6 points for the touchdown are bundled with the extra point, and the score is not updated until the extra point is complete. It seems this might result in bugs, as in the Oct 20, 2002 game between Dallas and Arizona. In the third quarter, with a score of Arizona 6 - Dallas 0, Dallas scored a touchdown (row 13900) but "aborted" the extra point (row 13901). The 6 points for the Cowboys are not recorded in the data. The game eventually went to overtime, with the Cardinals kicking a winning field goal in OT for a final score of Arizona 9 - Dallas 6, but the data here records it as Arizona 6 - Dallas 0.
gavinlynch 14y ago
Amazing!!! Thanks to www.advancednflstats.com for doing all the leg-work. Highly recommend their site too. Their in-game win probability statistics are always a must-have for me on game-day ^_^
- tghw 14y ago
  I really like his 4th Down analysis: http://www.advancednflstats.com/2009/09/4th-down-study-part-1.html http://www.advancednflstats.com/2009/09/4th-down-study-part-... The tl;dr version can be found at: http://www.advancednflstats.com/2010/05/4th-down-briefs.html http://www.advancednflstats.com/2010/05/4th-down-briefs.html The conclusion is that teams should go for it on 4th down much more often than they currently do. He also has a calculator where you can get the exact values: http://wp.advancednflstats.com/4thdncalc1.php http://wp.advancednflstats.com/4thdncalc1.php
  - yukoncornelius 14y ago
    The Patriots have used this analysis: http://www.math.toronto.edu/mpugh/Teaching/Sci199_03/Football_game_theory.htm http://www.math.toronto.edu/mpugh/Teaching/Sci199_03/Footbal... I also believe Belichek/Adams have funded some football economics research.
  - binxbolling 14y ago
    Could just be my imagination, but I felt like this season I saw many more teams go for it on 4th down (maybe some of this data can prove or disprove that?). Perhaps the impact of the above, or similar, analyses is finally being seen.
kevinburke 14y ago
I wrote a small wrapper around the 4th down calculator on that site, which should help you figure out if your team should go for it on 4th down: http://downanddistance.herokuapp.com/ http://downanddistance.herokuapp.com/
- stuff4ben 14y ago
  Need a bounds check or two in there. Tried setting the number of yards you need to 1 and the yards away from the endzone to 99 and it threw up a nice exception. Cool calculator though!
  - brianbreslin 14y ago
    As a Madden (game) aficionado my first thought was "ALWAYS go for it on 4th!" but then again I play super recklessly...
dude_abides 14y ago
Here is an idea: build a predictive model of an offensive coach that predicts the play he will call, given a game situation (and based on that, build a predictiveness quotient for a coach).
- fleaflicker 14y ago
  It doesn't work like that in practice. Football is very dependent on matchups. Coaches will vary gameplans from week-to-week to exploit weaknesses they see on film.
  - kaliblack 14y ago
    Agreed. Also, assuming you could create a successful predictive model the advantage would be short term. All coaches would hop on the bandwagon.
    - pseut 14y ago
      How optimistic of you.
  - dude_abides 14y ago
    Matchup would be a part of the model. My experience with predictive modeling in various domains has taught me that people tend to underestimate how predictive they are (NFL offensive/defensive coaches are no exception).
    - lftl 14y ago
      I'm interested in doing some predictive modeling for a couple of project ideas I've been kicking around. Are there any specific resources you would recommend as good starter material?
  - jpeterson 14y ago
    It could still be useful for finding tendencies, though.
deleted 14y ago
[deleted]
ImJasonH 14y ago
I've started uploading these CSVs to a public Google BigQuery dataset called [nfl], so you can run queries over them like this: SELECT off, COUNT(off) AS count FROM [nfl.2012reg] WHERE description CONTAINS "INTERCEPTED" GROUP BY off ORDER BY count DESC (This counts the number of plays that resulted in an interception by the team that threw the interception, sorted from most to fewest INTs)
- danvoell 14y ago
  I'm new to BigQuery, how do I access a public dataset? I ran the query and got the error Not Found: Dataset 578707073226:nfl
  - ImJasonH 14y ago
    Sorry, I didn't actually make it public it seems. Should work now.
    - danvoell 14y ago
      thanks!
- snake_plissken 14y ago
  mean reversion between ints for NFL live betting on next int? i smell greenbacks!
  - sethist 14y ago
    That sounds like the dictionary definition of the gambler's fallacy. If anything, the odds of an interception likely increase after a previous interception as it would be a sign of a defensive advantage over the QB. If only there was somewhere we could get the data to figure out for sure...
p4bl0 14y ago
As a French not interested in sports at all this would have made no sense at all to me before I watched the TV series The League [1]. Now I kind of enjoy the fact that these stats exist and are available in an open format, even if I don't really care myself. [1] http://www.imdb.com/title/tt1480684/ http://www.imdb.com/title/tt1480684/
danso 14y ago
This looks like great fun...Judging by some of the sample entries, it will also be an instructive example of the limitations of CSV and why serious analysts who want to work with unstructured data need to know a scripting language, or at least regexes. Sample description field: > 20020905_SF@NYG,1,59,20,NYG,SF,3,11,81,(14:20) (Shotgun) K.Collins pass intended for T.Barber INTERCEPTED by T.Parrish (M.Rumph) at NYG 29. T.Parrish to NYG 23 for 6 yards (T.Barber).,0,0,2002 In the comments section of the OP, someone posted this sample Excel function: =IF(ISNUMBER(SEARCH("right tackle",J2)),"rush",IF(ISNUMBER(SEARCH("right guard",J2)),"rush",IF(ISNUMBER(SEARCH("left guard",J2)),"rush",IF(ISNUMBER(SEARCH("up the middle",J2)),"rush",IF(ISNUMBER(SEARCH("left tackle",J2)),"rush",IF(ISNUMBER(SEARCH("left end",J2)),"rush",IF(ISNUMBER(SEARCH("right end",J2)),"rush",IF(ISNUMBER(SEARCH("pass",J2)),"pass",IF(ISNUMBER(SEARCH("kneel",J2)),"kneel",IF(ISNUMBER(SEARCH("punt",J2)),"punt",IF(ISNUMBER(SEARCH("kicks",J2)),"kickoff",IF(ISNUMBER(SEARCH("extra point",J2)),"extrapoint",IF(ISNUMBER(SEARCH("sacked",J2)),"sack",IF(ISNUMBER(SEARCH("PENALTY",J2)),"penalty",IF(ISNUMBER(SEARCH("field goal",J2)),"fieldgoal",IF(ISNUMBER(SEARCH("FUMBLES",J2)),"fumble",IF(ISNUMBER(SEARCH("spiked",J2)),"spike",IF(ISNUMBER(SEARCH("scrambles",J2)),"rush","rush")))))))))))))))))) Dear god, at what point do people finally realize that it's worth learning some simple scripting to work with text files?
- HelloMcFly 14y ago
  The Excel function looks ridiculous, but it probably didn't take more than 10 minutes to make, tops. Nested conditionals are easy. At any rate, what would you recommend most to accomplish the task? I'm learning Python and know R a bit, so I was just wondering how I was going to about combing through the data.
  - danso 14y ago
    Python or Ruby is fine...the main trick is to be able to process those fields with regular expressions...which, IIRC, requires throwing in VBscript if you were to handle it solely in Excel. Python and Ruby would also allow for more elegant-looking -- i.e. more maintainable -- functions to handle that field.
mcs 14y ago
wow
jredwards 14y ago
I've used data from Brian Burke's site before. I think it's the exact PBP data the NFL has, but you'll find that the structure and common phrasings change over the years. I had to write a lot of regular expressions and I was still catching edge cases for weeks. btw, pro-football-reference has pbp data now too, and it probably goes back a lot further, but I think they discourage mass scraping of their site.
danso 14y ago
There's a FAQ for this data that is on the site's main nav: http://www.advancednflstats.com/2007/02/contact.html http://www.advancednflstats.com/2007/02/contact.html Of particular interest: Where did you get your data? Most of my team data comes from open online sources such as espn.com, nfl.com, myway.com, and yahoo.com. It's easy for anyone to grab whatever they're interested in from those sites. My play-by-play data comes from a source that's not publicly available, and at this time I regret that I cannot share it. However, I am working hard to develop a way to spread the wealth. One of my biggest goals is to help create a larger, more open, and more collaborative community for football research. ---- There's no real terms of service so I'm curious as to the constraints in using this for commercial purposes. I most definitely want to use this for teaching purposes (how to text-mine, how to build a web app from data, etc) but want to know what terms the data can be redistributed.
- bendmorris 14y ago
  IANAL, but it has been ruled that NFL player names and statistics are protected by the First Amendment, i.e. no one "owns" it and anyone is free to use it for any purpose. http://blogs.trb.com/sports/custom/business/blog/2009/04/cbssportscom_wins_fantasy_game.html http://blogs.trb.com/sports/custom/business/blog/2009/04/cbs... However, you do have to get the data, and unauthorized access of computers (which constitutes trespassing) can be a legal gray area. I'd love to hear a lawyer weigh in on the legality of scraping the data directly from espn.com.
  - pseut 14y ago
    Last time I checked, the play by play data on espn.com was pretty error-ridden. This was three or so years ago, so it might have changed, and I was hypothetically interested in the score columns, so it may not matter depending on other hypothetical uses. But I'd hypothetically avoid scraping ESPN for that reason alone.
winstonian 14y ago
Would be great if there were columns: Head Coach - Offensive Coordinator - Defensive Coordinator - Formation - Play
asc76 14y ago
Wow. Simply. Wow.
activus 14y ago
It would be interesting to take this data and build an app around it for fantasy football. If you have all the tendencies, and how players like your player have played against certain teams, you could make better guesses on who to play.
- cacciatc 14y ago
  I did something like that for an AI class in college. We used FuzzyCLIPS to write an expert system for drafting fantasy football teams. A ruby script pulled the CSV data from some other site that had the previous three years of NFL data, and then converted the CSV to fact files which the system then read in. When all was said and done it worked, but made some pretty crappy draft picks! I should find that code....
petersalas 14y ago
This seems like as good a time as any to share something I've been working on which uses the same source data, even though it's pretty rough at the moment (slow, bad data, only currently goes through week 8 of 2012, etc.): http://nfl-query.herokuapp.com/ http://nfl-query.herokuapp.com/ The basic syntax is [stats] [conditions] : [row] / [column]. There's some autocompletion to try to make it possible to discover what is accepted. Examples: passing yards : team / season first downs / first down attempts : down / distance rushing yards min 100 rushing yards : player, game, quarter rushing yards / carries min 200 carries : player One of the biggest problems is that it's currently way too easy to shoot yourself in the foot by making a really slow query.
nchuhoai 14y ago
For my ML class, I used this dataset to play around with predicting plays. Im obviously no expert, but wanted to share my report anyways: https://www.dropbox.com/s/cy04oxaq83mxvoz/report.pdf https://www.dropbox.com/s/cy04oxaq83mxvoz/report.pdf
bdittmer 14y ago
Now all we need is some historical line & associated movement data...
eel 14y ago
FYI, for baseball fans, you can get similar data about each play in the MLB from http://www.retrosheet.org http://www.retrosheet.org
evanjacobs 14y ago
A bit OT, but I thought this might be a good opportunity to mention the upcoming SportsHackDay in Seattle from Feb 1-3 which culminates in a group viewing of the SuperBowl. http://sportshackday.com/ http://sportshackday.com/
bsims 14y ago
You should send this to the Buffalo Bill's new analytics department. Maybe a Hacker could get hired to an NFL team. http://www.nfl.com/news/story/0ap1000000121582/article/bill-polian-moneyball-does-not-work-in-the-nfl http://www.nfl.com/news/story/0ap1000000121582/article/bill-...
AlwaysBCoding 14y ago
God this is such interesting stuff. How do we still not have a fully featured open source NFL stats-rosters-game charting API? Who wouldn't want to contribute to that project? Other than cool data visualization stuff, the obvious implication is the potential to devise a profitable system to pick games against the spread. The guys at Football Outsiders have done a decent job at it and made a proprietary algorithm that picked games at 58% this year ( which is over the threshold you need to be profitable in Vegas ). But even those guys are still having some trouble getting access to and aggregating the data in a usable format. I really want to sit down and start playing around with some of this data so I appreciate you putting this together for everyone. The NFL needs an open source API and this is definitely a step in the right direction.
- burntsushi 14y ago
  I believe my library, nflgame [1], would fit the bill. Features all play-by-play data back to 2009, and includes the ability to track play-by-play data live. [1] - https://github.com/BurntSushi/nflgame https://github.com/BurntSushi/nflgame
burntsushi 14y ago
The CSV file format is nice, but if you're looking for a Python API to play with NFL stats without having to parse play-data fields, check out nflgame [1]. I've written up a quick primer. [2] It also includes the ability to get play-by-play statistics live. [1] - https://github.com/BurntSushi/nflgame https://github.com/BurntSushi/nflgame [2] - http://blog.burntsushi.net/nfl-live-statistics-with-python http://blog.burntsushi.net/nfl-live-statistics-with-python
patrickk 14y ago
Here's some soccer data, doesn't include play-by-play though (soccer generally isn't suited to that kind of breakdown, although Opta Sports do track it). http://www.football-data.co.uk/downloadm.php http://www.football-data.co.uk/downloadm.php Tons of European leagues, going back to 1993 in some cases. Here's some sites that give detailed stats and match reports: http://www.eplindex.com/ http://www.eplindex.com/ http://www.whoscored.com/ http://www.whoscored.com/ http://www.soccerstats.com/ http://www.soccerstats.com/ http://www.soccerway.com/ http://www.soccerway.com/ http://www.squawka.com/ http://www.squawka.com/ Man City use their petro-dollars to open up Opta Sports (detailed match stats) to all: http://www.mcfc.co.uk/the-club/mcfc-analytics http://www.mcfc.co.uk/the-club/mcfc-analytics Someone needs to compile stats equivalent to these NFL ones for european football! Hmmmm...
- joeycfan 14y ago
  I've found this to be the mother of all soccer sites http://www.bari91.com/ http://www.bari91.com/
crabasa 14y ago
If you live in the vicinity of Seattle, there is a sports-themed hackathon going on Superbowl weekend. Google, ESPN and a bunch of tech companies are sponsoring. The grand prize will be passes to the Sloan Sports Conference. More details to come: http://sportshackday.com http://sportshackday.com
wildster 14y ago
Shame it does not have player ids. I wonder how many players with the same surname and the same first initial play for the same team.
pmarsh 14y ago
There is a lot to have fun with here. I would imagine though that in a lot of NFL coaching rooms there has to be a balance between coaching and analysis. Like someone else said, it's about match-ups. Semi-related : http://profootballtalk.nbcsports.com/2013/01/03/polian-thinks-moneyball-wont-work-in-nfl/ http://profootballtalk.nbcsports.com/2013/01/03/polian-think...
grogenaut 14y ago
This is slightly off topic but does anyone know of a resource to get the odds at gametime historically for NFL games?
- glamp 14y ago
  https://www.dropbox.com/s/ikczgv737lllh0a/nfl_spreads_1985-2010.csv https://www.dropbox.com/s/ikczgv737lllh0a/nfl_spreads_1985-2...
  - grogenaut 14y ago
    thanks!
zempf 14y ago
There's an interface into this sort of play-by-play data (since 2000) at http://pro-football-reference.com/play-index/play_finder.cgi http://pro-football-reference.com/play-index/play_finder.cgi -- lets you do queries on down/distance/position on field/score differential, all that sort of stuff.
JohnFromBuffalo 14y ago
What ... no NHL? Oh ya.

421 points by edw519 14y ago

The comments on that are awesome too - great advice for parsing, categorizing, and such. I couldn't download 2010 though - "Sorry, we are unable to generate a view of the document at this time. Please try again later."

ScottWhigham 14y ago
If you click the little down arrow (top left), it will download the file. Just a heads-up in case others see this message as well.

euroclydon 14y ago

How many of you are thinking right now: I'm going to generate an HTML page for every game and throw ads on it? Be Honest!

pinchyfingers 14y ago
hush :)
DanBC 14y ago
Is anyone going to try a 'moneyball' style Fix_Your_Fantasy_League_LineUp site with ads?
- 404error 14y ago
  I might try to use the data to create mock drafts.

arscan 14y ago

I'm frankly surprised that this information is allowed to be distributed. I spent awhile in the financial services industry, and while it was really easy to obtain "public" information like stock quote data, I recall that we weren't allowed to simply scrape data from public sites... we had to pay a license fee to get a feed of the data if we were planning on repackaging & distributing it. It seems to me that the NFL would want to have exclusive rights to distribute this data and charge people a fee for access to it. Clearly I'm no expert in these legal affairs though.

aidenn0 14y ago
IANAL, but I asked one about this while ago; let's see if I can remember: It's complicated. The NFL broadcasts are copyrighted, and come with a statement that (among other things) distributing descriptions of the game is not allowed. That could be considered a derivative work. On the other hand, a live performance is generally not protected from copyright, so if you attend a live game to collect the data, you may be in the clear. The data isn't owned by the NFL, but all recordings of the games are, and so any data obtained by watching recordings of the games could potentially be controlled by the NFL.
- sethist 14y ago
  IANAL too, but the NFL lost a relatively recent court case regarding fantasy football that, as far as I am aware, made the leagues statistics public knowledge as long as you compile the information yourself. Therefore the main legal issue with this data would be the source.
- _delirium 14y ago
  It might not even violate the NFL's copyright if extracted from tapes. For one thing, something is only a "derived work" for copyright purposes if it's a "creative work" subject to copyright at all, and in the U.S., data sets comprising factual information aren't typically considered "creative". For another, it's not clear whether data about a recording is derived from the recording for copyright purposes. For example, a re-edit or mash-up of a film is clearly a derived work, but is a count of how many minutes each character speaks a derived work? Or is a Spotify-style algorithmic analysis of a song's musical style a derived work of the song? I wouldn't want to put a large bet on where exactly those lines are drawn, though.
kodablah 14y ago
The NFL is not near as strict as MLB (note how you never see an MLB highlight on youtube?) yet MLB allows http://retrosheet.org/ http://retrosheet.org/ to exist. I don't believe it's technically "dissemination". Edit: I should note that even http://gdx.mlb.com/components/game/mlb/ http://gdx.mlb.com/components/game/mlb/ contents are governed by http://gdx.mlb.com/components/copyright.txt http://gdx.mlb.com/components/copyright.txt which doesn't allow commercial use.

edw519 14y ago

From Line 42536 of the 2008 CSV file: 20090201_PIT@ARI,2,30,18,ARI,PIT,1,1,1,(:18) (Shotgun) K.Warner pass short middle intended for A.Boldin INTERCEPTED by J.Harrison at PIT 0. J.Harrison for 100 yards TOUCHDOWN. Super Bowl Record longest interception return yards. Penalty on ARZ-E.Brown Face Mask (15 Yards) declined. The Replay Assistant challenged the runner broke the plane ruling and the play was Upheld.,7,10,2008 They forgot: for(i=0;i<92;i++){yell('edw519','GO!')} Seriously, I had plans for the next 4 days, but I just scrapped them. Funny how jazzed I get when it's data that I can really relate to... I've already structured my data warehouse and started the loads. (I'll probably need a whole day just to parse the text in Field 10.) Then I'm going to build a Business Intelligence system on top of it. I will finally have the proof I need that I, not the offensive coordinator, should be texting each play to Coach Tomlin. See you guys on Monday. EDIT: OK, I'm back, but not for long. I'm having way too much fun with this... fleaflicker: Cool website & domain name. Thanks for the tips. I expect shortcomings in the data, but it looks like it's in a lot better shape than the usual free form enterprise quality/vendor/customer comments I usually have to parse. We'll see... MattSayer & sjs382: I don't plan to do any analysis. I prefer to build an app that enables others to do their own analyses, answering questions that nobody else is asking. Like "Which Steeler makes the most tackles on opposing runs or more than 5 yards when it's 3rd down and longer than 8 yards to go, the temperature is below 38, and edw519 is twirling his Terrible Towel clockwise?" jerf: Nice thought. I've spent years trying to earn enough money to buy the Pittsburgh Steelers just to fire the slackers and fumblers and win the Super Bowl every year. Maybe I should just take an easier route and solve that problem like any self-respecting hacker should: with data & logic. No Steeler game this weekend; I may have found my destiny </sarcasm>

MattSayar 14y ago
Please post your results, I'd love to read your analysis.
- sjs382 14y ago
  Agreed. I'd love to learn about the process, too. Not just the results/findings.
14y ago

tghw 14y ago

Looking through the 2002 season, there's an oddity around touchdowns and extra points. It seems that the 6 points for the touchdown are bundled with the extra point, and the score is not updated until the extra point is complete. It seems this might result in bugs, as in the Oct 20, 2002 game between Dallas and Arizona. In the third quarter, with a score of Arizona 6 - Dallas 0, Dallas scored a touchdown (row 13900) but "aborted" the extra point (row 13901). The 6 points for the Cowboys are not recorded in the data. The game eventually went to overtime, with the Cardinals kicking a winning field goal in OT for a final score of Arizona 9 - Dallas 6, but the data here records it as Arizona 6 - Dallas 0.

gavinlynch 14y ago

Amazing!!! Thanks to www.advancednflstats.com for doing all the leg-work. Highly recommend their site too. Their in-game win probability statistics are always a must-have for me on game-day ^_^

tghw 14y ago
I really like his 4th Down analysis: http://www.advancednflstats.com/2009/09/4th-down-study-part-1.html http://www.advancednflstats.com/2009/09/4th-down-study-part-... The tl;dr version can be found at: http://www.advancednflstats.com/2010/05/4th-down-briefs.html http://www.advancednflstats.com/2010/05/4th-down-briefs.html The conclusion is that teams should go for it on 4th down much more often than they currently do. He also has a calculator where you can get the exact values: http://wp.advancednflstats.com/4thdncalc1.php http://wp.advancednflstats.com/4thdncalc1.php
- yukoncornelius 14y ago
  The Patriots have used this analysis: http://www.math.toronto.edu/mpugh/Teaching/Sci199_03/Football_game_theory.htm http://www.math.toronto.edu/mpugh/Teaching/Sci199_03/Footbal... I also believe Belichek/Adams have funded some football economics research.
- binxbolling 14y ago
  Could just be my imagination, but I felt like this season I saw many more teams go for it on 4th down (maybe some of this data can prove or disprove that?). Perhaps the impact of the above, or similar, analyses is finally being seen.

kevinburke 14y ago

I wrote a small wrapper around the 4th down calculator on that site, which should help you figure out if your team should go for it on 4th down: http://downanddistance.herokuapp.com/ http://downanddistance.herokuapp.com/

stuff4ben 14y ago
Need a bounds check or two in there. Tried setting the number of yards you need to 1 and the yards away from the endzone to 99 and it threw up a nice exception. Cool calculator though!
- brianbreslin 14y ago
  As a Madden (game) aficionado my first thought was "ALWAYS go for it on 4th!" but then again I play super recklessly...

dude_abides 14y ago

Here is an idea: build a predictive model of an offensive coach that predicts the play he will call, given a game situation (and based on that, build a predictiveness quotient for a coach).

fleaflicker 14y ago
It doesn't work like that in practice. Football is very dependent on matchups. Coaches will vary gameplans from week-to-week to exploit weaknesses they see on film.
- kaliblack 14y ago
  Agreed. Also, assuming you could create a successful predictive model the advantage would be short term. All coaches would hop on the bandwagon.
  - pseut 14y ago
    How optimistic of you.
- dude_abides 14y ago
  Matchup would be a part of the model. My experience with predictive modeling in various domains has taught me that people tend to underestimate how predictive they are (NFL offensive/defensive coaches are no exception).
  - lftl 14y ago
    I'm interested in doing some predictive modeling for a couple of project ideas I've been kicking around. Are there any specific resources you would recommend as good starter material?
- jpeterson 14y ago
  It could still be useful for finding tendencies, though.

deleted 14y ago

[deleted]

ImJasonH 14y ago

I've started uploading these CSVs to a public Google BigQuery dataset called [nfl], so you can run queries over them like this: SELECT off, COUNT(off) AS count FROM [nfl.2012reg] WHERE description CONTAINS "INTERCEPTED" GROUP BY off ORDER BY count DESC (This counts the number of plays that resulted in an interception by the team that threw the interception, sorted from most to fewest INTs)

danvoell 14y ago
I'm new to BigQuery, how do I access a public dataset? I ran the query and got the error Not Found: Dataset 578707073226:nfl
- ImJasonH 14y ago
  Sorry, I didn't actually make it public it seems. Should work now.
  - danvoell 14y ago
    thanks!
snake_plissken 14y ago
mean reversion between ints for NFL live betting on next int? i smell greenbacks!
- sethist 14y ago
  That sounds like the dictionary definition of the gambler's fallacy. If anything, the odds of an interception likely increase after a previous interception as it would be a sign of a defensive advantage over the QB. If only there was somewhere we could get the data to figure out for sure...

p4bl0 14y ago

As a French not interested in sports at all this would have made no sense at all to me before I watched the TV series The League [1]. Now I kind of enjoy the fact that these stats exist and are available in an open format, even if I don't really care myself. [1] http://www.imdb.com/title/tt1480684/ http://www.imdb.com/title/tt1480684/

danso 14y ago

This looks like great fun...Judging by some of the sample entries, it will also be an instructive example of the limitations of CSV and why serious analysts who want to work with unstructured data need to know a scripting language, or at least regexes. Sample description field: > 20020905_SF@NYG,1,59,20,NYG,SF,3,11,81,(14:20) (Shotgun) K.Collins pass intended for T.Barber INTERCEPTED by T.Parrish (M.Rumph) at NYG 29. T.Parrish to NYG 23 for 6 yards (T.Barber).,0,0,2002 In the comments section of the OP, someone posted this sample Excel function: =IF(ISNUMBER(SEARCH("right tackle",J2)),"rush",IF(ISNUMBER(SEARCH("right guard",J2)),"rush",IF(ISNUMBER(SEARCH("left guard",J2)),"rush",IF(ISNUMBER(SEARCH("up the middle",J2)),"rush",IF(ISNUMBER(SEARCH("left tackle",J2)),"rush",IF(ISNUMBER(SEARCH("left end",J2)),"rush",IF(ISNUMBER(SEARCH("right end",J2)),"rush",IF(ISNUMBER(SEARCH("pass",J2)),"pass",IF(ISNUMBER(SEARCH("kneel",J2)),"kneel",IF(ISNUMBER(SEARCH("punt",J2)),"punt",IF(ISNUMBER(SEARCH("kicks",J2)),"kickoff",IF(ISNUMBER(SEARCH("extra point",J2)),"extrapoint",IF(ISNUMBER(SEARCH("sacked",J2)),"sack",IF(ISNUMBER(SEARCH("PENALTY",J2)),"penalty",IF(ISNUMBER(SEARCH("field goal",J2)),"fieldgoal",IF(ISNUMBER(SEARCH("FUMBLES",J2)),"fumble",IF(ISNUMBER(SEARCH("spiked",J2)),"spike",IF(ISNUMBER(SEARCH("scrambles",J2)),"rush","rush")))))))))))))))))) Dear god, at what point do people finally realize that it's worth learning some simple scripting to work with text files?

HelloMcFly 14y ago
The Excel function looks ridiculous, but it probably didn't take more than 10 minutes to make, tops. Nested conditionals are easy. At any rate, what would you recommend most to accomplish the task? I'm learning Python and know R a bit, so I was just wondering how I was going to about combing through the data.
- danso 14y ago
  Python or Ruby is fine...the main trick is to be able to process those fields with regular expressions...which, IIRC, requires throwing in VBscript if you were to handle it solely in Excel. Python and Ruby would also allow for more elegant-looking -- i.e. more maintainable -- functions to handle that field.

mcs 14y ago

wow

jredwards 14y ago

I've used data from Brian Burke's site before. I think it's the exact PBP data the NFL has, but you'll find that the structure and common phrasings change over the years. I had to write a lot of regular expressions and I was still catching edge cases for weeks. btw, pro-football-reference has pbp data now too, and it probably goes back a lot further, but I think they discourage mass scraping of their site.

danso 14y ago

There's a FAQ for this data that is on the site's main nav: http://www.advancednflstats.com/2007/02/contact.html http://www.advancednflstats.com/2007/02/contact.html Of particular interest: Where did you get your data? Most of my team data comes from open online sources such as espn.com, nfl.com, myway.com, and yahoo.com. It's easy for anyone to grab whatever they're interested in from those sites. My play-by-play data comes from a source that's not publicly available, and at this time I regret that I cannot share it. However, I am working hard to develop a way to spread the wealth. One of my biggest goals is to help create a larger, more open, and more collaborative community for football research. ---- There's no real terms of service so I'm curious as to the constraints in using this for commercial purposes. I most definitely want to use this for teaching purposes (how to text-mine, how to build a web app from data, etc) but want to know what terms the data can be redistributed.

bendmorris 14y ago
IANAL, but it has been ruled that NFL player names and statistics are protected by the First Amendment, i.e. no one "owns" it and anyone is free to use it for any purpose. http://blogs.trb.com/sports/custom/business/blog/2009/04/cbssportscom_wins_fantasy_game.html http://blogs.trb.com/sports/custom/business/blog/2009/04/cbs... However, you do have to get the data, and unauthorized access of computers (which constitutes trespassing) can be a legal gray area. I'd love to hear a lawyer weigh in on the legality of scraping the data directly from espn.com.
- pseut 14y ago
  Last time I checked, the play by play data on espn.com was pretty error-ridden. This was three or so years ago, so it might have changed, and I was hypothetically interested in the score columns, so it may not matter depending on other hypothetical uses. But I'd hypothetically avoid scraping ESPN for that reason alone.

winstonian 14y ago

Would be great if there were columns: Head Coach - Offensive Coordinator - Defensive Coordinator - Formation - Play

asc76 14y ago

Wow. Simply. Wow.

activus 14y ago

It would be interesting to take this data and build an app around it for fantasy football. If you have all the tendencies, and how players like your player have played against certain teams, you could make better guesses on who to play.

cacciatc 14y ago
I did something like that for an AI class in college. We used FuzzyCLIPS to write an expert system for drafting fantasy football teams. A ruby script pulled the CSV data from some other site that had the previous three years of NFL data, and then converted the CSV to fact files which the system then read in. When all was said and done it worked, but made some pretty crappy draft picks! I should find that code....

petersalas 14y ago

This seems like as good a time as any to share something I've been working on which uses the same source data, even though it's pretty rough at the moment (slow, bad data, only currently goes through week 8 of 2012, etc.): http://nfl-query.herokuapp.com/ http://nfl-query.herokuapp.com/ The basic syntax is [stats] [conditions] : [row] / [column]. There's some autocompletion to try to make it possible to discover what is accepted. Examples: passing yards : team / season first downs / first down attempts : down / distance rushing yards min 100 rushing yards : player, game, quarter rushing yards / carries min 200 carries : player One of the biggest problems is that it's currently way too easy to shoot yourself in the foot by making a really slow query.

nchuhoai 14y ago

For my ML class, I used this dataset to play around with predicting plays. Im obviously no expert, but wanted to share my report anyways: https://www.dropbox.com/s/cy04oxaq83mxvoz/report.pdf https://www.dropbox.com/s/cy04oxaq83mxvoz/report.pdf

bdittmer 14y ago

Now all we need is some historical line & associated movement data...

eel 14y ago

FYI, for baseball fans, you can get similar data about each play in the MLB from http://www.retrosheet.org http://www.retrosheet.org

evanjacobs 14y ago

A bit OT, but I thought this might be a good opportunity to mention the upcoming SportsHackDay in Seattle from Feb 1-3 which culminates in a group viewing of the SuperBowl. http://sportshackday.com/ http://sportshackday.com/

bsims 14y ago

You should send this to the Buffalo Bill's new analytics department. Maybe a Hacker could get hired to an NFL team. http://www.nfl.com/news/story/0ap1000000121582/article/bill-polian-moneyball-does-not-work-in-the-nfl http://www.nfl.com/news/story/0ap1000000121582/article/bill-...

AlwaysBCoding 14y ago

God this is such interesting stuff. How do we still not have a fully featured open source NFL stats-rosters-game charting API? Who wouldn't want to contribute to that project? Other than cool data visualization stuff, the obvious implication is the potential to devise a profitable system to pick games against the spread. The guys at Football Outsiders have done a decent job at it and made a proprietary algorithm that picked games at 58% this year ( which is over the threshold you need to be profitable in Vegas ). But even those guys are still having some trouble getting access to and aggregating the data in a usable format. I really want to sit down and start playing around with some of this data so I appreciate you putting this together for everyone. The NFL needs an open source API and this is definitely a step in the right direction.

burntsushi 14y ago
I believe my library, nflgame [1], would fit the bill. Features all play-by-play data back to 2009, and includes the ability to track play-by-play data live. [1] - https://github.com/BurntSushi/nflgame https://github.com/BurntSushi/nflgame

burntsushi 14y ago

The CSV file format is nice, but if you're looking for a Python API to play with NFL stats without having to parse play-data fields, check out nflgame [1]. I've written up a quick primer. [2] It also includes the ability to get play-by-play statistics live. [1] - https://github.com/BurntSushi/nflgame https://github.com/BurntSushi/nflgame [2] - http://blog.burntsushi.net/nfl-live-statistics-with-python http://blog.burntsushi.net/nfl-live-statistics-with-python

patrickk 14y ago

Here's some soccer data, doesn't include play-by-play though (soccer generally isn't suited to that kind of breakdown, although Opta Sports do track it). http://www.football-data.co.uk/downloadm.php http://www.football-data.co.uk/downloadm.php Tons of European leagues, going back to 1993 in some cases. Here's some sites that give detailed stats and match reports: http://www.eplindex.com/ http://www.eplindex.com/ http://www.whoscored.com/ http://www.whoscored.com/ http://www.soccerstats.com/ http://www.soccerstats.com/ http://www.soccerway.com/ http://www.soccerway.com/ http://www.squawka.com/ http://www.squawka.com/ Man City use their petro-dollars to open up Opta Sports (detailed match stats) to all: http://www.mcfc.co.uk/the-club/mcfc-analytics http://www.mcfc.co.uk/the-club/mcfc-analytics Someone needs to compile stats equivalent to these NFL ones for european football! Hmmmm...

joeycfan 14y ago
I've found this to be the mother of all soccer sites http://www.bari91.com/ http://www.bari91.com/

crabasa 14y ago

If you live in the vicinity of Seattle, there is a sports-themed hackathon going on Superbowl weekend. Google, ESPN and a bunch of tech companies are sponsoring. The grand prize will be passes to the Sloan Sports Conference. More details to come: http://sportshackday.com http://sportshackday.com

wildster 14y ago

Shame it does not have player ids. I wonder how many players with the same surname and the same first initial play for the same team.

pmarsh 14y ago

There is a lot to have fun with here. I would imagine though that in a lot of NFL coaching rooms there has to be a balance between coaching and analysis. Like someone else said, it's about match-ups. Semi-related : http://profootballtalk.nbcsports.com/2013/01/03/polian-thinks-moneyball-wont-work-in-nfl/ http://profootballtalk.nbcsports.com/2013/01/03/polian-think...

grogenaut 14y ago

This is slightly off topic but does anyone know of a resource to get the odds at gametime historically for NFL games?

glamp 14y ago
https://www.dropbox.com/s/ikczgv737lllh0a/nfl_spreads_1985-2010.csv https://www.dropbox.com/s/ikczgv737lllh0a/nfl_spreads_1985-2...
- grogenaut 14y ago
  thanks!

zempf 14y ago

There's an interface into this sort of play-by-play data (since 2000) at http://pro-football-reference.com/play-index/play_finder.cgi http://pro-football-reference.com/play-index/play_finder.cgi -- lets you do queries on down/distance/position on field/score differential, all that sort of stuff.

JohnFromBuffalo 14y ago

What ... no NHL? Oh ya.