3 ms·
have a love-hate relationship with the NHTSA database. On one hand, it covers 40 years of data, and unlike virtually every other long-running dataset in existen
by danso 10y ago
have a love-hate relationship with the NHTSA database. On one hand, it covers 40 years of data, and unlike virtually every other long-running dataset in existence, it keeps not only a relatively standardized schema, it publishes a all-in-one user manual that covers every year (and any changes between them); most places publish a new PDF for every year, which means you have to read through each one. Even better is that when they make changes or additions to their methodology, they publish an auxiliary dataset that you can use to retroactively update old datasets.
But then for some reason, they throw out all rationality when it comes to naming files and fields. It's as if there was employee who was old-school and was stuck in mindset of naming things under the 8-character all-caps limitation. And then someone else thought, "Fuck this", and went the entirely opposite direction by naming files with long titles and spaces, and the two employees had a passive-aggressive argument that never got resolved over the decades.
This makes scraping and collating the data a real pain in the ass. But compiling the data got a lot easier when I realized that wget works on FTP sites. Here's a mirror of the FTP site (around 1.7GB zipped) that I put up on Github (still working on getting everything together in one database): https://github.com/wgetsnaps/ftp.nhtsa.dot.gov--fars https://github.com/wgetsnaps/ftp.nhtsa.dot.gov--fars