3 ms·
Do you have any examples of government sites that are doing this? I have a side-hobby of setting up scrapers which pull scraped data into a git repository, pre
by simonw 7y ago
Do you have any examples of government sites that are doing this?
I have a side-hobby of setting up scrapers which pull scraped data into a git repository, precisely for this kind of thing. I'd be happy to set a few up.
Some of my posts about this technique (which I call "git scraping"): https://simonwillison.net/tags/gitscraping/ https://simonwillison.net/tags/gitscraping/
- Endlessly 7y agoNot sure if you’re asking for: (1) US coronavirus data in general, (2) examples of sources that do not log prior days data, or (3) source that “edit” prior data without noting the edits. (4) something else If (1) this page in the table under the column “sources” links to where the data came from: https://www.worldometers.info/coronavirus/country/us/ https://www.worldometers.info/coronavirus/country/us/
- simonw 7y ago2 and 3.
- jeromegv 7y agoGovernment of Ontario has been doing it. They now have some APIs that link to the number of cases per day, but for the first month of the pandemic, they were literally overwriting the page every day and when you look at github there was tons of researchers that came up with their own parser in order to save this data before it is deleted. I've also been running bash scripts every day to save an archive of some of those pages https://github.com/jeromegv/covid_data https://github.com/jeromegv/covid_data