4 ms·
Git is a key technology in this approach, because the value you get out of this form of scraping is the commit history - it's a way of turning a static source o
by simonw 3y ago
Git is a key technology in this approach, because the value you get out of this form of scraping is the commit history - it's a way of turning a static source of information into a record of how that information changed over time.
I think it's fine to use the term "scraping" to refer to downloading a JSON file.
These days an increasing number of websites work by serving up JSON which is then turned into HTML by a client-side JavaScript app. The JSON often isn't a formally documented API, but you can grab it directly to avoid the extra step of processing the HTML.
I do run Git scrapers that process HTML as well. A couple of examples:
scrape-san-mateo-fire-dispatch https://github.com/simonw/scrape-san-mateo-fire-dispatch https://github.com/simonw/scrape-san-mateo-fire-dispatch scrapes the HTML from http://www.firedispatch.com/iPhoneActiveIncident.asp?Agency=04100 http://www.firedispatch.com/iPhoneActiveIncident.asp?Agency=... and records both the original HTML and converted JSON in the repository.
scrape-hacker-news-by-domain https://github.com/simonw/scrape-hacker-news-by-domain https://github.com/simonw/scrape-hacker-news-by-domain uses my https://shot-scraper.datasette.io/ https://shot-scraper.datasette.io/ browser automation tool to convert an HTML page on Hacker News into JSON and save that to the repo. I wrote more about how that works here: https://simonwillison.net/2022/Dec/2/datasette-write-api/ https://simonwillison.net/2022/Dec/2/datasette-write-api/
That one's a particularly fun demo because it's currently capturing changes to the points and comment count on this thread - a recent example commit: https://github.com/simonw/scrape-hacker-news-by-domain/commit/a7e2f3fcc6dfa6a442a2f93608062c83e20512e8 https://github.com/simonw/scrape-hacker-news-by-domain/commi...
- gremlinsinc 3y agoI think the name is also a little ambiguous. I suggest maybe, commited-scraping, or time-scraping, or chronicle-scraping... and chatGPT could probably come up with something even better lol.
- layer8 3y ago“Periodic snapshotting” would be more accurate. That’s the usual term of the art.
- cachvico 3y agoIt's not scraping, you're just consuming an API. Scraping is when you're parsing human-readable content (HTML) and extracting data, as the parent comment correctly points out.
- simonw 3y agoI don't think that definition is universally agreed upon. I have had many conversations over the years where the term "scraping" referred to activities that didn't involve things like parsing HTML.
- fragmede 3y agoAgree to disagree. To me, scraping implies a level of fragility that hitting an endpoint that returns JSON does not have.
- simonw 3y agoUndocumented endpoints that return JSON are pretty fragile! One of the benefits of catching them in a Git repo is that it helps you spot when their structure changes in ways that may break code that you write on top of them.
- oneeyedpigeon 3y agoFrom the Wikipedia entry for "data scraping": > the key element that distinguishes data scraping from regular parsing is that the output being scraped is intended for display to an end-user, rather than as an input to another program Snapshotting JSON files can be incredibly useful, but I don't think you should call it "scraping".
- cxr 3y agoScraping also has, in some contexts, negative associations. In a project for a non-profit that I'm involved with that coincidentally was originally a remix of some of Simon's code for one of these "Git scraping" projects + Datasette, I recently made the decision to refer to it strictly as what it is: a crawler. I'm less warm at this point to the general idea behind the hack of dumping the resulting JSON crawl data to GitHub. It's a very roundabout way of approaching basically what something like TerminusDB was made for. It definitely feels like the main motivation was GitHub-specific stuff and not Git, really—namely, free jobs with GitHub Actions—and everything else flowed from that. It turns out that GitHub Actions proved to be too unreliable for executing on schedule, anyway, so we ported the crawler to JS with an eye towards using Cloudflare Workers and their cron triggers (which also come in a free flavor).