4 ms·

This is cool but the name is confusing. First of all, git is not being scraped nor is git being used to do any scraping, git is only used as the storage format

This is cool but the name is confusing. First of all, git is not being scraped nor is git being used to do any scraping, git is only used as the storage format for the snapshots. Second, there is no scraping happening at all. Scraping is when you parse a file intended for human display in order to extract the embedded unstructured data. The examples given are about periodically downloading an already structured json file and uploading it to github. No parsing is happening unless you count when he manually searches for the json file in the browser dev tools.

simonw 3y ago
Git is a key technology in this approach, because the value you get out of this form of scraping is the commit history - it's a way of turning a static source of information into a record of how that information changed over time. I think it's fine to use the term "scraping" to refer to downloading a JSON file. These days an increasing number of websites work by serving up JSON which is then turned into HTML by a client-side JavaScript app. The JSON often isn't a formally documented API, but you can grab it directly to avoid the extra step of processing the HTML. I do run Git scrapers that process HTML as well. A couple of examples: scrape-san-mateo-fire-dispatch https://github.com/simonw/scrape-san-mateo-fire-dispatch https://github.com/simonw/scrape-san-mateo-fire-dispatch scrapes the HTML from http://www.firedispatch.com/iPhoneActiveIncident.asp?Agency=04100 http://www.firedispatch.com/iPhoneActiveIncident.asp?Agency=... and records both the original HTML and converted JSON in the repository. scrape-hacker-news-by-domain https://github.com/simonw/scrape-hacker-news-by-domain https://github.com/simonw/scrape-hacker-news-by-domain uses my https://shot-scraper.datasette.io/ https://shot-scraper.datasette.io/ browser automation tool to convert an HTML page on Hacker News into JSON and save that to the repo. I wrote more about how that works here: https://simonwillison.net/2022/Dec/2/datasette-write-api/ https://simonwillison.net/2022/Dec/2/datasette-write-api/ That one's a particularly fun demo because it's currently capturing changes to the points and comment count on this thread - a recent example commit: https://github.com/simonw/scrape-hacker-news-by-domain/commit/a7e2f3fcc6dfa6a442a2f93608062c83e20512e8 https://github.com/simonw/scrape-hacker-news-by-domain/commi...
- gremlinsinc 3y ago
  I think the name is also a little ambiguous. I suggest maybe, commited-scraping, or time-scraping, or chronicle-scraping... and chatGPT could probably come up with something even better lol.
  - layer8 3y ago
    “Periodic snapshotting” would be more accurate. That’s the usual term of the art.
- cachvico 3y ago
  It's not scraping, you're just consuming an API. Scraping is when you're parsing human-readable content (HTML) and extracting data, as the parent comment correctly points out.
  - simonw 3y ago
    I don't think that definition is universally agreed upon. I have had many conversations over the years where the term "scraping" referred to activities that didn't involve things like parsing HTML.
    - fragmede 3y ago
      Agree to disagree. To me, scraping implies a level of fragility that hitting an endpoint that returns JSON does not have.
      - simonw 3y ago
        Undocumented endpoints that return JSON are pretty fragile! One of the benefits of catching them in a Git repo is that it helps you spot when their structure changes in ways that may break code that you write on top of them.
        oneeyedpigeon 3y ago
ants_everywhere 3y ago
I think "time-lapse" might be the right metaphor. With time-lapse photography, we see snapshots of a continuous process taken at regular intervals. I'm not sure what it has to do with git. It seems like any version control system would work. Or, really, the main use of git here is that GitHub is effectively being used as a free database. The snapshots and timestamps are enough to see the changes, regardless of storage format.

simonw 3y ago

Git is a key technology in this approach, because the value you get out of this form of scraping is the commit history - it's a way of turning a static source of information into a record of how that information changed over time. I think it's fine to use the term "scraping" to refer to downloading a JSON file. These days an increasing number of websites work by serving up JSON which is then turned into HTML by a client-side JavaScript app. The JSON often isn't a formally documented API, but you can grab it directly to avoid the extra step of processing the HTML. I do run Git scrapers that process HTML as well. A couple of examples: scrape-san-mateo-fire-dispatch https://github.com/simonw/scrape-san-mateo-fire-dispatch https://github.com/simonw/scrape-san-mateo-fire-dispatch scrapes the HTML from http://www.firedispatch.com/iPhoneActiveIncident.asp?Agency=04100 http://www.firedispatch.com/iPhoneActiveIncident.asp?Agency=... and records both the original HTML and converted JSON in the repository. scrape-hacker-news-by-domain https://github.com/simonw/scrape-hacker-news-by-domain https://github.com/simonw/scrape-hacker-news-by-domain uses my https://shot-scraper.datasette.io/ https://shot-scraper.datasette.io/ browser automation tool to convert an HTML page on Hacker News into JSON and save that to the repo. I wrote more about how that works here: https://simonwillison.net/2022/Dec/2/datasette-write-api/ https://simonwillison.net/2022/Dec/2/datasette-write-api/ That one's a particularly fun demo because it's currently capturing changes to the points and comment count on this thread - a recent example commit: https://github.com/simonw/scrape-hacker-news-by-domain/commit/a7e2f3fcc6dfa6a442a2f93608062c83e20512e8 https://github.com/simonw/scrape-hacker-news-by-domain/commi...

gremlinsinc 3y ago
I think the name is also a little ambiguous. I suggest maybe, commited-scraping, or time-scraping, or chronicle-scraping... and chatGPT could probably come up with something even better lol.
- layer8 3y ago
  “Periodic snapshotting” would be more accurate. That’s the usual term of the art.
cachvico 3y ago
It's not scraping, you're just consuming an API. Scraping is when you're parsing human-readable content (HTML) and extracting data, as the parent comment correctly points out.
- simonw 3y ago
  I don't think that definition is universally agreed upon. I have had many conversations over the years where the term "scraping" referred to activities that didn't involve things like parsing HTML.
  - fragmede 3y ago
    Agree to disagree. To me, scraping implies a level of fragility that hitting an endpoint that returns JSON does not have.
    - simonw 3y ago
      Undocumented endpoints that return JSON are pretty fragile! One of the benefits of catching them in a Git repo is that it helps you spot when their structure changes in ways that may break code that you write on top of them.
      - oneeyedpigeon 3y ago

ants_everywhere 3y ago

I think "time-lapse" might be the right metaphor. With time-lapse photography, we see snapshots of a continuous process taken at regular intervals. I'm not sure what it has to do with git. It seems like any version control system would work. Or, really, the main use of git here is that GitHub is effectively being used as a free database. The snapshots and timestamps are enough to see the changes, regardless of storage format.