3 ms·

I'm using a webscraper that extracts data from some APIs, and stores all the data in a git repo periodically. Then you can see the data changing with tools like

• simonw 4y ago
  I love that pattern. I call it "git scraping" and I've been writing about it for a few years (and building tools too): https://simonwillison.net/series/git-scraping/ https://simonwillison.net/series/git-scraping/
  • ukd1 4y ago
    Interesting - I've been doing similar for a while (not writing about it though), but generally "self-commit" - i.e. the code in the repo commits back to itself, usuualy using github actions, or gitlab. E.g. https://github.com/ukd1/lvms-events https://github.com/ukd1/lvms-events
    • simonw 4y ago
      That's a great example of the same pattern, really neat! If you like you could tag it git-scraping to add it to this list: https://github.com/topics/git-scraping https://github.com/topics/git-scraping
      • ukd1 4y ago
        Thanks, and never thought of that; will do!