8 ms·
Git scraping: track changes over time by scraping to a Git repository
- simonw 6y agoOne of my favourite things about Git scraping is that, thanks to CI providers like GitHub Actions which are free for open projects, it costs nothing to run. https://github.com/simonw/ca-fires-history https://github.com/simonw/ca-fires-history for example scrapes every 20 minutes and runs on the free GitHub Actions plan. I've run these on free Travis CI and Circle CI plans in the past as well. If you want to run a private Git scraper you can pay money to do so, which seems very reasonable to me.
- dilly_li 6y agoNifty! Do you plan to use those commit history in any way? Is it just for bookkeeping?
- simonw 6y agoI'm hoping someone uses this data for something interesting - it may end up being me! I ran the same kind of thing against PG&E outage data last year and used it to generate different visualizations: https://simonwillison.net/2019/Oct/10/pge-outages/ https://simonwillison.net/2019/Oct/10/pge-outages/
- mh- 6y agoreally appreciated your PGE stuff last year. I hope we won't need it, but if we do.. I hope you'll do it again this year. :)
- polote 6y agoThat's a very nice hack of github actions. The strength of that is to have super reliable storage but the risk is to loose everything because you don't respect the Github terms of use. Here is the term of use of Actions if anyone interested : https://docs.github.com/en/free-pro-team@latest/github/site-policy/github-additional-product-terms#5-actions-and-packages https://docs.github.com/en/free-pro-team@latest/github/site-...
- simonw 6y agoI tend to run these against things that are unlikely to complain about it - most government data sources are OK in my experience. If you're scraping against the terms of service of something you'll probably want to pay for a private repository.
- drhayes9 6y agoThe relevant section seems to be "Additionally, Actions should not be used for...any other activity unrelated to the production, testing, deployment, or publication of the software project associated with the repository where GitHub Actions are used." Don't get me wrong, I absolutely adore this usage of GH actions. But isn't this a TOS violation? I've thought about using GH actions as a generalized cron for online stuff and that feels like it walks the line (e.g. re-generating my static site via a Netlify webhook so it can update comments or whatnot). I feel okay about it because the static site is what's contained in the repo.
- swyx 6y ago> super reliable i've been scraping daily for most of this year, and fwiw gh actions fail pretty regularly. here's my actions log: https://github.com/sw-yx/gh-action-data-scraping/actions?query=is%3Afailure https://github.com/sw-yx/gh-action-data-scraping/actions?que... so if you want reliable data scraping you'll actually have to build in some sort of retry capability so as not to lose data. just fyi for anyone here
- RvdV 6y agoWow, that's a cool idea! I wonder, how do you handle bad responses (e.g. HTTP 400/500) from the server? It might mess up the logs twice if it's committed...
- mackorone 6y agoI wrote something like this for archiving public Spotify playlists: https://github.com/mackorone/spotify-playlist-archive https://github.com/mackorone/spotify-playlist-archive It’s nice to know I can easily find songs I liked based on when I listened to them.
- simonw 6y agoThat's such a great example of this pattern in action. Would you be up for adding the git-scraping topic so it shows up on https://github.com/topics/git-scraping https://github.com/topics/git-scraping ?
- mdaniel 6y agoI actually suggested to GitHub back when the topics feature first came out that I thought the ability to submit topic recommendations to repos would be beneficial, since maybe some repo owners don't know about topics, or don't know which ones would help their discovery, or maybe (like a lot of PRs) would accept the suggestion but can't be bothered to do the legwork Now that ".github" is a thing, putting the topics in there would allow a real PR workflow for suggestions
- simonw 6y agoI'd love to be able to do that.
- mackorone 6y agoDone!
- 13rac1 6y agoYes! Great idea! I've been working on a similar, but different project recently. Using screenshots instead of git. I wanted to see the change in air quality due to the wildfires over time, so I'm making time-lapse videos: PurpleAir Air Quality 7200X Time-Lapse - Glass Fire - San Francisco Bay Area - 9/28-10/3 https://youtu.be/CFT-EIEmzfM https://youtu.be/CFT-EIEmzfM
- wferrell 6y agoHow did you capture all these screenshots? Great video!
- victor9000 6y agoYou can use Puppeteer to drive the web scraping and use page.screenshot() to capture images. You can find a quick example in the docs below: https://github.com/puppeteer/puppeteer/blob/v5.3.1/docs/api.md#class-page https://github.com/puppeteer/puppeteer/blob/v5.3.1/docs/api....
- 13rac1 6y agoThanks! Yes, it's puppeteer, plus imagemagick to add transparency using an overlay on an empty map, and ffmpeg to make the 60fps video. A lot of data, currently have over 200GB.
- scollet 6y agoScarifying! Thanks!
- welder 6y agoPurpleair also has a JSON api if you want the raw data behind those screenshots.
- patrickdevivo 6y agoThis is really cool, and coupled with a tool like https://github.com/augmentable-dev/askgit https://github.com/augmentable-dev/askgit, could yield some interesting results. It could produce a table of specific values/text changing over time, measure frequency of changes, etc
- yoz 6y agoWow, finding this tool has made my day - thank you for making it! I did a quick skim through the code, and it appears that askgit reads the repo into a SQLite DB first, then queries that. Is that right? Also, have you looked at Fossil for an alternative VCS for this work, or is that not a consideration here?
- arkadiyt 6y agoI wrote a similar git scraper for tracking bug bounty programs and their scopes. Here's the data, updated hourly: https://github.com/arkadiyt/bounty-targets-data https://github.com/arkadiyt/bounty-targets-data And the code: https://github.com/arkadiyt/bounty-targets https://github.com/arkadiyt/bounty-targets
- simonw 6y agoThat's a fantastic example - and it looks like you've had that running for three years now! https://github.com/arkadiyt/bounty-targets-data https://github.com/arkadiyt/bounty-targets-data
- adamzapasnik 6y agoHaha, I've written a scraper for docker tags for Elixir images this way. I'm even gonna open source it this weekend...
- mdaniel 6y agoI just don't understand why docker hub doesn't have Atom or RSS feeds; it's the perfect situation for it: infrequent(?) changes that a large audience wants to be notified about
- breck 6y agoGit is eating the database world.
- vamps 6y agofor the good
- rekwah 6y agoA nice complement to this would be tooling that treats file revisions as time series data, e.g. plot the fire contained percentage over time. Any existing tools out there?
- simonw 6y agoNot that I've seen. I've written code to do that using the GitPython library in the past, but I had to write it custom for each project - e.g. https://simonwillison.net/2019/Oct/10/pge-outages/ https://simonwillison.net/2019/Oct/10/pge-outages/
- dec0dedab0de 6y agoI used to use git for backing up configs for network gear, most of which involved some sort of scraping because they didn't have APIs. It never really caught on with the rest of the network team, but I thought it was a great idea.
- ClikeX 6y agoSounds like Infrastructure as Code to me. Shame it didn't catch on.
- gabereiser 6y agoSo funny because I’ve proposed the same thing! Not necessarily git, just a merkle tree in general to diff changes to state over time. Super interesting read! The trade off being more cpu required to build a picture of state than storage requirements due to having to walk the tree.
- baby 6y agoSo using git instead of a database when a database would be more suited?
- jetpackjoe 6y agoCan you expand on why a database would be more well suited? It seems to me that if the goal is to track changes over time, git is very well suited to that.
- baby 6y agogit is not suited to search through the different fields that have changed (it's content agnostic). A database allows you to plot and search efficiently.
- gitgud 6y agoWell you could say folders are like "tables" and files are like "fields" within those tables... The main difference between git and a traditional DB is querying and relationships between fields
- Frost1x 6y agoDidn't cha know, innovation in the tech industry is all about using things differently than intended and building new compositions that result in the same end goals as older less sophisticated compositions.
- yoz 6y agoGit is a database. It's designed for storing the changes made to files over time. But hey, if there's a different kind of database that you think would be better here, I'd be fascinated to see it. Just make sure that it can 1. handle major structural changes to the files being tracked (e.g. JSON schema revisions) 2. store and retrieve these changes as efficiently as git 3. do the above with as little work required by the user Good luck!
- 6y ago
- jtwaleson 6y agoI do this for FDA databases. Great to keep track of competitors and new product categories. Internal repo though.
- VWWHFSfQ 6y agoIt's kinda neat but a pretty obvious TOS violation. I'm not sure github is really interested in people taking advantage of their free service to run their web scraper bots.
- jxf 6y agoQ: What part of this is a TOS violation?
- swyx 6y agofor what its worth, when i did it (https://www.swyx.io/github-scraping/ https://www.swyx.io/github-scraping/) and tweeted about it (https://twitter.com/swyx/status/1219739210434994177 https://twitter.com/swyx/status/1219739210434994177), github employees replied supportively. i really doubt they care unless you are running massive scale.
- polote 6y agofor now they are probably going to tolerate any usage even if you harm them in order to be the leader and kill the competition, then things will change. 101 monopoly
- ric2b 6y agoIf by github employees you mean the devs, I expect them to like it. It's the managers and lawyers that you need to worry about.
- tomhallett 6y agoI've needed something like this for trying to wrangle third party javascript libraries (which you load from their servers): * tracking changes * kicking off a selenium test run when their js changes * including their js into your js bundle to reduce network calls (git scrape => cd pipeline) Very cool!
- jldugger 6y agoI used to do this for job postings in target companies and a few other purposes in heady era of 2008. One of the hassles is that some pages do not neatly contain their useful data in json, so the tiny deploy script becomes a little more complicated to support the various steps: 1. Collection: logins, curl 2. Pretty print: jq / tidy 3. Selectors: Beautiful Soup / jq 4. Annotation: svn / git commit The Github Actions angle is new and welcome though.
- jldugger 6y agoOh, and I should mention, it's pretty cool to see how companies change job ads on a daily basis. Beyond just the spelling fixes, I've seen places advertise help wanted postings for like a day and remove them the next day on a regular basis. Or tack on a 'on call 24/7/365' statement. Or reuse a job posting but tack on the word 'Manager.'
- simonw 6y agoThat's so interesting. Next time I'm job seeking I'll definitely look into doing this.
- peterburkimsher 6y agoIf you're interested in Australia or New Zealand, we can chat. I have a scraper running on my spare laptop now, and am going to try porting it to Github Actions so it continues without me. The source data is all HTML not JSON though, and I have to scrape pages, then parse job IDs, and then re-scrape the job listings themselves. Having it as a SQLite database is more helpful than the default search: e.g. all jobs that don't include the phrase "right to live and work in this location", all jobs that have email addresses, GROUP BY advertiser - features I wish but don't expect would ever be added to the source site.
- jldugger 6y agoIn retrospect it mostly didn't work. What I really learned from this is that help-wanted pages are a formality. I figure places that hire people who read HN pay recruiters, and applying directly with the company less often 'skips the line' and more often finds a direct line to the trashcan, since h1b hiring laws require employers try to find local talent work first.
- LukeEF 6y agoTerminusDB (co founder here) was partially inspired by the git scraping approach to the revision control for data problem. We built a database that gives you all of the functionality of git, but in a database so you can query and with a commit graph etc. Has git semantics for clone, fork, rebase, merge and the other major functions. We store & share deltas and use succinct data structures that allow us to pass around in-memory DBs that you can then query in place. We're relatively new, but happy to see versioning data wherever it might land. Open source forever so really trying to be git for data. Technical white paper on our structure: https://github.com/terminusdb/terminusdb-server/blob/dev/docs/whitepaper/terminusdb.pdf https://github.com/terminusdb/terminusdb-server/blob/dev/doc...
- LukeEF 6y agoOne comment on using git, isn't there a pull:push bottle neck that means it can't service any workload that has any write rate greater than (say) one write every 30 seconds. Write is subject to interpretation of course given commit != push (and i am unashamedly suggesting that a DB is better)
- Quenty 6y agoVery cool that it’s based off of RDF triples. I’m really curious about your product. What does performance look like? The TerminusDB website says it’s in memory? Can you have a database larger than your RAM?
- LukeEF 6y agoRDF triples turn out to be a crucial part of the architecture as they make describing deltas really straightforward. It is just these triples were added and these triples were taken away. Performance is good - you get a degradation in query time as you build more appended layers, but you can squash these to a single plane to speed up. Often we have a query branch where the layers are optimized for query and another branch with all the commit history in place. We are working on something we call Delta roll ups at the moment - these are like squashes that keep the history. Hopefully you'll soon be able to automate the roll ups to keep query performance at a specified level (something like the vacuum cleaner in postgres). It is in-memory, so you are limited to what's in RAM for querying, but it persists to disk, and we are betting that memory is going to get bigger and cheaper over the next while.
- asjo 6y agoI've been recording changes to the three lists of approved first names in Denmark in git for the past 8 years: https://koldfront.dk/git/godkendtefornavne/about/ https://koldfront.dk/git/godkendtefornavne/about/ I'm using cron on my cupboard server to update regularly rather than relying on Microsoft GitHub, and a standard post-update hook that emails me the diff when there are changes, so I can check out what new, funny, misspelled firstnames have been approved. Git works really nice for this kind of use, as long as you make sure you store the data in a "stable" and "diffable" format.
- ivan_ah 6y agoAn excellent tool that can be used in conjunction with this approach is the word-level diff option for git diffs: `git diff --color-words` Rather than processing entire lines (useful for code), it shows individual word changes, so it's great for tracking data as in the screenshots in the article and even better for text (output produced is similar to latexdiff for those familiar). I highly recommend... I have a git alias setup for this, and use it almost every day.
- simonw 6y agoThat's a great tip, thanks. Looks like you can use that option with 'git show' too: git show --color-words
- ivan_ah 6y agoOh cool. I just learned a new git verb today! (Previously I always used `git diff <commit-1> <commit>` but `git show <commit>` is much simpler.
- globular-toast 6y agomagit users can enable this in all diffs by setting the variable `magit-diff-refine-hunk'.
- mark_l_watson 6y agoThanks! I have written my last half dozen books using markdown + leanpub.com, keeping my book artifacts in GitHub. I am going to try this!
- KajMagnus 6y ago> [I] use it almost every day. What do you use it for? I'm wondering, what web pages / what information is so interesting, so one looks for changes almost every day :-) Is it maybe for finding out if contents linked by Minireference text books, change in significant ways? Or move to different URLs?
- ivan_ah 6y ago
- klysm 6y agoI’ve done a similar thing with APIs that return JSON. I write a small script which strictly queries the API and dumps the raw JSON into Postgres. Any structured data can be derived from the raw results (optionally with triggers) but what’s key is you will never reject a response from the API - anything goes. A nice of extension of this is having a distinct process using listen/notify on a derived table to drive notifications. I used this design to notify me of apartment listings changing and it was quite reliable.
- johnnyapol 6y agoA course scheduler for my school, QuACS ( https://github.com/quacs/quacs https://github.com/quacs/quacs ) was built entirely around git scraping. Around 7:30AM EST each morning, a GitHub Action is kicked off to scrape various things (class hour schedule, course catalog, faculty directory, etc) and produce a commit containing the current data. Then, the website is rebuilt (also using GitHub Actions) and hosted on GitHub pages. The only thing that's not static about the site is a call is made to our Student Information System (SIS) to retrieve current enrollment numbers in courses for seats. It's helpful as the data is now open for everyone in a nice format, so some other projects that utilize course data have been taking advantage of it.
- simonw 6y agoThat's really smart. I found the scraping action here: https://github.com/quacs/quacs-data/blob/master/.github/workflows/scrape-data.yml https://github.com/quacs/quacs-data/blob/master/.github/work... Looks like it's even tracking covid data released by the school: https://github.com/quacs/quacs-data/commits/master/covid.json https://github.com/quacs/quacs-data/commits/master/covid.jso...
- mitchtbaum 6y agoTorchbear makes this very simple using the Gut version control system. It's Speakeasy https://github.com/foundpatternscellar/run-tests/blob/master/packages/stackoverflow_scrapper/actions/scrape_stackoverflow.lua https://github.com/foundpatternscellar/run-tests/blob/master... https://github.com/speakeasy-engine/gut https://github.com/speakeasy-engine/gut Much better even to parse the data in Lightouch and save it in ContentDB
- nunez 6y agoThis is a reliably solid way of finding passwords and access keys
- williamsmj 6y agoI use this pattern to scrape the Redfin and Zillow estimates for the value of my house: https://github.com/williamsmj/real-estate-scrape/ https://github.com/williamsmj/real-estate-scrape/.
- tuananh 6y agothe problem with this is it's not easy to pull data out from git. eg: i want to visualize teh changes over time.
- crazysim 6y agoA decade ago, I had a professor who would update their syllabus site and not reliably tell the class. Wget in spider mode, a cron job, a Bitbucket repo with email notifications on push, and a Google Group helped out a lot!
- hardwaresofton 6y agoThis is also a relatively simple way to run a useful one-page informational site if you combine with Github Pages (or even just push to an S3 bucket/netlify/whatever). You can even have search for your one-page informational site with libraries like lunr[0]. A lot of websites that are data driven (or community contribution driven) would benefit from this kind of model. One thing I was thinking of building was a For example an open database of laptops, similar to some other sites out there. Once you've got the data all in one place you can also start doing some interesting analytical queries with CI and maybe add a second page with interesting trends. [0]: https://lunrjs.com/guides/getting_started.html https://lunrjs.com/guides/getting_started.html
- metaodi 6y agoWe are using this approach to scrape the COVID-19 data in Switzerland. https://github.com/openZH/covid_19 https://github.com/openZH/covid_19 There is a common runner script and a matrix configuration to run this script for each Swiss canton (see https://github.com/openZH/covid_19/blob/master/.github/workflows/run_scrapers.yml https://github.com/openZH/covid_19/blob/master/.github/workf...). The scrapers report errors to our slack channel and we even have workflows to deactivate/re-activate a single scraper, since it's common that one fails and we need to fix it (to stop spamming the slack channel with error messages).
- jstrieb 6y agoI've used a similar technique to scrape GitHub profile statistics data and record it over time. It seems to work really well, particularly since the GitHub API doesn't record project view data for a period longer than 14 days. https://github.com/jstrieb/github-stats https://github.com/jstrieb/github-stats
- matchilling 6y agoNice, I didn't know that "Git scraping" was actually a term, but I used this technique to build an API which allows fetching HMRC exchange rates in JSON format. https://github.com/matchilling/hmrc-exchange-rates https://github.com/matchilling/hmrc-exchange-rates
- Hackbraten 6y agoSomewhat related Twitter thread from a few days ago: https://twitter.com/Programazing/status/1313479031069835266 https://twitter.com/Programazing/status/1313479031069835266
- im1983 6y agoNice. Being thinking on doing the same for an ETF composite I’ve been following. I’d love to see how it changes over time