4 ms·
If you're going to run something like this, I thoroughly recommend using Git for it. You can have your cron do something like this: curl https://internal.
by simonw 3y ago
If you're going to run something like this, I thoroughly recommend using Git for it.
You can have your cron do something like this:
curl https://internal.corp/employees.txt > employees.txt
git add employees.txt
git commit -m "Automated: $(date -u)" || exit 0
The || exit 0 should ensure no errors even if there is nothing to commit
Now you have a commit history of every change made to that source of information - just run "git log" to view it.
I run this kind of thing on scheduled GitHub Actions all the time, see https://simonwillison.net/2020/Oct/9/git-scraping/ https://simonwillison.net/2020/Oct/9/git-scraping/
- jacquesm 3y agoThat's clever, thank you! I will definitely use this.
- MarkSweep 3y agoMaybe add a ‘| sort’ in there for determinism. But yeah, git is an underrated database for this type of small scale data.
- svat 3y agoI do something similar but instead of `|| exit 0` I use `--allow-empty` on the `git commit`. I don't mind the empty commits this creates, as they let me know that there was a successful automated run that happened to be empty, rather than having failed to run for whatever reason.
- michaelcampbell 3y agoThanks for this, much more "intent revealing" than my (up to today) standard practice of `... || true` to keep my `set -e` from killing my script for this "error-but-not-really"
- GauntletWizard 3y agoI've been seriously considering using Git for all sorts of oft-changing-but-rarely-majorly data. Lists of books in my bookshelves. The other problem is that I sorta want transactional-database features on top of these things. Git does this well. I also want fast indexing on parts. Git does not do this well. I am considering writing a "standard" for the dumping of sqlite to git, so that I can just delegate this out; Any transaction can be expressed as a git commit, and I can run both at once for both the durability and the reasonable indexing; The sqlite database can be re-created and reindexed whenever, and it also sorta works for backups... Definitely just spinning my wheels, though. We'll see where databases take us next.
- rofrol 3y agoMaybe this https://github.com/dolthub/dolt https://github.com/dolthub/dolt
- GauntletWizard 3y agoDolt is really close and yet just doesn't feel right; I don't want to "commit" between transactions, I want every transaction to be a commit.
- timsehn 3y agoYou can do this with a setting: https://docs.dolthub.com/sql-reference/version-control/dolt-sysvars#dolt_transaction_commit https://docs.dolthub.com/sql-reference/version-control/dolt-...
- GauntletWizard 3y agoThat's awesome. I will have to try that out.
- ElectricalUnion 3y agoWould fossil fit this bill? Somewhat git-compatible, based on sqlite3.
- JonChesterfield 3y agoSpecifically the fossil repo is a sqlite database. I have a cron script stashing whatever is currently in an obsidian vault into a fossil repo. There's a fossil addremove command that makes that very easy. Thus distributed backups of said vault.
- Zambyte 3y agoWhat does "somewhat git-compatible" mean? Can it sometimes use existing git repos, or is the mental model close enough to pick up without learning much?
- hiAndrewQuinn 3y agoI used your exact technique to start generating a diffable archive for the Finnish easy language news broadcast. It's been a huge help in gathering high quality comprehensible input for me, thanks! https://github.com/hiAndrewQuinn/selkouutiset-scrape/ https://github.com/hiAndrewQuinn/selkouutiset-scrape/
- itronitron 3y agoA colleague of mine would do a sort of temporal-network analysis of this data to see which people either enjoyed working with (or for) each other or which did not, based on how people would switch groups over time.
- cutemonster 3y agoDid s/he use the analysis results for anything? (If you know)
- itronitron 3y agoI'm fairly certain they used it to gauge future project opportunities as our work was often multi-disciplinary across departments. It was probably also useful for validating people's stories if they came up. For example, if someone wanted to switch into your group 'A' from group 'C' and they had been working on a project with X, Y, and Z, then you could get a strong sense of why that might feel that way.
- bandie91 3y agoif git is-there-anything-to-commit; then git commit ...; fi 'is-there-anything-to-commit' is aliased to '! [ -n "$(git status -s --porcelain --untracked-files=no)" ]'