3 ms·

Imagine the possibilities of using GitHub actions as a way to trigger events based on new data being saved. This could be used to sync data across multiple git

Imagine the possibilities of using GitHub actions as a way to trigger events based on new data being saved. This could be used to sync data across multiple git databases.

_alternator_ 1y ago
This is built in to git as “git hooks”; GitHub not required. But don’t do it. Postgres has a system to trigger commands based on SQL patterns (eg on commit, when updating a row in this table, etc). Much more powerful and maintainable since it is designed for this. The fact is that basically every data structure can be abused as a database (python dicts, flat files, wave in mercury traveling down a long tube). Don’t reinvent the wheel, learn to use a power tool like Postgres.
- bitmasher9 1y ago
  I don’t think my humor of recreating serverless architecture came off very well in my original comment.
  - _alternator_ 1y ago
    Lol, I see the joke now.
jauntywundrkind 1y ago
I never got around to part 2, but I made a git-auto-commit program (using Watchman), with one of my use cases being capturing the history of my logseq markdown notes. I never did the next part but yeah, I was hoping to write some agents running remotely to actively follow the repo & then rewrite history with better commit names. It was pre AI boom, but auto-summarizers & taggers running as their own agents were a fantasy ask, something that felt like the architecture would have been condusive for if/when the disaggregated work became possible. Git as eventsourcing is kind of a ground floor. Being able to decoupled the processing is the loftier hope for centralizing around git imo. There's a lot of people yelling pretty loudly 'git would never work' 'the performance wouldn't be enough' 'write your own app specific version control if you need it', but being in the world's best ecosystem with the world's best tools for version controlled things feels like a no brainer. Theres so many fun distributed computing tasks folks get up to, so many ways of doing sync, but we developers already know a really good standard option and the world of tools around it is enviable beyond imagining. Performance might also have some big upsides, by virtue of processing being so seamlessly decentealizable here too. I do think that the giant "huge tables of all intermixed customer data" monoliths / dataliths that most apps use is not a good fit, won't scale nicely. But if you can give up some coherency, and take advantage of the scale out ability, it lets any machine any process loosely couple it's way into the transaction processing or analysis pipelines: it becomes more interesting and compelling. I've worked at a place where each customer had their own data tables, and BlueSky/atproto'a Personal Data Store'a one-sqlite-per-user also rather mirrors this architecture, of having lots of data stores, then separate systems doing any necessary aggregation with more typical big table architectures): scaling a lot of gits feels like the place where git would win. Wandering off possible core architectural upsides briefly: biggest likely challenge I see is figuring out how to save state nicely into files. To really take advantage of diffs and merges, we need to walk away from treating git like a blob store, and imo likely walk away from JSON stores too. Finer grained updates, having the file system exposes the structure of the data feels quasi-mamdatory, if you want to really let git help you store data: you have to come down to its level, leverage the filesystem to file your data. Json2dir for example nicely expands a single JSON into a directory tree of simple files, which git would be far more able to mall and track. https://github.com/alurm/json2dir https://github.com/alurm/json2dir https://news.ycombinator.com/item?id=44840307 https://news.ycombinator.com/item?id=44840307

Imagine the possibilities of using GitHub actions as a way to trigger events based on new data being saved. This could be used to sync data across multiple git databases.

_alternator_ 1y ago

This is built in to git as “git hooks”; GitHub not required. But don’t do it. Postgres has a system to trigger commands based on SQL patterns (eg on commit, when updating a row in this table, etc). Much more powerful and maintainable since it is designed for this. The fact is that basically every data structure can be abused as a database (python dicts, flat files, wave in mercury traveling down a long tube). Don’t reinvent the wheel, learn to use a power tool like Postgres.

bitmasher9 1y ago
I don’t think my humor of recreating serverless architecture came off very well in my original comment.
- _alternator_ 1y ago
  Lol, I see the joke now.

jauntywundrkind 1y ago

I never got around to part 2, but I made a git-auto-commit program (using Watchman), with one of my use cases being capturing the history of my logseq markdown notes. I never did the next part but yeah, I was hoping to write some agents running remotely to actively follow the repo & then rewrite history with better commit names. It was pre AI boom, but auto-summarizers & taggers running as their own agents were a fantasy ask, something that felt like the architecture would have been condusive for if/when the disaggregated work became possible. Git as eventsourcing is kind of a ground floor. Being able to decoupled the processing is the loftier hope for centralizing around git imo. There's a lot of people yelling pretty loudly 'git would never work' 'the performance wouldn't be enough' 'write your own app specific version control if you need it', but being in the world's best ecosystem with the world's best tools for version controlled things feels like a no brainer. Theres so many fun distributed computing tasks folks get up to, so many ways of doing sync, but we developers already know a really good standard option and the world of tools around it is enviable beyond imagining. Performance might also have some big upsides, by virtue of processing being so seamlessly decentealizable here too. I do think that the giant "huge tables of all intermixed customer data" monoliths / dataliths that most apps use is not a good fit, won't scale nicely. But if you can give up some coherency, and take advantage of the scale out ability, it lets any machine any process loosely couple it's way into the transaction processing or analysis pipelines: it becomes more interesting and compelling. I've worked at a place where each customer had their own data tables, and BlueSky/atproto'a Personal Data Store'a one-sqlite-per-user also rather mirrors this architecture, of having lots of data stores, then separate systems doing any necessary aggregation with more typical big table architectures): scaling a lot of gits feels like the place where git would win. Wandering off possible core architectural upsides briefly: biggest likely challenge I see is figuring out how to save state nicely into files. To really take advantage of diffs and merges, we need to walk away from treating git like a blob store, and imo likely walk away from JSON stores too. Finer grained updates, having the file system exposes the structure of the data feels quasi-mamdatory, if you want to really let git help you store data: you have to come down to its level, leverage the filesystem to file your data. Json2dir for example nicely expands a single JSON into a directory tree of simple files, which git would be far more able to mall and track. https://github.com/alurm/json2dir https://github.com/alurm/json2dir https://news.ycombinator.com/item?id=44840307 https://news.ycombinator.com/item?id=44840307