3 ms·

I've been seriously considering using Git for all sorts of oft-changing-but-rarely-majorly data. Lists of books in my bookshelves. The other problem is that I

I've been seriously considering using Git for all sorts of oft-changing-but-rarely-majorly data. Lists of books in my bookshelves. The other problem is that I sorta want transactional-database features on top of these things. Git does this well. I also want fast indexing on parts. Git does not do this well. I am considering writing a "standard" for the dumping of sqlite to git, so that I can just delegate this out; Any transaction can be expressed as a git commit, and I can run both at once for both the durability and the reasonable indexing; The sqlite database can be re-created and reindexed whenever, and it also sorta works for backups... Definitely just spinning my wheels, though. We'll see where databases take us next.

rofrol 3y ago
Maybe this https://github.com/dolthub/dolt https://github.com/dolthub/dolt
- GauntletWizard 3y ago
  Dolt is really close and yet just doesn't feel right; I don't want to "commit" between transactions, I want every transaction to be a commit.
  - timsehn 3y ago
    You can do this with a setting: https://docs.dolthub.com/sql-reference/version-control/dolt-sysvars#dolt_transaction_commit https://docs.dolthub.com/sql-reference/version-control/dolt-...
    - GauntletWizard 3y ago
      That's awesome. I will have to try that out.
ElectricalUnion 3y ago
Would fossil fit this bill? Somewhat git-compatible, based on sqlite3.
- JonChesterfield 3y ago
  Specifically the fossil repo is a sqlite database. I have a cron script stashing whatever is currently in an obsidian vault into a fossil repo. There's a fossil addremove command that makes that very easy. Thus distributed backups of said vault.
- Zambyte 3y ago
  What does "somewhat git-compatible" mean? Can it sometimes use existing git repos, or is the mental model close enough to pick up without learning much?
  - ElectricalUnion 3y ago
    > is the mental model close enough to pick up without learning much? I would say git core concepts are pretty similar to fossil concepts, but actual plumbing implementation details are pretty distinct. The major difference that I remember from a day-to-day "git porcelain" perspective is that rebases and other types of history rewriting are very discouraged. For a Rosetta stone of somewhat comparable commands: https://fossil-scm.org/home/doc/trunk/www/gitusers.md https://fossil-scm.org/home/doc/trunk/www/gitusers.md > Can it sometimes use existing git repos You can, but it's kinda a lot of really slow busywork. And you lose some of the not-file-dvcs features of Fossil, but it is possible. This page explains how: https://www.fossil-scm.org/home/doc/trunk/www/mirrortogithub.md https://www.fossil-scm.org/home/doc/trunk/www/mirrortogithub...
jarofgreen 3y ago
In your git & sqlite setup, I'm not sure which way round you are thinking - which is the SSOT (Single Source Of Truth) and which is the handy cache. I've been working on a tool that treats the git repo as the SSOT then lets you dump out all kinds of formats for data work including a sqlite DB. I haven't had as much time as I would like on it but it's at https://pypi.org/project/DataTig/ https://pypi.org/project/DataTig/
rrrrrrrrrrrryan 3y ago
> oft-changing-but-rarely-majorly data I think you're referring to SCDs, and there are plenty of well-defined ways to track these within relational databases: https://en.wikipedia.org/wiki/Slowly_changing_dimension https://en.wikipedia.org/wiki/Slowly_changing_dimension Why git?
GauntletWizard 3y ago
If anyone is coming in late, I have discovered https://immudb.io/ https://immudb.io/ , which claims to be a lot of what I'm looking for. Transactional SQL database, indexed, yet immutable data and verifiable chains of transactions.

rofrol 3y ago

Maybe this https://github.com/dolthub/dolt https://github.com/dolthub/dolt

GauntletWizard 3y ago
Dolt is really close and yet just doesn't feel right; I don't want to "commit" between transactions, I want every transaction to be a commit.
- timsehn 3y ago
  You can do this with a setting: https://docs.dolthub.com/sql-reference/version-control/dolt-sysvars#dolt_transaction_commit https://docs.dolthub.com/sql-reference/version-control/dolt-...
  - GauntletWizard 3y ago
    That's awesome. I will have to try that out.

ElectricalUnion 3y ago

Would fossil fit this bill? Somewhat git-compatible, based on sqlite3.

JonChesterfield 3y ago
Specifically the fossil repo is a sqlite database. I have a cron script stashing whatever is currently in an obsidian vault into a fossil repo. There's a fossil addremove command that makes that very easy. Thus distributed backups of said vault.
Zambyte 3y ago
What does "somewhat git-compatible" mean? Can it sometimes use existing git repos, or is the mental model close enough to pick up without learning much?
- ElectricalUnion 3y ago
  > is the mental model close enough to pick up without learning much? I would say git core concepts are pretty similar to fossil concepts, but actual plumbing implementation details are pretty distinct. The major difference that I remember from a day-to-day "git porcelain" perspective is that rebases and other types of history rewriting are very discouraged. For a Rosetta stone of somewhat comparable commands: https://fossil-scm.org/home/doc/trunk/www/gitusers.md https://fossil-scm.org/home/doc/trunk/www/gitusers.md > Can it sometimes use existing git repos You can, but it's kinda a lot of really slow busywork. And you lose some of the not-file-dvcs features of Fossil, but it is possible. This page explains how: https://www.fossil-scm.org/home/doc/trunk/www/mirrortogithub.md https://www.fossil-scm.org/home/doc/trunk/www/mirrortogithub...

jarofgreen 3y ago

In your git & sqlite setup, I'm not sure which way round you are thinking - which is the SSOT (Single Source Of Truth) and which is the handy cache. I've been working on a tool that treats the git repo as the SSOT then lets you dump out all kinds of formats for data work including a sqlite DB. I haven't had as much time as I would like on it but it's at https://pypi.org/project/DataTig/ https://pypi.org/project/DataTig/

rrrrrrrrrrrryan 3y ago

> oft-changing-but-rarely-majorly data I think you're referring to SCDs, and there are plenty of well-defined ways to track these within relational databases: https://en.wikipedia.org/wiki/Slowly_changing_dimension https://en.wikipedia.org/wiki/Slowly_changing_dimension Why git?

GauntletWizard 3y ago

If anyone is coming in late, I have discovered https://immudb.io/ https://immudb.io/ , which claims to be a lot of what I'm looking for. Transactional SQL database, indexed, yet immutable data and verifiable chains of transactions.