3 ms·

I think his arguments apply more closely to SQLite databases. They're not directly human readable, but boy are there a lot of tools for working with them.

askvictor 3y ago
We have a use case where we effectively need to have a relational database, but in git. The database doesn't change much, but when it does, references between tables may need to be updated. But we need to easily be able to see diffs between different versions. We're trying an SQLite DB, with exports to CSV as part of CI - the CSV files are human-readable and diff'able. It's also worth noting that SQLite can ingest CSV files into memory and perform queries on them directly - if the files are not too large, it's possible to bypass the sqlite format entirely.
- zachmu 3y ago
  Somebody already said this, but we built exactly this and it's called Dolt. https://github.com/dolthub/dolt https://github.com/dolthub/dolt Would love to hear how it addresses your use case or falls short.
- john-shaffer 3y ago
  Have you considered https://github.com/dolthub/dolt https://github.com/dolthub/dolt for your use case?
- thenbe 3y ago
  > we need to easily be able to see diffs between different versions Can git attributes help in this case? It allows you to teach git how to diff binary files using external tools. Here [0] is a demonstration for teaching git to produce an "image diff" for *.png files using exiftool. You can do something similar for *.sqlite files by adding these 3 lines [1] [2]. The sqlite3 cli needs to be installed. Alternatively, there's a tool that might also fit the bill called datafold/data-diff [3]. I'm pretty sure I originally heard of it on a HN thread so those comments may offer even more alternative solutions. [0]: https://youtu.be/Md44rcw13k4?t=540 https://youtu.be/Md44rcw13k4?t=540 [the relevant timestamp is @ 9:00] [1]: https://github.com/kriansa/dotfiles/blob/7a8c1b11b06378b8ca8ac1931c2859d70c956df2/modules/git/include/.gitconfig#L47-L48 https://github.com/kriansa/dotfiles/blob/7a8c1b11b06378b8ca8... [2]: https://github.com/kriansa/dotfiles/blob/7a8c1b11b06378b8ca8ac1931c2859d70c956df2/modules/git/include/.gitattributes#L35 https://github.com/kriansa/dotfiles/blob/7a8c1b11b06378b8ca8... [3]: https://github.com/datafold/data-diff https://github.com/datafold/data-diff

askvictor 3y ago

We have a use case where we effectively need to have a relational database, but in git. The database doesn't change much, but when it does, references between tables may need to be updated. But we need to easily be able to see diffs between different versions. We're trying an SQLite DB, with exports to CSV as part of CI - the CSV files are human-readable and diff'able. It's also worth noting that SQLite can ingest CSV files into memory and perform queries on them directly - if the files are not too large, it's possible to bypass the sqlite format entirely.

zachmu 3y ago
Somebody already said this, but we built exactly this and it's called Dolt. https://github.com/dolthub/dolt https://github.com/dolthub/dolt Would love to hear how it addresses your use case or falls short.
john-shaffer 3y ago
Have you considered https://github.com/dolthub/dolt https://github.com/dolthub/dolt for your use case?
thenbe 3y ago
> we need to easily be able to see diffs between different versions Can git attributes help in this case? It allows you to teach git how to diff binary files using external tools. Here [0] is a demonstration for teaching git to produce an "image diff" for *.png files using exiftool. You can do something similar for *.sqlite files by adding these 3 lines [1] [2]. The sqlite3 cli needs to be installed. Alternatively, there's a tool that might also fit the bill called datafold/data-diff [3]. I'm pretty sure I originally heard of it on a HN thread so those comments may offer even more alternative solutions. [0]: https://youtu.be/Md44rcw13k4?t=540 https://youtu.be/Md44rcw13k4?t=540 [the relevant timestamp is @ 9:00] [1]: https://github.com/kriansa/dotfiles/blob/7a8c1b11b06378b8ca8ac1931c2859d70c956df2/modules/git/include/.gitconfig#L47-L48 https://github.com/kriansa/dotfiles/blob/7a8c1b11b06378b8ca8... [2]: https://github.com/kriansa/dotfiles/blob/7a8c1b11b06378b8ca8ac1931c2859d70c956df2/modules/git/include/.gitattributes#L35 https://github.com/kriansa/dotfiles/blob/7a8c1b11b06378b8ca8... [3]: https://github.com/datafold/data-diff https://github.com/datafold/data-diff