3 ms·

Are people really adding their database dumps to git? Does that scale? Without this trick, what was the point?

upon_drumhead 3y ago
Version control? I have a pile of static-ish SQLite databases for unit tests to run against. I plan on using this for when we modify those databases, it’s legible what the change is.
samox 3y ago
The entire database, no. But a table with slowly changing dimensions, or a table that "should not" change, git is a simple solution for snapshotting.
noirscape 3y ago
Not scale, no, but git also tends to work pretty well for certain data projects; given git is just a VCS it can track any kind of file (to varying degrees of usefulness - git really doesn't like files that massively reorder themselves when the editor saves them like MSWord or certain FXML generators, it completely mucks up the diffs). For hobby projects, I've absolutely committed an SQLITE database in the past to serve as an HTTP cache; I was relying on an external API with some pretty hefty usage limits but at the same time was also certain that the data I was requesting from that API would never change, so I could just keep all responses around and reuse them forever (and it was easier to cache the response than to write a bespoke serialization format for converting them). Yeah, it's technically a build artifact (because it's not code) but it's better than getting 429 errors on every new dev machine for an hour + it was just hobbyist code, nothing meant for professional use.
skadamat 3y ago
Out of the box no, but you can solve the scaling git problem by improving the deduplication: https://xethub.com/assets/docs/faq/how-xet-deduplication-works https://xethub.com/assets/docs/faq/how-xet-deduplication-wor... We're focused on this problem at XetHub. - Llama2 git repo with ~600 GB of model files: https://xethub.com/XetHub/Llama2 https://xethub.com/XetHub/Llama2 - RedPajama git repo with 3.4 terabytes of data: https://xethub.com/XetHub/RedPajama-Data-1T https://xethub.com/XetHub/RedPajama-Data-1T

Version control? I have a pile of static-ish SQLite databases for unit tests to run against. I plan on using this for when we modify those databases, it’s legible what the change is.

samox 3y ago

The entire database, no. But a table with slowly changing dimensions, or a table that "should not" change, git is a simple solution for snapshotting.

noirscape 3y ago

Not scale, no, but git also tends to work pretty well for certain data projects; given git is just a VCS it can track any kind of file (to varying degrees of usefulness - git really doesn't like files that massively reorder themselves when the editor saves them like MSWord or certain FXML generators, it completely mucks up the diffs). For hobby projects, I've absolutely committed an SQLITE database in the past to serve as an HTTP cache; I was relying on an external API with some pretty hefty usage limits but at the same time was also certain that the data I was requesting from that API would never change, so I could just keep all responses around and reuse them forever (and it was easier to cache the response than to write a bespoke serialization format for converting them). Yeah, it's technically a build artifact (because it's not code) but it's better than getting 429 errors on every new dev machine for an hour + it was just hobbyist code, nothing meant for professional use.

skadamat 3y ago

Out of the box no, but you can solve the scaling git problem by improving the deduplication: https://xethub.com/assets/docs/faq/how-xet-deduplication-works https://xethub.com/assets/docs/faq/how-xet-deduplication-wor... We're focused on this problem at XetHub. - Llama2 git repo with ~600 GB of model files: https://xethub.com/XetHub/Llama2 https://xethub.com/XetHub/Llama2 - RedPajama git repo with 3.4 terabytes of data: https://xethub.com/XetHub/RedPajama-Data-1T https://xethub.com/XetHub/RedPajama-Data-1T