4 ms·
Unfortunately, Git isn't a good way to collaborate on massive csv files for two reasons a) git stores changes at the file level, rather than row level, b) for m
by picardo 13y ago
Unfortunately, Git isn't a good way to collaborate on massive csv files for two reasons a) git stores changes at the file level, rather than row level, b) for massive datasets, you quickly run into file size limits. That's why I am looking forward to dat[1], a collaborative DVCS that's going to blow Git off the water for this kind of tasks, even if it achieves only half of its objectives.
[1] https://github.com/maxogden/dat https://github.com/maxogden/dat
- dchest 13y agogit stores changes at the file level, rather than row level It applies delta compression when storing blobs in pack files.
- rpedela 13y agogit does not handle large files well. Actually none of the open-source VCS do a good job. I think that was picardo's main point.
- zhemao 13y agoThis isn't precisely true. Git stores changes at the "blob" level. Large plaintext files (like a CSV) are generally split into several blobs. You are correct that git works badly for binary files, like images and compiled executables, But plaintext files like CSVs should be OK.