4 ms·
This useful for large binaries?
by causal 2y ago
This useful for large binaries?
- natsucks 2y agoWould appreciate a good answer to this question. I deal with large medical imaging data (DICOM) and i cannot tell whether it's worth it and/or feasible.
- thangngoc89 2y agoIt's very much feasible. I'm currently using DVC for DICOM, the repo has growth to about 5TB of small dcm files (less than < 100KB each). We use a NFS mounted NAS for development but the DVC's cache needs to be on the NVMe, otherwise performance would be terrible.
- tomnicholas1 2y agoYou should look at Icechunk. Your imaging data is structured (it's a multidimensional array), so it should be possible be to represent it as "Virtual Zarr". Then you could commit it to an Icechunk store. https://earthmover.io/blog/icechunk https://earthmover.io/blog/icechunk
- dmpetrov 2y agoYes. And if you track transformations of the binaries or ml training
- mkbehbehani 2y agoYes, I’ve been using it for about a year to populate databases with a reference DB dump. The current file is about 18GB. I use cloudflare R2 as the backing store so even though it’s being pulled very frequently the cloudflare bill is a few bucks per month.
- kbumsik 2y agoLarge files are good, but it may have performance issues with many (millions) small files.