3 ms·
Yes, its performance is rather poor and there can be a lot of headaches with caching (especially if you're using a file system that doesn't support reflinks). F
by woodson 2y ago
Yes, its performance is rather poor and there can be a lot of headaches with caching (especially if you're using a file system that doesn't support reflinks). For large sharded datasets (e.g. WebDataset), you're better off with other solutions, especially when your ML pipeline can stream them directly from object storage.
- dmpetrov 2y agoRight, DVC caches data for consistency and reproducibility. If caching is not needed and streaming required, we've created a sister tool DataChain. It's even supports WebDataset and can stream from tar archives and filter images by metadata. WebDataset example: https://github.com/iterative/datachain/blob/main/examples/multimodal/wds_filtered.py https://github.com/iterative/datachain/blob/main/examples/mu...
- notrealyme123 2y agoThank you! Thats news to me. I will absolutely give it a try