4 ms·
I had a lot of problems when using it with a dataset of many jpg Files. The indexing for every dvc status took many minutes to check every file. Caching did no
by notrealyme123 2y ago
I had a lot of problems when using it with a dataset of many jpg Files.
The indexing for every dvc status took many minutes to check every file. Caching did not work.
Sadly I had to let go of it.
- woodson 2y agoYes, its performance is rather poor and there can be a lot of headaches with caching (especially if you're using a file system that doesn't support reflinks). For large sharded datasets (e.g. WebDataset), you're better off with other solutions, especially when your ML pipeline can stream them directly from object storage.
- dmpetrov 2y agoRight, DVC caches data for consistency and reproducibility. If caching is not needed and streaming required, we've created a sister tool DataChain. It's even supports WebDataset and can stream from tar archives and filter images by metadata. WebDataset example: https://github.com/iterative/datachain/blob/main/examples/multimodal/wds_filtered.py https://github.com/iterative/datachain/blob/main/examples/mu...
- notrealyme123 2y agoThank you! Thats news to me. I will absolutely give it a try