3 ms·

The HuggingFace datasets python package is free and can handle images inside Apache Arrow just fine. Combined with a few Exos, NVMes, and dm-cache, storing TBs

The HuggingFace datasets python package is free and can handle images inside Apache Arrow just fine. Combined with a few Exos, NVMes, and dm-cache, storing TBs of data becomes very affordable and I have direct 10G access and can mmap on the network share. In my opinion, this space will be won based on price per throughput. But you're charging $1k monthly for 1TB? And my hobbyist speech recognition projects (30TB) vastly exceed your enterprise plan? That discrepancy makes it challenging to believe that your team has as much experience with actual AI projects, as the reference to self-driving cars would suggest. Anything that easily fits onto a consumer NAS is probably not an enterprise data lake. A one-time €1700 purchase (equivalent to 2 months of your service) will buy me a TS-464-4G with 32TB RAID1 and 5 years of warranty.

avrionov 4y ago
How do you store your data set? And do you have any open source releases?
- fxtentacle 4y ago
  "HuggingFace datasets" is an open source Python package: https://github.com/huggingface/datasets/ https://github.com/huggingface/datasets/ And they also have ready-to-use scripts for A LOT of the usual datasets: https://huggingface.co/datasets https://huggingface.co/datasets including LAION 400M and LAION 2B: https://huggingface.co/datasets/laion/laion2B-en https://huggingface.co/datasets/laion/laion2B-en
  - davidbuniat 4y ago
    thanks for the context and links! I replied to your comment slightly above in one comment. :)
- davidbuniat 4y ago
  You can store your data either remotely or locally (see here on how https://docs.activeloop.ai/getting-started/creating-datasets-manually https://docs.activeloop.ai/getting-started/creating-datasets...). You can then visualize your datasets if their stored on our cloud, in AWS/GCP, or you can drag and drop your local dataset in Deep Lake format into our UI (https://docs.activeloop.ai/dataset-visualization https://docs.activeloop.ai/dataset-visualization) We do, with version control, Python based dataloader and dataset format being open source! Please check out https://github.com/activeloopai/deeplake https://github.com/activeloopai/deeplake.
davidbuniat 4y ago
Re: HF - we know them and admire their work (primarily, until very recently, focused on NLP, while we focus mostly on CV). As mentioned in the post, a large part of Deep Lake, including the Python-based dataloader and dataset format, is open source as well - https://github.com/activeloopai/deeplake https://github.com/activeloopai/deeplake. Likewise, we curate a list of large open source datasets here -> https://datasets.activeloop.ai/docs/ml/ https://datasets.activeloop.ai/docs/ml/, but our main thing isn't aggregating datasets (focus for HF datasets), but rather providing people with a way to manage their data efficiently. That being said, all of the 125+ public datasets we have are available in seconds with one line of code. :) We haven't benchmarked against HF datasets in a while, but Deep Lake's dataloader is much, much faster in third-party benchmarks (see this https://arxiv.org/pdf/2209.13705 https://arxiv.org/pdf/2209.13705 and here for an older version, that was much slower than what we have now, see this: https://pasteboard.co/la3DmCUR2iFb.png https://pasteboard.co/la3DmCUR2iFb.png). HF under the hood uses Git-LFS (to the best of my knowledge) and is not opinionated on formats, so LAION just dumps Parquet files on their storage. While your setup would work for a few TBs, scaling to PB would be tricky including maintaining your own infrastructure. And yep, as you said NAS/NFS would neither be able to handle the scale (especially writes with 1k workers). I am also slightly curious about your use of mmap files with image/video compressed data (as zero-copy won’t happen) unless you decompress inside the GPU ;), but would love to learn more from you! Re: pricing thanks for the feedback, storage is one component and customly priced for PB-scale workloads.
- fxtentacle 4y ago
  I was referring specifically to this page: https://www.activeloop.ai/pricing/ https://www.activeloop.ai/pricing/ which says "Deep Lake Enterprise" and then "10TB of managed data (total)". To me, that read as if the Enterprise plan is limited to a maximum of 10TB.
  - davidbuniat 4y ago
    Ah, thank you so much for noticing this! this is a very important piece of feedback (we will fix it shortly). What we meant was the 10TB is the first tier of the enterprise plans, and the rest are more custom-billed (typically because those also require other custom integrations, etc). If you find any other points of confusion, please send them our way, and we will fix it, the community has been instrumental over the years in iterating on the product! :)
npRandom 4y ago
Full Disclosure: I'm a ML Team Lead at DagsHub. TL;DR DagsHub lets you stream datasets from any repo you can access for free. We have open-source datasets for various tasks and domains (image, video, MRI, audio, etc.) that you can use, or upload yours and stream it. Learn more - https://dagshub.com/docs/feature_guide/direct_data_access/ https://dagshub.com/docs/feature_guide/direct_data_access/ How does it work? Every DagsHub repo comes with a configured remote storage, where users can host models, datasets, or any other large file. We recently added a new capability to our open-source client and free-to-use API that enables the streaming of files stored on DagsHub Storage. It enables access to any dataset stored on DagsHub, stream it to your machine, version it, and upload it to your DagsHub repo - all from your python code. I think the coolest part of this feature is that it doesn't require any modifications to your code base or data format. You can find more info about it here -> https://dagshub.com/docs/feature_guide/direct_data_access/ https://dagshub.com/docs/feature_guide/direct_data_access/ Feel free to reach out if you have any other questions (:

avrionov 4y ago

How do you store your data set? And do you have any open source releases?

fxtentacle 4y ago
"HuggingFace datasets" is an open source Python package: https://github.com/huggingface/datasets/ https://github.com/huggingface/datasets/ And they also have ready-to-use scripts for A LOT of the usual datasets: https://huggingface.co/datasets https://huggingface.co/datasets including LAION 400M and LAION 2B: https://huggingface.co/datasets/laion/laion2B-en https://huggingface.co/datasets/laion/laion2B-en
- davidbuniat 4y ago
  thanks for the context and links! I replied to your comment slightly above in one comment. :)
davidbuniat 4y ago
You can store your data either remotely or locally (see here on how https://docs.activeloop.ai/getting-started/creating-datasets-manually https://docs.activeloop.ai/getting-started/creating-datasets...). You can then visualize your datasets if their stored on our cloud, in AWS/GCP, or you can drag and drop your local dataset in Deep Lake format into our UI (https://docs.activeloop.ai/dataset-visualization https://docs.activeloop.ai/dataset-visualization) We do, with version control, Python based dataloader and dataset format being open source! Please check out https://github.com/activeloopai/deeplake https://github.com/activeloopai/deeplake.

davidbuniat 4y ago

Re: HF - we know them and admire their work (primarily, until very recently, focused on NLP, while we focus mostly on CV). As mentioned in the post, a large part of Deep Lake, including the Python-based dataloader and dataset format, is open source as well - https://github.com/activeloopai/deeplake https://github.com/activeloopai/deeplake. Likewise, we curate a list of large open source datasets here -> https://datasets.activeloop.ai/docs/ml/ https://datasets.activeloop.ai/docs/ml/, but our main thing isn't aggregating datasets (focus for HF datasets), but rather providing people with a way to manage their data efficiently. That being said, all of the 125+ public datasets we have are available in seconds with one line of code. :) We haven't benchmarked against HF datasets in a while, but Deep Lake's dataloader is much, much faster in third-party benchmarks (see this https://arxiv.org/pdf/2209.13705 https://arxiv.org/pdf/2209.13705 and here for an older version, that was much slower than what we have now, see this: https://pasteboard.co/la3DmCUR2iFb.png https://pasteboard.co/la3DmCUR2iFb.png). HF under the hood uses Git-LFS (to the best of my knowledge) and is not opinionated on formats, so LAION just dumps Parquet files on their storage. While your setup would work for a few TBs, scaling to PB would be tricky including maintaining your own infrastructure. And yep, as you said NAS/NFS would neither be able to handle the scale (especially writes with 1k workers). I am also slightly curious about your use of mmap files with image/video compressed data (as zero-copy won’t happen) unless you decompress inside the GPU ;), but would love to learn more from you! Re: pricing thanks for the feedback, storage is one component and customly priced for PB-scale workloads.

fxtentacle 4y ago
I was referring specifically to this page: https://www.activeloop.ai/pricing/ https://www.activeloop.ai/pricing/ which says "Deep Lake Enterprise" and then "10TB of managed data (total)". To me, that read as if the Enterprise plan is limited to a maximum of 10TB.
- davidbuniat 4y ago
  Ah, thank you so much for noticing this! this is a very important piece of feedback (we will fix it shortly). What we meant was the 10TB is the first tier of the enterprise plans, and the rest are more custom-billed (typically because those also require other custom integrations, etc). If you find any other points of confusion, please send them our way, and we will fix it, the community has been instrumental over the years in iterating on the product! :)

npRandom 4y ago

Full Disclosure: I'm a ML Team Lead at DagsHub. TL;DR DagsHub lets you stream datasets from any repo you can access for free. We have open-source datasets for various tasks and domains (image, video, MRI, audio, etc.) that you can use, or upload yours and stream it. Learn more - https://dagshub.com/docs/feature_guide/direct_data_access/ https://dagshub.com/docs/feature_guide/direct_data_access/ How does it work? Every DagsHub repo comes with a configured remote storage, where users can host models, datasets, or any other large file. We recently added a new capability to our open-source client and free-to-use API that enables the streaming of files stored on DagsHub Storage. It enables access to any dataset stored on DagsHub, stream it to your machine, version it, and upload it to your DagsHub repo - all from your python code. I think the coolest part of this feature is that it doesn't require any modifications to your code base or data format. You can find more info about it here -> https://dagshub.com/docs/feature_guide/direct_data_access/ https://dagshub.com/docs/feature_guide/direct_data_access/ Feel free to reach out if you have any other questions (: