4 ms·
How do you store your data set? And do you have any open source releases?
by avrionov 4y ago
How do you store your data set? And do you have any open source releases?
- fxtentacle 4y ago"HuggingFace datasets" is an open source Python package: https://github.com/huggingface/datasets/ https://github.com/huggingface/datasets/ And they also have ready-to-use scripts for A LOT of the usual datasets: https://huggingface.co/datasets https://huggingface.co/datasets including LAION 400M and LAION 2B: https://huggingface.co/datasets/laion/laion2B-en https://huggingface.co/datasets/laion/laion2B-en
- davidbuniat 4y agothanks for the context and links! I replied to your comment slightly above in one comment. :)
- davidbuniat 4y agoYou can store your data either remotely or locally (see here on how https://docs.activeloop.ai/getting-started/creating-datasets-manually https://docs.activeloop.ai/getting-started/creating-datasets...). You can then visualize your datasets if their stored on our cloud, in AWS/GCP, or you can drag and drop your local dataset in Deep Lake format into our UI (https://docs.activeloop.ai/dataset-visualization https://docs.activeloop.ai/dataset-visualization) We do, with version control, Python based dataloader and dataset format being open source! Please check out https://github.com/activeloopai/deeplake https://github.com/activeloopai/deeplake.