3 ms·

I've been observing and trying out all sorts of different solutions in this space (dvc, git-annex, git-lfs, quilt, just rsync, etc), and so far haven't been sup

I've been observing and trying out all sorts of different solutions in this space (dvc, git-annex, git-lfs, quilt, just rsync, etc), and so far haven't been super happy with any of them. So it's nice to see more alternatives. Some questions/observations: - Python API is great and definitely more useful than CLI when it comes to regular use. But CLI would also be nice for some operations, such as import/export, see history metadata, etc. - Just a nit, but if I didn't already by know about activeloop I might be less interested due to the "deep lake" branding. There's already so much "data lake" stuff that I have no interest in since it's historically not useful for image data. - In academic contexts, it's typical to have a static dataset and always use that. In my use cases, typically I have an every-growing "raw" dataset and extract/preprocess subsets periodically for (re)training, annotation, etc. I typically consider these subsets ephemeral, as I can regenerate on demand. Last time I tried activeloop, it seemed like it was more geared towards the static dataset use case, but browsing the docs now it seems like there's more consideration of the latter case, so I'll have to look at that. - In the examples I've seen, it seems like if a JPEG image is added to the dataset with jpeg compression, it first gets decompressed, so it would have to be recompressed - a lossy operation, right? (edit: formatting)

davidbuniat 4y ago
There's indeed plenty of tools out there, so the next few years are very interesting in terms of seeing which approach gets adopted by the wider audience. Feedback from the community (inc. feedback from you!) is instrumental in designing a tool that works well and addresses all the needs, so thanks a lot for your input re: CLI and branding. :) Good observation, we've made sure that datasets can evolve better as you go. Specifically for your use case, you can query subsets of data and materialize it on the fly to be streamed, and then go back to a specific dataset "view" (i.e. saved query), as needed. See how this works at around 5:50 here - https://youtu.be/SxsofpSIw3k https://youtu.be/SxsofpSIw3k As for your last question, when the jpeg is appended using its file path, the compressed bytes get stored in the dataset without decompression/recompression. When the data is accessed as a numpy array, then the jpeg bytes are decompressed. For researchers in Academia, our Growth plan is free. Since you work at a startup, the trial for Growth plan is for two weeks. If you want access, hit us up in the Community slack (slack.activeloop.ai - or you can just test the querying on public activeloop datasets!)
- davidbuniat 4y ago
  One more thing regarding the compression -> you can both deeplake.link() to your raw data lake without touching it, or use deeplake.read() which preservers the compression as long as it matches with tensor default compression. https://docs.deeplake.ai/en/latest/deeplake.html?highlight=link#deeplake.link https://docs.deeplake.ai/en/latest/deeplake.html?highlight=l... https://docs.deeplake.ai/en/latest/deeplake.html#deeplake.read https://docs.deeplake.ai/en/latest/deeplake.html#deeplake.re...
  - dimatura 4y ago
    thanks for the pointer, I'll definitely give deeplake a closer look
    - davidbuniat 4y ago
      my pleasure, let us know if you have any feedback, would love to see you succeed wtih Deep Lake. :)

davidbuniat 4y ago

There's indeed plenty of tools out there, so the next few years are very interesting in terms of seeing which approach gets adopted by the wider audience. Feedback from the community (inc. feedback from you!) is instrumental in designing a tool that works well and addresses all the needs, so thanks a lot for your input re: CLI and branding. :) Good observation, we've made sure that datasets can evolve better as you go. Specifically for your use case, you can query subsets of data and materialize it on the fly to be streamed, and then go back to a specific dataset "view" (i.e. saved query), as needed. See how this works at around 5:50 here - https://youtu.be/SxsofpSIw3k https://youtu.be/SxsofpSIw3k As for your last question, when the jpeg is appended using its file path, the compressed bytes get stored in the dataset without decompression/recompression. When the data is accessed as a numpy array, then the jpeg bytes are decompressed. For researchers in Academia, our Growth plan is free. Since you work at a startup, the trial for Growth plan is for two weeks. If you want access, hit us up in the Community slack (slack.activeloop.ai - or you can just test the querying on public activeloop datasets!)

davidbuniat 4y ago
One more thing regarding the compression -> you can both deeplake.link() to your raw data lake without touching it, or use deeplake.read() which preservers the compression as long as it matches with tensor default compression. https://docs.deeplake.ai/en/latest/deeplake.html?highlight=link#deeplake.link https://docs.deeplake.ai/en/latest/deeplake.html?highlight=l... https://docs.deeplake.ai/en/latest/deeplake.html#deeplake.read https://docs.deeplake.ai/en/latest/deeplake.html#deeplake.re...
- dimatura 4y ago
  thanks for the pointer, I'll definitely give deeplake a closer look
  - davidbuniat 4y ago
    my pleasure, let us know if you have any feedback, would love to see you succeed wtih Deep Lake. :)