3 ms·
I am not your target audience but I have been thinking of building a very minified version of this using [0] Pooch and [1] S3FS. Right now we spend a lot of ti
by whinvik 2y ago
I am not your target audience but I have been thinking of building a very minified version of this using [0] Pooch and [1] S3FS.
Right now we spend a lot of time downloading various stuff from HTTP or S3 links and then figuring out folder structures to keep them in our S3 buckets. Pooch really simplifies the caching for this by having a deterministic path on your local storage for downloaded files, but has no S3 backend.
So a combination of 2 would be to just have 1 call to a link that would embed the caching both locally and on our S3 buckets deterministically.
[0] https://www.fatiando.org/pooch/latest/ https://www.fatiando.org/pooch/latest/
[1] https://s3fs.readthedocs.io/en/latest/ https://s3fs.readthedocs.io/en/latest/
- huntaub 2y agoI think this is a great insight, and something that I think about often. The challenge that I see is that the scientist archetype (whether it's data science, AI researcher, or anything else) isn't really interested in doing software development for these kinds of things. They just want the data to be there, and it's super nice to be able to click through the S3 console to be able to see and share the data their using. I think that what you're doing is a great idea for folks who are accessing their data primarily through Python programs!