5 ms·
I was thinking of something similar for applications. Just earlier today I needed a large dataset for some machine learning with Jupyter notebook in a git repo.
by QasimK 9y ago
I was thinking of something similar for applications. Just earlier today I needed a large dataset for some machine learning with Jupyter notebook in a git repo. I don't want to store the dataset in the git repo due to the size, and I don't really want to deal with "downloading the file if it isn't already downloaded into some location".
What would be awesome is something like this little bit of hypothetical Python code:
import ipfs
file = ipfs.file(FILE_HASH, keep=True, share=True) # THIS LINE
with open(file) as fin:
# Do stuff like a normal file.
No need to distinguish between local files and cloud files! You can choose whether to keep the file permanently or temporarily, and whether to share it back to other IPFS nodes. You would probably need a system service to do this seamlessly as you suggested.
Sure, it might take a some time to download it at first - but you have to deal with that either way. On subsequent runs, it all just works. (I assume for popular enough files you don't have to worry about the availability of the file, especially if people tend to share back by default).
The big benefit being it is simpler to share datasets and actually use datasets - you can do both at the same time with one little file content identifier (FILE_HASH above).
- Jtsummers 9y agoIPFS includes a way to mount it as a filesystem. So if you know the IPNS URL for some data, or the hash, you can access it like accessing any other file. If it's already downloaded then there shouldn't be any overhead versus just opening a file, if it hasn't it'll download it. You oughtn't need any extra library or logic for it.
- QasimK 9y agoThat's brilliant! I think that means it should also support reading the file in chunks, as it gets downloaded, automatically. I wonder how purging less important files is/could be handled through this? It would be good if there was some way to say, "this file is likely only going to be used temporarily (or just once), so feel free to delete it if disk space is required". That hinting might not be necessary as IPFS could use some least-recently used algorithm, but there should be some way to at least pin files, "this is really important, don't delete this".
- voltagex_ 9y agoWindows has been experimenting with variations of remote-files-as-local via OneDrive for some time now. I really think it needs to be baked in to the filesystem or at least the file browser to be successful, and be absolutely transparent for the application that's requesting the file.