3 ms·
1) Are there any ways to do this in a performant manner? Afaik, since the tarball is still sequential with no way to jump around, interacting with this filesyst
by ctecte 5y ago
1) Are there any ways to do this in a performant manner? Afaik, since the tarball is still sequential with no way to jump around, interacting with this filesystem would be fairly slow right?
2) Agreed, still need to look more into this, although it's more involved and entails changing more of our pipeline for packaging and distributing these images.
3) I need to look into how this would handle files that haven't been downloaded yet. From what I know about httpfs filesystems, I'm not sure how much would need to be done to let this block until file needed is downloaded vs the normal behavior of calling out to get the file being requested.
5) Could you go into more detail here? Not sure I understand how file boundaries, etc can help with latency/bandwidth.
- hawski 5y agoAd 3. With squashfs via httpfs it would probably be fast enough. I remember years ago httpfs capable of booting livecd iso and it was performant enough. I'm not current with this knowledge, but is there a httpfs, that would download increasingly cache the file while doing range requests as it gets read requests for certain parts of the file?
- tarasglek 5y ago3) it just blocks and prioritizes that part of download 5) if you can chuck the file so chunks that get downloaded are whole files..less likely to do multiple requests for a single file 6)I forgot to do the best part of the optimization..do feedback-guided-optimization. You can then repack the squashfs/tar file to have all the frequently-accessed files together
- ctecte 5y agoAhh got it, this makes a lot of sense, maybe something to try next hackathon :)