ctecte

7 ms·

1.
by ctecte 5y ago
Ahh got it, this makes a lot of sense, maybe something to try next hackathon :)
2.
by ctecte 5y ago
This was written in golang. Specifically channels and io.Pipe objects made it pretty straight forward to "glue" different phases of download and extraction together into the final solution.
3.
by ctecte 5y ago
Oh nice! Yeah this is basically the same design/intuition that this project also started out with (I also just wanted some way to combine the benefits of aria2c with piping). Wish I found this earlier lol. I only later decided to look
4.
by ctecte 5y ago
I did look into performing stuff like md5 inline while streaming the download, however it tended to slow down the extraction significantly. Other less rigorous checks like crc32 are still possible inline though with minimal performance redu
5.
by ctecte 5y ago
Company wanted to see if there was any interest in it before putting in effort to open source :/ Hoping this is enough to release it on Github.
6.
by ctecte 5y ago
1) Are there any ways to do this in a performant manner? Afaik, since the tarball is still sequential with no way to jump around, interacting with this filesystem would be fairly slow right? 2) Agreed, still need to look more into this, alt
7.
by ctecte 5y ago
Had not considered that. Reading about squashfs I see that it's readonly. Do you know if that's fine when mounting the directory for an LXC container? Also I wonder how much performance impact (if any) there might be on spark jobs
8.
by ctecte 5y ago
Author here, it's true that GNU tar auto-detects compression when specifying a local tarball to extract. However when piping a file stream to tar to extract it actually doesn't auto-detect and fails if you don't explicitly sp
9.
Speeding up LXC container pull by up to 3x(databricks.com)
97 points by ctecte 5y ago|22 comments
10.
Fastar, CLI to speed up tar extraction(databricks.com)
4 points by ctecte 5y ago|1 comments