3 ms·
Can you use process substitution? b2 upload_file bucket <(tar -cj huge-directory) archive.tar.bz2 The argument the command sees will be something like "/d
by jorams 6y ago
Can you use process substitution?
b2 upload_file bucket <(tar -cj huge-directory) archive.tar.bz2
The argument the command sees will be something like "/dev/fd/42", and the shell will provide the output of tar through that file.
- neurostimulant 6y agoThanks for the idea! I'm going to try this.
- neurostimulant 6y agoDoes process substitution actually wrote the content to disk first or not? The information on internet I found seem to be conflicting on this. If it's actually writing the data into disk first, then it's probably won't solve my problem (limited disk i/o). Afaik writing to pipe won't result in saving the data to disk temporarily. I guess the only way to know is to try it out on my system and see how it performs.
- Ineentho 6y agoIf process substitution doesn't work, shouldn't /dev/stdin work? I haven't tried it, but as long as b2 doesn't try to check the file size before uploading I don't see why it wouldn't work: b2 upload_file bucket /dev/stdin < file
- ADefenestrator 6y agoIt definitely doesn't write it to disk first. It's basically a pipe() under the hood, but exposed as a file descriptor. Downside is that seeking doesn't work, but that shouldn't affect your case.
- neurostimulant 6y agoSo apparently process substitution doesn't work. The b2 client is probably trying to read file size or something, keep failing with "ERROR: Invalid upload source: /dev/fd/xx". Maybe their api requires knowing the filesize upfront instead of allowing "streaming" upload.