5 ms·

Huh, I thought it already had this! Must have mixed it up with a different object storage service (maybe DigitalOcean?). I've been using B2 for backup storage

Huh, I thought it already had this! Must have mixed it up with a different object storage service (maybe DigitalOcean?). I've been using B2 for backup storage for some personal projects. It doesn't necessarily do anything "better" than S3 from what I've seen, but never having to log into AWS's dashboard is a reward enough on its own. They do have a command-line client that's a quick PIP install, so you can do something like: b2 upload-file bucket-name /path/to/file remote-filename Which is, of course, nice for backups.

Hamuko 6y ago
>Huh, I thought it already had this! Same since it seems to be on the new storage service launch checklist right after "buy hard drives".
neurostimulant 6y ago
I really wish the B2 client support uploading file from Unix pipe. It would be nice to be able to archive a huge directory into a tar.bz archive and directly pipe the result into the B2 client without having to save the archive into disk first. Currently I have to save the tar.bz archive to disk first before uploading to balance. Took several hours to do so (huge spinning disks array, not as fast as ssd), while uploading to B2 is blazing fast. Saving the archive to ramdrive essentially solved this, but as the data grows I don't have enough memory to spare anymore for a ram drive that can fit the whole archive.
- jorams 6y ago
  Can you use process substitution? b2 upload_file bucket <(tar -cj huge-directory) archive.tar.bz2 The argument the command sees will be something like "/dev/fd/42", and the shell will provide the output of tar through that file.
  - neurostimulant 6y ago
    Thanks for the idea! I'm going to try this.
  - neurostimulant 6y ago
    Does process substitution actually wrote the content to disk first or not? The information on internet I found seem to be conflicting on this. If it's actually writing the data into disk first, then it's probably won't solve my problem (limited disk i/o). Afaik writing to pipe won't result in saving the data to disk temporarily. I guess the only way to know is to try it out on my system and see how it performs.
    - Ineentho 6y ago
      If process substitution doesn't work, shouldn't /dev/stdin work? I haven't tried it, but as long as b2 doesn't try to check the file size before uploading I don't see why it wouldn't work: b2 upload_file bucket /dev/stdin < file
    - ADefenestrator 6y ago
      It definitely doesn't write it to disk first. It's basically a pipe() under the hood, but exposed as a file descriptor. Downside is that seeking doesn't work, but that shouldn't affect your case.
  - neurostimulant 6y ago
    So apparently process substitution doesn't work. The b2 client is probably trying to read file size or something, keep failing with "ERROR: Invalid upload source: /dev/fd/xx". Maybe their api requires knowing the filesize upfront instead of allowing "streaming" upload.

Hamuko 6y ago

>Huh, I thought it already had this! Same since it seems to be on the new storage service launch checklist right after "buy hard drives".

neurostimulant 6y ago

I really wish the B2 client support uploading file from Unix pipe. It would be nice to be able to archive a huge directory into a tar.bz archive and directly pipe the result into the B2 client without having to save the archive into disk first. Currently I have to save the tar.bz archive to disk first before uploading to balance. Took several hours to do so (huge spinning disks array, not as fast as ssd), while uploading to B2 is blazing fast. Saving the archive to ramdrive essentially solved this, but as the data grows I don't have enough memory to spare anymore for a ram drive that can fit the whole archive.

jorams 6y ago
Can you use process substitution? b2 upload_file bucket <(tar -cj huge-directory) archive.tar.bz2 The argument the command sees will be something like "/dev/fd/42", and the shell will provide the output of tar through that file.
- neurostimulant 6y ago
  Thanks for the idea! I'm going to try this.
- neurostimulant 6y ago
  Does process substitution actually wrote the content to disk first or not? The information on internet I found seem to be conflicting on this. If it's actually writing the data into disk first, then it's probably won't solve my problem (limited disk i/o). Afaik writing to pipe won't result in saving the data to disk temporarily. I guess the only way to know is to try it out on my system and see how it performs.
  - Ineentho 6y ago
    If process substitution doesn't work, shouldn't /dev/stdin work? I haven't tried it, but as long as b2 doesn't try to check the file size before uploading I don't see why it wouldn't work: b2 upload_file bucket /dev/stdin < file
  - ADefenestrator 6y ago
    It definitely doesn't write it to disk first. It's basically a pipe() under the hood, but exposed as a file descriptor. Downside is that seeking doesn't work, but that shouldn't affect your case.
- neurostimulant 6y ago
  So apparently process substitution doesn't work. The b2 client is probably trying to read file size or something, keep failing with "ERROR: Invalid upload source: /dev/fd/xx". Maybe their api requires knowing the filesize upfront instead of allowing "streaming" upload.