4 ms·

Because of the lack of fast and reliable s3 clients out there, I built https://github.com/rlmcpherson/s3gof3r https://github.com/rlmcpherson/s3gof3r, that can d

Because of the lack of fast and reliable s3 clients out there, I built https://github.com/rlmcpherson/s3gof3r https://github.com/rlmcpherson/s3gof3r, that can do over 1 Gbps with ease, both multipart uploads and parallelized downloads. The killer feature, though, is streaming that enables things like gof3r get -b <bucket> -k <key> | tar -x to extract directories tarred directories or any other streaming application. It also provides end-to-end md5 integrity checking. For objects over a few GB, I haven't found anything matching in speed or reliability.

zargon 13y ago
From when I was looking into doing something similar, I recall s3 needing to know the content-length of upload parts up front. How do you handle that for streaming? Do you buffer in memory up to the max part size so that you can give the correct content-length header for the last part? I ask because my uses would include low-memory VMs so I'm curious about the memory overhead.
- rmcpherson 13y ago
  You don't need the content length of what you are uploading for multipart uploads, see (http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadInitiate.html http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadIniti...). For each part of a multipart upload that is sent, however, you do need to include the content-length header, so that may be what you are referring to. These have a minimum size set by amazon of 5 MB. With the https://github.com/rlmcpherson/s3gof3r https://github.com/rlmcpherson/s3gof3r the memory overhead of a streaming upload is approximately part size * concurrent uploads (plus a couple of buffers in the pool). So, for example, if you configure the part size to 20 MB and set concurrent uploads to 10, that would be about 220 MB of memory usage.
  - zargon 13y ago
    Yeah, having to buffer the stream in order to set the correct value for the content-length header on the last part of the multipart upload is what I was referring to. Your example using a 20MB part size is quite feasible. Thanks! Is it safe to use the master branch? Do you have build instructions? I haven't used go before.
    - rmcpherson 13y ago
      We use the version in the master branch at CodeGuard to transfer many terabytes into and out of S3 daily. While I hesitate to call it "production-ready", as that means different things to different people, we do use it in production with no issues. If you want to try it without having to install go, there are statically linked binaries for OSX and Linux Amd/64 linked on the github page that should run on any distro without having to install any dependencies. If you have any issues or questions, feel free to contact me at the email in my HN profile. I hope it works for your use case! :)
    - rmcpherson 13y ago
      I missed your question about build instructions, sorry. I'll add them to the readme, but it's just "go get github.com/rlmcpherson/s3gof3r/gof3r" for the CLI once you have installed go. Usage instructions are in the readme already.

zargon 13y ago

From when I was looking into doing something similar, I recall s3 needing to know the content-length of upload parts up front. How do you handle that for streaming? Do you buffer in memory up to the max part size so that you can give the correct content-length header for the last part? I ask because my uses would include low-memory VMs so I'm curious about the memory overhead.

rmcpherson 13y ago
You don't need the content length of what you are uploading for multipart uploads, see (http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadInitiate.html http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadIniti...). For each part of a multipart upload that is sent, however, you do need to include the content-length header, so that may be what you are referring to. These have a minimum size set by amazon of 5 MB. With the https://github.com/rlmcpherson/s3gof3r https://github.com/rlmcpherson/s3gof3r the memory overhead of a streaming upload is approximately part size * concurrent uploads (plus a couple of buffers in the pool). So, for example, if you configure the part size to 20 MB and set concurrent uploads to 10, that would be about 220 MB of memory usage.
- zargon 13y ago
  Yeah, having to buffer the stream in order to set the correct value for the content-length header on the last part of the multipart upload is what I was referring to. Your example using a 20MB part size is quite feasible. Thanks! Is it safe to use the master branch? Do you have build instructions? I haven't used go before.
  - rmcpherson 13y ago
    We use the version in the master branch at CodeGuard to transfer many terabytes into and out of S3 daily. While I hesitate to call it "production-ready", as that means different things to different people, we do use it in production with no issues. If you want to try it without having to install go, there are statically linked binaries for OSX and Linux Amd/64 linked on the github page that should run on any distro without having to install any dependencies. If you have any issues or questions, feel free to contact me at the email in my HN profile. I hope it works for your use case! :)
  - rmcpherson 13y ago
    I missed your question about build instructions, sorry. I'll add them to the readme, but it's just "go get github.com/rlmcpherson/s3gof3r/gof3r" for the CLI once you have installed go. Usage instructions are in the readme already.