4 ms·
If you are pulling large files from S3 we have found that they can be sped up by requesting multiple ranges simultaneously. It is easy to hit 5Gb/s or 10Gb/s on
by lowbloodsugar 11y ago
If you are pulling large files from S3 we have found that they can be sped up by requesting multiple ranges simultaneously. It is easy to hit 5Gb/s or 10Gb/s on instances with the necessary bandwidth, accessing a single file, or multiple files. We have not encountered a limit on S3 itself. YMMV.
- hrez 11y agoExcellent https://github.com/rlmcpherson/s3gof3r https://github.com/rlmcpherson/s3gof3r is my tool of choice for "fast, parallelized, pipelined streaming access to Amazon S3." If you want to saturate network bandwidth with S3 that's the one tool I know that can do it.