4 ms·

Does anybody have a sense of how fast uploading to S3 can get? I'm on a gigabit link here in Singapore, uploading to Singapore instance of S3 (via ARQ) - and I

Does anybody have a sense of how fast uploading to S3 can get? I'm on a gigabit link here in Singapore, uploading to Singapore instance of S3 (via ARQ) - and I'm disappointed that I rarely see better than about 40 Mbits/second.

MBCook 11y ago
Testing in my office (central US) to N. Virginia and Oregon was getting about 90 Mbit/sec. I'm guessing that's the cap on our connection, not an Amazon limit.
philliphaydon 11y ago
I'm in Singapore, uploading to Oregon, 100mb, took minutes , I changed to upload via signed cloud front requests, takes 15s to upload 100mb now...
- ghshephard 11y ago
  I presume you mean 100 MB (or just plain 100 MBytes) - so, 53 megabits/second. So a little better, but nowhere near 1 gigabit (which I presume you have, it's so cheap here - not that it seems to be useful for anything other than great speedtest results)
  - philliphaydon 11y ago
    I use homeplugs because the router and computer are in different rooms. So I only get about ~300mbit from my room. I've never actually tried from next to the router. My point was just that I get better results using CloudFront than using S3 or Multi Upload S3. Sharing what I can :)
twistedpair 11y ago
Make sure you chunk it and use the S3 multi-part upload API. That way, you can break into N chunks, and then upload N in parallel, to get closer to saturating your uplink. The same works for downloads, where you can use byte range GET's to essentially do multi-part download.
hrez 11y ago
try https://github.com/rlmcpherson/s3gof3r https://github.com/rlmcpherson/s3gof3r you can often get wire speed
lox 11y ago
I wrote a benchmark a while back: https://github.com/lox/s3-bench https://github.com/lox/s3-bench
bagels 11y ago
900-950 Mbps on a 1000 Mbps line, but only if you use 20 threads and multipart
kalleboo 11y ago
I live in southern Japan, with a single thread to a Tokyo bucket I get around 200-300 MBit, which is about what my WiFi network maxes out at edit: uploading to a Singapore bucket I get around 90 Mbit

MBCook 11y ago

Testing in my office (central US) to N. Virginia and Oregon was getting about 90 Mbit/sec. I'm guessing that's the cap on our connection, not an Amazon limit.

philliphaydon 11y ago

I'm in Singapore, uploading to Oregon, 100mb, took minutes , I changed to upload via signed cloud front requests, takes 15s to upload 100mb now...

ghshephard 11y ago
I presume you mean 100 MB (or just plain 100 MBytes) - so, 53 megabits/second. So a little better, but nowhere near 1 gigabit (which I presume you have, it's so cheap here - not that it seems to be useful for anything other than great speedtest results)
- philliphaydon 11y ago
  I use homeplugs because the router and computer are in different rooms. So I only get about ~300mbit from my room. I've never actually tried from next to the router. My point was just that I get better results using CloudFront than using S3 or Multi Upload S3. Sharing what I can :)

twistedpair 11y ago

Make sure you chunk it and use the S3 multi-part upload API. That way, you can break into N chunks, and then upload N in parallel, to get closer to saturating your uplink. The same works for downloads, where you can use byte range GET's to essentially do multi-part download.

hrez 11y ago

try https://github.com/rlmcpherson/s3gof3r https://github.com/rlmcpherson/s3gof3r you can often get wire speed

lox 11y ago

I wrote a benchmark a while back: https://github.com/lox/s3-bench https://github.com/lox/s3-bench

bagels 11y ago

900-950 Mbps on a 1000 Mbps line, but only if you use 20 threads and multipart

kalleboo 11y ago

I live in southern Japan, with a single thread to a Tokyo bucket I get around 200-300 MBit, which is about what my WiFi network maxes out at edit: uploading to a Singapore bucket I get around 90 Mbit