9 ms·
Amazon S3 Transfer Acceleration
- nxzero 10y ago"What's a Snowball?" Animated YouTube by Amazon explains: https://youtube.com/watch?v=9uc2DSZ1wL8 https://youtube.com/watch?v=9uc2DSZ1wL8
- shortstuffsushi 10y agoEven with that, I don't get it. "Moving lots of data either requires a huge pipe, or a ton of storage disks." With that, they offer their Snowball device, which, if I'm understanding correctly, holds up to 50TB (now 80TB), which they physically ship to you, and then you ship back to them. How does this fix either of the constraints (disk space / connection pipe)?
- nxzero 10y agoWhen the last time you moved 10TB+ from onsite to offsite and: - how long did it take, - how was latency/connectivity/etc, - was it encrypted, - how much did the hardware cost - etc.
- brianwawok 10y agoSo you have 50TB of images to send me and a 10mbit internet pipe. Is it faster to FTP me the file or mail it on a hard drive?
- shortstuffsushi 10y agoTo you and @nxzero, I'm not debating whether it's better or worse, I'm trying to find out if I'm actually understanding it right. From some of the other comments, it appears that it is just a matter of them sending you a physical disk -- it makes sense now.
- brianwawok 10y agoYes they mail you a fancy disk, you fill and mail back.
- tacticus 10y agoit's 428 days to ship 50TB of stuff on a 10Mb/s pipe. 5 days if you have 1Gb/s This assumes you get perfect utilisation of the connection (which never happens) so i would probably round up to 450 to 500 days on a 10Mb/s pipe
- extra88 10y agoWith Snowball, you don't need a huge pipe because your data is being sneaker-netted to one of their data centers. You don't need a ton of storage disks because they're lending Snowball to you in the short-term as a means of copying your data to S3 (and/or Glacier) for the long-term. If you can spare a 1 gigabit connection to saturate with S3 uploads, you can send 50TB in about two weeks. It takes about a week to request a Snowball, have it arrive, you fill it (takes about a day, assuming you have a 10Gbit connection for Snowball), you ship it back, they copy the contents to AWS storage. If you don't have a spare 1 gigabit connection, the speed is that much better. Even if you don't have 10Gbit hardware to fill Snowball with, a local, dedicated 1 gigabit connection to Snowball would be much more reliable.
- mabbo 10y agoDoes it seriously take a week to get a snowball? You'd think with all the logistics Amazon owns...
- chrisseaton 10y agoI think they meant a week to request it, receive it, fill it, send it back, upload it to S3.
- extra88 10y agoA week to go from requesting a Snowball to having your data in the cloud. For example, place the request Monday morning, receive it Wednesday afternoon, immediately get started filling it, ship it back Thursday afternoon, they receive it on Saturday, hook it up, your data is in your AWS storage by the end of the day Sunday.
- notatoad 10y agoit doesn't fix the constraint about needing a ton of disks, but it solves it for you - they're loaning you a big pile of disk space to use for the transfer.
- Cozumel 10y agoI went to try it but don't see it as an option!
- vaibs 10y agoIt will be released by tomorrow.
- Cozumel 10y agoThanks!
- negrash 10y agoYou can enable in CloudBerry Explorer for Amazon S3 - http://www.cloudberrylab.com/blog/amazon-s3-transfer-acceleration-support-in-cloudberry-explorer-v4-6/ http://www.cloudberrylab.com/blog/amazon-s3-transfer-acceler...
- toomuchtodo 10y agoTL;DR Newer, larger "snowball" appliance (current = 50TB and newer 80TB version), as well as access in additional international regions. Network optimized S3 uploads (faster inbound transfers) for an extra 4 cents/GB. Sidenote: I'm curious if you can get the same speedup for inbound transfers by using a Cloudfront distribution in front of your S3 bucket for no additional charge (no cost for inbound transfer on either service).
- nnx 10y agoEven though their Amazon S3 Transfer Acceleration Speed Comparison page [1] do test uploads only, this new feature is documented to accelerate downloads as well. I guess optimized WAN network routing between S3 regions and their edge locations usually beats regular internet routing. [1] http://s3-accelerate-speedtest.s3-accelerate.amazonaws.com/en/accelerate-speed-comparsion.html http://s3-accelerate-speedtest.s3-accelerate.amazonaws.com/e...
- toomuchtodo 10y agoI didn't mention that because it wasn't in the official blog post, but you're correct. My assumption is that AWS has their own dedicated fiber/private connectivity between edges and regions.
- manigandham 10y agoIt is mentioned, although in only 1 tiny instance: > With today’s announcement, the edge network also helps to accelerate data transfers in to and out of Amazon S3.
- ladon86 10y agoYou can't upload through Cloudfront. It's a good analogy though, you can think of this feature as a "CDN for uploads", if you'll excuse the abuse of the CDN acronym.
- toomuchtodo 10y ago
- nxzero 10y agoWhat's the fastest & cheapest way to get data off Amazon's cloud services?
- morgante 10y agoIt depends entirely on the quantity of data. For huge datasets, Snowball will be fastest & cheapest (literally shipping data).
- nxzero 10y agoGreat, thanks, here's the data plan to grab: https://aws.amazon.com/datasets/ https://aws.amazon.com/datasets/ For downloading & uploading very large datasets, would it make more sense to proxy the through Amazon to a Snowball and ship it. Very possible is the answer is it depends, just trying to get a sense of how data transfer via Amazon is works and is priced.
- GeechieCloud 10y agoAlso worth checking out this when thinking about speed - https://aws.amazon.com/importexport/faqs/#when-to-use https://aws.amazon.com/importexport/faqs/#when-to-use
- vvanders 10y agoAs they used to say, never underestimate the bandwidth of a van filled with harddrives.
- cortesoft 10y agoNever underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.
- nxzero 10y agoAlways IP over Avian Carriers: https://en.m.wikipedia.org/wiki/IP_over_Avian_Carriers https://en.m.wikipedia.org/wiki/IP_over_Avian_Carriers && Sneakernets https://en.m.wikipedia.org/wiki/Sneakernet https://en.m.wikipedia.org/wiki/Sneakernet
- sudhirj 10y agoLooks like they're using the Cloudfront infra to have files sent to s3 via cloudfront. Uploads to a cloudfront edge down the street will be a lot faster than the s3 datacenter. Will this allow the HTTP sliding windows to be optimized more in any way? Any idea if this could approach UDT upload speeds for large files?
- narfz 10y agois there a way to run the benchmark from CLI? i would like to run the benchmark on our servers to see if we can get an improvement.
- ghshephard 10y agoDoes anybody have a sense of how fast uploading to S3 can get? I'm on a gigabit link here in Singapore, uploading to Singapore instance of S3 (via ARQ) - and I'm disappointed that I rarely see better than about 40 Mbits/second.
- MBCook 10y agoTesting in my office (central US) to N. Virginia and Oregon was getting about 90 Mbit/sec. I'm guessing that's the cap on our connection, not an Amazon limit.
- philliphaydon 10y agoI'm in Singapore, uploading to Oregon, 100mb, took minutes , I changed to upload via signed cloud front requests, takes 15s to upload 100mb now...
- ghshephard 10y agoI presume you mean 100 MB (or just plain 100 MBytes) - so, 53 megabits/second. So a little better, but nowhere near 1 gigabit (which I presume you have, it's so cheap here - not that it seems to be useful for anything other than great speedtest results)
- philliphaydon 10y agoI use homeplugs because the router and computer are in different rooms. So I only get about ~300mbit from my room. I've never actually tried from next to the router. My point was just that I get better results using CloudFront than using S3 or Multi Upload S3. Sharing what I can :)
- twistedpair 10y agoMake sure you chunk it and use the S3 multi-part upload API. That way, you can break into N chunks, and then upload N in parallel, to get closer to saturating your uplink. The same works for downloads, where you can use byte range GET's to essentially do multi-part download.
- disbelief 10y agoAnyone know if the quoted cost of $0.04/GB for Transfer Acceleration is instead of the $0.03/GB for standard S3 ingress, or in addition to it? Ie. is the final cost per GB $0.07 or $0.04 when using Transfer Acceleration? Edit: as pointed out to me below, there actually is no ingress cost for standard S3 uploading. The $0.03/GB is monthly storage. So the cost of this service is $0.04/GB instead of $0.00.
- agrajag 10y agoData transfer into S3 is free - it's the storage that's $0.03/GB-Month
- koolba 10y ago> Anyone know if the quoted cost of $0.04/GB for Transfer Acceleration is instead of the $0.03/GB for standard S3 ingress, or in addition to it? Ie. is the final cost per GB $0.07 or $0.04 when using Transfer Acceleration? S3 ingress is free. You're thinking of storage pricing.
- disbelief 10y agoAh yes of course, thanks. I took too brief a glance at the S3 pricing page.
- negrash 10y agoAmazon S3 Transfer Acceleration is natively supported in FREE CloudBerry Explorer http://www.cloudberrylab.com/blog/amazon-s3-transfer-acceleration-support-in-cloudberry-explorer-v4-6/ http://www.cloudberrylab.com/blog/amazon-s3-transfer-acceler... and CloudBerry Backup http://www.cloudberrylab.com/blog/amazon-s3-transfer-acceleration-in-cloudberry-backup-4-8-2/ http://www.cloudberrylab.com/blog/amazon-s3-transfer-acceler... from day one. Check it out!
- strick 10y agoStill no ipv6 for S3?
- ec109685 10y agoDo you have IPv6 only devices you'd like to support?
- sandGorgon 10y agoI'm not seeing this in my buckets. is this going to roll out gradually ?
- tylermac1 10y agoIt's turned on tomorrow.
- johnp_ 10y agoSecure Connection Failed An error occurred during a connection to aws.amazon.com. Peer attempted old style (potentially vulnerable) handshake. Error code: SSL_ERROR_UNSAFE_NEGOTIATION Really,Amazon? https://www.ssllabs.com/ssltest/analyze.html?d=aws.amazon.com https://www.ssllabs.com/ssltest/analyze.html?d=aws.amazon.co... No PFS, TLS 1.0 only, No HSTS or HPKP ??
- nbevans 10y agoI wish Azure would improve Blob Storage. It is really a shadow of S3's capabilities.
- s9ix 10y agoDid anyone else have some pretty terrible results from the speed tester? I was pretty optimistic seeing their sample screenshot - am I missing something? http://i.imgur.com/UwNBGf3.png http://i.imgur.com/UwNBGf3.png
- OniBait 10y agoIt says on the page: "Note: In general, the farther away you are from an Amazon S3 region, the higher the speed improvement you can expect from using Amazon S3 Transfer Acceleration. If you see similar speed results with and without the acceleration, your upload bandwidth or a system constraint might be limiting your speed." So I suspect you might be having that problem in your test.
- aroch 10y agoIf it makes you feel better, the accelerated ones are slower for me! http://i.imgur.com/n6DhyHj.png http://i.imgur.com/n6DhyHj.png
- CobrastanJorji 10y agoSame here, got 11%-24% slower in most regions, and only 3% faster in US-EAST-1. Sao Paulo got 60% faster, though. Regardless of my results, I've gotta hand it to Amazon on a really kickass speed comparison page. That's a great user experience. It immediately tells me whether I should be using this or not from my particular corporate network without requiring me to waste a day or so trying it out.
- thspimpolds 10y agoWe have 20Gbit of Direct Connect with public peering on. This causes upload for us to be 40% slower in primary region and 2-5% slower in the other regions in the US. We only get any benefit going to APAC, granted we aren't your "typical" or "ideal" user for this
- kalleboo 10y agoMy ISP hates Europe it seems... http://i.imgur.com/66123xv.png http://i.imgur.com/66123xv.png The accelerated speeds I were getting hovered around 250-300 MBps
- dantiberian 10y agoDo I understand this correctly that Cloudfront is still the better option for fronting static assets like JS and CSS?
- EwanToo 10y agoYes, cloudfront gives you far better control over things like headers, routing for different paths, error pages, etc, than vanilla s3 http access does, and is almost certainly lower latency compared to s3 (assuming your files aren't GB-sized)
- Benjamin_Dobell 10y agoYou don't actually have a choice: https://news.ycombinator.com/item?id=11531357 https://news.ycombinator.com/item?id=11531357
- mryan 10y agoThat is not correct. You can host a website from an S3 bucket without using CloudFront. Although of course, downloads will come directly from the S3 bucket and not from a CF edge location.
- Benjamin_Dobell 10y agoHuh, just what did you think I was saying? Of course you can put CloudFront in front of S3 website hosting!!! You can't use S3 Transfer Acceleration with S3 Static Website Hosting.
- mryan 10y agoMy apologies, I misread your comment.
- kalleboo 10y agoThat's only if you want a fancy CNAME. If you're just hosting static assets, you might not care what the domain name is
- Benjamin_Dobell 10y ago> The name of the bucket used for Transfer Acceleration must be DNS-compliant and must not contain periods ("."). Source: http://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration.html#transfer-acceleration-getting-started http://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acce... > The bucket names must match the names of the website that you are hosting. For example, to host your example.com website on Amazon S3, you would create a bucket named example.com. Source: http://docs.aws.amazon.com/AmazonS3/latest/dev/website-hosting-custom-domain-walkthrough.html http://docs.aws.amazon.com/AmazonS3/latest/dev/website-hosti... ... Looks like they don't want you replacing CloudFront with Amazon S3 Transfer Acceleration - which is a royal pain for some of my use cases (internal facing websites that don't need a full CDN).
- re 10y agoS3 Website Hosting is distinct from CloudFront; you can create a CloudFront distribution backed by a bucket with any name. You can combine the two, but it isn't required. http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/MigrateS3ToCloudFront.html http://docs.aws.amazon.com/AmazonCloudFront/latest/Developer... http://docs.aws.amazon.com/gettingstarted/latest/swh/website-hosting-intro.html http://docs.aws.amazon.com/gettingstarted/latest/swh/website...
- EwanToo 10y agoI think it's simply wildcard SSL getting in the way, if you put a . in the bucket name, the DNS record becomes a.b.c.s3-xxxx.amazonaws.com The wildcard SSL certificate they use only matches the c.s3-xxxx.amazonaws.com part of the sub domain
- peteretep 10y agoStill no uploading of gzip'd data to S3, amirite?
- mryan 10y agoWhat stops you from uploading gzip'd data to S3?
- peteretep 10y agoIt stays gzip'd once you've uploaded it.
- mryan 10y agoWell to be fair, that is not the same as "you can not upload gzip'd content". Your complaint seems to be "S3 doesn't automatically gunzip gzip files that I upload", which sounds like the desired behaviour to me. i.e. if I ever upload gzipped content to S3, it is because I want it to be served compressed over HTTP, or because I am moving a compressed backup file to S3. In neither case would gunzipping be desirable, although I do appreciate that compressing during the transfer, and decompressing on the other end, could save some bytes during transfers.
- peteretep 10y agoMy complaint is that I can't upload a gzip'd file and have it decompressed on their end, yes. Yes, the other things I could have meant would have been odd, so I'm not sure why you spent time typing them out.
- mryan 10y ago> Yes, the other things I could have meant would have been odd, so I'm not sure why you spent time typing them out. It seemed clear that you were either confused about what features were supported by S3, or not expressing your complaint correctly. The reason behind typing them out was trying to politely point this out, while describing some of the real-world use cases for using gzip'd files in S3 (partially to highlight why it would not make sense for S3 to gunzip files automatically). I felt this would be a more useful comment than "you are wrong, you can upload gzip'd files to S3", or just downvoting your comment for being incorrect.
- itaifrenkel 10y agoIs the improvement also applicable when uploading files from the s3 management console into an accelerated bucket?