4 ms·

What's the fastest & cheapest way to get data off Amazon's cloud services?

by nxzero 11y ago

morgante 11y ago
It depends entirely on the quantity of data. For huge datasets, Snowball will be fastest & cheapest (literally shipping data).
- nxzero 11y ago
  Great, thanks, here's the data plan to grab: https://aws.amazon.com/datasets/ https://aws.amazon.com/datasets/ For downloading & uploading very large datasets, would it make more sense to proxy the through Amazon to a Snowball and ship it. Very possible is the answer is it depends, just trying to get a sense of how data transfer via Amazon is works and is priced.
  - GeechieCloud 11y ago
    Also worth checking out this when thinking about speed - https://aws.amazon.com/importexport/faqs/#when-to-use https://aws.amazon.com/importexport/faqs/#when-to-use
- vvanders 11y ago
  As they used to say, never underestimate the bandwidth of a van filled with harddrives.
  - cortesoft 11y ago
    Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.
    - nxzero 11y ago
      Always IP over Avian Carriers: https://en.m.wikipedia.org/wiki/IP_over_Avian_Carriers https://en.m.wikipedia.org/wiki/IP_over_Avian_Carriers && Sneakernets https://en.m.wikipedia.org/wiki/Sneakernet https://en.m.wikipedia.org/wiki/Sneakernet
jewel 11y ago
I know of three ways to get data out of amazon's cloud. Whether they are the fastest depends on how much data you're talking about. 1) Download the data and pay egress bandwidth charges. Starts at $90/TB, but gets cheaper with more usage. 2) Use import/export snowball to have disks shipped to you. This is $30/TB + $250 per 80TB device. 3) Use direct connect to connect fiber directly to the region. Costs $1620/mo for each 10G line + $30/TB + an unknown amount to your fiber provider. It's too bad that export traffic can't be marked as "low priority" with a cheaper cost. I imagine there are times at night when utilization is low and big export jobs could be run during those times. (It's obviously not in AWS's interest to make it cheap to get data out of their data centers.)
rsync 11y ago
"What's the fastest & cheapest way to get data off Amazon's cloud services?" We[1] may have a good answer for you ... Let's assume you have terabytes of data, otherwise there's no difficulty, right ? So, when you combine the HN readers discount pricing for 10TB datasets, which is 4c/GB/mo. with the fact that we support 's3cmd' in our environment: ssh user@rsync.net s3cmd get s3://rsync/mscdex.exe ... and the fact that we have no other charges (no transfer/usage/bandwidth charges) ... ... and the fact that we have 10gbps connectivity through he.net ... It's possible that we would be a good fit. You fire up a 10TB account for $400/mo and you issue s3cmd commands, over SSH, on your rsync.net account, which "pulls" the data from S3, at pretty much whatever speed Amazon can throw at us.[2] Email us. [1] rsync.net [2] http://arstechnica.com/information-technology/2015/12/rsync-net-zfs-replication-to-the-cloud-is-finally-here-and-its-fast/ http://arstechnica.com/information-technology/2015/12/rsync-... ... scroll down to the part where they do the speed tests ...

What's the fastest & cheapest way to get data off Amazon's cloud services?

by nxzero 11y ago

What's the fastest & cheapest way to get data off Amazon's cloud services?

It depends entirely on the quantity of data. For huge datasets, Snowball will be fastest & cheapest (literally shipping data).

nxzero 11y ago
Great, thanks, here's the data plan to grab: https://aws.amazon.com/datasets/ https://aws.amazon.com/datasets/ For downloading & uploading very large datasets, would it make more sense to proxy the through Amazon to a Snowball and ship it. Very possible is the answer is it depends, just trying to get a sense of how data transfer via Amazon is works and is priced.
- GeechieCloud 11y ago
  Also worth checking out this when thinking about speed - https://aws.amazon.com/importexport/faqs/#when-to-use https://aws.amazon.com/importexport/faqs/#when-to-use
vvanders 11y ago
As they used to say, never underestimate the bandwidth of a van filled with harddrives.
- cortesoft 11y ago
  Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.
  - nxzero 11y ago
    Always IP over Avian Carriers: https://en.m.wikipedia.org/wiki/IP_over_Avian_Carriers https://en.m.wikipedia.org/wiki/IP_over_Avian_Carriers && Sneakernets https://en.m.wikipedia.org/wiki/Sneakernet https://en.m.wikipedia.org/wiki/Sneakernet

jewel 11y ago

I know of three ways to get data out of amazon's cloud. Whether they are the fastest depends on how much data you're talking about. 1) Download the data and pay egress bandwidth charges. Starts at $90/TB, but gets cheaper with more usage. 2) Use import/export snowball to have disks shipped to you. This is $30/TB + $250 per 80TB device. 3) Use direct connect to connect fiber directly to the region. Costs $1620/mo for each 10G line + $30/TB + an unknown amount to your fiber provider. It's too bad that export traffic can't be marked as "low priority" with a cheaper cost. I imagine there are times at night when utilization is low and big export jobs could be run during those times. (It's obviously not in AWS's interest to make it cheap to get data out of their data centers.)

rsync 11y ago

"What's the fastest & cheapest way to get data off Amazon's cloud services?" We[1] may have a good answer for you ... Let's assume you have terabytes of data, otherwise there's no difficulty, right ? So, when you combine the HN readers discount pricing for 10TB datasets, which is 4c/GB/mo. with the fact that we support 's3cmd' in our environment: ssh user@rsync.net s3cmd get s3://rsync/mscdex.exe ... and the fact that we have no other charges (no transfer/usage/bandwidth charges) ... ... and the fact that we have 10gbps connectivity through he.net ... It's possible that we would be a good fit. You fire up a 10TB account for $400/mo and you issue s3cmd commands, over SSH, on your rsync.net account, which "pulls" the data from S3, at pretty much whatever speed Amazon can throw at us.[2] Email us. [1] rsync.net [2] http://arstechnica.com/information-technology/2015/12/rsync-net-zfs-replication-to-the-cloud-is-finally-here-and-its-fast/ http://arstechnica.com/information-technology/2015/12/rsync-... ... scroll down to the part where they do the speed tests ...