6 ms·
It would be nice if they managed the transfer themselves via AWS Snowball. Sure, they would have upfront costs, but based on what I spend on AWS monthly, it's p
by ap22213 10y ago
It would be nice if they managed the transfer themselves via AWS Snowball. Sure, they would have upfront costs, but based on what I spend on AWS monthly, it's probably worth it to them.
- dekhn 10y agoWhy would you want to move data between AWS and GCP using a hard drive? The networks between the two cloud providers are immense.
- cobookman 10y agoIt'd avoid expensive egress costs.
- deleted 10y ago[deleted]
- ranman 10y agoAt some point (many 1000s of terrabytes) it is faster and cheaper to ship the data on physical devices than it is to send it over the network. (I work for AWS)
- dekhn 10y agoYup, and I work for Google. We routinely copy petabytes between AWS and GCP. It's not currently more efficient to ship petabytes if you include the time to copy to the device and then recover it.
- vmasto 10y agoGiven that Amazon currently offers this as a service (two different services actually, snowball and snowmobile for exabytes), i wonder what gives? Is one less informed than the other or what am I missing something? Why such disparity in opinion?
- dekhn 10y agoGoogle also offers this as a service; I've work on high performance networking file transfers. my experience is that most people who move data get very low utilization compared to the actual throughput of the network. People typically use one TCP connection, one process. high performance data transfers use thousands of TCP connections and thousands of processes. Many other people underestimate the time/labor effort of dealing with a snowball.
- vmasto 10y agoSo the question then is what are those services for?
- dgacmu 10y agoWe used a box of disks to get about 20 terabytes out of Amazon to CMU. It ended up being about 50℅ cheaper (from memory - may be off a bit) because we did not account for any employee costs. Startup, running on fumes, none of us drawing a salary, etc.
- dekhn 10y ago"It ended up cheaper because we did not account for employee costs". You basically just proved my point: things aren't cheaper if you have to factor in employee cost.
- dgacmu 10y agoTechnically, that's a logical fallacy:. A&B->true does not mean !B->false. But, really, I'm not trying to prove or disprove your point. Just noting that there was a situation for us where disk made sense, and we were satisfied with the outcome. Spending 4 hours of person time to save a thousand dollars was reasonable for us in a way it probably wouldn't be for many real companies, because we had comparatively little money and we're willing to work for peanuts. (Note that I actually share your bias in this one. I both use GCP for my personal stuff and I'm writing this from a Google cafe. :-)
- mvitorino 10y ago"Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway."
- dekhn 10y agoI never do. Nor do I underestimate the bandwidth between cloud providers.
- mvitorino 10y agoWhat magnitude are we talking about? Genuinely curious.
- brianwawok 10y agoHah. What about about the egress cost? Getting 100 PB out of AWS is not cheap. (maybe it is cheap in actual cost, but not in what the end user has to pay to AWS cost).
- paulddraper 10y agohttps://techcrunch.com/2016/11/30/amazon-will-truck-your-massive-piles-of-data-to-the-cloud-with-an-18-wheeler/ https://techcrunch.com/2016/11/30/amazon-will-truck-your-mas...
- mvitorino 10y agoAwesome :)
- dx034 10y ago> All this power fuels one terabyte per-second of data transfer speeds across a 40 gigabyte per-second connection. How is that supposed to work? Do they mean several 40gbit/s connections? Or did I misunderstand sth here?
- paulddraper 10y agoYes, several.
- _wdh7 10y agoThe networks between the two cloud providers are also immensely expensive to use.
- Terretta 10y agoWhen we would roll out additional datacenters for our video delivery network, we shipped new storage to old DCs, loaded, then shipped to new DCs. At meaningful volumes, literally months faster than using tier 1 backbone, and a fraction of the cost as well.
- ap22213 10y agoI didn't realize that they have direct connections. The AWS data center down the street (Virginia) is directly connected to some google cloud datacenter? Sorry - I'm generally ignorant of datacenter technology. So a back of envelope calculation says that it would take around 10 days to transfer 100 PiB over Gigabit ethernet. When you say immense, do you mean faster than that?
- dekhn 10y agoI would say at least multiple 10Gbit interfaces if you wanted to transfer the data.
- NetStrikeForce 10y agoThey probably meet at some exchange very close to the DCs
- mikecb 10y agoGoogle's new NoVa DC will probably be in/around the equinix right down the street from AWS.
- paulddraper 10y agoTransferring 100 PiB at 10 Gbps (1.3 GiB/s) takes 121 days.