6 ms·
I want to try out Google, but they need to make it easier to try it out. I have petabytes of data in S3 that I would need to move first (at least some of it).
by ap22213 10y ago
I want to try out Google, but they need to make it easier to try it out. I have petabytes of data in S3 that I would need to move first (at least some of it).
- mvitorino 10y agoYou can try "Cloud Storage Transfer": `Transfer data to your Cloud Storage buckets from Amazon Simple Storage Service (S3), HTTP/HTTPS servers or other buckets. You can schedule once-off or daily transfers, and you can filter files based on name prefix and when they were changed.` https://cloud.google.com/storage/transfer/ https://cloud.google.com/storage/transfer/
- ap22213 10y agoIt would be nice if they managed the transfer themselves via AWS Snowball. Sure, they would have upfront costs, but based on what I spend on AWS monthly, it's probably worth it to them.
- dekhn 10y agoWhy would you want to move data between AWS and GCP using a hard drive? The networks between the two cloud providers are immense.
- cobookman 10y agoIt'd avoid expensive egress costs.
- deleted 10y ago[deleted]
- ranman 10y agoAt some point (many 1000s of terrabytes) it is faster and cheaper to ship the data on physical devices than it is to send it over the network. (I work for AWS)
- dekhn 10y agoYup, and I work for Google. We routinely copy petabytes between AWS and GCP. It's not currently more efficient to ship petabytes if you include the time to copy to the device and then recover it.
- vmasto 10y agoGiven that Amazon currently offers this as a service (two different services actually, snowball and snowmobile for exabytes), i wonder what gives? Is one less informed than the other or what am I missing something? Why such disparity in opinion?
- dekhn 10y agoGoogle also offers this as a service; I've work on high performance networking file transfers. my experience is that most people who move data get very low utilization compared to the actual throughput of the network. People typically use one TCP connection, one process. high performance data transfers use thousands of TCP connections and thousands of processes. Many other people underestimate the time/labor effort of dealing with a snowball.
- vmasto 10y agoSo the question then is what are those services for?
- dgacmu 10y agoWe used a box of disks to get about 20 terabytes out of Amazon to CMU. It ended up being about 50℅ cheaper (from memory - may be off a bit) because we did not account for any employee costs. Startup, running on fumes, none of us drawing a salary, etc.
- dekhn 10y ago"It ended up cheaper because we did not account for employee costs". You basically just proved my point: things aren't cheaper if you have to factor in employee cost.
- mvitorino 10y ago"Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway."
- dekhn 10y agoI never do. Nor do I underestimate the bandwidth between cloud providers.
- mvitorino 10y agoWhat magnitude are we talking about? Genuinely curious.
- brianwawok 10y agoHah. What about about the egress cost? Getting 100 PB out of AWS is not cheap. (maybe it is cheap in actual cost, but not in what the end user has to pay to AWS cost).
- paulddraper 10y agohttps://techcrunch.com/2016/11/30/amazon-will-truck-your-massive-piles-of-data-to-the-cloud-with-an-18-wheeler/ https://techcrunch.com/2016/11/30/amazon-will-truck-your-mas...
- mvitorino 10y agoAwesome :)
- dx034 10y ago> All this power fuels one terabyte per-second of data transfer speeds across a 40 gigabyte per-second connection. How is that supposed to work? Do they mean several 40gbit/s connections? Or did I misunderstand sth here?
- paulddraper 10y agoYes, several.
- _wdh7 10y agoThe networks between the two cloud providers are also immensely expensive to use.
- Terretta 10y agoWhen we would roll out additional datacenters for our video delivery network, we shipped new storage to old DCs, loaded, then shipped to new DCs. At meaningful volumes, literally months faster than using tier 1 backbone, and a fraction of the cost as well.
- ap22213 10y agoI didn't realize that they have direct connections. The AWS data center down the street (Virginia) is directly connected to some google cloud datacenter? Sorry - I'm generally ignorant of datacenter technology. So a back of envelope calculation says that it would take around 10 days to transfer 100 PiB over Gigabit ethernet. When you say immense, do you mean faster than that?
- dekhn 10y agoI would say at least multiple 10Gbit interfaces if you wanted to transfer the data.
- NetStrikeForce 10y agoThey probably meet at some exchange very close to the DCs
- mikecb 10y agoGoogle's new NoVa DC will probably be in/around the equinix right down the street from AWS.
- paulddraper 10y agoTransferring 100 PiB at 10 Gbps (1.3 GiB/s) takes 121 days.
- deleted 10y ago[deleted]
- cobookman 10y agoEmail me at bookman@google.com and I'll be sure to start internal conversations around what we can do to help you out.