4 ms·

I work on Google Cloud and have an obvious bias here, but having thought about this for a couple year now, this does not seem like a good or cost-effective way

I work on Google Cloud and have an obvious bias here, but having thought about this for a couple year now, this does not seem like a good or cost-effective way to move that much data. In particular if you have ~100PB of data to move and you are in a location that can be reached by a 45-foot long shipping container on a truck with access to the ~350KW of power it takes to run Snowmobile, you're clearly not somewhere _completely_ inaccessible. Given that 100PB of data will cost you $400k/mo in storage costs on AWS Glacier (before the discounts that you'll obviously negotiate for), even relatively remote locations become "accessible" for 100 Gbps+ fiber or microwave connectivity, and _very_ remote locations for tens of Gbps. The Snowmobile itself is only 1 Tb/s, so if you think you're going to save time this work, consider how long it takes to move it to you and then move it to an AWS facility versus the time it takes to fill it. I don't get where this is the right call for _any_ customers, even as a one-off. I'd love it if someone from AWS could tell me where my math is off on this.

snowwrestler 7y ago
It reads to me like you're saying: if you're going to spend $400k/month on storage cost, why not pay for a network buildout too? I don't think that allocating X per month for op-ex implies that you also have Y amount of money to spend on cap-ex... especially if you don't own the location. 350kw of power is just another trailer to pull up.
- jsolson 7y ago
  I am assuming that anyone who managed to accumulate 100 PB of data (or close enough to make this viable) is still accumulating data at a solid clip, or likely has a lot _more_ than just 100 PB of data
KaiserPro 7y ago
its about timing. Say I have datacentre in london, Sure I could plumb in 4 100gig lines. But they are typically on a year lease and have a long lead time. Plus, once its on the datatruck its AWS's problem.
mike_d 7y ago
There is your first mistake: thinking the customer wants to solve the problem in a a good or cost-effective way. Snowmobile was built because a pre-sales engineer sarcastically said backing up a truck full of hard drives to a customers datacenter would be the fastest way to import X PB of data. It's probably loading from a tape silo and spent 18 months at the customer site.
- milesward 7y ago
  No, it was built because a solutions architect sat next to a film producer on a plane and talked about how to solve for his unique needs.
  - jsolson 7y ago
    Miles, I can't tell if you're messing with me or just stating what actually happened :)
a012 7y ago
Obligatory XKCD https://what-if.xkcd.com/31/ https://what-if.xkcd.com/31/
jsjohnst 7y ago
> even relatively remote locations become "accessible" for 100 Gbps+ fiber or microwave connectivity, and _very_ remote locations for tens of Gbps. I’m guessing: 1. You’ve never priced out said circuits before 2. Never seen the lead times and minimum contract periods on them 3. Don’t realize in general how hard it is to actually max out circuits like that. Shipping / freight company is generally still a preferred solution to moving very very large amounts of data even in 2020. Your own employer still ships large datasets that way too. Here’s a link from 2007: https://royal.pingdom.com/fedex-still-faster-than-the-internet/ https://royal.pingdom.com/fedex-still-faster-than-the-intern... But from contacts I know inside Google, it’s still done regularly.
- jsolson 7y ago
  Pretty good idea on #3 (and for much much much larger links), and a decent idea for #1. If I had 1-10 PB to move I could totally see most people using FedEx to ship JBODs around. For small transfers it makes sense, and we also have a product for this. Internally I'd use our WAN for those, which is where this now starts to make sense: Amazon does not have anywhere near the global WAN capacity that we do, nor so far have they had a good reason to have that capacity. If you've got the global capacity to allow a customer to peer to your nearest location (and your nearest location likely isn't too far away for most of the world), fiber makes sense for large (~EB-scale, which I assume anyone with 100PB+ is well on their way to 1EB+) amounts of data. We have a product for these peerings (primarily for customers in colos): https://cloud.google.com/interconnect/docs/concepts/dedicated-overview https://cloud.google.com/interconnect/docs/concepts/dedicate...
  - wildfire 7y ago
    That product looks exactly like AWS Direct Connect Also, when I go to: https://www.infrastructure.aws/ https://www.infrastructure.aws/ (annoyingly I can't link to the network) it says 100Gbps of WAN capacity around the AWS global network. I did a quick search and wasn't able to find what it is for GCP -- do you have a link handy?
greglindahl 7y ago
I guess you didn't notice that Google Cloud has a thing that competes with Snowball? https://cloud.google.com/transfer-appliance https://cloud.google.com/transfer-appliance Also, if you want to know an extreme edge case that can't (currently) be done any other way, consider the astronomical observatories at the South Pole.
- jsolson 7y ago
  I said Snowmobile doesn't make sense. Snowball makes total sense. The economics are entirely different. Antarctica was the only place that came to mind, but I wasn't sure about the costs of getting a shipping container in / out.
  - greglindahl 7y ago
    For the south pole, they use C-130s, and not standard shipping containers. You can see the little ones that they use in the photo at the bottom of https://www.nsf.gov/geo/opp/support/lc130.jsp https://www.nsf.gov/geo/opp/support/lc130.jsp
    - jsolson 7y ago
      Right, but Snowmobile is packaged as a shipping container because it has a small datacenter inside, so unless you can fit the assembled 45ft container in the plane, I don't think this works? Those are big fucking planes, so maybe you can?

snowwrestler 7y ago

It reads to me like you're saying: if you're going to spend $400k/month on storage cost, why not pay for a network buildout too? I don't think that allocating X per month for op-ex implies that you also have Y amount of money to spend on cap-ex... especially if you don't own the location. 350kw of power is just another trailer to pull up.

jsolson 7y ago
I am assuming that anyone who managed to accumulate 100 PB of data (or close enough to make this viable) is still accumulating data at a solid clip, or likely has a lot _more_ than just 100 PB of data

KaiserPro 7y ago

its about timing. Say I have datacentre in london, Sure I could plumb in 4 100gig lines. But they are typically on a year lease and have a long lead time. Plus, once its on the datatruck its AWS's problem.

mike_d 7y ago

There is your first mistake: thinking the customer wants to solve the problem in a a good or cost-effective way. Snowmobile was built because a pre-sales engineer sarcastically said backing up a truck full of hard drives to a customers datacenter would be the fastest way to import X PB of data. It's probably loading from a tape silo and spent 18 months at the customer site.

milesward 7y ago
No, it was built because a solutions architect sat next to a film producer on a plane and talked about how to solve for his unique needs.
- jsolson 7y ago
  Miles, I can't tell if you're messing with me or just stating what actually happened :)

a012 7y ago

Obligatory XKCD https://what-if.xkcd.com/31/ https://what-if.xkcd.com/31/

jsjohnst 7y ago

> even relatively remote locations become "accessible" for 100 Gbps+ fiber or microwave connectivity, and _very_ remote locations for tens of Gbps. I’m guessing: 1. You’ve never priced out said circuits before 2. Never seen the lead times and minimum contract periods on them 3. Don’t realize in general how hard it is to actually max out circuits like that. Shipping / freight company is generally still a preferred solution to moving very very large amounts of data even in 2020. Your own employer still ships large datasets that way too. Here’s a link from 2007: https://royal.pingdom.com/fedex-still-faster-than-the-internet/ https://royal.pingdom.com/fedex-still-faster-than-the-intern... But from contacts I know inside Google, it’s still done regularly.

jsolson 7y ago
Pretty good idea on #3 (and for much much much larger links), and a decent idea for #1. If I had 1-10 PB to move I could totally see most people using FedEx to ship JBODs around. For small transfers it makes sense, and we also have a product for this. Internally I'd use our WAN for those, which is where this now starts to make sense: Amazon does not have anywhere near the global WAN capacity that we do, nor so far have they had a good reason to have that capacity. If you've got the global capacity to allow a customer to peer to your nearest location (and your nearest location likely isn't too far away for most of the world), fiber makes sense for large (~EB-scale, which I assume anyone with 100PB+ is well on their way to 1EB+) amounts of data. We have a product for these peerings (primarily for customers in colos): https://cloud.google.com/interconnect/docs/concepts/dedicated-overview https://cloud.google.com/interconnect/docs/concepts/dedicate...
- wildfire 7y ago
  That product looks exactly like AWS Direct Connect Also, when I go to: https://www.infrastructure.aws/ https://www.infrastructure.aws/ (annoyingly I can't link to the network) it says 100Gbps of WAN capacity around the AWS global network. I did a quick search and wasn't able to find what it is for GCP -- do you have a link handy?

greglindahl 7y ago

I guess you didn't notice that Google Cloud has a thing that competes with Snowball? https://cloud.google.com/transfer-appliance https://cloud.google.com/transfer-appliance Also, if you want to know an extreme edge case that can't (currently) be done any other way, consider the astronomical observatories at the South Pole.

jsolson 7y ago
I said Snowmobile doesn't make sense. Snowball makes total sense. The economics are entirely different. Antarctica was the only place that came to mind, but I wasn't sure about the costs of getting a shipping container in / out.
- greglindahl 7y ago
  For the south pole, they use C-130s, and not standard shipping containers. You can see the little ones that they use in the photo at the bottom of https://www.nsf.gov/geo/opp/support/lc130.jsp https://www.nsf.gov/geo/opp/support/lc130.jsp
  - jsolson 7y ago
    Right, but Snowmobile is packaged as a shipping container because it has a small datacenter inside, so unless you can fit the assembled 45ft container in the plane, I don't think this works? Those are big fucking planes, so maybe you can?