4 ms·

Disclosure: I work on Google Cloud. I wouldn't throw the baby out with the bathwater. Across Google, we rely almost exclusively on networked storage (Colossus)

by boulos 9y ago

Disclosure: I work on Google Cloud. I wouldn't throw the baby out with the bathwater. Across Google, we rely almost exclusively on networked storage (Colossus) unless we need extremely high-performance local flash. Being able to separate compute from storage is a huge part of our ability to both scale out, as well as to do live migration for GCE (where Persistent Disk, our equivalent of EBS is built upon Colossus). Persistent Disk has never had an outage like the EBS one, but I attribute that to the Colossus and underlying teams having run this at Google for a really long time. Fwiw, the AWS folks have also massively improved EBS over the years. You can still worry, but I prefer to think about overall MTBF rather than consider networked storage in particular as the plague :).

mattbillenstein 9y ago
To each their own. I don't like getting paged at 3am, so I choose to remove complexity and external dependencies where I can. And not operating at Google-scale, this is one of those things I can live without.
butwhythough 9y ago
Disclosure: I use both gce and ec2 Unfortunately unlike EBS, persistent disks (Colossus) in Google cloud share a network plane with the VM. To quote the docs. "Each persistent disk write operation contributes to your virtual machine instance's cumulative network egress cap." https://cloud.google.com/compute/docs/disks/performance https://cloud.google.com/compute/docs/disks/performance If you haven't noticed the mtu in Google cloud is < 1500 vs AWS where you can get jumbo frames (9k). I have no reason to believe persistent disks are different. Enabling live migrations? You mean the choice of having an instance terminated with zero notice or migrate with a 60 second warning if you subscribe to the right API? Oh yeah and this is happening constantly. Live migrations (and by connection VM attrition), persistent disks, and network performance are my least favorite aspects of google cloud today. That being said google cloud does have alot of advantages over ec2. These are just not them.
- cube2222 9y ago
  No, that's not that actually. On google cloud platform your virtual machine usually gets migrated between machines without any downtime. Aren't you talking about preemptible instances?
  - butwhythough 9y ago
    One would think right? We are not running with preemptible instances. Yes I have verified all the scheduling flags for our instances. Preemptible: false OnMaintenance: MIGRATE (only other option is terminate which happens far too often for any sizable shop to use and you lose the 60 second warning) Key word being usually. Even if it's not a complete failure we have seen all sorts of fun from packet loss/latency spikes to seg faults. I have no doubt things will improve but right now pd, network, and live migrations are pain points for our group in gce.
    - sulam 9y ago
      Full disclosure: evaluating GCP pretty heavily -- and can support what you're saying from conversations with existing customers (multiple). Live migrations cause both latency/cpu slowdowns (brownouts) and actual (short) interruptions (blackouts). This isn't even a secret, it's fairly well-documented: https://cloud.google.com/compute/docs/instances/live-migration https://cloud.google.com/compute/docs/instances/live-migrati...
- boulos 9y ago
  First, I'm sorry you've had a bad time. We track our VM MTBF closely, so hearing "this is happening constantly" is really worrying. Feel free to reach out to me (email in profile), or Support so we can dig into your experience. If something is wrong, we should diagnose and fix it. Can you say why you prefer explicitly separated egress caps? We let our networking egress be shared between all sources of traffic on purpose, because it lets you go full throttle rather than hardcap on "flavor". That is, why restrict someone that doesn't write to disk much by "stealing" several Gbps for the PD/EBS they aren't going to use? Finally, it's true that our MTU is too damn low. But that also isn't particularly material for PD: when the guest issues a write, we handle it all behind the scenes (it's not like your guest sees the write get fragmented into packets).
ddorian43 9y ago
You saying bigtable doesn't work with local disks ?
- londons_explore 9y ago
  Correct. Big table is all built on remote network filesystems, with certain stuff preloaded into ram.

by boulos 9y ago

mattbillenstein 9y ago

To each their own. I don't like getting paged at 3am, so I choose to remove complexity and external dependencies where I can. And not operating at Google-scale, this is one of those things I can live without.

butwhythough 9y ago

Disclosure: I use both gce and ec2 Unfortunately unlike EBS, persistent disks (Colossus) in Google cloud share a network plane with the VM. To quote the docs. "Each persistent disk write operation contributes to your virtual machine instance's cumulative network egress cap." https://cloud.google.com/compute/docs/disks/performance https://cloud.google.com/compute/docs/disks/performance If you haven't noticed the mtu in Google cloud is < 1500 vs AWS where you can get jumbo frames (9k). I have no reason to believe persistent disks are different. Enabling live migrations? You mean the choice of having an instance terminated with zero notice or migrate with a 60 second warning if you subscribe to the right API? Oh yeah and this is happening constantly. Live migrations (and by connection VM attrition), persistent disks, and network performance are my least favorite aspects of google cloud today. That being said google cloud does have alot of advantages over ec2. These are just not them.

cube2222 9y ago
No, that's not that actually. On google cloud platform your virtual machine usually gets migrated between machines without any downtime. Aren't you talking about preemptible instances?
- butwhythough 9y ago
  One would think right? We are not running with preemptible instances. Yes I have verified all the scheduling flags for our instances. Preemptible: false OnMaintenance: MIGRATE (only other option is terminate which happens far too often for any sizable shop to use and you lose the 60 second warning) Key word being usually. Even if it's not a complete failure we have seen all sorts of fun from packet loss/latency spikes to seg faults. I have no doubt things will improve but right now pd, network, and live migrations are pain points for our group in gce.
  - sulam 9y ago
    Full disclosure: evaluating GCP pretty heavily -- and can support what you're saying from conversations with existing customers (multiple). Live migrations cause both latency/cpu slowdowns (brownouts) and actual (short) interruptions (blackouts). This isn't even a secret, it's fairly well-documented: https://cloud.google.com/compute/docs/instances/live-migration https://cloud.google.com/compute/docs/instances/live-migrati...
boulos 9y ago
First, I'm sorry you've had a bad time. We track our VM MTBF closely, so hearing "this is happening constantly" is really worrying. Feel free to reach out to me (email in profile), or Support so we can dig into your experience. If something is wrong, we should diagnose and fix it. Can you say why you prefer explicitly separated egress caps? We let our networking egress be shared between all sources of traffic on purpose, because it lets you go full throttle rather than hardcap on "flavor". That is, why restrict someone that doesn't write to disk much by "stealing" several Gbps for the PD/EBS they aren't going to use? Finally, it's true that our MTU is too damn low. But that also isn't particularly material for PD: when the guest issues a write, we handle it all behind the scenes (it's not like your guest sees the write get fragmented into packets).

ddorian43 9y ago

You saying bigtable doesn't work with local disks ?

londons_explore 9y ago
Correct. Big table is all built on remote network filesystems, with certain stuff preloaded into ram.