4 ms·

Disclosure: I use both gce and ec2 Unfortunately unlike EBS, persistent disks (Colossus) in Google cloud share a network plane with the VM. To quote the docs.

Disclosure: I use both gce and ec2 Unfortunately unlike EBS, persistent disks (Colossus) in Google cloud share a network plane with the VM. To quote the docs. "Each persistent disk write operation contributes to your virtual machine instance's cumulative network egress cap." https://cloud.google.com/compute/docs/disks/performance https://cloud.google.com/compute/docs/disks/performance If you haven't noticed the mtu in Google cloud is < 1500 vs AWS where you can get jumbo frames (9k). I have no reason to believe persistent disks are different. Enabling live migrations? You mean the choice of having an instance terminated with zero notice or migrate with a 60 second warning if you subscribe to the right API? Oh yeah and this is happening constantly. Live migrations (and by connection VM attrition), persistent disks, and network performance are my least favorite aspects of google cloud today. That being said google cloud does have alot of advantages over ec2. These are just not them.

cube2222 9y ago
No, that's not that actually. On google cloud platform your virtual machine usually gets migrated between machines without any downtime. Aren't you talking about preemptible instances?
- butwhythough 9y ago
  One would think right? We are not running with preemptible instances. Yes I have verified all the scheduling flags for our instances. Preemptible: false OnMaintenance: MIGRATE (only other option is terminate which happens far too often for any sizable shop to use and you lose the 60 second warning) Key word being usually. Even if it's not a complete failure we have seen all sorts of fun from packet loss/latency spikes to seg faults. I have no doubt things will improve but right now pd, network, and live migrations are pain points for our group in gce.
  - sulam 9y ago
    Full disclosure: evaluating GCP pretty heavily -- and can support what you're saying from conversations with existing customers (multiple). Live migrations cause both latency/cpu slowdowns (brownouts) and actual (short) interruptions (blackouts). This isn't even a secret, it's fairly well-documented: https://cloud.google.com/compute/docs/instances/live-migration https://cloud.google.com/compute/docs/instances/live-migrati...
boulos 9y ago
First, I'm sorry you've had a bad time. We track our VM MTBF closely, so hearing "this is happening constantly" is really worrying. Feel free to reach out to me (email in profile), or Support so we can dig into your experience. If something is wrong, we should diagnose and fix it. Can you say why you prefer explicitly separated egress caps? We let our networking egress be shared between all sources of traffic on purpose, because it lets you go full throttle rather than hardcap on "flavor". That is, why restrict someone that doesn't write to disk much by "stealing" several Gbps for the PD/EBS they aren't going to use? Finally, it's true that our MTU is too damn low. But that also isn't particularly material for PD: when the guest issues a write, we handle it all behind the scenes (it's not like your guest sees the write get fragmented into packets).
- butwhythough 9y ago
  Thank you for the fresh perspective and the thoughtful reply. I also appreciate the offer to reach out. Rest assured we are actively engaged during the events and have found support incredibly responsive. At a high level none of these are reasons for us to stop investing in our google cloud stack, nor have they caused any major outages. Think of them as quality of life comments. Live migrations To clarify "this is happening constantly". I meant we see live migrations happen frequently through the day. Going back the last 24 hours I see "hundreds" of migrations. We do have days where we will see 3 or 4x this number. The majority of these were successful and our logging, probes, and graphs show nothing exciting. On the positive side of things, we have noticed a marked improvement in migration times, probe failures, and instance fatalities in the last 6 months. Where before we would regularly see live migrations take upwards of 15 minutes or longer. They are now at or under 2 minutes (With only a handful of exceptions barely worth mentioning). I do appreciate the facility of live migration and the proactive approach googles takes to host maintenance. The 60 second notification window is just too damn short for some of our services to properly drain themselves. So instead, we hold on to our butts and hope for the best on those boxes. If there was one improvement to live migration it would be to have the option of a 15 minute (or even 30 minutes.. am I being greedy?) notice. Networking egress caps shared between instance and persistent disk The edge case that hurts here is when you have a high bandwidth service that also writes a lot of data to a PD disk and the comcast style burst bandwidth throttling that happens on the instances (This is pure speculation and may have improved since last time this was investigated. The observation at the time was the throttling is a bit too efficient and hit the instance disproportionately). We have since migrated to either local ssd or tmpfs disks for these type of hosts in gce. Their sister services are still running fine in ec2 on ebs instances. MTU Yay! 1500 would be better, 9k would be great. This hurts when connectivity starts to see an increase in packet loss and the corresponding increase in packet re-transmission (and latency). Not to mention the overhead these extra packets incur (Those 20-60+ bytes just in headers add up quick). TLDR wishlist Longer notification window before live migration actually starts PD-optimized instances 1500/9k MTU

cube2222 9y ago

No, that's not that actually. On google cloud platform your virtual machine usually gets migrated between machines without any downtime. Aren't you talking about preemptible instances?

butwhythough 9y ago
One would think right? We are not running with preemptible instances. Yes I have verified all the scheduling flags for our instances. Preemptible: false OnMaintenance: MIGRATE (only other option is terminate which happens far too often for any sizable shop to use and you lose the 60 second warning) Key word being usually. Even if it's not a complete failure we have seen all sorts of fun from packet loss/latency spikes to seg faults. I have no doubt things will improve but right now pd, network, and live migrations are pain points for our group in gce.
- sulam 9y ago
  Full disclosure: evaluating GCP pretty heavily -- and can support what you're saying from conversations with existing customers (multiple). Live migrations cause both latency/cpu slowdowns (brownouts) and actual (short) interruptions (blackouts). This isn't even a secret, it's fairly well-documented: https://cloud.google.com/compute/docs/instances/live-migration https://cloud.google.com/compute/docs/instances/live-migrati...

boulos 9y ago

First, I'm sorry you've had a bad time. We track our VM MTBF closely, so hearing "this is happening constantly" is really worrying. Feel free to reach out to me (email in profile), or Support so we can dig into your experience. If something is wrong, we should diagnose and fix it. Can you say why you prefer explicitly separated egress caps? We let our networking egress be shared between all sources of traffic on purpose, because it lets you go full throttle rather than hardcap on "flavor". That is, why restrict someone that doesn't write to disk much by "stealing" several Gbps for the PD/EBS they aren't going to use? Finally, it's true that our MTU is too damn low. But that also isn't particularly material for PD: when the guest issues a write, we handle it all behind the scenes (it's not like your guest sees the write get fragmented into packets).

butwhythough 9y ago
Thank you for the fresh perspective and the thoughtful reply. I also appreciate the offer to reach out. Rest assured we are actively engaged during the events and have found support incredibly responsive. At a high level none of these are reasons for us to stop investing in our google cloud stack, nor have they caused any major outages. Think of them as quality of life comments. Live migrations To clarify "this is happening constantly". I meant we see live migrations happen frequently through the day. Going back the last 24 hours I see "hundreds" of migrations. We do have days where we will see 3 or 4x this number. The majority of these were successful and our logging, probes, and graphs show nothing exciting. On the positive side of things, we have noticed a marked improvement in migration times, probe failures, and instance fatalities in the last 6 months. Where before we would regularly see live migrations take upwards of 15 minutes or longer. They are now at or under 2 minutes (With only a handful of exceptions barely worth mentioning). I do appreciate the facility of live migration and the proactive approach googles takes to host maintenance. The 60 second notification window is just too damn short for some of our services to properly drain themselves. So instead, we hold on to our butts and hope for the best on those boxes. If there was one improvement to live migration it would be to have the option of a 15 minute (or even 30 minutes.. am I being greedy?) notice. Networking egress caps shared between instance and persistent disk The edge case that hurts here is when you have a high bandwidth service that also writes a lot of data to a PD disk and the comcast style burst bandwidth throttling that happens on the instances (This is pure speculation and may have improved since last time this was investigated. The observation at the time was the throttling is a bit too efficient and hit the instance disproportionately). We have since migrated to either local ssd or tmpfs disks for these type of hosts in gce. Their sister services are still running fine in ec2 on ebs instances. MTU Yay! 1500 would be better, 9k would be great. This hurts when connectivity starts to see an increase in packet loss and the corresponding increase in packet re-transmission (and latency). Not to mention the overhead these extra packets incur (Those 20-60+ bytes just in headers add up quick). TLDR wishlist Longer notification window before live migration actually starts PD-optimized instances 1500/9k MTU