3 ms·

First, I'm sorry you've had a bad time. We track our VM MTBF closely, so hearing "this is happening constantly" is really worrying. Feel free to reach out to me

by boulos 9y ago

First, I'm sorry you've had a bad time. We track our VM MTBF closely, so hearing "this is happening constantly" is really worrying. Feel free to reach out to me (email in profile), or Support so we can dig into your experience. If something is wrong, we should diagnose and fix it. Can you say why you prefer explicitly separated egress caps? We let our networking egress be shared between all sources of traffic on purpose, because it lets you go full throttle rather than hardcap on "flavor". That is, why restrict someone that doesn't write to disk much by "stealing" several Gbps for the PD/EBS they aren't going to use? Finally, it's true that our MTU is too damn low. But that also isn't particularly material for PD: when the guest issues a write, we handle it all behind the scenes (it's not like your guest sees the write get fragmented into packets).

butwhythough 9y ago
Thank you for the fresh perspective and the thoughtful reply. I also appreciate the offer to reach out. Rest assured we are actively engaged during the events and have found support incredibly responsive. At a high level none of these are reasons for us to stop investing in our google cloud stack, nor have they caused any major outages. Think of them as quality of life comments. Live migrations To clarify "this is happening constantly". I meant we see live migrations happen frequently through the day. Going back the last 24 hours I see "hundreds" of migrations. We do have days where we will see 3 or 4x this number. The majority of these were successful and our logging, probes, and graphs show nothing exciting. On the positive side of things, we have noticed a marked improvement in migration times, probe failures, and instance fatalities in the last 6 months. Where before we would regularly see live migrations take upwards of 15 minutes or longer. They are now at or under 2 minutes (With only a handful of exceptions barely worth mentioning). I do appreciate the facility of live migration and the proactive approach googles takes to host maintenance. The 60 second notification window is just too damn short for some of our services to properly drain themselves. So instead, we hold on to our butts and hope for the best on those boxes. If there was one improvement to live migration it would be to have the option of a 15 minute (or even 30 minutes.. am I being greedy?) notice. Networking egress caps shared between instance and persistent disk The edge case that hurts here is when you have a high bandwidth service that also writes a lot of data to a PD disk and the comcast style burst bandwidth throttling that happens on the instances (This is pure speculation and may have improved since last time this was investigated. The observation at the time was the throttling is a bit too efficient and hit the instance disproportionately). We have since migrated to either local ssd or tmpfs disks for these type of hosts in gce. Their sister services are still running fine in ec2 on ebs instances. MTU Yay! 1500 would be better, 9k would be great. This hurts when connectivity starts to see an increase in packet loss and the corresponding increase in packet re-transmission (and latency). Not to mention the overhead these extra packets incur (Those 20-60+ bytes just in headers add up quick). TLDR wishlist Longer notification window before live migration actually starts PD-optimized instances 1500/9k MTU

by boulos 9y ago

butwhythough 9y ago

Thank you for the fresh perspective and the thoughtful reply. I also appreciate the offer to reach out. Rest assured we are actively engaged during the events and have found support incredibly responsive. At a high level none of these are reasons for us to stop investing in our google cloud stack, nor have they caused any major outages. Think of them as quality of life comments. Live migrations To clarify "this is happening constantly". I meant we see live migrations happen frequently through the day. Going back the last 24 hours I see "hundreds" of migrations. We do have days where we will see 3 or 4x this number. The majority of these were successful and our logging, probes, and graphs show nothing exciting. On the positive side of things, we have noticed a marked improvement in migration times, probe failures, and instance fatalities in the last 6 months. Where before we would regularly see live migrations take upwards of 15 minutes or longer. They are now at or under 2 minutes (With only a handful of exceptions barely worth mentioning). I do appreciate the facility of live migration and the proactive approach googles takes to host maintenance. The 60 second notification window is just too damn short for some of our services to properly drain themselves. So instead, we hold on to our butts and hope for the best on those boxes. If there was one improvement to live migration it would be to have the option of a 15 minute (or even 30 minutes.. am I being greedy?) notice. Networking egress caps shared between instance and persistent disk The edge case that hurts here is when you have a high bandwidth service that also writes a lot of data to a PD disk and the comcast style burst bandwidth throttling that happens on the instances (This is pure speculation and may have improved since last time this was investigated. The observation at the time was the throttling is a bit too efficient and hit the instance disproportionately). We have since migrated to either local ssd or tmpfs disks for these type of hosts in gce. Their sister services are still running fine in ec2 on ebs instances. MTU Yay! 1500 would be better, 9k would be great. This hurts when connectivity starts to see an increase in packet loss and the corresponding increase in packet re-transmission (and latency). Not to mention the overhead these extra packets incur (Those 20-60+ bytes just in headers add up quick). TLDR wishlist Longer notification window before live migration actually starts PD-optimized instances 1500/9k MTU