4 ms·

Just less than an hour ago I got 10+ alerts that my monitoring agents on GCE machines are unreachable for 5 mins again and networking is down. I see this at lea

Just less than an hour ago I got 10+ alerts that my monitoring agents on GCE machines are unreachable for 5 mins again and networking is down. I see this at least once a month, separately the cloud SQL has weird outages again about once-twice a month. I also run some real hardware machines in a datacenter and have none of these problems. I am not impressed with Google's cloud.

boulos 11y ago
Sorry to hear that; which zone are you in? I don't see any notice of an outage, but I'd be happy to investigate for you (I work on Compute Engine). To your other issue with Cloud SQL, there is sadly a mandatory downtime with every release they do (often weekly). The team is working on it and knows it's unacceptable for serving workloads. Edit: Here's a link to the maintenance note in the FAQ for reference - https://cloud.google.com/sql/faq#maintenancerestart https://cloud.google.com/sql/faq#maintenancerestart
- vladimirralev 11y ago
  Very strange, you should be able to observe these in almost any monitor. I see this in multiple projects/networks. Outage today was on us-central1-a. I got alerts from multiple agents - new relic, zabbix and ssh monitor so I don't think it can be blamed on tools or something like this. It happened in two bursts, this is the first one http://imgur.com/XdzuJ2r http://imgur.com/XdzuJ2r with messages from both zabbix and new relic, the time zone in the emails is EEST. These are all different machines, no duplication.
  - boulos 11y ago
    This is what I get for not being in SRE (I don't get paged, I just poll our incident lists, etc.): there was definitely a blip at that time, people are on it right now looking into the root cause (but it was automatically remediated within just a couple minutes as you saw). Regardless, sorry for the blip and for being wrong about the all clear.
mrmondo 11y ago
Googles not alone in this, we got a lot of unexplained seemingly random outages when trying azure - it ended up being so unpredictably unreliable we found it unusable for anything important not to mention they never updated their status page.
buffoon 11y ago
Just remember when you contract something out to a third party, you're not subject to their SLA level but their general level of competence. This can be quite a bit lower than the SLA, even if they are high profile or expensive. If you're lucky they'll bung you a discount as compensation but don't rely on that. I speak with experience with Google, AWS and Rackspace.

boulos 11y ago

Sorry to hear that; which zone are you in? I don't see any notice of an outage, but I'd be happy to investigate for you (I work on Compute Engine). To your other issue with Cloud SQL, there is sadly a mandatory downtime with every release they do (often weekly). The team is working on it and knows it's unacceptable for serving workloads. Edit: Here's a link to the maintenance note in the FAQ for reference - https://cloud.google.com/sql/faq#maintenancerestart https://cloud.google.com/sql/faq#maintenancerestart

vladimirralev 11y ago
Very strange, you should be able to observe these in almost any monitor. I see this in multiple projects/networks. Outage today was on us-central1-a. I got alerts from multiple agents - new relic, zabbix and ssh monitor so I don't think it can be blamed on tools or something like this. It happened in two bursts, this is the first one http://imgur.com/XdzuJ2r http://imgur.com/XdzuJ2r with messages from both zabbix and new relic, the time zone in the emails is EEST. These are all different machines, no duplication.
- boulos 11y ago
  This is what I get for not being in SRE (I don't get paged, I just poll our incident lists, etc.): there was definitely a blip at that time, people are on it right now looking into the root cause (but it was automatically remediated within just a couple minutes as you saw). Regardless, sorry for the blip and for being wrong about the all clear.

mrmondo 11y ago

Googles not alone in this, we got a lot of unexplained seemingly random outages when trying azure - it ended up being so unpredictably unreliable we found it unusable for anything important not to mention they never updated their status page.

buffoon 11y ago

Just remember when you contract something out to a third party, you're not subject to their SLA level but their general level of competence. This can be quite a bit lower than the SLA, even if they are high profile or expensive. If you're lucky they'll bung you a discount as compensation but don't rely on that. I speak with experience with Google, AWS and Rackspace.