5 ms·

Google Compute Engine Is Down

113 points by jread 12y ago

jread 12y ago
All our status VMs are unreachable: https://cloudharmony.com/status-for-google https://cloudharmony.com/status-for-google
dvh 12y ago
"Incident began at 2015-02-19 15:59" Is that in future or am I missing something?
- waitingkuo 12y ago
  Same here... Let's see what will happen
- tazjin 12y ago
  That seems to be local time in Hong Kong. Edit: Just noticed that it says all times are US Pacific. Hm, then it is in the future.
- pronoiac 12y ago
  The times on the left, standalone, are Pacific. In the text on the right, UTC. Edit: uh, wait, this is wrong. It's edited (possibly) or I'm just bleary-eyed (more likely).
- rey12rey 12y ago
  I believe it's a typo as all other references seem to point to 22:59 Feb 18 2015. It's 22:59 Feb 18 2015 also for Google Cloud SQL https://status.cloud.google.com/incident/cloud-sql/17006 https://status.cloud.google.com/incident/cloud-sql/17006 Edit: It turns out this was the case. Updated now.
lern_too_spel 12y ago
Google Cloud Platform is a cluster, in the http://www.urbandictionary.com/define.php?term=Cluster&defid=1424405 http://www.urbandictionary.com/define.php?term=Cluster&defid... sense. Amazon eats their own dogfood with AWS services, while Google does not, and no amount of marketing is going to make up for that. Every time I give it a try, I find random bugs everywhere, and I know there are not enough internal engineers feeling the pain to get those bugs fixed.
- jgreen10 12y ago
  The difference is really just in maturity. AWS started years earlier and has had time to work out the kinks. GCE will get there, but it'll need time and experience.
  - EdwardDiego 12y ago
    Yep, exactly. This GCE outage comes at a great time to help me answer the question, should I use GCE over AWS? Probably not yet.
- deleted 12y ago
  [deleted]
  - ngrilly 12y ago
    Do you have any fact that support your claim?
    - hueving 12y ago
      The way Google deploys applications is fundamentally different than the model they expose to customers. Google's cloud platform runs on top of their real resource management system, not the other way around.
      - vertex-four 12y ago
        Amazon didn't run entirely on AWS until four years after AWS was created: https://twitter.com/cvwadored/status/81400058624475136 https://twitter.com/cvwadored/status/81400058624475136
        esrauch 12y ago
        I worked there in 2011 and I'm pretty sure not everything was on AWS at that time. I don't know how to square that with the tweet: my own understanding could be wrong, or it could be a misleading stat like 100% of public traffic hits a server on AWS infrastructure but not all of the other backend services were.
jsprogrammer 12y ago
0:30 passed and no update? Who would have guessed that? Anyone?
- thezilch 12y ago
  There are updates and clear times for when to expect the next update.
  - jsprogrammer 12y ago
    Right, when I posted it said the next update would occur at 0:30, but it was after 0:30 and there was no update.
deleted 12y ago
[deleted]
nateweiss 12y ago
We found that we could still go into their console, SSH into the boxes (from the browser-based thing), and reboot the boxes from there. When they came back up, they worked. May have just been good luck, just happening to come back up on hardware that's not behind the bad routers or whatever it is. Edit: Also, we found that our instances were reachable (they mostly provide a JSON-based API over HTTP), in the sense that they were getting the incoming HTTP/HTTPS traffic that they normally do. But the responses were not getting back out to whoever requested them... lost on the way back out or something.
- sixbit 12y ago
  This worked for me too, about an hour ago. So not just good luck it seems :-)
sixbit 12y ago
Ssh'ing from the gce web console seemed to make my instances reachable. Afterward ssh from terminal and web access worked.
mnml_ 12y ago
I was going to ask them If they will give compensations, but I can't contact them as I don't have "silver support" :/
iamspoilt 12y ago
The page says that all issues are resolved now.
nitinics 12y ago
On any post incident reports - shall we ever come out of using the most common and ambiguous technology lingo "network issue". If it were identified a network issue (Route black hole, prefix hijacking, resources depletion or consistency issues etc) then you probably know enough to elaborate on the specifics.
aceperry 12y ago
Preliminary cause is described here: https://status.cloud.google.com/incident/compute/15045 https://status.cloud.google.com/incident/compute/15045

113 points by jread 12y ago

jread 12y ago

All our status VMs are unreachable: https://cloudharmony.com/status-for-google https://cloudharmony.com/status-for-google

dvh 12y ago

"Incident began at 2015-02-19 15:59" Is that in future or am I missing something?

waitingkuo 12y ago
Same here... Let's see what will happen
tazjin 12y ago
That seems to be local time in Hong Kong. Edit: Just noticed that it says all times are US Pacific. Hm, then it is in the future.
pronoiac 12y ago
The times on the left, standalone, are Pacific. In the text on the right, UTC. Edit: uh, wait, this is wrong. It's edited (possibly) or I'm just bleary-eyed (more likely).
rey12rey 12y ago
I believe it's a typo as all other references seem to point to 22:59 Feb 18 2015. It's 22:59 Feb 18 2015 also for Google Cloud SQL https://status.cloud.google.com/incident/cloud-sql/17006 https://status.cloud.google.com/incident/cloud-sql/17006 Edit: It turns out this was the case. Updated now.

lern_too_spel 12y ago

Google Cloud Platform is a cluster, in the http://www.urbandictionary.com/define.php?term=Cluster&defid=1424405 http://www.urbandictionary.com/define.php?term=Cluster&defid... sense. Amazon eats their own dogfood with AWS services, while Google does not, and no amount of marketing is going to make up for that. Every time I give it a try, I find random bugs everywhere, and I know there are not enough internal engineers feeling the pain to get those bugs fixed.

jgreen10 12y ago
The difference is really just in maturity. AWS started years earlier and has had time to work out the kinks. GCE will get there, but it'll need time and experience.
- EdwardDiego 12y ago
  Yep, exactly. This GCE outage comes at a great time to help me answer the question, should I use GCE over AWS? Probably not yet.
deleted 12y ago
[deleted]
- ngrilly 12y ago
  Do you have any fact that support your claim?
  - hueving 12y ago
    The way Google deploys applications is fundamentally different than the model they expose to customers. Google's cloud platform runs on top of their real resource management system, not the other way around.
    - vertex-four 12y ago
      Amazon didn't run entirely on AWS until four years after AWS was created: https://twitter.com/cvwadored/status/81400058624475136 https://twitter.com/cvwadored/status/81400058624475136
      - esrauch 12y ago
        I worked there in 2011 and I'm pretty sure not everything was on AWS at that time. I don't know how to square that with the tweet: my own understanding could be wrong, or it could be a misleading stat like 100% of public traffic hits a server on AWS infrastructure but not all of the other backend services were.

jsprogrammer 12y ago

0:30 passed and no update? Who would have guessed that? Anyone?

thezilch 12y ago
There are updates and clear times for when to expect the next update.
- jsprogrammer 12y ago
  Right, when I posted it said the next update would occur at 0:30, but it was after 0:30 and there was no update.

deleted 12y ago

[deleted]

nateweiss 12y ago

We found that we could still go into their console, SSH into the boxes (from the browser-based thing), and reboot the boxes from there. When they came back up, they worked. May have just been good luck, just happening to come back up on hardware that's not behind the bad routers or whatever it is. Edit: Also, we found that our instances were reachable (they mostly provide a JSON-based API over HTTP), in the sense that they were getting the incoming HTTP/HTTPS traffic that they normally do. But the responses were not getting back out to whoever requested them... lost on the way back out or something.

sixbit 12y ago
This worked for me too, about an hour ago. So not just good luck it seems :-)

sixbit 12y ago

Ssh'ing from the gce web console seemed to make my instances reachable. Afterward ssh from terminal and web access worked.

mnml_ 12y ago

I was going to ask them If they will give compensations, but I can't contact them as I don't have "silver support" :/

iamspoilt 12y ago

The page says that all issues are resolved now.

nitinics 12y ago

On any post incident reports - shall we ever come out of using the most common and ambiguous technology lingo "network issue". If it were identified a network issue (Route black hole, prefix hijacking, resources depletion or consistency issues etc) then you probably know enough to elaborate on the specifics.

aceperry 12y ago

Preliminary cause is described here: https://status.cloud.google.com/incident/compute/15045 https://status.cloud.google.com/incident/compute/15045