3 ms·

I admire the transparency but I don't pretend for a second it's the whole story. This happened during work hours and if they indeed did get notified so fast, I'

by smakz 17y ago

I admire the transparency but I don't pretend for a second it's the whole story. This happened during work hours and if they indeed did get notified so fast, I'm wondering why it took over 90 minutes to recover. Also, the outage, for me anyway, seemed to last much longer then the stated 100 minutes. I seem to remember being unable to access GMail for a span of about 3 hours today.

sriramk 17y ago
[Disclaimer: I work for Microsoft] I'm super impressed that they responded so quickly. When a first notification comes in, it is rarely with a fully fleshed out diagnosis. It is usually some runner/monitoring tool sending you an alert saying something is busted. If you get lucky, you had enough monitoring tools across the system to atleast get close to the source. If not, you have some debugging ahead of you. And then, you need to figure out what a good fix is and given that the entire system is in a wobbly state, a bad fix could make the situation worse. And then, you figure out how to rollout the fix and actually make the change. If you're smart, you'll do it in a staged manner and be able to roll back the moment something goes wrong. In short, these things take time. Going from notification to working fix in 90 minutes for what sounds like a nasty network hardware issue is very good.
- seldo 17y ago
  Agreed. 100 minutes from first detection through diagnosis to solution is an excellent turnaround time, and although others have mocked them for plugging their architecture in the post, it's a credit to their architecture that the simplest solution -- flood the system with additional capacity -- was available and so quick to implement. [um, I work for Yahoo, but do you really need a disclaimer on a compliment?]
  - sriramk 17y ago
    I try to mention that whenever a major company which competes with Microsoft in some way is involved. Even the most innocent of statements have a way of getting badly misinterpreted, taken out of context, etc.
- jodrellblank 17y ago
  It sounds like nothing to do with network hardware, they're using 'router' to describe their software that directs gmail HTTP requests around.
- deleted 17y ago
  [deleted]
- smakz 17y ago
  I just re-read the article and I was mistaken. I thought they determined it was a capacity problem immediately, but it sounds like that's actually what took so long to figure out. I work for a web company, and a 100 minute outage is considered absolutely outrageous, but of course it happens.

by smakz 17y ago

sriramk 17y ago

[Disclaimer: I work for Microsoft] I'm super impressed that they responded so quickly. When a first notification comes in, it is rarely with a fully fleshed out diagnosis. It is usually some runner/monitoring tool sending you an alert saying something is busted. If you get lucky, you had enough monitoring tools across the system to atleast get close to the source. If not, you have some debugging ahead of you. And then, you need to figure out what a good fix is and given that the entire system is in a wobbly state, a bad fix could make the situation worse. And then, you figure out how to rollout the fix and actually make the change. If you're smart, you'll do it in a staged manner and be able to roll back the moment something goes wrong. In short, these things take time. Going from notification to working fix in 90 minutes for what sounds like a nasty network hardware issue is very good.

seldo 17y ago
Agreed. 100 minutes from first detection through diagnosis to solution is an excellent turnaround time, and although others have mocked them for plugging their architecture in the post, it's a credit to their architecture that the simplest solution -- flood the system with additional capacity -- was available and so quick to implement. [um, I work for Yahoo, but do you really need a disclaimer on a compliment?]
- sriramk 17y ago
  I try to mention that whenever a major company which competes with Microsoft in some way is involved. Even the most innocent of statements have a way of getting badly misinterpreted, taken out of context, etc.
jodrellblank 17y ago
It sounds like nothing to do with network hardware, they're using 'router' to describe their software that directs gmail HTTP requests around.
deleted 17y ago
[deleted]
smakz 17y ago
I just re-read the article and I was mistaken. I thought they determined it was a capacity problem immediately, but it sounds like that's actually what took so long to figure out. I work for a web company, and a 100 minute outage is considered absolutely outrageous, but of course it happens.