4 ms·

A Richter scale for outages

37 points by smcgivern 12y ago

genmon 12y ago
Author of linked blog post here -- would love to hear what you think!
- superpatosainz 12y ago
  It's a very interesting concept, because now in 2015 I've seen more and more things using technology and computers where they shouldn't be used. People seem to not understand that the gimmick feature that needs a computer just adds a billion moving parts (one for each transistor, statement of code, etc) that may fail. I was in the process of buying a new TV and I couldn't find a good ol' TV, nope, all there was "super awesome smart voice commanded can see the weather TVs"... so I bought one intending to just use it for cable (I still used netflix in my computer) and after a whole damn day of updating firmwares and waiting for it to boot (I miss the CRT days)... some capacitors that were in the backlight circuit blew. Thank you very much Samsung. (Btw another clear example of misusing technology was yesterday's post about trains using GPS for "smart door controls") Oh also, the scale in your post is more of a Mercali than Richter.
  - genmon 12y ago
    I didn't realise there was an effect-focused earthquake scale, that's definitely more what I meant
  - laumars 12y ago
    Yes, thank you!! The KISS approach seems all but forgotten in gadgets these days. It's perverse that now that chips have never been cheaper, technology has become more expensive in terms of longevity since chips are thrown into everything and often for no good reason and then your equipment bricks at a whim because the skill level now required to fix faulty electronics is several orders of magnitude greater than the competence of your average geek. And does this added complexity improve usability? Of course not. It's just consumer masturbation so the Jones's can show off. Then there's the whole topic of security now that every device is a mini-computer. However I will concede the hypocrisy of this post given I'm sending it from a smart phone. But all good arguments deserve one exception to the rule ;)
- Semiapies 12y ago
  Logarithmic scales make sense for things with clear, quantifiable measures in some particular dimension. This scale both includes a large number of qualitative measures that aren't particularly obviously different orders of magnitude and conflates severity of effect and time to fix.
  - genmon 12y ago
    I'd love for systems theory (cybernetics or whatever) to have gotten to the point where we could have an actual measure of disruption. Severity seems to come from both what the outage is, and also from our failure to build resilient systems... it's earthquake tremors vs whether we followed the building codes. But unlike earthquakes, it's really hard to tease those two apart
  - superpatosainz 12y ago
    There's one like what you say: Mercali.
- atlbeer 12y ago
  An example that sprung to mind while reading was the arson/suicide attack on Chicago's air traffic control facility late last year http://en.wikipedia.org/wiki/Chicago_Air_Route_Traffic_Control_Center#2014_fire http://en.wikipedia.org/wiki/Chicago_Air_Route_Traffic_Contr...
steakejjs 12y ago
Would a possible example of a large magnitude event be the 2012 Virginia Derecho[1] ? Does anyone remember this evening as a day of large outages? I wasn't able to experience the event as an internet user having lost power. This is the type of event I would EXPECT to get a big magnitude. [1]http://en.wikipedia.org/wiki/June_2012_North_American_derecho#Virginia http://en.wikipedia.org/wiki/June_2012_North_American_derech...
- genmon 12y ago
  that's the kind of thing. the one that comes to mind for me is the 9 hour AT&T long distance outage of 1990: http://www.mit.edu/hacker/part1.html http://www.mit.edu/hacker/part1.html -- it's these kind of emergent outages that really interest me
JamesBaxter 12y ago
I like the idea for internal discussion of problems but I don't think we'll see usage of a scale like this on say a corporate support Twitter account. It will end up with the Hacker groups of the world bragging about the 10s they've caused major technology companies and competing for scores...
nuclear_eclipse 12y ago
> 2.0 ... Facebook down ... For some users, this more like a 9.0. We've literally seen people call 911 because they can't access Facebook during an outage.
poulsbohemian 12y ago
Corporate America often has major issues that even internally they can't decide where it rates on the scale - and corporate communications is not about to let that information out. Outright outages for major corps or infrastructure is rare, but performance degradation is so constant for some companies that stumbling along in constant fire-fighting mode is considered normal, IE: something like a 4.0 on this scale is every day. A significant portion of my time over the last ten years has been consulting to large enterprises to solve their production problems and try to ensure they don't happen again. It's shocking how broken things can be before a dime will be spent to improve anything.
- genmon 12y ago
  I get super concerned when I hear reports like this -- what I'd like to hear is that the Netflix chaos monkey method is being adopted for critical infrastructure to increase resilience. Instead I can totally imagine a software fault that brings transit across a city down for a few days. I think we had a software fault in UK air traffic control recently that knocked out flying for about half a day -- who knows how much of this is going on
United857 12y ago
Why would "collapse of minor network requiring rebuild. e.g. recent Sony hack that meant no computers, printers, or existing network infrastructure could be re-used without manual check of each item." rate a 6.0 (affect a single organization) whereas "Minor network freeze but can be recovered with a reboot; broad human inconvenience without threat. e.g. regional ATM network down for a day, cellular network down for a day for single operator." is 4.0 (affect many members of the public in a region)
- genmon 12y ago
  Good point. My examples should ideally be the same scale. For 4.0, I think it's about a single point of failure that, when rectified, recovery is simple. For 6.0, some kind of permanent and cascading problem. The difference between the server being down, and accidentally shipping an update that bricks all clients.
Thrymr 12y ago
> Like the Richter magnitude scale Obsolete? The Richter scale is historically important, but in modern usage has been replaced by moment magnitude [0]. As others have pointed out, in any case this subjective scale has more in common with the Mercalli intensity scale [1]. [0] http://en.wikipedia.org/wiki/Moment_magnitude_scale http://en.wikipedia.org/wiki/Moment_magnitude_scale [1] http://en.wikipedia.org/wiki/Mercalli_intensity_scale http://en.wikipedia.org/wiki/Mercalli_intensity_scale

37 points by smcgivern 12y ago

genmon 12y ago

Author of linked blog post here -- would love to hear what you think!

superpatosainz 12y ago
It's a very interesting concept, because now in 2015 I've seen more and more things using technology and computers where they shouldn't be used. People seem to not understand that the gimmick feature that needs a computer just adds a billion moving parts (one for each transistor, statement of code, etc) that may fail. I was in the process of buying a new TV and I couldn't find a good ol' TV, nope, all there was "super awesome smart voice commanded can see the weather TVs"... so I bought one intending to just use it for cable (I still used netflix in my computer) and after a whole damn day of updating firmwares and waiting for it to boot (I miss the CRT days)... some capacitors that were in the backlight circuit blew. Thank you very much Samsung. (Btw another clear example of misusing technology was yesterday's post about trains using GPS for "smart door controls") Oh also, the scale in your post is more of a Mercali than Richter.
- genmon 12y ago
  I didn't realise there was an effect-focused earthquake scale, that's definitely more what I meant
- laumars 12y ago
  Yes, thank you!! The KISS approach seems all but forgotten in gadgets these days. It's perverse that now that chips have never been cheaper, technology has become more expensive in terms of longevity since chips are thrown into everything and often for no good reason and then your equipment bricks at a whim because the skill level now required to fix faulty electronics is several orders of magnitude greater than the competence of your average geek. And does this added complexity improve usability? Of course not. It's just consumer masturbation so the Jones's can show off. Then there's the whole topic of security now that every device is a mini-computer. However I will concede the hypocrisy of this post given I'm sending it from a smart phone. But all good arguments deserve one exception to the rule ;)
Semiapies 12y ago
Logarithmic scales make sense for things with clear, quantifiable measures in some particular dimension. This scale both includes a large number of qualitative measures that aren't particularly obviously different orders of magnitude and conflates severity of effect and time to fix.
- genmon 12y ago
  I'd love for systems theory (cybernetics or whatever) to have gotten to the point where we could have an actual measure of disruption. Severity seems to come from both what the outage is, and also from our failure to build resilient systems... it's earthquake tremors vs whether we followed the building codes. But unlike earthquakes, it's really hard to tease those two apart
- superpatosainz 12y ago
  There's one like what you say: Mercali.
atlbeer 12y ago
An example that sprung to mind while reading was the arson/suicide attack on Chicago's air traffic control facility late last year http://en.wikipedia.org/wiki/Chicago_Air_Route_Traffic_Control_Center#2014_fire http://en.wikipedia.org/wiki/Chicago_Air_Route_Traffic_Contr...

steakejjs 12y ago

Would a possible example of a large magnitude event be the 2012 Virginia Derecho[1] ? Does anyone remember this evening as a day of large outages? I wasn't able to experience the event as an internet user having lost power. This is the type of event I would EXPECT to get a big magnitude. [1]http://en.wikipedia.org/wiki/June_2012_North_American_derecho#Virginia http://en.wikipedia.org/wiki/June_2012_North_American_derech...

genmon 12y ago
that's the kind of thing. the one that comes to mind for me is the 9 hour AT&T long distance outage of 1990: http://www.mit.edu/hacker/part1.html http://www.mit.edu/hacker/part1.html -- it's these kind of emergent outages that really interest me

JamesBaxter 12y ago

I like the idea for internal discussion of problems but I don't think we'll see usage of a scale like this on say a corporate support Twitter account. It will end up with the Hacker groups of the world bragging about the 10s they've caused major technology companies and competing for scores...

nuclear_eclipse 12y ago

> 2.0 ... Facebook down ... For some users, this more like a 9.0. We've literally seen people call 911 because they can't access Facebook during an outage.

poulsbohemian 12y ago

Corporate America often has major issues that even internally they can't decide where it rates on the scale - and corporate communications is not about to let that information out. Outright outages for major corps or infrastructure is rare, but performance degradation is so constant for some companies that stumbling along in constant fire-fighting mode is considered normal, IE: something like a 4.0 on this scale is every day. A significant portion of my time over the last ten years has been consulting to large enterprises to solve their production problems and try to ensure they don't happen again. It's shocking how broken things can be before a dime will be spent to improve anything.

genmon 12y ago
I get super concerned when I hear reports like this -- what I'd like to hear is that the Netflix chaos monkey method is being adopted for critical infrastructure to increase resilience. Instead I can totally imagine a software fault that brings transit across a city down for a few days. I think we had a software fault in UK air traffic control recently that knocked out flying for about half a day -- who knows how much of this is going on

United857 12y ago

Why would "collapse of minor network requiring rebuild. e.g. recent Sony hack that meant no computers, printers, or existing network infrastructure could be re-used without manual check of each item." rate a 6.0 (affect a single organization) whereas "Minor network freeze but can be recovered with a reboot; broad human inconvenience without threat. e.g. regional ATM network down for a day, cellular network down for a day for single operator." is 4.0 (affect many members of the public in a region)

genmon 12y ago
Good point. My examples should ideally be the same scale. For 4.0, I think it's about a single point of failure that, when rectified, recovery is simple. For 6.0, some kind of permanent and cascading problem. The difference between the server being down, and accidentally shipping an update that bricks all clients.

Thrymr 12y ago

> Like the Richter magnitude scale Obsolete? The Richter scale is historically important, but in modern usage has been replaced by moment magnitude [0]. As others have pointed out, in any case this subjective scale has more in common with the Mercalli intensity scale [1]. [0] http://en.wikipedia.org/wiki/Moment_magnitude_scale http://en.wikipedia.org/wiki/Moment_magnitude_scale [1] http://en.wikipedia.org/wiki/Mercalli_intensity_scale http://en.wikipedia.org/wiki/Mercalli_intensity_scale