5 ms·

It would be fascinating to read about what bug took down a system with one outage in 20 years. I remember reading in the chubby paper (from google) that user er

It would be fascinating to read about what bug took down a system with one outage in 20 years. I remember reading in the chubby paper (from google) that user error was the cause of more than half the downtime incidents. Wonder if that's the case here too.

tim333 9y ago
Yeah, I recall last year "BA’s £150,000,000 outage was caused by someone turning computers on and off too quickly" I think companies often try to stay quite about the dumb stuff.
- makmanalp 9y ago
  Well - I wouldn't call it dumb stuff. After all, it's only a matter of time one of us does something truly stupid :-) Even more so when under stress and pressure when shit hits the fan, which is usually when human operators have to intervene. It's part of building reliable systems to reduce the chances of operator error. See: the Hawaii missile alert bug! It must be truly terrifying to sit at that particular keyboard typing in any command.
  - colechristensen 9y ago
    Read "Inviting Disaster" for a lot more about this topic. Many very high profile disasters were caused by operator error or more precisely complex systems not designed for what you might call failure ergonomics.
dx034 9y ago
If I remember correctly, a similar outage in the UK was due to increased flight volumes. The system had a hardcoded limit somewhere which caused issues as flight volume increased. The software was old enough that they weren't aware of that issue beforehand. I could imagine something similar happening here. It definitely shows how well software can work with the right practices. Only one outage in 20 years and that one only caused a reduction of 10% in capacity. Don't think many companies can match that.
- colechristensen 9y ago
  Anyone could, few care to. They really are the wrong practices for a whole lot of things. The level of care, design, and verification just isn't necessary for applications with few or fixable consequences. It is a bit sad that nearly nothing these days strives for that kind of excellence.
- carlmr 9y ago
  >It definitely shows how well software can work with the right practices. Only one outage in 20 years and that one only caused a reduction of 10% in capacity. Don't think many companies can match that. This is what happens when you build software with the same meticulousness as other engineering disciplines. However a lot of software is so much more complex than what other disciplines can build (because you can build anything you can imagine), coupled with early deadlines and profit pressure, that it's unrealistic for most software to be developed this way. You easily have a cost factor of 100x in time as well as money.
  - Cthulhu_ 9y ago
    I'm talking out of my ass here, but I can imagine that the focus for this software system was very sharp and hasn't changed (much) in the past 20 years. When you have a product with tight focus, you can polish the shit out of it and make it last 20+ years. A bit of the Unix mentality. Most products today - and stuff on HN is at the forefront of that - is much more about selling a product to a lot of people, often in highly contested markets. That is, if you make a product, focus on just the core and don't do anything else, you'll stagnate and die. Then again / on the other hand, there's Dropbox whose core functionality has not changed as far as I can tell for a decade - it still does the exact same thing as when I first installed it. Spotify whose IPO was today that doesn't seem to change its core model. In both cases though, I don't know where they put all their development capacity; probably on back-end systems / scalability, marketing, and new applications (like dropbox creating a google docs competitor).
  - xcvbxzas 9y ago
    I see this sentiment a lot. Intuitively it seems like it should be true, but I don't think the case is really quite so clear cut. The costs involve way more than just the initial development. Maintenance eats up a huge, perhaps even a majority, of the total cost as well. And outages or other failures can be very expensive too. It's also important to keep in mind that this isn't an all or nothing situation. We can have software that is more reliable without asking that it chug away without issue for a decade, or anywhere near as long as we expect bridges or buildings to last. The process of developing more reliable software isn't necessarily more expensive than less reliable software. It can even be cheaper. I'm struggling to find the links (maybe somebody else has them handy, or I'll edit them in if I find them), but there have been a few case studies done a few years back by companies that moved to using Ada. In addition to the benefits of more reliable software, they also found development costs were better or at least no worse than C. I know that isn't exactly the language to compare to these days, but as I said these were done some time ago. This is just my own argument, but I suspect that's because the same problems that ultimately cause problems after release also cause problems during development. With a more reliable programming system/environment, problems that might show up later during development are shown to be an issue immediately. This means the issue doesn't need to be tracked down, which can take some serious time. The developers are even fresh on problem area. Personally speaking, I've been totally won over by Ada. It ain't perfect, but it's a hell of a lot better than anything else I've seen - and I've looked a lot. In my own projects (mainly personal or for school admittedly) development is much easier and ultimately quicker. I don't have to spend a day tracking down a weird bug because the compiler let's me know about the issue as soon as I try to cause it.

tim333 9y ago

Yeah, I recall last year "BA’s £150,000,000 outage was caused by someone turning computers on and off too quickly" I think companies often try to stay quite about the dumb stuff.

makmanalp 9y ago
Well - I wouldn't call it dumb stuff. After all, it's only a matter of time one of us does something truly stupid :-) Even more so when under stress and pressure when shit hits the fan, which is usually when human operators have to intervene. It's part of building reliable systems to reduce the chances of operator error. See: the Hawaii missile alert bug! It must be truly terrifying to sit at that particular keyboard typing in any command.
- colechristensen 9y ago
  Read "Inviting Disaster" for a lot more about this topic. Many very high profile disasters were caused by operator error or more precisely complex systems not designed for what you might call failure ergonomics.

dx034 9y ago

If I remember correctly, a similar outage in the UK was due to increased flight volumes. The system had a hardcoded limit somewhere which caused issues as flight volume increased. The software was old enough that they weren't aware of that issue beforehand. I could imagine something similar happening here. It definitely shows how well software can work with the right practices. Only one outage in 20 years and that one only caused a reduction of 10% in capacity. Don't think many companies can match that.

colechristensen 9y ago
Anyone could, few care to. They really are the wrong practices for a whole lot of things. The level of care, design, and verification just isn't necessary for applications with few or fixable consequences. It is a bit sad that nearly nothing these days strives for that kind of excellence.
carlmr 9y ago
>It definitely shows how well software can work with the right practices. Only one outage in 20 years and that one only caused a reduction of 10% in capacity. Don't think many companies can match that. This is what happens when you build software with the same meticulousness as other engineering disciplines. However a lot of software is so much more complex than what other disciplines can build (because you can build anything you can imagine), coupled with early deadlines and profit pressure, that it's unrealistic for most software to be developed this way. You easily have a cost factor of 100x in time as well as money.
- Cthulhu_ 9y ago
  I'm talking out of my ass here, but I can imagine that the focus for this software system was very sharp and hasn't changed (much) in the past 20 years. When you have a product with tight focus, you can polish the shit out of it and make it last 20+ years. A bit of the Unix mentality. Most products today - and stuff on HN is at the forefront of that - is much more about selling a product to a lot of people, often in highly contested markets. That is, if you make a product, focus on just the core and don't do anything else, you'll stagnate and die. Then again / on the other hand, there's Dropbox whose core functionality has not changed as far as I can tell for a decade - it still does the exact same thing as when I first installed it. Spotify whose IPO was today that doesn't seem to change its core model. In both cases though, I don't know where they put all their development capacity; probably on back-end systems / scalability, marketing, and new applications (like dropbox creating a google docs competitor).
- xcvbxzas 9y ago
  I see this sentiment a lot. Intuitively it seems like it should be true, but I don't think the case is really quite so clear cut. The costs involve way more than just the initial development. Maintenance eats up a huge, perhaps even a majority, of the total cost as well. And outages or other failures can be very expensive too. It's also important to keep in mind that this isn't an all or nothing situation. We can have software that is more reliable without asking that it chug away without issue for a decade, or anywhere near as long as we expect bridges or buildings to last. The process of developing more reliable software isn't necessarily more expensive than less reliable software. It can even be cheaper. I'm struggling to find the links (maybe somebody else has them handy, or I'll edit them in if I find them), but there have been a few case studies done a few years back by companies that moved to using Ada. In addition to the benefits of more reliable software, they also found development costs were better or at least no worse than C. I know that isn't exactly the language to compare to these days, but as I said these were done some time ago. This is just my own argument, but I suspect that's because the same problems that ultimately cause problems after release also cause problems during development. With a more reliable programming system/environment, problems that might show up later during development are shown to be an issue immediately. This means the issue doesn't need to be tracked down, which can take some serious time. The developers are even fresh on problem area. Personally speaking, I've been totally won over by Ada. It ain't perfect, but it's a hell of a lot better than anything else I've seen - and I've looked a lot. In my own projects (mainly personal or for school admittedly) development is much easier and ultimately quicker. I don't have to spend a day tracking down a weird bug because the compiler let's me know about the issue as soon as I try to cause it.