4 ms·

Even private failures (outages, data loss) involving enterprise systems can cost an enterprise business gigantic sums of money directly or in fines or in custom

by Hoff 17y ago

Even private failures (outages, data loss) involving enterprise systems can cost an enterprise business gigantic sums of money directly or in fines or in customer perception or (for some applications) can put patients' health and lives at risk. Public outages can involve fires and building collapses. Various of the customers I've worked with over the years have calculated their direct losses at thousands or more dollars per minute of outage. This is a whole different world view from a web outage or a PC reboot; of how to upgrade an application or a server environment without restarting it. Up-times here can be measured in decades. Your development processes and your management accordingly become conservative. Sometimes hidebound. This is a whole different world from what most folks are accustomed to working in, too. This in terms of scale and cost and speeds and feeds involved; thirty million lines of even the best source code around is not the most agile of platforms ever constructed, but it can be critical to the operations and profits of a business. There are numbers of (quiet) successes here and there have been many successful COTS migrations; I've worked on a number of these migrations and application replacements over the years, and these projects just don't get the press attention of the screw-ups. And yes, I'm aware of both successes and screw-ups. And of one box that was up and running for seventeen years.

jbert 17y ago
So the silver bullet is to reduce the cost of failure? So that you can allow failure, which seems to be a necessary requirement for fast iteration and hence fast development? i.e. the hard thing to do is to find a way to slice off 5 or 10% of the problem space so you can have a fast-iterating pilot system which can be a little flakey. As it beds down into maturity after it's period of rapid development (and higher-than-acceptable downtime), the new system can then be rolled out to the wider audience. So the trick is to designate an "early adopter" section of your enterprise (maybe one branch office), give them a free reign and then scale their good solutions enterprise-wide.
- Hoff 17y ago
  The easy stuff usually gets sliced off and dealt with. The folks working on these boxes are Not Dumb. When you're dealing with financial transactions or stock trading or with medical records, you're either able to deal with the fire hose of data, or with the uptime requirements, or the scale of the data. Or not. These servers and clusters are very different than what most folks are accustomed to dealing with; vastly larger servers, application environments, communications, storage. And unfortunately for upgrades and migrations and incremental work, usually also involving (often fragile and ill-documented and poorly understood) interconnections all over the place. And critical.
  - sailormoon 17y ago
    The folks working on these boxes are Not Dumb. No-one's saying they're dumb. They're saying they're Doing It Wrong. And the things you're talking about - banks and stock - are edge cases of the enterprise world. I would actually argue that most stock markets are actually very well implemented. Banks usually have competent IT too, since IT is a core competency of theirs. These are edge cases though; most corporate IT is orders of magnitude less sexy than real time trading systems.
  - jbert 17y ago
    > When you're dealing with financial transactions or stock trading or with medical records, you're either able to deal with the fire hose of data, or with the uptime requirements, or the scale of the data. Or not. But you might find that there is one stock market with a smaller trade volume than the others. Or one of the 50 hospitals in your district is smaller. If you can peel off a bunch of "early adopters" and make them guinea pigs, you can get some of the benefits of rapid development. You either need the buy in of the early adopters or be able to push them around. This happens all the time. The UK used to test their laws on the Scots before rolling them out to the rest of the country.
paul 17y ago
They estimate the costs of downtime, but do they estimate the cost of having bad systems?
- wglb 17y ago
  No, they don't. And what is not said here is that they don't acknowledge failures, even when they are obvious, thus depriving the organization a chance to learn from the failure.
wglb 17y ago
Then again there are businesses which don't lose money on a per minute basis. Consider an insurance company. Its cycles (except for web portals) are monthly. And how much money does your bank lose when you have to stand in line for their system to reboot? Maybe not all that much. Then there are businesses, like ITA which is used to sell seats on airplanes, where there could be an effect of cascading downtime could cost money all along the line. Much more likely is the project failure cost. Dozens or hundreds of programmers working on a large project that gets cancelled. This is the real nub of enterprise software risk.
- whatusername 17y ago
  If my banks ATM's start rebooting more than about once then I'm finding a new bank. The Fail Whale is fine for twitter - less so when there's real $$ involved.

by Hoff 17y ago

jbert 17y ago

So the silver bullet is to reduce the cost of failure? So that you can allow failure, which seems to be a necessary requirement for fast iteration and hence fast development? i.e. the hard thing to do is to find a way to slice off 5 or 10% of the problem space so you can have a fast-iterating pilot system which can be a little flakey. As it beds down into maturity after it's period of rapid development (and higher-than-acceptable downtime), the new system can then be rolled out to the wider audience. So the trick is to designate an "early adopter" section of your enterprise (maybe one branch office), give them a free reign and then scale their good solutions enterprise-wide.

Hoff 17y ago
The easy stuff usually gets sliced off and dealt with. The folks working on these boxes are Not Dumb. When you're dealing with financial transactions or stock trading or with medical records, you're either able to deal with the fire hose of data, or with the uptime requirements, or the scale of the data. Or not. These servers and clusters are very different than what most folks are accustomed to dealing with; vastly larger servers, application environments, communications, storage. And unfortunately for upgrades and migrations and incremental work, usually also involving (often fragile and ill-documented and poorly understood) interconnections all over the place. And critical.
- sailormoon 17y ago
  The folks working on these boxes are Not Dumb. No-one's saying they're dumb. They're saying they're Doing It Wrong. And the things you're talking about - banks and stock - are edge cases of the enterprise world. I would actually argue that most stock markets are actually very well implemented. Banks usually have competent IT too, since IT is a core competency of theirs. These are edge cases though; most corporate IT is orders of magnitude less sexy than real time trading systems.
- jbert 17y ago
  > When you're dealing with financial transactions or stock trading or with medical records, you're either able to deal with the fire hose of data, or with the uptime requirements, or the scale of the data. Or not. But you might find that there is one stock market with a smaller trade volume than the others. Or one of the 50 hospitals in your district is smaller. If you can peel off a bunch of "early adopters" and make them guinea pigs, you can get some of the benefits of rapid development. You either need the buy in of the early adopters or be able to push them around. This happens all the time. The UK used to test their laws on the Scots before rolling them out to the rest of the country.

paul 17y ago

They estimate the costs of downtime, but do they estimate the cost of having bad systems?

wglb 17y ago
No, they don't. And what is not said here is that they don't acknowledge failures, even when they are obvious, thus depriving the organization a chance to learn from the failure.

wglb 17y ago

Then again there are businesses which don't lose money on a per minute basis. Consider an insurance company. Its cycles (except for web portals) are monthly. And how much money does your bank lose when you have to stand in line for their system to reboot? Maybe not all that much. Then there are businesses, like ITA which is used to sell seats on airplanes, where there could be an effect of cascading downtime could cost money all along the line. Much more likely is the project failure cost. Dozens or hundreds of programmers working on a large project that gets cancelled. This is the real nub of enterprise software risk.

whatusername 17y ago
If my banks ATM's start rebooting more than about once then I'm finding a new bank. The Fail Whale is fine for twitter - less so when there's real $$ involved.