7 ms·

Doing It Wrong

114 points by gthank 17y ago

arohner 17y ago
Everything tbray says is correct, but I had the feeling I'd heard this before. Then I realized, this post is a combination of Gall's law "all working complex systems evolved from working simple systems", and Joel's advice of "never outsource core competencies" http://en.wikipedia.org/wiki/Galls_law http://en.wikipedia.org/wiki/Galls_law http://www.joelonsoftware.com/articles/fog0000000007.html http://www.joelonsoftware.com/articles/fog0000000007.html
garyrichardson 17y ago
And this is why I prefer to work for small companies/startups vs. large companies.
Hoff 17y ago
Even private failures (outages, data loss) involving enterprise systems can cost an enterprise business gigantic sums of money directly or in fines or in customer perception or (for some applications) can put patients' health and lives at risk. Public outages can involve fires and building collapses. Various of the customers I've worked with over the years have calculated their direct losses at thousands or more dollars per minute of outage. This is a whole different world view from a web outage or a PC reboot; of how to upgrade an application or a server environment without restarting it. Up-times here can be measured in decades. Your development processes and your management accordingly become conservative. Sometimes hidebound. This is a whole different world from what most folks are accustomed to working in, too. This in terms of scale and cost and speeds and feeds involved; thirty million lines of even the best source code around is not the most agile of platforms ever constructed, but it can be critical to the operations and profits of a business. There are numbers of (quiet) successes here and there have been many successful COTS migrations; I've worked on a number of these migrations and application replacements over the years, and these projects just don't get the press attention of the screw-ups. And yes, I'm aware of both successes and screw-ups. And of one box that was up and running for seventeen years.
- jbert 17y ago
  So the silver bullet is to reduce the cost of failure? So that you can allow failure, which seems to be a necessary requirement for fast iteration and hence fast development? i.e. the hard thing to do is to find a way to slice off 5 or 10% of the problem space so you can have a fast-iterating pilot system which can be a little flakey. As it beds down into maturity after it's period of rapid development (and higher-than-acceptable downtime), the new system can then be rolled out to the wider audience. So the trick is to designate an "early adopter" section of your enterprise (maybe one branch office), give them a free reign and then scale their good solutions enterprise-wide.
  - Hoff 17y ago
    The easy stuff usually gets sliced off and dealt with. The folks working on these boxes are Not Dumb. When you're dealing with financial transactions or stock trading or with medical records, you're either able to deal with the fire hose of data, or with the uptime requirements, or the scale of the data. Or not. These servers and clusters are very different than what most folks are accustomed to dealing with; vastly larger servers, application environments, communications, storage. And unfortunately for upgrades and migrations and incremental work, usually also involving (often fragile and ill-documented and poorly understood) interconnections all over the place. And critical.
    - sailormoon 17y ago
      The folks working on these boxes are Not Dumb. No-one's saying they're dumb. They're saying they're Doing It Wrong. And the things you're talking about - banks and stock - are edge cases of the enterprise world. I would actually argue that most stock markets are actually very well implemented. Banks usually have competent IT too, since IT is a core competency of theirs. These are edge cases though; most corporate IT is orders of magnitude less sexy than real time trading systems.
    - jbert 17y ago
      > When you're dealing with financial transactions or stock trading or with medical records, you're either able to deal with the fire hose of data, or with the uptime requirements, or the scale of the data. Or not. But you might find that there is one stock market with a smaller trade volume than the others. Or one of the 50 hospitals in your district is smaller. If you can peel off a bunch of "early adopters" and make them guinea pigs, you can get some of the benefits of rapid development. You either need the buy in of the early adopters or be able to push them around. This happens all the time. The UK used to test their laws on the Scots before rolling them out to the rest of the country.
aristus 17y ago
I think the author is misled by survivor bias. How many Basecamp clones have been released and failed? How many more have been planned, half-implemented, and shelved? Also, how many "enterprise" development teams can commit to creating and supporting one and only one (1) app over a multi-year period? How many can depend on thousands and millions of users to discover bugs, do load testing, penetration testing, usability, etc etc? How many "massages" has Flickr had in the last 5 years? Is that uptime acceptable to a bank? How many shoe retailers and state governments even know how to recruit and retain the type of talent that produces great software? How many could admit they don't?
- fauigerzigerk 17y ago
  Ok but I think his point is that in the enterprise software space even the survivors fail in the sense that users hate them.
  - michaelneale 17y ago
    He then oddly goes on to talk about the buy vs build argument for things like Oracle/SAP financials/ERP - which are deeply hated by those that use them (granted, may not be able to make that kind of BigCo HR/financial stuff fun !) - but what is interesting is that those systems are not really off the shelf. They are usually multi-million dollar "implementations" and sometimes (seriously!) "re-implementations" of them - which cost orders of magnitude more then most IT bespoke projects - often due to licencing costs. This makes me think that the numbers make things look better then they are. For example: if the cost of an setup for SAP financials is 2mil, and 400K "implementation", then a blow out of 100% in the "implementation" cost is not, percentage wise, as big a deal as if it was a bespoke project (ie sans the licence cost) - as the bulk of the cost is a fixed licence. This is also why building (as in skyscraper style) blow outs don't appear as huge - they labour cost is generally a relatively smaller portion of the project cost then the materials compared to IT projects.
- benhoyt 17y ago
  Is that uptime [Flickr's] acceptable to a bank? I would love a Flickr-like bank. Banks are only open about 8 hours out of 24, and (at least here in NZ) even when I do an online money transfer, it only goes through in their batch that midnight. I would love a front-end to our bank system that did things right, especially instant and easy transfers. I guess you could say PayPal tries to do this, but you have to get money in and out of PayPal, and they have high fees.
ricardo 17y ago
"Of course, we’re not in the Promised Land yet. I’m actually surprised that Salesforce isn’t a lot bigger than it is; a variety of things are holding back the migration to the utility model" Many organizations still attempt to 'build' their own highly customized CRM system using SFDC as a starting point. It doesn't matter if they buy a system off the shelf, the IT organizations for big companies will manage to make these projects just as unsuccessful as their built from scratch counterparts. The success of these implementations, and in turn the companies that sell the systems like SFDC, are dependent on how willing their own customers are to change their internal processes to match what the system does well. The biggest project failures for these systems occur when these organizations refuse to change how they do business and instead spend millions of dollars in customization fees to the vendor. SFDC isn't immune to the failures that the author describes, they just experience it from a different perspective.
fauigerzigerk 17y ago
He definately has a point, but then again to make an accounting package as simple as twitter you'd have to make the tax code as simple as twitter too :-)
JoelMcCracken 17y ago
I think the author is missing the point. Large, overly complex, ineffective systems a symptom of corporate culture. It isn't something that you can just fix without fixing the underlying cause.
- michaelneale 17y ago
  Exactly - it is NOT a technology problem. Well other then vendors capitalising on the corporate culture in selling "solutions" which just make things worse.
njharman 17y ago
The design of a system will reflect the organization that creates it.
jmatt 17y ago
This is right on. I've worked at big and medium size companies in the past decade. I worked at a startup that was acquired by a big company. I've implemented enterprise systems from design through support for governments and dow30 companies. Those of you that think that these systems are symptom of the corporate culture are wrong. I'm not defending the poor corporate cultures that I saw. I am defending the argument that better technologies and processes could lead to better systems. From a process standpoint the problem is it's counter intuitive to most management that smaller, higher paid teams using obscure* technology could out produce the highly polished cogs-in-the-machine process. For example, this usually isn't true for (most) mechanical engineering projects. We all know it as fact on HN but it doesn't apply to many enterprise industries. I remember times where as a team we were denied the ability to create scripts or tools to do work for us. Or we were denied the use of open source tools. Instead we spent days doing it manually. Then, when it needed to be done for the next project, we would again do it manually. We couldn't push a small but critical change without completing a multi-day code move process. When we did write scripts, IT had to approve them which meant you were at their mercy. There were many more of these sorts of poor process decisions. And there were smart capable programmers working there, often pushing to make the right decision. When I run into old co-workers many still believe it's impossible to implement enterprise code in something like ruby, python, lisp, haskell, etc. It's Java or C#. Common reasoning - What happens if I need a new Clojure or Lisp programmer!? Or, where will ruby be in 20 years? We know today where Microsoft support will be in the future. Look at Microsoft's support plans. Etc. I'll add a little bit of my own opinion in here too. It is my hope that eventually there will be a few elegant generic enterprise solutions where DSLs are implemented for business analysts. It's insane how much of this work is still done by programmers. How many more times does a programmer have to code a { main-menu customer organization admin ... } window or custom approval process, etc. (Yes, I know this has been attempted often without success, it's still a good idea. And I think in the end it'll happen. It just takes time to figure out what will work) * obscure to management. EDIT: Grammar
- extension 17y ago
  Common reasoning - What happens if I need a new Clojure or Lisp programmer!? That's a reasonable argument when read as "What happens to our Erlang+Ruby crystal palace when management takes over and talented devs won't touch us with a ten foot pole and we are forced to hire Java monkeys who think lambda is something you barbecue on a skewer?"
sailormoon 17y ago
The time between having an idea and its public launch is measured in days not months, weeks not years. I like and totally agree with most of the article but this myth annoys me. Exactly none of the the huge complex web apps he namedrops could have been made in under a week. And even if they were, that time omits what must have been an extended period of contemplation and conceptual design by a definite "rock star" level programmer.
- mbrubeck 17y ago
  Sure, creating an entire new product can take a lot longer, but almost everyone working in web startups has the experience of launching a new feature (for an existing site) the same day it was first suggested.
  - sailormoon 17y ago
    Sure, if it's a small feature that fits in well with the existing structure. But Tbray's talking like people can just knock up facebook or basecamp from scratch, in a matter of days. No-one can do that. Oh yeah sure, you might be able to make a decent stab at re-implementing one of them in a few days but it's a different story when you have to just make it up. Anyway I do agree with him generally, just that I see this stuff a lot. Good web developers are stunningly productive compared to your average .NET corporate seat-warmer but they're not magic!
k7d 17y ago
Been there for quite some time. I can't agree more - a lot of things are wrong in enterprise world. Couple big ones from my experience: - Process-centric culture. Most enterprises have processes for making processes. Process always complicates everything. And most people inside will blindly protect the process, so it's really difficult to question it. - Artificially complicated solutions from "big vendors" (read Oracle, IBM etc). They keep adding new "features" over an over only to have something to sell constantly. I have never seen an enterprise software being simplified. A lot of people inside enterprise will again protect these since their jobs may become redundant if simpler solutions are introduced.
statictype 17y ago
Really, the problem boils down to this: Enterprise software is not written for end-users. It's written for companies. It's not marketed to the people who will actually use it and it's not paid for by the people who are going to use it. So it should come as no surprise that the incentives and motives that shape enterprise software are completely orthogonal to the incentives for creating good software.

114 points by gthank 17y ago

Everything tbray says is correct, but I had the feeling I'd heard this before. Then I realized, this post is a combination of Gall's law "all working complex systems evolved from working simple systems", and Joel's advice of "never outsource core competencies" http://en.wikipedia.org/wiki/Galls_law http://en.wikipedia.org/wiki/Galls_law http://www.joelonsoftware.com/articles/fog0000000007.html http://www.joelonsoftware.com/articles/fog0000000007.html

garyrichardson 17y ago

And this is why I prefer to work for small companies/startups vs. large companies.

Hoff 17y ago

Even private failures (outages, data loss) involving enterprise systems can cost an enterprise business gigantic sums of money directly or in fines or in customer perception or (for some applications) can put patients' health and lives at risk. Public outages can involve fires and building collapses. Various of the customers I've worked with over the years have calculated their direct losses at thousands or more dollars per minute of outage. This is a whole different world view from a web outage or a PC reboot; of how to upgrade an application or a server environment without restarting it. Up-times here can be measured in decades. Your development processes and your management accordingly become conservative. Sometimes hidebound. This is a whole different world from what most folks are accustomed to working in, too. This in terms of scale and cost and speeds and feeds involved; thirty million lines of even the best source code around is not the most agile of platforms ever constructed, but it can be critical to the operations and profits of a business. There are numbers of (quiet) successes here and there have been many successful COTS migrations; I've worked on a number of these migrations and application replacements over the years, and these projects just don't get the press attention of the screw-ups. And yes, I'm aware of both successes and screw-ups. And of one box that was up and running for seventeen years.

jbert 17y ago
So the silver bullet is to reduce the cost of failure? So that you can allow failure, which seems to be a necessary requirement for fast iteration and hence fast development? i.e. the hard thing to do is to find a way to slice off 5 or 10% of the problem space so you can have a fast-iterating pilot system which can be a little flakey. As it beds down into maturity after it's period of rapid development (and higher-than-acceptable downtime), the new system can then be rolled out to the wider audience. So the trick is to designate an "early adopter" section of your enterprise (maybe one branch office), give them a free reign and then scale their good solutions enterprise-wide.
- Hoff 17y ago
  The easy stuff usually gets sliced off and dealt with. The folks working on these boxes are Not Dumb. When you're dealing with financial transactions or stock trading or with medical records, you're either able to deal with the fire hose of data, or with the uptime requirements, or the scale of the data. Or not. These servers and clusters are very different than what most folks are accustomed to dealing with; vastly larger servers, application environments, communications, storage. And unfortunately for upgrades and migrations and incremental work, usually also involving (often fragile and ill-documented and poorly understood) interconnections all over the place. And critical.
  - sailormoon 17y ago
    The folks working on these boxes are Not Dumb. No-one's saying they're dumb. They're saying they're Doing It Wrong. And the things you're talking about - banks and stock - are edge cases of the enterprise world. I would actually argue that most stock markets are actually very well implemented. Banks usually have competent IT too, since IT is a core competency of theirs. These are edge cases though; most corporate IT is orders of magnitude less sexy than real time trading systems.
  - jbert 17y ago
    > When you're dealing with financial transactions or stock trading or with medical records, you're either able to deal with the fire hose of data, or with the uptime requirements, or the scale of the data. Or not. But you might find that there is one stock market with a smaller trade volume than the others. Or one of the 50 hospitals in your district is smaller. If you can peel off a bunch of "early adopters" and make them guinea pigs, you can get some of the benefits of rapid development. You either need the buy in of the early adopters or be able to push them around. This happens all the time. The UK used to test their laws on the Scots before rolling them out to the rest of the country.

aristus 17y ago

I think the author is misled by survivor bias. How many Basecamp clones have been released and failed? How many more have been planned, half-implemented, and shelved? Also, how many "enterprise" development teams can commit to creating and supporting one and only one (1) app over a multi-year period? How many can depend on thousands and millions of users to discover bugs, do load testing, penetration testing, usability, etc etc? How many "massages" has Flickr had in the last 5 years? Is that uptime acceptable to a bank? How many shoe retailers and state governments even know how to recruit and retain the type of talent that produces great software? How many could admit they don't?

fauigerzigerk 17y ago
Ok but I think his point is that in the enterprise software space even the survivors fail in the sense that users hate them.
- michaelneale 17y ago
  He then oddly goes on to talk about the buy vs build argument for things like Oracle/SAP financials/ERP - which are deeply hated by those that use them (granted, may not be able to make that kind of BigCo HR/financial stuff fun !) - but what is interesting is that those systems are not really off the shelf. They are usually multi-million dollar "implementations" and sometimes (seriously!) "re-implementations" of them - which cost orders of magnitude more then most IT bespoke projects - often due to licencing costs. This makes me think that the numbers make things look better then they are. For example: if the cost of an setup for SAP financials is 2mil, and 400K "implementation", then a blow out of 100% in the "implementation" cost is not, percentage wise, as big a deal as if it was a bespoke project (ie sans the licence cost) - as the bulk of the cost is a fixed licence. This is also why building (as in skyscraper style) blow outs don't appear as huge - they labour cost is generally a relatively smaller portion of the project cost then the materials compared to IT projects.
benhoyt 17y ago
Is that uptime [Flickr's] acceptable to a bank? I would love a Flickr-like bank. Banks are only open about 8 hours out of 24, and (at least here in NZ) even when I do an online money transfer, it only goes through in their batch that midnight. I would love a front-end to our bank system that did things right, especially instant and easy transfers. I guess you could say PayPal tries to do this, but you have to get money in and out of PayPal, and they have high fees.

ricardo 17y ago

"Of course, we’re not in the Promised Land yet. I’m actually surprised that Salesforce isn’t a lot bigger than it is; a variety of things are holding back the migration to the utility model" Many organizations still attempt to 'build' their own highly customized CRM system using SFDC as a starting point. It doesn't matter if they buy a system off the shelf, the IT organizations for big companies will manage to make these projects just as unsuccessful as their built from scratch counterparts. The success of these implementations, and in turn the companies that sell the systems like SFDC, are dependent on how willing their own customers are to change their internal processes to match what the system does well. The biggest project failures for these systems occur when these organizations refuse to change how they do business and instead spend millions of dollars in customization fees to the vendor. SFDC isn't immune to the failures that the author describes, they just experience it from a different perspective.

fauigerzigerk 17y ago

He definately has a point, but then again to make an accounting package as simple as twitter you'd have to make the tax code as simple as twitter too :-)

JoelMcCracken 17y ago

I think the author is missing the point. Large, overly complex, ineffective systems a symptom of corporate culture. It isn't something that you can just fix without fixing the underlying cause.

michaelneale 17y ago
Exactly - it is NOT a technology problem. Well other then vendors capitalising on the corporate culture in selling "solutions" which just make things worse.

njharman 17y ago

The design of a system will reflect the organization that creates it.

jmatt 17y ago

This is right on. I've worked at big and medium size companies in the past decade. I worked at a startup that was acquired by a big company. I've implemented enterprise systems from design through support for governments and dow30 companies. Those of you that think that these systems are symptom of the corporate culture are wrong. I'm not defending the poor corporate cultures that I saw. I am defending the argument that better technologies and processes could lead to better systems. From a process standpoint the problem is it's counter intuitive to most management that smaller, higher paid teams using obscure* technology could out produce the highly polished cogs-in-the-machine process. For example, this usually isn't true for (most) mechanical engineering projects. We all know it as fact on HN but it doesn't apply to many enterprise industries. I remember times where as a team we were denied the ability to create scripts or tools to do work for us. Or we were denied the use of open source tools. Instead we spent days doing it manually. Then, when it needed to be done for the next project, we would again do it manually. We couldn't push a small but critical change without completing a multi-day code move process. When we did write scripts, IT had to approve them which meant you were at their mercy. There were many more of these sorts of poor process decisions. And there were smart capable programmers working there, often pushing to make the right decision. When I run into old co-workers many still believe it's impossible to implement enterprise code in something like ruby, python, lisp, haskell, etc. It's Java or C#. Common reasoning - What happens if I need a new Clojure or Lisp programmer!? Or, where will ruby be in 20 years? We know today where Microsoft support will be in the future. Look at Microsoft's support plans. Etc. I'll add a little bit of my own opinion in here too. It is my hope that eventually there will be a few elegant generic enterprise solutions where DSLs are implemented for business analysts. It's insane how much of this work is still done by programmers. How many more times does a programmer have to code a { main-menu customer organization admin ... } window or custom approval process, etc. (Yes, I know this has been attempted often without success, it's still a good idea. And I think in the end it'll happen. It just takes time to figure out what will work) * obscure to management. EDIT: Grammar

extension 17y ago
Common reasoning - What happens if I need a new Clojure or Lisp programmer!? That's a reasonable argument when read as "What happens to our Erlang+Ruby crystal palace when management takes over and talented devs won't touch us with a ten foot pole and we are forced to hire Java monkeys who think lambda is something you barbecue on a skewer?"

sailormoon 17y ago

The time between having an idea and its public launch is measured in days not months, weeks not years. I like and totally agree with most of the article but this myth annoys me. Exactly none of the the huge complex web apps he namedrops could have been made in under a week. And even if they were, that time omits what must have been an extended period of contemplation and conceptual design by a definite "rock star" level programmer.

mbrubeck 17y ago
Sure, creating an entire new product can take a lot longer, but almost everyone working in web startups has the experience of launching a new feature (for an existing site) the same day it was first suggested.
- sailormoon 17y ago
  Sure, if it's a small feature that fits in well with the existing structure. But Tbray's talking like people can just knock up facebook or basecamp from scratch, in a matter of days. No-one can do that. Oh yeah sure, you might be able to make a decent stab at re-implementing one of them in a few days but it's a different story when you have to just make it up. Anyway I do agree with him generally, just that I see this stuff a lot. Good web developers are stunningly productive compared to your average .NET corporate seat-warmer but they're not magic!

k7d 17y ago

Been there for quite some time. I can't agree more - a lot of things are wrong in enterprise world. Couple big ones from my experience: - Process-centric culture. Most enterprises have processes for making processes. Process always complicates everything. And most people inside will blindly protect the process, so it's really difficult to question it. - Artificially complicated solutions from "big vendors" (read Oracle, IBM etc). They keep adding new "features" over an over only to have something to sell constantly. I have never seen an enterprise software being simplified. A lot of people inside enterprise will again protect these since their jobs may become redundant if simpler solutions are introduced.

statictype 17y ago

Really, the problem boils down to this: Enterprise software is not written for end-users. It's written for companies. It's not marketed to the people who will actually use it and it's not paid for by the people who are going to use it. So it should come as no surprise that the incentives and motives that shape enterprise software are completely orthogonal to the incentives for creating good software.