6 ms·

This is a great example of how technical debt 'kills'. It's not a murder, it's negligence and a slow demise. I went through one of these projects. The tech deb

This is a great example of how technical debt 'kills'. It's not a murder, it's negligence and a slow demise. I went through one of these projects. The tech debt was never as bad as you describe, but it was a small company operating on a short runway. It also taught me an unfortunate lesson about non-technical founders and the dangers of outsourced code. The MVP for the company had been bought off the shelf. It worked fine, but the code was abstruse and utterly resistant to change. As the price (in time and dollars) of change requests grew, they sensibly in-housed development. Unfortunately, their clients had some idea what to expect in terms of features per day and dollar. Requests like "let us use our logo and custom color scheme" turned out to be serious challenges since every color and style decision was clumsily hardcoded, so we took far too long to achieve them. Ultimately, we ended up a contract behind - bringing in business to fund delivering on the previous request. Most startups operate under the gun like that (with either fundraising or contracts), but they start there and labor to escape. We started solvent, and had no clear plan to break out of tech debt - a rebuild would have been too slow, 'working smarter' wasn't viable, and expanding the tech team would have come too late and too costly. So, we died. Not because we couldn't do work, but because we couldn't do it at a competitive speed.

ryandrake 10y ago
Seen this a lot. A lot of companies think they are "product" companies, but due to their unwillingness to push back on customers, they become custom engineering shops, bolting on little one-off mods to their project over and over to appease bad customers (or to appease POTENTIAL customers who haven't even bought the product yet). Stop me when you recognize this one: "Hey your product is great, but we really want something that does [totally different thing]. If you just add that thing, we will pay for all the NRE and you can sell it to others as part of your product! Win win!" Advice to junior developers: If you hear such talk in the hallway, RUN!
- ayuvar 10y ago
  I think every one of my former employers who have failed, did so by doing those 'customs.' The last one even spun off a dedicated team that built (hacked) prototype customs in order to secure sales, then threw away the prototype and, after collecting the commission, told the new customers that it would take several years to get what they just saw in production but in the meantime we can do our existing product with some mods. I imagine the pressure to accept these deals is immense though. Why let an innocuous little feature request hold up such a great deal?
  - cerved 10y ago
    Sounds like poor sales to me
- Bartweiss 10y ago
  Ouch, that's almost word-for-word from the company that died of debt. It was enterprise sales, so customization was unavoidable, but no one was differentiating between big and small changes, or big and small buyers. The product was desperately struggling to do ~3 things at once, and still being sold to potential buyers on the promise of a fourth thing it would do "soon".
  - jrs235 10y ago
    Enterprise customers which require enterprise sales require enterprise pricing. If appropriate enterprise pricing is not in place then you risk an enterprise failure.
- pavel_lishin 10y ago
  "Let us use our logo" doesn't seem like an unreasonable request.
  - nicostouch 10y ago
    I remember my first day on the job too.
  - wayn3 10y ago
    thats one line of code on some part of some html template if done in a non-idiotic fashion (speaking to the javascript overcomplicators)
    - Bartweiss 10y ago
      Heh, yep. And it was one line of code, after several hundred lines had been torn out and rearranged to ensure that different clients could insert their own pictures of different sizes without everything exploding. The whole team was desperately trying to force enough flexibility into the software that one-line changes could be made in <10 lines, instead of >100.
  - Bartweiss 10y ago
    And it wasn't! That was part of the problem: the sales people couldn't push back on most requests because they were often quite reasonable. When they were more demanding, it was usually from a large prospective buyer so we had to bend over backwards. The result was that we had huge tasks to do with no (current) revenue, and small tasks to do that took 10x as long as they should have. Since servicing existing revenue streams (even on reasonable requests) became so time-consuming, handling big enterprise demands became totally untenable.
    - johnnyhillbilly 10y ago
      Sounds like you needed better sales people.
      - 10y ago
- kernelbandwidth 10y ago
  We're an enterprise software shop, which necessarily means we do a lot of custom work, but we're careful to consider what we'll do. My mentor is an old hand that been through multiple exits, and in every meeting we have, he hammers this point. You're either a product shop, or a professional services shop, and if you don't know which one you are (or you believe wrongly) you die. Simple as that. The deeper you get into the consequences of knowing (or even forcing) which you are, the more implications it has for everything from product design to business strategy, and it's extraordinary how such a simple seeming thing effects such a vast amount of the company.
  - rvdavis 10y ago
    Can you expand on what your mentor defined as a "product shop" vs "professional services shop"?
    - kernelbandwidth 10y ago
      Sure, but the answer is pretty trivial: If you spend more than half your time on customization, you're a professional services shop. He also added that if you're a product shop doing less than 70% off-the-shelf, you're probably screwed, while 90% off the shelf is really the ideal (again, enterprise software). I think the more interesting question is "what counts as professional services?" This gets much trickier, for example when you start building out APIs to make second- or third-party integrations easier, is that "product" or "professional services"? It certainly seems like product building, but if you're doing for a customer's use, it gets real blurry real fast. If you're not using that API internally, you're almost certainly on the professional services side. If you do use it internally, is it rock solid enough that you can support and expose it without that support becoming professional services? Drawing sharp lines aside, this all probably seems kind of trivial, but the first time I ran through our product design with him and we discussed this, I went back and radically re-thought a lot of our strategy, particularly at the customer interfaces.
      - rvdavis 10y ago
        Great explanation, thanks!
- andrewclunn 10y ago
  Working for a company like that, BUT they allowed me to completely rewrite 3 of the tools from scratch in a more modular fashion so that I could do these things without having to modify the old code bases. Now there are two other applications that I still have to support (and were written by a consulting company we no longer contract through). It's night and day. So this isn't really the worst thing if you're given the authority and power to take full control of an application and rebuild it and take ownership of it. Of course, this doesn't really apply to junior devs.
- dagw 10y ago
  Last company I worked for did that to great success. All our customer got a custom version of our product tailored to their needs and their project. At least half our customers where doing something that needed at least one new feature that we didn't currently have. If you build your business model around that it is not necessarily a problematic model.
lewisl9029 10y ago
This part in Rich Hickey's Simple Made Easy talk [1] had a lasting impression on me. It really drove home the point on how a build up of complexity (one of the most common forms of tech debt, and one of the hardest to avoid) can eventually "kill" a project in exactly the way you described, slowly and painfully: "But I have all this speed. I'm agile. I'm fast. You know, this easy stuff is making my life good because I have a lot of speed." What kind of runner can run as fast as they possibly can from the very start of a race? [Audience reply: Sprinter] Right, only somebody who runs really short races, okay? But of course, we are programmers, and we are smarter than runners, apparently, because we know how to fix that problem, right? We just fire the starting pistol every hundred yards and call it a new sprint. ...It's my contention, based on experience, that if you ignore complexity, you will slow down. You will invariably slow down over the long haul. ...if you focus on ease, you will be able to go as fast as possible from the beginning of the race. But no matter what technology you use, or sprints or firing pistols, or whatever, the complexity will eventually kill you. It will kill you in a way that will make every sprint accomplish less. Most sprints will be about completely redoing things you've already done. And the net effect is you're not moving forward in any significant way. [1] https://github.com/matthiasn/talk-transcripts/blob/master/Hickey_Rich/SimpleMadeEasy.md https://github.com/matthiasn/talk-transcripts/blob/master/Hi...

ryandrake 10y ago

Seen this a lot. A lot of companies think they are "product" companies, but due to their unwillingness to push back on customers, they become custom engineering shops, bolting on little one-off mods to their project over and over to appease bad customers (or to appease POTENTIAL customers who haven't even bought the product yet). Stop me when you recognize this one: "Hey your product is great, but we really want something that does [totally different thing]. If you just add that thing, we will pay for all the NRE and you can sell it to others as part of your product! Win win!" Advice to junior developers: If you hear such talk in the hallway, RUN!

ayuvar 10y ago
I think every one of my former employers who have failed, did so by doing those 'customs.' The last one even spun off a dedicated team that built (hacked) prototype customs in order to secure sales, then threw away the prototype and, after collecting the commission, told the new customers that it would take several years to get what they just saw in production but in the meantime we can do our existing product with some mods. I imagine the pressure to accept these deals is immense though. Why let an innocuous little feature request hold up such a great deal?
- cerved 10y ago
  Sounds like poor sales to me
Bartweiss 10y ago
Ouch, that's almost word-for-word from the company that died of debt. It was enterprise sales, so customization was unavoidable, but no one was differentiating between big and small changes, or big and small buyers. The product was desperately struggling to do ~3 things at once, and still being sold to potential buyers on the promise of a fourth thing it would do "soon".
- jrs235 10y ago
  Enterprise customers which require enterprise sales require enterprise pricing. If appropriate enterprise pricing is not in place then you risk an enterprise failure.
pavel_lishin 10y ago
"Let us use our logo" doesn't seem like an unreasonable request.
- nicostouch 10y ago
  I remember my first day on the job too.
- wayn3 10y ago
  thats one line of code on some part of some html template if done in a non-idiotic fashion (speaking to the javascript overcomplicators)
  - Bartweiss 10y ago
    Heh, yep. And it was one line of code, after several hundred lines had been torn out and rearranged to ensure that different clients could insert their own pictures of different sizes without everything exploding. The whole team was desperately trying to force enough flexibility into the software that one-line changes could be made in <10 lines, instead of >100.
- Bartweiss 10y ago
  And it wasn't! That was part of the problem: the sales people couldn't push back on most requests because they were often quite reasonable. When they were more demanding, it was usually from a large prospective buyer so we had to bend over backwards. The result was that we had huge tasks to do with no (current) revenue, and small tasks to do that took 10x as long as they should have. Since servicing existing revenue streams (even on reasonable requests) became so time-consuming, handling big enterprise demands became totally untenable.
  - johnnyhillbilly 10y ago
    Sounds like you needed better sales people.
    - 10y ago
kernelbandwidth 10y ago
We're an enterprise software shop, which necessarily means we do a lot of custom work, but we're careful to consider what we'll do. My mentor is an old hand that been through multiple exits, and in every meeting we have, he hammers this point. You're either a product shop, or a professional services shop, and if you don't know which one you are (or you believe wrongly) you die. Simple as that. The deeper you get into the consequences of knowing (or even forcing) which you are, the more implications it has for everything from product design to business strategy, and it's extraordinary how such a simple seeming thing effects such a vast amount of the company.
- rvdavis 10y ago
  Can you expand on what your mentor defined as a "product shop" vs "professional services shop"?
  - kernelbandwidth 10y ago
    Sure, but the answer is pretty trivial: If you spend more than half your time on customization, you're a professional services shop. He also added that if you're a product shop doing less than 70% off-the-shelf, you're probably screwed, while 90% off the shelf is really the ideal (again, enterprise software). I think the more interesting question is "what counts as professional services?" This gets much trickier, for example when you start building out APIs to make second- or third-party integrations easier, is that "product" or "professional services"? It certainly seems like product building, but if you're doing for a customer's use, it gets real blurry real fast. If you're not using that API internally, you're almost certainly on the professional services side. If you do use it internally, is it rock solid enough that you can support and expose it without that support becoming professional services? Drawing sharp lines aside, this all probably seems kind of trivial, but the first time I ran through our product design with him and we discussed this, I went back and radically re-thought a lot of our strategy, particularly at the customer interfaces.
    - rvdavis 10y ago
      Great explanation, thanks!
andrewclunn 10y ago
Working for a company like that, BUT they allowed me to completely rewrite 3 of the tools from scratch in a more modular fashion so that I could do these things without having to modify the old code bases. Now there are two other applications that I still have to support (and were written by a consulting company we no longer contract through). It's night and day. So this isn't really the worst thing if you're given the authority and power to take full control of an application and rebuild it and take ownership of it. Of course, this doesn't really apply to junior devs.
dagw 10y ago
Last company I worked for did that to great success. All our customer got a custom version of our product tailored to their needs and their project. At least half our customers where doing something that needed at least one new feature that we didn't currently have. If you build your business model around that it is not necessarily a problematic model.

lewisl9029 10y ago

This part in Rich Hickey's Simple Made Easy talk [1] had a lasting impression on me. It really drove home the point on how a build up of complexity (one of the most common forms of tech debt, and one of the hardest to avoid) can eventually "kill" a project in exactly the way you described, slowly and painfully: "But I have all this speed. I'm agile. I'm fast. You know, this easy stuff is making my life good because I have a lot of speed." What kind of runner can run as fast as they possibly can from the very start of a race? [Audience reply: Sprinter] Right, only somebody who runs really short races, okay? But of course, we are programmers, and we are smarter than runners, apparently, because we know how to fix that problem, right? We just fire the starting pistol every hundred yards and call it a new sprint. ...It's my contention, based on experience, that if you ignore complexity, you will slow down. You will invariably slow down over the long haul. ...if you focus on ease, you will be able to go as fast as possible from the beginning of the race. But no matter what technology you use, or sprints or firing pistols, or whatever, the complexity will eventually kill you. It will kill you in a way that will make every sprint accomplish less. Most sprints will be about completely redoing things you've already done. And the net effect is you're not moving forward in any significant way. [1] https://github.com/matthiasn/talk-transcripts/blob/master/Hickey_Rich/SimpleMadeEasy.md https://github.com/matthiasn/talk-transcripts/blob/master/Hi...