8 ms·

Yes and no. The project wasn't killed specifically because "you have technical debt". It was killed because there was no way for anyone to be effective with t

Yes and no. The project wasn't killed specifically because "you have technical debt". It was killed because there was no way for anyone to be effective with the combination of poor undocumented code. "We need to change the email message that goes out when someone registers". This took a team of (4?) people 5 calendar days to change. As a contractor, I had to vpn in to one system, then remote desktop over another vpn to another system. Building web apps, these dev systems were not allowed to talk to the internet at all, so things like pulling external dependencies (security libraries, templating libraries, etc) was impossible - pretty much everything was handrolled, largely due to this restriction. The last big killer was that the system was not passing accessibility audits. Trying to determine where to make a change to any single element would take minutes to hours, vs seconds to minutes you'd normally expect. Much of the 'templates' used were the result of a SQL statement joining 12 tables (html_meta, html_form, html_link, html_grid, etc) and complex concat()s, so adding a page or making a change might take an hour to track down the appropriate collection of tables, then figure out a SQL script to run, then send it to the person who had permissions to make updates to the SQL, then wait and see. Did the technical debt itself kill the project? Technically no, but the inability to do anything productive in a reasonable amount of time forced the project to shut down.

dx034 10y ago
Templates stored across a database is probably the worst thing I've seen repeatedly across projects. Just because a database can store everything doesn't mean it has to. Some people really seem(ed) to have an allergy to plain files for storage. A plain file with OS level caching will beat most (if not all) databases for static content. But doesn't sound as fancy, so it's probably harder to charge a lot of money for it.
- mgkimsal 10y ago
  Template in one database table I can live with (pros and cons, multiple front-ends, etc). One template broken up in to 12 tables requiring an 100+ line SQL statement with concat()s and HTML interspersed is insane. Had there been an API or utilities with it to manage it, it might have been manageable, but nope - just "write some queries". Also, just repeated your comment to a friend who said "that's the worst thing you've seen? can i have your job?" :)
  - nicostouch 10y ago
    (blown away by all the responses to my original question!!!) Your story here makes me laugh if only because of a very painfully familiar memory. Luckily this wasn't a big production system but rather an internal tool (that I guess clients did also use but it wasn't part of 'production' per se) that was written entirely in perl_cgi filled with cryptic regular expressions written in complete spaghetti code and it would concatenate together entire webpages that had bits of them rendered by including the contents of files strewn all over the file system and of course the logic to concatenate all the html together was strewn across a fistful of files which were in disparate locations. In short I was once asked to make a simple change to some html and after 5 days of reading through perl_cgi and developing a pure hatred for Larry Wall, I decided to do a java re-write that took 3 days. I mean... crikey. Haha.
    - RUG3Y 10y ago
      We have a similar application in PHP. By the time I've traced through all of the included files that are touched by a particular function, I've forgotten what I'm looking for. It's truly a nightmare.
- rbirkby 10y ago
  Wait until you see a Turing complete DSL programming language stored line-by-line in rows in a database table and executed by pl/SQL using cursors, locking the entire execution to prevent concurrency.
  - nicostouch 10y ago
    This is why we can't develop nice things.
- Terr_ 10y ago
  I'm dealing with one internal project where this happens because there's an artificial IT/build distinction between "emergency" code push and "casual" raw database change. This means lots of business-rule crap gets softcoded into the database or ini files (increasing complexity and bug-risk) just to support a hypothetical future where somebody needs it changed without a full sprint cycle.
- bandrami 10y ago
  And you aren't kidding about "repeatedly". Personally I associate it with the late 1990s/early oughts and ColdFusion; I think one of the early CF frameworks really encouraged it, and it kind of just stuck from there, particularly in Government web work. But it's probably wider than that...
jkingsbery 10y ago
This has been my experience. Since technical debt is hard to measure, it's more a case of a series of unwise technical decisions leading to a lack of productivity. Due to tight schedules, short-cuts are taken which lead to more unwise technical decisions, and you have a death-spiral.
- Bahamut 10y ago
  Isn't this precisely technical debt? Unless you want to split hairs and call this a technical massacre...
  - Bartweiss 10y ago
    It is, but the question is what "killed" by technical debt means. It's uncommon but not unheard of for code to reach the point of "we can't do that". Mostly, though, the proximate cause of death is a funding shortage or management decision to shutdown. Technical debt is just driving the cost overruns or inefficiencies that kill the project.
    - Bahamut 10y ago
      I don't disagree that the root cause in that situation is not engineering in most situations and that it usually is an indication of a symptom, but attempting to change the meaning of the term itself is not a great approach for communicating that.
pyb 10y ago
I've also got experience of this kind of situation (See my other comment) I think you can definitely call that technical debt.
robodale 10y ago
This. It's not like there is a sign saying "Technical Debt Required to Proceed"...but rather the slow death from a thousand cuts to productivity caused by having to analyze every potential system, process, template, stored procedure, etc, etc...to make any stable(ish) change. Even if things are loosely coupled and not dependent on each other...you still have to go in and make those changes. Telling this to a room full of non-understanding management is a whole different challenge...
oelmekki 10y ago
This account kind of comfort me in what I think of technical debt : most of time, the problem is most likely lack of documentation than anything else. I don't see how a big project could be coded without containing anything specific to the project. And even then, the architecture by itself is unique and deserves documentation.
Bartweiss 10y ago
This is a great example of how technical debt 'kills'. It's not a murder, it's negligence and a slow demise. I went through one of these projects. The tech debt was never as bad as you describe, but it was a small company operating on a short runway. It also taught me an unfortunate lesson about non-technical founders and the dangers of outsourced code. The MVP for the company had been bought off the shelf. It worked fine, but the code was abstruse and utterly resistant to change. As the price (in time and dollars) of change requests grew, they sensibly in-housed development. Unfortunately, their clients had some idea what to expect in terms of features per day and dollar. Requests like "let us use our logo and custom color scheme" turned out to be serious challenges since every color and style decision was clumsily hardcoded, so we took far too long to achieve them. Ultimately, we ended up a contract behind - bringing in business to fund delivering on the previous request. Most startups operate under the gun like that (with either fundraising or contracts), but they start there and labor to escape. We started solvent, and had no clear plan to break out of tech debt - a rebuild would have been too slow, 'working smarter' wasn't viable, and expanding the tech team would have come too late and too costly. So, we died. Not because we couldn't do work, but because we couldn't do it at a competitive speed.
- ryandrake 10y ago
  Seen this a lot. A lot of companies think they are "product" companies, but due to their unwillingness to push back on customers, they become custom engineering shops, bolting on little one-off mods to their project over and over to appease bad customers (or to appease POTENTIAL customers who haven't even bought the product yet). Stop me when you recognize this one: "Hey your product is great, but we really want something that does [totally different thing]. If you just add that thing, we will pay for all the NRE and you can sell it to others as part of your product! Win win!" Advice to junior developers: If you hear such talk in the hallway, RUN!
  - ayuvar 10y ago
    I think every one of my former employers who have failed, did so by doing those 'customs.' The last one even spun off a dedicated team that built (hacked) prototype customs in order to secure sales, then threw away the prototype and, after collecting the commission, told the new customers that it would take several years to get what they just saw in production but in the meantime we can do our existing product with some mods. I imagine the pressure to accept these deals is immense though. Why let an innocuous little feature request hold up such a great deal?
ionwake 10y ago
I think this is classed as lethal technical debt
mgkimsal 10y ago
ACK - I FORGOT THE BEST BIT... (well, maybe not best, but...) No one could install anything locally - everything had to be done on their locked down remote systems (some were Amazon remote desktops). For the accessibility testing, the auditing company used JAWS. The company I was contracting to had one license (or so I was told) so I couldn't have one. We actually tried to install JAWS on an Amazon desktop, but it just crashed the entire virtual desktop, requiring re-imaging. That happened twice, so we gave up. So, the proposed workflow was, I'd make a change, push code, email someone to move that code to a system that an internal tester could look at it. I'd get an email back, then email the internal tester that the code was ready to go look at. The internal tester would go to the screen(s) in question, using JAWS, then "tell me what JAWS said". That would often take several hours or a day. I was then supposed to make changes based on that feedback, then repeat the cycle until things were 'fixed', then we'd ask the auditing company for another test, which they'd schedule for 2 weeks in the future. Then we'd wait. During the first iteration of this part, sr mgrs kept asking me "when will this be done?". I kept trying to explain that we didn't even know what "done" was - the auditing company just had blind folks that would use the system with JAWS enabled and if they felt it was usable, they'd say so, otherwise, they'd report back "hey, this isn't usable", and we'd have to start digging in again.

dx034 10y ago

Templates stored across a database is probably the worst thing I've seen repeatedly across projects. Just because a database can store everything doesn't mean it has to. Some people really seem(ed) to have an allergy to plain files for storage. A plain file with OS level caching will beat most (if not all) databases for static content. But doesn't sound as fancy, so it's probably harder to charge a lot of money for it.

mgkimsal 10y ago
Template in one database table I can live with (pros and cons, multiple front-ends, etc). One template broken up in to 12 tables requiring an 100+ line SQL statement with concat()s and HTML interspersed is insane. Had there been an API or utilities with it to manage it, it might have been manageable, but nope - just "write some queries". Also, just repeated your comment to a friend who said "that's the worst thing you've seen? can i have your job?" :)
- nicostouch 10y ago
  (blown away by all the responses to my original question!!!) Your story here makes me laugh if only because of a very painfully familiar memory. Luckily this wasn't a big production system but rather an internal tool (that I guess clients did also use but it wasn't part of 'production' per se) that was written entirely in perl_cgi filled with cryptic regular expressions written in complete spaghetti code and it would concatenate together entire webpages that had bits of them rendered by including the contents of files strewn all over the file system and of course the logic to concatenate all the html together was strewn across a fistful of files which were in disparate locations. In short I was once asked to make a simple change to some html and after 5 days of reading through perl_cgi and developing a pure hatred for Larry Wall, I decided to do a java re-write that took 3 days. I mean... crikey. Haha.
  - RUG3Y 10y ago
    We have a similar application in PHP. By the time I've traced through all of the included files that are touched by a particular function, I've forgotten what I'm looking for. It's truly a nightmare.
rbirkby 10y ago
Wait until you see a Turing complete DSL programming language stored line-by-line in rows in a database table and executed by pl/SQL using cursors, locking the entire execution to prevent concurrency.
- nicostouch 10y ago
  This is why we can't develop nice things.
Terr_ 10y ago
I'm dealing with one internal project where this happens because there's an artificial IT/build distinction between "emergency" code push and "casual" raw database change. This means lots of business-rule crap gets softcoded into the database or ini files (increasing complexity and bug-risk) just to support a hypothetical future where somebody needs it changed without a full sprint cycle.
bandrami 10y ago
And you aren't kidding about "repeatedly". Personally I associate it with the late 1990s/early oughts and ColdFusion; I think one of the early CF frameworks really encouraged it, and it kind of just stuck from there, particularly in Government web work. But it's probably wider than that...

jkingsbery 10y ago

This has been my experience. Since technical debt is hard to measure, it's more a case of a series of unwise technical decisions leading to a lack of productivity. Due to tight schedules, short-cuts are taken which lead to more unwise technical decisions, and you have a death-spiral.

Bahamut 10y ago
Isn't this precisely technical debt? Unless you want to split hairs and call this a technical massacre...
- Bartweiss 10y ago
  It is, but the question is what "killed" by technical debt means. It's uncommon but not unheard of for code to reach the point of "we can't do that". Mostly, though, the proximate cause of death is a funding shortage or management decision to shutdown. Technical debt is just driving the cost overruns or inefficiencies that kill the project.
  - Bahamut 10y ago
    I don't disagree that the root cause in that situation is not engineering in most situations and that it usually is an indication of a symptom, but attempting to change the meaning of the term itself is not a great approach for communicating that.

pyb 10y ago

I've also got experience of this kind of situation (See my other comment) I think you can definitely call that technical debt.

robodale 10y ago

This. It's not like there is a sign saying "Technical Debt Required to Proceed"...but rather the slow death from a thousand cuts to productivity caused by having to analyze every potential system, process, template, stored procedure, etc, etc...to make any stable(ish) change. Even if things are loosely coupled and not dependent on each other...you still have to go in and make those changes. Telling this to a room full of non-understanding management is a whole different challenge...

oelmekki 10y ago

This account kind of comfort me in what I think of technical debt : most of time, the problem is most likely lack of documentation than anything else. I don't see how a big project could be coded without containing anything specific to the project. And even then, the architecture by itself is unique and deserves documentation.

Bartweiss 10y ago

This is a great example of how technical debt 'kills'. It's not a murder, it's negligence and a slow demise. I went through one of these projects. The tech debt was never as bad as you describe, but it was a small company operating on a short runway. It also taught me an unfortunate lesson about non-technical founders and the dangers of outsourced code. The MVP for the company had been bought off the shelf. It worked fine, but the code was abstruse and utterly resistant to change. As the price (in time and dollars) of change requests grew, they sensibly in-housed development. Unfortunately, their clients had some idea what to expect in terms of features per day and dollar. Requests like "let us use our logo and custom color scheme" turned out to be serious challenges since every color and style decision was clumsily hardcoded, so we took far too long to achieve them. Ultimately, we ended up a contract behind - bringing in business to fund delivering on the previous request. Most startups operate under the gun like that (with either fundraising or contracts), but they start there and labor to escape. We started solvent, and had no clear plan to break out of tech debt - a rebuild would have been too slow, 'working smarter' wasn't viable, and expanding the tech team would have come too late and too costly. So, we died. Not because we couldn't do work, but because we couldn't do it at a competitive speed.

ryandrake 10y ago
Seen this a lot. A lot of companies think they are "product" companies, but due to their unwillingness to push back on customers, they become custom engineering shops, bolting on little one-off mods to their project over and over to appease bad customers (or to appease POTENTIAL customers who haven't even bought the product yet). Stop me when you recognize this one: "Hey your product is great, but we really want something that does [totally different thing]. If you just add that thing, we will pay for all the NRE and you can sell it to others as part of your product! Win win!" Advice to junior developers: If you hear such talk in the hallway, RUN!
- ayuvar 10y ago
  I think every one of my former employers who have failed, did so by doing those 'customs.' The last one even spun off a dedicated team that built (hacked) prototype customs in order to secure sales, then threw away the prototype and, after collecting the commission, told the new customers that it would take several years to get what they just saw in production but in the meantime we can do our existing product with some mods. I imagine the pressure to accept these deals is immense though. Why let an innocuous little feature request hold up such a great deal?

ionwake 10y ago

I think this is classed as lethal technical debt

mgkimsal 10y ago

ACK - I FORGOT THE BEST BIT... (well, maybe not best, but...) No one could install anything locally - everything had to be done on their locked down remote systems (some were Amazon remote desktops). For the accessibility testing, the auditing company used JAWS. The company I was contracting to had one license (or so I was told) so I couldn't have one. We actually tried to install JAWS on an Amazon desktop, but it just crashed the entire virtual desktop, requiring re-imaging. That happened twice, so we gave up. So, the proposed workflow was, I'd make a change, push code, email someone to move that code to a system that an internal tester could look at it. I'd get an email back, then email the internal tester that the code was ready to go look at. The internal tester would go to the screen(s) in question, using JAWS, then "tell me what JAWS said". That would often take several hours or a day. I was then supposed to make changes based on that feedback, then repeat the cycle until things were 'fixed', then we'd ask the auditing company for another test, which they'd schedule for 2 weeks in the future. Then we'd wait. During the first iteration of this part, sr mgrs kept asking me "when will this be done?". I kept trying to explain that we didn't even know what "done" was - the auditing company just had blind folks that would use the system with JAWS enabled and if they felt it was usable, they'd say so, otherwise, they'd report back "hey, this isn't usable", and we'd have to start digging in again.