3 ms·

Hmm, that's not what I was taught about Agile vs Waterfall. It was about the number of design cycles – (discussion with client / design / programming / bugfix

Hmm, that's not what I was taught about Agile vs Waterfall. It was about the number of design cycles – (discussion with client / design / programming / bugfixing / client feedback) – before the final version was out : only 1 for the most rigid waterfall, and usually weekly cycles for the most agile. So it was not so much about any technical debt, but rather about the lack of understanding of what the client wanted (often the ignorance of the client himself of what he actually wanted). Incidentally, the most radical agile cycle would involve throwing out all the code on each cycle, which would of course throw away all the technical debt in the process (and hopefully prevent from making the same technical debt mistakes twice).

Jtsummers 6y ago
Your understanding is correct. Waterfall (may it die a fiery death) is at an extreme that assumes it's possible to know everything upfront. What most people fail to understand (who like it) is that Waterfall fails to scale. Agile is (almost) at the other extreme. Waterfall cannot scale beyond either well-understood domains (a web shop that uses RoR could probably apply Waterfall to a new project at this point, with over a decade of experience for each member of the team) or smaller systems (I'd say no bigger than around 100k SLOC of C code, beyond that your system usually becomes too complex) or short time lines (less than 3-6 months, which implies smaller systems). The assumption of complete and proper understanding will bite you on either a larger project or novel domain (novel to you or the world) or a longer timeline. Agile (I'll take Scrum's 1-2 week cycle + extreme programming) aims to shorten the feedback loop so you know that not only have you built what you thought you built (verification) but also what your customer wanted you to build (validation) on a regular basis. If you're wrong with Waterfall, you've wasted years. If you're wrong with Agile, you've wasted weeks. And boy can you be wrong with Waterfall. I joined at the tail end of a 5-year Waterfall project, and they had fucked it up. Over 300 people had worked on it, and it was the wrong damn thing (had half the features needed by the time it shipped, and those half barely worked, and if they worked they were too slow to actually be useful). 1500 years, let's say an average of 75 years per lifetime. That was 20 human lifetimes wasted. And let's not talk about the billions of dollars that went into it. (300 is probably a low estimate, once you get into the subs of the subs of the subcontractors it's hard to get a good tally.)
jasonwatkinspdx 6y ago
Ward talks about this occasionally, but generally there's a frustration among the original agile manifesto authors about how the principles they were after became distorted by the consultant industry that spawned behind the terms. The financial metaphor above is definitely what Ward meant when coining the phrase technical debt, and it dovetails into agile in the sense one should be making conscious decisions about balancing upfront investment vs uncertain goals. Technical debt can be an opportunity in the sense of doing the learning on the cheap, but you can't run up that debt indefinitely.
crdrost 6y ago
> So it was not so much about any technical debt, but rather about the lack of understanding of what the client wanted (often the ignorance of the client himself of what he actually wanted). I cannot emphasize enough how much I want you to read, watch, listen to Ward Cunningham in his own words. :) If you know what technical debt really is, then this sentence kind of sticks out as a sore thumb, as it is literally saying "So it was not so much about any technical debt, but rather about the technical debt." Technical debt viewed this way is not just “hurdles in my code that I don’t like to jump over.” It is a subset of those hurdles and kludges and nicks and grumbles. What subset? Precisely the subset that corresponds to our lack of up-front understanding of what we were building when we were at first building it. Put another way, every single problem we can solve well has at least one (maybe many) perspective from which that solution seems “relatively straightforward.” If you can specify that perspective in a way that a computer can understand it, you can think of that as “creating an algebra” for that problem and similar problems, or perhaps a “domain-specific language.” Technical debt is precisely the mismatch between the language you are programming in, and the domain you are programming for. Syntactically, this "language" might be some programming language like Java. But semantically, you get to define your own classes and methods and create your own little world inside that language, where things can be combined and chained together, and the richness and algebraic predictability of that little world informs the spaghettification and robustness of your code to implement the algorithm within that world. Viewed this way, a lot of programming today spends absolutely no investment into that "little world" and just consists of writing FORTRAN. Just pages and pages and pages of FORTRAN. I mean, you're writing it in some other language, maybe Python, maybe Java: but you do not have any sort of little algebraic world that you phrase your problem in, you just say "fetch this input, fetch that input, begin subroutine with vaguely suggestive name to label the following steps, compare the two inputs, fetch other data as needed, perform these side-effects, modify these variables, return to the beginning of some loop." Very verbose individual commands being operated on individual data structures. So I tell you "make sure that if I press this button a hundred times you don't start the job a hundred times, only one job per project name should be running at any given time, but if I press the button while a job is ongoing you should still probably trust that I made a mistake when I first pushed the button and now I want to abort if possible and retry with the latest version" and because you don't have a rich vocabulary for that you're stuck programming something in a "redis assembly language" or whatever, redis_job_flag = "job_id_for_" + project.name; current_job = redis.get(redis_job_flag); if (current_job) { redis_complete_flag = "job_is_complete_" + current_job; add_followup = redis.setnx(redis_complete_flag, "afterwards=run_again", 60*minutes); if (!add_followup) { followup = redis.get(redis_complete_flag); if (followup == "afterwards=run_again") { // someone else got there first return; } else { // race condition, it completed before we could // set the followup attempt, so we need to retry // this method to restart it return recurse(); } } else { // ok we can trust that they will redo what they just did return; } } // if we're here then we did not see any prior jobs. my_id = generate_random_id() try_to_setnx = redis.setnx(redis_job_flag, my_id, 60*minutes); if (!try_to_setnx) { // race condition, someone else got started before me. // better still set a follow-up just in case. return recurse(); } // hooray! I am the unique one running the job. redis_complete_flag = "job_is_complete_" + my_id; looping = true while (looping) { doTheActualThing(); really_done = redis.setnx(redis_complete_flag, "complete", 60*minutes); if (really_done) { redis.unset(redis_job_flag); looping = false; } else { // let someone else tell us to rerun again redis.unset(redis_complete_flag); // bug we ran into where someone made us rerun jobs // for an hour, we reset the timeout for this here redis.set(redis_job_flag, my_id, 60*minutes); } } and then this requirement is restated for other buttons and that gets copy/pasted everywhere and other mutations get inserted in various places. And I think what Ward would tell you is not "never write all those lines of code" but that this eventually needs a refactor so that it matches the business language, repeatableDebounce("project_jobs", project.name, doTheActualThing); // repeatableDebounce defined elsewhere, // it is part of our "little world" Basically, Ward wants you to start doing "aspect-oriented programming" because it turns out that is a felicitous perspective from which to view these requirements about needing to cache whether jobs are in-transit and if they are then to retry them and whatever else. Technical debt is the noise in the above implementation being pasted and tweaked across the entire codebase because we didn't have this perspective when we were starting.

Jtsummers 6y ago

Your understanding is correct. Waterfall (may it die a fiery death) is at an extreme that assumes it's possible to know everything upfront. What most people fail to understand (who like it) is that Waterfall fails to scale. Agile is (almost) at the other extreme. Waterfall cannot scale beyond either well-understood domains (a web shop that uses RoR could probably apply Waterfall to a new project at this point, with over a decade of experience for each member of the team) or smaller systems (I'd say no bigger than around 100k SLOC of C code, beyond that your system usually becomes too complex) or short time lines (less than 3-6 months, which implies smaller systems). The assumption of complete and proper understanding will bite you on either a larger project or novel domain (novel to you or the world) or a longer timeline. Agile (I'll take Scrum's 1-2 week cycle + extreme programming) aims to shorten the feedback loop so you know that not only have you built what you thought you built (verification) but also what your customer wanted you to build (validation) on a regular basis. If you're wrong with Waterfall, you've wasted years. If you're wrong with Agile, you've wasted weeks. And boy can you be wrong with Waterfall. I joined at the tail end of a 5-year Waterfall project, and they had fucked it up. Over 300 people had worked on it, and it was the wrong damn thing (had half the features needed by the time it shipped, and those half barely worked, and if they worked they were too slow to actually be useful). 1500 years, let's say an average of 75 years per lifetime. That was 20 human lifetimes wasted. And let's not talk about the billions of dollars that went into it. (300 is probably a low estimate, once you get into the subs of the subs of the subcontractors it's hard to get a good tally.)

jasonwatkinspdx 6y ago

Ward talks about this occasionally, but generally there's a frustration among the original agile manifesto authors about how the principles they were after became distorted by the consultant industry that spawned behind the terms. The financial metaphor above is definitely what Ward meant when coining the phrase technical debt, and it dovetails into agile in the sense one should be making conscious decisions about balancing upfront investment vs uncertain goals. Technical debt can be an opportunity in the sense of doing the learning on the cheap, but you can't run up that debt indefinitely.

crdrost 6y ago

> So it was not so much about any technical debt, but rather about the lack of understanding of what the client wanted (often the ignorance of the client himself of what he actually wanted). I cannot emphasize enough how much I want you to read, watch, listen to Ward Cunningham in his own words. :) If you know what technical debt really is, then this sentence kind of sticks out as a sore thumb, as it is literally saying "So it was not so much about any technical debt, but rather about the technical debt." Technical debt viewed this way is not just “hurdles in my code that I don’t like to jump over.” It is a subset of those hurdles and kludges and nicks and grumbles. What subset? Precisely the subset that corresponds to our lack of up-front understanding of what we were building when we were at first building it. Put another way, every single problem we can solve well has at least one (maybe many) perspective from which that solution seems “relatively straightforward.” If you can specify that perspective in a way that a computer can understand it, you can think of that as “creating an algebra” for that problem and similar problems, or perhaps a “domain-specific language.” Technical debt is precisely the mismatch between the language you are programming in, and the domain you are programming for. Syntactically, this "language" might be some programming language like Java. But semantically, you get to define your own classes and methods and create your own little world inside that language, where things can be combined and chained together, and the richness and algebraic predictability of that little world informs the spaghettification and robustness of your code to implement the algorithm within that world. Viewed this way, a lot of programming today spends absolutely no investment into that "little world" and just consists of writing FORTRAN. Just pages and pages and pages of FORTRAN. I mean, you're writing it in some other language, maybe Python, maybe Java: but you do not have any sort of little algebraic world that you phrase your problem in, you just say "fetch this input, fetch that input, begin subroutine with vaguely suggestive name to label the following steps, compare the two inputs, fetch other data as needed, perform these side-effects, modify these variables, return to the beginning of some loop." Very verbose individual commands being operated on individual data structures. So I tell you "make sure that if I press this button a hundred times you don't start the job a hundred times, only one job per project name should be running at any given time, but if I press the button while a job is ongoing you should still probably trust that I made a mistake when I first pushed the button and now I want to abort if possible and retry with the latest version" and because you don't have a rich vocabulary for that you're stuck programming something in a "redis assembly language" or whatever, redis_job_flag = "job_id_for_" + project.name; current_job = redis.get(redis_job_flag); if (current_job) { redis_complete_flag = "job_is_complete_" + current_job; add_followup = redis.setnx(redis_complete_flag, "afterwards=run_again", 60*minutes); if (!add_followup) { followup = redis.get(redis_complete_flag); if (followup == "afterwards=run_again") { // someone else got there first return; } else { // race condition, it completed before we could // set the followup attempt, so we need to retry // this method to restart it return recurse(); } } else { // ok we can trust that they will redo what they just did return; } } // if we're here then we did not see any prior jobs. my_id = generate_random_id() try_to_setnx = redis.setnx(redis_job_flag, my_id, 60*minutes); if (!try_to_setnx) { // race condition, someone else got started before me. // better still set a follow-up just in case. return recurse(); } // hooray! I am the unique one running the job. redis_complete_flag = "job_is_complete_" + my_id; looping = true while (looping) { doTheActualThing(); really_done = redis.setnx(redis_complete_flag, "complete", 60*minutes); if (really_done) { redis.unset(redis_job_flag); looping = false; } else { // let someone else tell us to rerun again redis.unset(redis_complete_flag); // bug we ran into where someone made us rerun jobs // for an hour, we reset the timeout for this here redis.set(redis_job_flag, my_id, 60*minutes); } } and then this requirement is restated for other buttons and that gets copy/pasted everywhere and other mutations get inserted in various places. And I think what Ward would tell you is not "never write all those lines of code" but that this eventually needs a refactor so that it matches the business language, repeatableDebounce("project_jobs", project.name, doTheActualThing); // repeatableDebounce defined elsewhere, // it is part of our "little world" Basically, Ward wants you to start doing "aspect-oriented programming" because it turns out that is a felicitous perspective from which to view these requirements about needing to cache whether jobs are in-transit and if they are then to retry them and whatever else. Technical debt is the noise in the above implementation being pasted and tweaked across the entire codebase because we didn't have this perspective when we were starting.