6 ms·

The problem I’ve run into is that when you’re iterating fast, writing code takes double the time when you also have to write the tests. Unit tests are still ea

by wenc 4y ago

The problem I’ve run into is that when you’re iterating fast, writing code takes double the time when you also have to write the tests. Unit tests are still easy to write but most complex software have many parts that combine combinatorially and writing integration tests requires lots of mocking. This investment pays off when the design is stable but when business requirements are not that stable this becomes very expensive. Some tests are actually very hard to write — I once led a project that where the code had both cloud and on-prem API calls (and called Twilio). Some of those environments were outside our control but we still had to make sure they we handled their failure modes. The testing code was very difficult to write and I wished we’d waited until we stabilized the code before attempting to test. There were too many rabbit holes that we naturally got rid of as we iterated and testing was like a ball and chain that made everything super laborious. TDD also represents a kind of first order thinking that assumes that if the individual parts are correct, the whole will likely be correct. It’s not wrong but it’s also very expensive to achieve. Software does have higher order effects. It’s like the old car analogy. American car makers used to believe that if you QC every part and make unit tolerances tight, you’ll get a good car on final assembly (unit tests). This is true if you can get it right all the time but it made US car manufacturing very expensive because it required perfection at every step. Ironically Japanese carmakers eschewed this and allowed loose unit tolerances, but made sure the final build tolerance worked even when the individual unit tolerances had variation. They found this made manufacturing less expensive and still produced very high quality (arguably higher quality since the assembly was rigid where it had to be, and flexible where it had to be). This is craftsman thinking vs strict precision thinking. This method is called “functional build” and Ford was the first US carmaker to adopt it. It eventually came to be adopted by all car makers. https://www.gardnerweb.com/articles/building-better-vehicles-via-functional-build https://www.gardnerweb.com/articles/building-better-vehicles...

pmarreck 4y ago
> writing code takes double the time when you also have to write the tests this time is more than made up for by the usual subsequent loss of debugging, refactoring and maintenance time, in my experience, at least for anything actively being used and updated
- wenc 4y ago
  In theory, I agree. In practice, at least for my projects, the results are mixed.
- tsimionescu 4y ago
  Yes, if you were right about the requirements, even if they weren't well specified. But if it turns out you implemented the wrong thing (either because the requirements simply changed for external reasons, or because you missed some fundamental aspect), then you wouldn't have had to debug, refractor or maintain that initial code, and the initial tests will probably be completely useless even if you end up salvaging some of the initial implementation.
  - twic 4y ago
    No, that's a separate issue, that eschewing TDD doesn't help you with. With TDD, the inner programming loop is: 1. form a belief about requirements 2. write a test to express that belief 3. write code to make that test pass Without TDD, the loop is: 1. form a belief about requirements 2. write code to express that belief 3. futz around with manual testing, REPLs, and after-the-fact testing until you're sufficiently happy that the code actually does express that belief And in my experience, the former loop is faster at producing working code.
    - minimeme 4y ago
      That's my experience also! It's all about faster feedback and confidence the tests provide.
    - ipaddr 4y ago
      It usually works out like.. form a belief about a requirement write a test test fails write code test fails add debug info to code test fails no debug showing call code directly and see debug code change assert test fails rewrite test test succeed output test class data.. false positive checking null equals null rewrite test test passes forget original purpose and stare at green passing tests with pride.
somewhereoutth 4y ago
> TDD also represents a kind of first order thinking that assumes that if the individual parts are correct, the whole will likely be correct. It’s not wrong In fact it is not just wrong, but very wrong, as your auto example shows. Unfortunately engineers are not trained/socialised to think as holistically as perhaps they should be.
- hbn 4y ago
  If individual parts being correct meant the whole thing will be correct, that means if you have a good sturdy propeller and you put it on top of your working car, then you have a working helicopter.
- kazinator 4y ago
  The non-strawman interpretation of TDD is the converse: if the individual parts are not right, then the whole will probably be garbage. It's worth it to apply TDD to the pieces to which TDD is applicable. If not strict TDD than at least "test first" weak TDD. The best candidates for TDD are libraries that implement pure data transformations with minimal integration with anything else. (I suspect that the rabid TDD advocates mostly work in areas where the majority of the code is like that. CRUD work with predictable control and data flows.)
  - wenc 4y ago
    Yes. Agree about TDD being more suited to low dependency software like CRUD apps or self contained libraries. Also sometimes even if the individual parts aren’t right, the whole can still work. Consider a function that handles all cases except for one that is rare, and testing for that case is expensive. The overall system however can be written to provide mitigations upon composing — eg each individual function does a sanity check on its inputs. The individual function itself might be wrong (incomplete) but in the larger system, it is inconsequential. Test effort is not a 1:1. Sometimes the test can be many times as complicated to write and maintain as the function being tested because it has to generate all the corner cases (and has to regenerate them if anything changes upstream). If you’re testing a function in the middle of a very complex data pipeline, you have regenerate all the artifacts upstream. Whereas sometimes an untested function can be written in such a way where it is inherently correct from first principles. An extreme analogy would be the Collatz conjecture. If you start by first writing the test, you’d be writing an almost infinite corpus of tests — on the flip side, writing the Collatz function is extremely simple and correct up to large finite number.
1123581321 4y ago
The automaker analogy is a better fit for the “practice” of not handling errors on the assumption a function can’t return an unexpected value. TDD is actually quite good at manufacturing methods to reasonable tolerance, which the Japanese did require. Higher level tests ensure the functional output is correct and typically don’t have built in any reliance on unit tests.
bostik 4y ago
> Some tests are actually very hard to write — I once led a project that where the code had both cloud and on-prem API calls I believe that this is a fundamental problem of testing in all distributed systems: you are trying to test and validate for emergent behaviour. The other term we have for such systems is: chaotic. Good luck with that. In fact, I have begun to suspect that the way we even think about software testing is backwards. Instead of test scenarios we should be thinking in failure scenarios - and try to subject our software to as much of those as possible. Define the bounding box of the failure universe, and allow computer to generate the testing scenarios within. EXPECT that all software within will eventually fail, but as long as it survives beyond set thresholds, it gets a green light. In a way... we'd need something like a bastard hybrid of fuzzing, chaos testing, soak testing, SRE principles and probabilistic outcomes.
- steve_gh 4y ago
  >I believe that this is a fundamental problem of testing in all distributed systems: you are trying to test and validate for emergent behaviour. The other term we have for such systems is: chaotic. Good luck with that Emergent behaviour is complex, not chaotic. Chaos comes from sensitive dependence on initial conditions. Complexity is associated with non-ergodic statistics (i.e. sampling across time gives different results to sampling across space).
  - bostik 4y ago
    Thank you for the correction. And indeed, "complex" would have been the right term. My bad.
- throwawaymaths 4y ago
  I work in Erlang virtual machine (elixir) and I am regularly writing tests against common distributed systems failures? You don't need property tests (or jeppsen maelstrom - style fuzzing) for your 95% scenarios. Distributed systems are not magically failure prone.
dathanb82 4y ago
I can't remember the last time the speed at which I could physically produce code was the bottleneck in a project. It's all about design and thinking through and documenting the edge cases, and coming up with new edge cases and going back to the design. By the time we know what we're going to write, writing the code isn't the bottleneck, and even if it takes twice as long, that's fine, especially since I generally end up designing a more usable interface as a result of using it (in my tests) as it's being built.
ParetoOptimal 4y ago
> The problem I’ve run into is that when you’re iterating fast, writing code takes double the time when you also have to write the tests. That was the time it took to actually write working code for that feature. The version of "working code" that took 50% as long was just a con to fool people into thinking you'd finished until they move onto other things and a "perfectly acceptable" regression is discovered.
- discreteevent 4y ago
  The reason someone is iterating fast is usually because they are trying to discover the best solution to a problem by building things. Once they have found this then they can write "working code". But they don't want to have to write tests for all the approaches that didn't work and will be thrown away after the prototyping phase.
majikandy 4y ago
> The problem I’ve run into is that when you’re iterating fast, writing code takes double the time when you also have to write the tests. The times I have believed this myself, often turned out to be wrong when the full cost of development was taken into account. And I came back to the code later wishing I had tests around it. So you end up TDDing only the bug fix and exercising that part of the code with the failing test and then the code correction.

by wenc 4y ago

pmarreck 4y ago

> writing code takes double the time when you also have to write the tests this time is more than made up for by the usual subsequent loss of debugging, refactoring and maintenance time, in my experience, at least for anything actively being used and updated

wenc 4y ago
In theory, I agree. In practice, at least for my projects, the results are mixed.
tsimionescu 4y ago
Yes, if you were right about the requirements, even if they weren't well specified. But if it turns out you implemented the wrong thing (either because the requirements simply changed for external reasons, or because you missed some fundamental aspect), then you wouldn't have had to debug, refractor or maintain that initial code, and the initial tests will probably be completely useless even if you end up salvaging some of the initial implementation.
- twic 4y ago
  No, that's a separate issue, that eschewing TDD doesn't help you with. With TDD, the inner programming loop is: 1. form a belief about requirements 2. write a test to express that belief 3. write code to make that test pass Without TDD, the loop is: 1. form a belief about requirements 2. write code to express that belief 3. futz around with manual testing, REPLs, and after-the-fact testing until you're sufficiently happy that the code actually does express that belief And in my experience, the former loop is faster at producing working code.
  - minimeme 4y ago
    That's my experience also! It's all about faster feedback and confidence the tests provide.
  - ipaddr 4y ago
    It usually works out like.. form a belief about a requirement write a test test fails write code test fails add debug info to code test fails no debug showing call code directly and see debug code change assert test fails rewrite test test succeed output test class data.. false positive checking null equals null rewrite test test passes forget original purpose and stare at green passing tests with pride.

somewhereoutth 4y ago

> TDD also represents a kind of first order thinking that assumes that if the individual parts are correct, the whole will likely be correct. It’s not wrong In fact it is not just wrong, but very wrong, as your auto example shows. Unfortunately engineers are not trained/socialised to think as holistically as perhaps they should be.

hbn 4y ago
If individual parts being correct meant the whole thing will be correct, that means if you have a good sturdy propeller and you put it on top of your working car, then you have a working helicopter.
kazinator 4y ago
The non-strawman interpretation of TDD is the converse: if the individual parts are not right, then the whole will probably be garbage. It's worth it to apply TDD to the pieces to which TDD is applicable. If not strict TDD than at least "test first" weak TDD. The best candidates for TDD are libraries that implement pure data transformations with minimal integration with anything else. (I suspect that the rabid TDD advocates mostly work in areas where the majority of the code is like that. CRUD work with predictable control and data flows.)
- wenc 4y ago
  Yes. Agree about TDD being more suited to low dependency software like CRUD apps or self contained libraries. Also sometimes even if the individual parts aren’t right, the whole can still work. Consider a function that handles all cases except for one that is rare, and testing for that case is expensive. The overall system however can be written to provide mitigations upon composing — eg each individual function does a sanity check on its inputs. The individual function itself might be wrong (incomplete) but in the larger system, it is inconsequential. Test effort is not a 1:1. Sometimes the test can be many times as complicated to write and maintain as the function being tested because it has to generate all the corner cases (and has to regenerate them if anything changes upstream). If you’re testing a function in the middle of a very complex data pipeline, you have regenerate all the artifacts upstream. Whereas sometimes an untested function can be written in such a way where it is inherently correct from first principles. An extreme analogy would be the Collatz conjecture. If you start by first writing the test, you’d be writing an almost infinite corpus of tests — on the flip side, writing the Collatz function is extremely simple and correct up to large finite number.

1123581321 4y ago

The automaker analogy is a better fit for the “practice” of not handling errors on the assumption a function can’t return an unexpected value. TDD is actually quite good at manufacturing methods to reasonable tolerance, which the Japanese did require. Higher level tests ensure the functional output is correct and typically don’t have built in any reliance on unit tests.

bostik 4y ago

> Some tests are actually very hard to write — I once led a project that where the code had both cloud and on-prem API calls I believe that this is a fundamental problem of testing in all distributed systems: you are trying to test and validate for emergent behaviour. The other term we have for such systems is: chaotic. Good luck with that. In fact, I have begun to suspect that the way we even think about software testing is backwards. Instead of test scenarios we should be thinking in failure scenarios - and try to subject our software to as much of those as possible. Define the bounding box of the failure universe, and allow computer to generate the testing scenarios within. EXPECT that all software within will eventually fail, but as long as it survives beyond set thresholds, it gets a green light. In a way... we'd need something like a bastard hybrid of fuzzing, chaos testing, soak testing, SRE principles and probabilistic outcomes.

steve_gh 4y ago
>I believe that this is a fundamental problem of testing in all distributed systems: you are trying to test and validate for emergent behaviour. The other term we have for such systems is: chaotic. Good luck with that Emergent behaviour is complex, not chaotic. Chaos comes from sensitive dependence on initial conditions. Complexity is associated with non-ergodic statistics (i.e. sampling across time gives different results to sampling across space).
- bostik 4y ago
  Thank you for the correction. And indeed, "complex" would have been the right term. My bad.
throwawaymaths 4y ago
I work in Erlang virtual machine (elixir) and I am regularly writing tests against common distributed systems failures? You don't need property tests (or jeppsen maelstrom - style fuzzing) for your 95% scenarios. Distributed systems are not magically failure prone.

dathanb82 4y ago

I can't remember the last time the speed at which I could physically produce code was the bottleneck in a project. It's all about design and thinking through and documenting the edge cases, and coming up with new edge cases and going back to the design. By the time we know what we're going to write, writing the code isn't the bottleneck, and even if it takes twice as long, that's fine, especially since I generally end up designing a more usable interface as a result of using it (in my tests) as it's being built.

ParetoOptimal 4y ago

> The problem I’ve run into is that when you’re iterating fast, writing code takes double the time when you also have to write the tests. That was the time it took to actually write working code for that feature. The version of "working code" that took 50% as long was just a con to fool people into thinking you'd finished until they move onto other things and a "perfectly acceptable" regression is discovered.

discreteevent 4y ago
The reason someone is iterating fast is usually because they are trying to discover the best solution to a problem by building things. Once they have found this then they can write "working code". But they don't want to have to write tests for all the approaches that didn't work and will be thrown away after the prototyping phase.

majikandy 4y ago

> The problem I’ve run into is that when you’re iterating fast, writing code takes double the time when you also have to write the tests. The times I have believed this myself, often turned out to be wrong when the full cost of development was taken into account. And I came back to the code later wishing I had tests around it. So you end up TDDing only the bug fix and exercising that part of the code with the failing test and then the code correction.