4 ms·

It’s always going to be orders of magnitude more difficult/expensive to retrofit testability. I’d argue that they were successful despite their approach not bec

by beny23 6y ago

It’s always going to be orders of magnitude more difficult/expensive to retrofit testability. I’d argue that they were successful despite their approach not because of it. I actually have a lot of time for their design choices (monolith, scale up as much as possible, etc, not everything needs to be distributed microservices) but to find doing that by sacrificing testing is surprising. I really don’t think it’s a model to follow.

BlargMcLarg 6y ago
It is when time to market with a feature-complete product is your primary concern, and testability isn't shown to be helping that metric for the developer's specific case. Until customers start caring a lot about development slowing down orders of magnitude down the line and hold their providers accountable, development will continue cutting corners where possible. Unfortunately, testability in particular is one aspect which doesn't have amazing support in regards to how much value it adds.
- beny23 6y ago
  I think that the “we can go quicker by not testing” is a fallacy. If you do TDD right it will make you quicker, not slower in the short term. Not just the long term...
  - BlargMcLarg 6y ago
    Emphasis on "I think". Unfortunately, your thinking hasn't had a whole lot of support, as studies have found both findings in favor of and arguments against TDD, both short term and long term. Furthermore, I would ask you to name the contexts in which you believe TDD to be better in the short term. Things differ when a lot of critical and non-trivial data-modifying code exists compared to an interface which changes requirements by the week, to name two extremes on opposite ends. Things also change when the codebase grows beyond proportions for the devs to actively manage within their heads, or when TDD starts serving as a way to document your codebase for new people coming in. There are plenty of situations one can imagine these conditions not being present, and where a developer with a relatively high degree of correct output would slow themselves down with TDD, the same way some people need to overhead of pulling out their calculator and confirming the answer while someone else calculates it in their head without confirmation. While I won't deny TDD can, or at least should, have benefits in the long term, it is fairly extreme to claim testing is almost always better without the proof to back it up.
    - sgtnoodle 6y ago
      I wouldn't have thought "testing is always good" would be a controversial claim! I could make the claim that dogs shouldn't eat potting soil and most folk would probably agree with me. I don't have any proof that it's actually bad though. My dog gobbled up some potting soil today and he will probably be just fine. I still think he's a doofus, though. I personally write significantly higher quality code, faster, when I spend a few hours thinking through the problem in a document beforehand. One of the things to think about is how to best test it, and most of the time that means a unit test. Knowing that when starting to write the code influences the interfaces and structures that I come up with. Anecdotally, one recent project at work required writing a relatively small, 100% self-contained program with significantly less serious / more silly requirements than the typical type of code I write. I initially didn't plan on writing any unit tests, and about 500 lines of code later I had the program working 99% correctly. There was one bit of overtly complex code that was a bottleneck, though, and it made the overall program 30x too slow for its purpose. The fix was to replace a conceptually simple iterative solver with some brain-melting closed form equations. About 8 hours of iteration and manual testing later, I was pretty sure I had correctly handled all the edge cases. My brain was melted, though, and I couldn't even remember half of the edge cases I had encountered. It took me another 20 minutes to write a unit test that compared the output of the iterative solver with the output of the closed-form solver. Had I done that from the beginning, I probably would have saved 4 hours and not melted my brain. The thing with writing code and testing is that, if you're in the habit of it, unit tests typically take a less-than-linear amount of effort vs. writing the actual code. Once you have tests, they decouple code complexity from development and maintenance, and you can therefore achieve a more sophisticated overall solution than otherwise with the same resources. If you don't develop testing as a skill, though, then those benefits don't materialize because you're inefficient at it. So, while it may be true that a given project or team wouldn't benefit from prioritizing test coverage, it's seemingly due to lack of expertise rather than a lack of obvious benefit. If I'm working on a home improvement project and my toddler wants to help tighten a screw, I wouldn't hand her my impact driver and expect her to successfully use it let alone not hurt herself, despite it being an unambiguously more efficient tool for driving screws than her little green plastic hammer.
      - 6y ago
blowski 6y ago
I think you're making exactly the same point. It's not a model to follow because it was a way of working that fit their requirements at that point in time. Given a different company, project, team, ecosystem, etc, the decision may well be different. Thus, you shouldn't follow their model unless you are identical in every way. What you should follow is the way they arrived at a decision, not the decision itself.
marcosdumay 6y ago
> It’s always going to be orders of magnitude more difficult/expensive to retrofit testability. I can't take this any other way than a huge overstatement. I'm yet to find any such example where retrofitting it is orders of magnitude harder. Worst case it requires understanding the entire system again, and adding instrumentation here or there. That can be almost as hard as writing the entire system again, but usually it's much easier, because all the exploratory work has been done and tests are merely formal. I'd say that correcting the bugs you'll find with those tests is way more work than adding the tests, and maybe they would be easier if they were discovered earlier. But again, the "orders of magnitude more work" is an overstatement.
- kybernetikos 6y ago
  I read that statement as orders of magnitude more difficult/expensive to retrofit testability as opposed to making the system testable at the start, not as opposed to creating the system.
  - beny23 6y ago
    Yes that’s what I meant
- beny23 6y ago
  Speaking from experience, if you have to go back after many years of development, where most of the team that made the original design choices has left and want to make changes without breaking the system and discover that half of the system has no tests then it’s going to be difficult to even make the smallest of changes. When you then decide that you need to retrofit good tests, it always becomes an exercise in trying to reverse engineer the requirements (because half the documentation has been lost or is so hopelessly out of date it may well be lost), then yes, making something well tested is very hard. Having said that, it often worth doing it because how else are you going to have the confidence that you don’t break the system with every little change, but it very much is going to require much more effort than if it had been done in the first place.

by beny23 6y ago

BlargMcLarg 6y ago

It is when time to market with a feature-complete product is your primary concern, and testability isn't shown to be helping that metric for the developer's specific case. Until customers start caring a lot about development slowing down orders of magnitude down the line and hold their providers accountable, development will continue cutting corners where possible. Unfortunately, testability in particular is one aspect which doesn't have amazing support in regards to how much value it adds.

beny23 6y ago
I think that the “we can go quicker by not testing” is a fallacy. If you do TDD right it will make you quicker, not slower in the short term. Not just the long term...
- BlargMcLarg 6y ago
  Emphasis on "I think". Unfortunately, your thinking hasn't had a whole lot of support, as studies have found both findings in favor of and arguments against TDD, both short term and long term. Furthermore, I would ask you to name the contexts in which you believe TDD to be better in the short term. Things differ when a lot of critical and non-trivial data-modifying code exists compared to an interface which changes requirements by the week, to name two extremes on opposite ends. Things also change when the codebase grows beyond proportions for the devs to actively manage within their heads, or when TDD starts serving as a way to document your codebase for new people coming in. There are plenty of situations one can imagine these conditions not being present, and where a developer with a relatively high degree of correct output would slow themselves down with TDD, the same way some people need to overhead of pulling out their calculator and confirming the answer while someone else calculates it in their head without confirmation. While I won't deny TDD can, or at least should, have benefits in the long term, it is fairly extreme to claim testing is almost always better without the proof to back it up.
  - sgtnoodle 6y ago
    I wouldn't have thought "testing is always good" would be a controversial claim! I could make the claim that dogs shouldn't eat potting soil and most folk would probably agree with me. I don't have any proof that it's actually bad though. My dog gobbled up some potting soil today and he will probably be just fine. I still think he's a doofus, though. I personally write significantly higher quality code, faster, when I spend a few hours thinking through the problem in a document beforehand. One of the things to think about is how to best test it, and most of the time that means a unit test. Knowing that when starting to write the code influences the interfaces and structures that I come up with. Anecdotally, one recent project at work required writing a relatively small, 100% self-contained program with significantly less serious / more silly requirements than the typical type of code I write. I initially didn't plan on writing any unit tests, and about 500 lines of code later I had the program working 99% correctly. There was one bit of overtly complex code that was a bottleneck, though, and it made the overall program 30x too slow for its purpose. The fix was to replace a conceptually simple iterative solver with some brain-melting closed form equations. About 8 hours of iteration and manual testing later, I was pretty sure I had correctly handled all the edge cases. My brain was melted, though, and I couldn't even remember half of the edge cases I had encountered. It took me another 20 minutes to write a unit test that compared the output of the iterative solver with the output of the closed-form solver. Had I done that from the beginning, I probably would have saved 4 hours and not melted my brain. The thing with writing code and testing is that, if you're in the habit of it, unit tests typically take a less-than-linear amount of effort vs. writing the actual code. Once you have tests, they decouple code complexity from development and maintenance, and you can therefore achieve a more sophisticated overall solution than otherwise with the same resources. If you don't develop testing as a skill, though, then those benefits don't materialize because you're inefficient at it. So, while it may be true that a given project or team wouldn't benefit from prioritizing test coverage, it's seemingly due to lack of expertise rather than a lack of obvious benefit. If I'm working on a home improvement project and my toddler wants to help tighten a screw, I wouldn't hand her my impact driver and expect her to successfully use it let alone not hurt herself, despite it being an unambiguously more efficient tool for driving screws than her little green plastic hammer.
    - 6y ago

blowski 6y ago

I think you're making exactly the same point. It's not a model to follow because it was a way of working that fit their requirements at that point in time. Given a different company, project, team, ecosystem, etc, the decision may well be different. Thus, you shouldn't follow their model unless you are identical in every way. What you should follow is the way they arrived at a decision, not the decision itself.

marcosdumay 6y ago

> It’s always going to be orders of magnitude more difficult/expensive to retrofit testability. I can't take this any other way than a huge overstatement. I'm yet to find any such example where retrofitting it is orders of magnitude harder. Worst case it requires understanding the entire system again, and adding instrumentation here or there. That can be almost as hard as writing the entire system again, but usually it's much easier, because all the exploratory work has been done and tests are merely formal. I'd say that correcting the bugs you'll find with those tests is way more work than adding the tests, and maybe they would be easier if they were discovered earlier. But again, the "orders of magnitude more work" is an overstatement.

kybernetikos 6y ago
I read that statement as orders of magnitude more difficult/expensive to retrofit testability as opposed to making the system testable at the start, not as opposed to creating the system.
- beny23 6y ago
  Yes that’s what I meant
beny23 6y ago
Speaking from experience, if you have to go back after many years of development, where most of the team that made the original design choices has left and want to make changes without breaking the system and discover that half of the system has no tests then it’s going to be difficult to even make the smallest of changes. When you then decide that you need to retrofit good tests, it always becomes an exercise in trying to reverse engineer the requirements (because half the documentation has been lost or is so hopelessly out of date it may well be lost), then yes, making something well tested is very hard. Having said that, it often worth doing it because how else are you going to have the confidence that you don’t break the system with every little change, but it very much is going to require much more effort than if it had been done in the first place.