6 ms·

In my opinion, the idea that you can have well-tested code OR performance is tosh. If you don’t have good testing and do it regularly how can you refactor your

by beny23 6y ago

In my opinion, the idea that you can have well-tested code OR performance is tosh. If you don’t have good testing and do it regularly how can you refactor your application for speed if you’re not going to be sure that you’re going to introduce a regression? And I think the stackoverflow team know it: > Currently, we’re trying to change this. We’re actively trying to write more tests and make our code more testable. It’s an engineering goal we aim to achieve, but the changes needed are significant. This makes it sound like a lofty goal but I think it just sounds more like. “Oh crap, we’re sitting on a mountain of tech debt” No I know you can’t always expect to arrive at a project and find the sunny uplands of TDD and continuous delivery but if you do get to “optimised for speed” code (to me that’s a synonym for tech debt that’s slowing down delivery) all you can do is slowly add what should have been done in the first place (tests).

hinkley 6y ago
Very, very few people know how to make cheap tests. I have one guy who writes fixtures that couples all of his tests. When I need to add functionality, first I have to replace his fixtures with basic stubs, just so I can add another bit of functionality. And we have two people who copy his coding style. Probably to fit in. I think, based on my own experiences and working with people learning to write tests, that 'simple' tests often feel more like the pejorative meaning of the word. They get uncomfortable, like you're debasing yourself to write such boring code. Surely I can write more interesting tests. Yes, you can write more interesting tests if your goal is for nobody to ever touch your code. Mine, mine, mine. The best tests, I know where my new code went wrong simply by reading the test output (not the test). If I refactor I should be able to maintain that coverage even if the order or structure of things gets completely switched around. Complex tests begat complex tests. If your old test is 2-3 lines of code, I just delete it and write a new one that does the same thing.
- beny23 6y ago
  Oh absolutely, keeping things simple should never be restricted to just production code. Testing code should be simple too. The idea that tests inhibits refactoring I find problematic though. Good tests enable refactoring. If the tests are too complex for anyone to change the code, the the code is too complex. Any complex system should be composed of simple components. Now this applies to monoliths as well as microservices. I agree it is easy to fall into the trap of thinking that tests are boring though as devs become more experienced there should be less of that thinking, hopefully.
  - CraigJPerry 6y ago
    >> If the tests are too complex for anyone to change the code, the the code is too complex In enterprise software, the pattern i see is specifically coupling rather than garden variety complexity. There’s a really pervasive idea that unit testing means 1:1 mapping of test class to production code class. To put it another way, there’s this really common but terrible idea that tests should map to your current implementation rather than the behaviour you want. Worse, some check in test cases that should only exist locally during development, they do things like directly test private implementations. A related issue is no one ever deletes tests. Testing done well, is utterly transformational. I took a piece of enterprise software, it wasn’t badly written per se but it was hard to release. 80%+ of releases were rolled back due to issues found in production. For each release, 1 of 7 “experts” were required to review your change to this component (to make it worse it changed fairly frequently). The problem was only 2 of those 7 had a track record of only approving good changes. I’m ashamed to say i was one of the other 5 - i reviewed and approved a couple of changes that had to be rolled back from prod. So, i thought sod this and i grabbed the stingy nettles. The code was not written to be testable but fast forward 16,000 (i had a small team) test cases which execute in around 16 minutes and over the past years that software continues to be released as frequently but has NEVER had a prod rollback since. The 7 experts idea is disbanded, it’s just a normal pull request these days. Lead time on changes is about 1/3 what it was - all the faffing about exploring model output in excel for hours if not days is gone, it’s just all captured as tests. It wasn’t cheap to do this but back of the envelope calculations, it repaid within 3 months just on lack of prod outages.
    - travisjungroth 6y ago
      > A related issue is no one ever deletes tests. I think this is a huge problem. I’m not sure why it exists. “If we have more tests it will be harder to change the code.” is an argument people make sincerely. Then... just delete the tests. If your argument against having these tests is you can imagine a point where they become a burden, just keep in mind that it’s really, really fast to delete code. “Untested” is a state we can get back to at any moment. Maybe the tests will provide a lot of value until then. Maybe if we do a big refactor we’ll delete the old tests that are too low level and make new ones. That’s fine.
- azeirah 6y ago
  I never got to understand testing. It's a thing that every developer is supposed to know and do well somehow. I always end up writing tests for my programs, but with all the stubs and fixtures and weird stuff I get into hairy problems that I don't understand. Tests are supposed to be helpful, but writing good tests is ridiculously hard. Or maybe it's just because I'm a perfectionist, and I should be happy with crappy tests that at least give me a little bit of information about regressions
  - wccrawford 6y ago
    It's one of those things where people who really believe in it are very vocal and will shout over anyone who dares to think there are any problems. Not everyone loves them. But, as I'm a test-lover, I have to say they're pretty amazing if you can get everyone on-board and writing good tests. As for how hard they are... It depends on how the code is structured. If the code is designed to be testable, it's pretty easy to write simple tests. Integrations get a little tougher, but as long as you know what the other side should return, it's pretty easy to stub them out. Testing existing code that was not designed to be testable is a lot harder. It typically will introduce a lot of side effects and won't be properly separated. It can be tested, but IMO just isn't worth the hassle. I've found that you end up spending as much time writing tests as you would have fixing the bugs that arise... And code doesn't get used forever. Modern dev culture insists that someone call for a rewrite eventually, no matter how good the code is, unless you're working in a system that already has decades of legacy code. My advice would be to start a small project of your own (or wait until you get a new project at work that's from scratch) and write tests from the start. It'll teach you a ton about testing and help you get the system off the ground quickly and solidly. I did this for the last big server-side system I wrote and the tests helped tremendously. The bugs that QA found were generally things that I didn't consider or that they designed wrong. The tests helped me get everything working and keep it working while they changed all kinds of things. Sadly, once that project ended up in the hands of other devs, the tests slowly died and I was the only one trying to fix them. I eventually went to another project for a while and came back to find the tests basically didn't work at all, and they died completely at that point. But they did their job at the start, and were totally worth it.
tempodox 6y ago
> This makes it sound like a lofty goal… Sales pitch 101. Sell your weaknesses as strengths.
- KineticLensman 6y ago
  Also true in interviews. When asked about their weaknesses, some candidates who have thought this through might say something like "I spend too much time trying to help struggling colleagues" or similar.
tsimionescu 6y ago
You're missing the development time dimension. The point of the SO post is that, given a limited development time, they chose to focus on performance and sacrifice testability. Many years later now, they are choosing the opposite. This doesn't make their initial decision wrong. The site exists and has thrived, and has achieved all of its performance goals. So the decision was correct. If it has now become too hard to maintain, perhaps the decision to sacrifice some performance to achieve testability will be correct as well (since it seems they haven't finished yet, we can't judge the results).
- beny23 6y ago
  It’s always going to be orders of magnitude more difficult/expensive to retrofit testability. I’d argue that they were successful despite their approach not because of it. I actually have a lot of time for their design choices (monolith, scale up as much as possible, etc, not everything needs to be distributed microservices) but to find doing that by sacrificing testing is surprising. I really don’t think it’s a model to follow.
  - BlargMcLarg 6y ago
    It is when time to market with a feature-complete product is your primary concern, and testability isn't shown to be helping that metric for the developer's specific case. Until customers start caring a lot about development slowing down orders of magnitude down the line and hold their providers accountable, development will continue cutting corners where possible. Unfortunately, testability in particular is one aspect which doesn't have amazing support in regards to how much value it adds.
    - beny23 6y ago
      I think that the “we can go quicker by not testing” is a fallacy. If you do TDD right it will make you quicker, not slower in the short term. Not just the long term...
      - BlargMcLarg 6y ago
atoav 6y ago
This believe got shattered when I learned Rust out of fun. Not only was the resulting code faster, it was is also incredible resistant against stupid changes. If you go in 6 months after and make a change where you forget a tiny little implementation detail, the thing fails, reminds you of that and once you fixed it, just works. Coming from a dynamic programming language I thought Rust was going to be my "if it must be fast, and is allowed to take longer"-language, but instead it is my "if my life would depend on it I would choose this"-language.
ploxiln 6y ago
What does StackOverflow really have left to deliver, though? They made a fairly snappy and robust web application which has very popular functionality, that serves giant boatloads of traffic, with a relatively small team of developers and a relatively small cluster of servers. They didn't have twitter-fail-whale like incidents as far as I recall. It seems they proved that you don't need lots of good unit tests, you can out-perform the whole industry without that. (I say this as someone who doesn't care for Windows or C# - I still recognize superb execution.) At this point they just feel sheepish and apologetic about it.

by beny23 6y ago

hinkley 6y ago

Very, very few people know how to make cheap tests. I have one guy who writes fixtures that couples all of his tests. When I need to add functionality, first I have to replace his fixtures with basic stubs, just so I can add another bit of functionality. And we have two people who copy his coding style. Probably to fit in. I think, based on my own experiences and working with people learning to write tests, that 'simple' tests often feel more like the pejorative meaning of the word. They get uncomfortable, like you're debasing yourself to write such boring code. Surely I can write more interesting tests. Yes, you can write more interesting tests if your goal is for nobody to ever touch your code. Mine, mine, mine. The best tests, I know where my new code went wrong simply by reading the test output (not the test). If I refactor I should be able to maintain that coverage even if the order or structure of things gets completely switched around. Complex tests begat complex tests. If your old test is 2-3 lines of code, I just delete it and write a new one that does the same thing.

beny23 6y ago
Oh absolutely, keeping things simple should never be restricted to just production code. Testing code should be simple too. The idea that tests inhibits refactoring I find problematic though. Good tests enable refactoring. If the tests are too complex for anyone to change the code, the the code is too complex. Any complex system should be composed of simple components. Now this applies to monoliths as well as microservices. I agree it is easy to fall into the trap of thinking that tests are boring though as devs become more experienced there should be less of that thinking, hopefully.
- CraigJPerry 6y ago
  >> If the tests are too complex for anyone to change the code, the the code is too complex In enterprise software, the pattern i see is specifically coupling rather than garden variety complexity. There’s a really pervasive idea that unit testing means 1:1 mapping of test class to production code class. To put it another way, there’s this really common but terrible idea that tests should map to your current implementation rather than the behaviour you want. Worse, some check in test cases that should only exist locally during development, they do things like directly test private implementations. A related issue is no one ever deletes tests. Testing done well, is utterly transformational. I took a piece of enterprise software, it wasn’t badly written per se but it was hard to release. 80%+ of releases were rolled back due to issues found in production. For each release, 1 of 7 “experts” were required to review your change to this component (to make it worse it changed fairly frequently). The problem was only 2 of those 7 had a track record of only approving good changes. I’m ashamed to say i was one of the other 5 - i reviewed and approved a couple of changes that had to be rolled back from prod. So, i thought sod this and i grabbed the stingy nettles. The code was not written to be testable but fast forward 16,000 (i had a small team) test cases which execute in around 16 minutes and over the past years that software continues to be released as frequently but has NEVER had a prod rollback since. The 7 experts idea is disbanded, it’s just a normal pull request these days. Lead time on changes is about 1/3 what it was - all the faffing about exploring model output in excel for hours if not days is gone, it’s just all captured as tests. It wasn’t cheap to do this but back of the envelope calculations, it repaid within 3 months just on lack of prod outages.
  - travisjungroth 6y ago
    > A related issue is no one ever deletes tests. I think this is a huge problem. I’m not sure why it exists. “If we have more tests it will be harder to change the code.” is an argument people make sincerely. Then... just delete the tests. If your argument against having these tests is you can imagine a point where they become a burden, just keep in mind that it’s really, really fast to delete code. “Untested” is a state we can get back to at any moment. Maybe the tests will provide a lot of value until then. Maybe if we do a big refactor we’ll delete the old tests that are too low level and make new ones. That’s fine.
azeirah 6y ago
I never got to understand testing. It's a thing that every developer is supposed to know and do well somehow. I always end up writing tests for my programs, but with all the stubs and fixtures and weird stuff I get into hairy problems that I don't understand. Tests are supposed to be helpful, but writing good tests is ridiculously hard. Or maybe it's just because I'm a perfectionist, and I should be happy with crappy tests that at least give me a little bit of information about regressions
- wccrawford 6y ago
  It's one of those things where people who really believe in it are very vocal and will shout over anyone who dares to think there are any problems. Not everyone loves them. But, as I'm a test-lover, I have to say they're pretty amazing if you can get everyone on-board and writing good tests. As for how hard they are... It depends on how the code is structured. If the code is designed to be testable, it's pretty easy to write simple tests. Integrations get a little tougher, but as long as you know what the other side should return, it's pretty easy to stub them out. Testing existing code that was not designed to be testable is a lot harder. It typically will introduce a lot of side effects and won't be properly separated. It can be tested, but IMO just isn't worth the hassle. I've found that you end up spending as much time writing tests as you would have fixing the bugs that arise... And code doesn't get used forever. Modern dev culture insists that someone call for a rewrite eventually, no matter how good the code is, unless you're working in a system that already has decades of legacy code. My advice would be to start a small project of your own (or wait until you get a new project at work that's from scratch) and write tests from the start. It'll teach you a ton about testing and help you get the system off the ground quickly and solidly. I did this for the last big server-side system I wrote and the tests helped tremendously. The bugs that QA found were generally things that I didn't consider or that they designed wrong. The tests helped me get everything working and keep it working while they changed all kinds of things. Sadly, once that project ended up in the hands of other devs, the tests slowly died and I was the only one trying to fix them. I eventually went to another project for a while and came back to find the tests basically didn't work at all, and they died completely at that point. But they did their job at the start, and were totally worth it.

tempodox 6y ago

> This makes it sound like a lofty goal… Sales pitch 101. Sell your weaknesses as strengths.

KineticLensman 6y ago
Also true in interviews. When asked about their weaknesses, some candidates who have thought this through might say something like "I spend too much time trying to help struggling colleagues" or similar.

tsimionescu 6y ago

You're missing the development time dimension. The point of the SO post is that, given a limited development time, they chose to focus on performance and sacrifice testability. Many years later now, they are choosing the opposite. This doesn't make their initial decision wrong. The site exists and has thrived, and has achieved all of its performance goals. So the decision was correct. If it has now become too hard to maintain, perhaps the decision to sacrifice some performance to achieve testability will be correct as well (since it seems they haven't finished yet, we can't judge the results).

beny23 6y ago
It’s always going to be orders of magnitude more difficult/expensive to retrofit testability. I’d argue that they were successful despite their approach not because of it. I actually have a lot of time for their design choices (monolith, scale up as much as possible, etc, not everything needs to be distributed microservices) but to find doing that by sacrificing testing is surprising. I really don’t think it’s a model to follow.
- BlargMcLarg 6y ago
  It is when time to market with a feature-complete product is your primary concern, and testability isn't shown to be helping that metric for the developer's specific case. Until customers start caring a lot about development slowing down orders of magnitude down the line and hold their providers accountable, development will continue cutting corners where possible. Unfortunately, testability in particular is one aspect which doesn't have amazing support in regards to how much value it adds.
  - beny23 6y ago
    I think that the “we can go quicker by not testing” is a fallacy. If you do TDD right it will make you quicker, not slower in the short term. Not just the long term...
    - BlargMcLarg 6y ago

atoav 6y ago

This believe got shattered when I learned Rust out of fun. Not only was the resulting code faster, it was is also incredible resistant against stupid changes. If you go in 6 months after and make a change where you forget a tiny little implementation detail, the thing fails, reminds you of that and once you fixed it, just works. Coming from a dynamic programming language I thought Rust was going to be my "if it must be fast, and is allowed to take longer"-language, but instead it is my "if my life would depend on it I would choose this"-language.

ploxiln 6y ago

What does StackOverflow really have left to deliver, though? They made a fairly snappy and robust web application which has very popular functionality, that serves giant boatloads of traffic, with a relatively small team of developers and a relatively small cluster of servers. They didn't have twitter-fail-whale like incidents as far as I recall. It seems they proved that you don't need lots of good unit tests, you can out-perform the whole industry without that. (I say this as someone who doesn't care for Windows or C# - I still recognize superb execution.) At this point they just feel sheepish and apologetic about it.