6 ms·

I'm shocked by how many developers check in code that passes the tests but they have not actually tested to make sure it works. Also, I'm tired of people not f

I'm shocked by how many developers check in code that passes the tests but they have not actually tested to make sure it works. Also, I'm tired of people not factoring in (and not honestly reporting) the time and frustration spent yak shaving to keep testing infrastructures working. I believe that it's because folks convince themselves that to acknowledge time lost to the ritual preparation for testing is a kind of weakness because other people aren't having those problems because surely you'd hear more about it. In reality, you can only write tests to cover the cases you anticipate. Correlating test coverage with reliability can be deadly; instead of losing sleep, periodically make sure that you can restore your backups to a production state and maybe even run some drills to see how your team responds when an unanticipated problem arises.

Gigachad 3y ago
Tests are unbelievably useful for updating libraries. Every time I update rails I see a ton of specs fail all over the app highlighting breaking changes not mentioned in the docs. Stuff that is impossible to anticipate otherwise.
- bschwindHN 3y ago
  This is where a nice type system and compilation checks in CI are very useful to have.
  - Gigachad 3y ago
    I agree a type system does wipe out 80% of the tests you need, but I still feel like the 20% is useful. You can just write tests that verify the output looks right without having to run every single line of code to make sure there are no typos or type issues.
    - reissbaker 3y ago
      Yeah, I think that's one of the driving forces away from high code coverage / dogmatic TDD: dynamically typed languages used to be a lot more popular, and you really do need incredible amounts of testing to keep those working reliably in the long run. Now that typed languages are more popular — while some of the tricky logic parts are still worth testing — you can wipe out a lot of the rest because it simply won't compile when, say, you passed a nullable variable to a function that didn't expect it.
  - 1propionyl 3y ago
    Types don't replace tests. Tests don't replace types. For either statement to be true, they would have to be entirely equivalent, at which point it would be a distinction without a difference to complain about. And last I checked no type system automatically generates and minimizes failure cases. And last I checked no test system can be used to formally prove anything about the behavior of code.
    - MrJohz 3y ago
      The two are definitely not equivalent, but they overlap. Types can replace tests - if an edge case exists but I can define it out of existence with my type system, then I no longer need tests for that case. And likewise, tests can replace types - I can use dependent type systems to define invariants in my system but at a certain point it becomes so unwieldy that it's easier to use tests to define those invariants less formally. In the end, they're just two different tools that both serve the overall aim of software correctness. Demonstrating correctness with one tool may be easier but less thorough, or more complicated but more precise, or whatever else. In this context, if you can get the type system to help you, you'll be able to get away with a lot fewer tests because you don't need to test things like "does this function call a method that actually exists?" or "what happens when this function is passed bad data?".
    - JoeyJoJoJr 3y ago
      Why does the statement need to be either true or false, and why would tests versus a type system have to be entirely equivalent? The parent comment mentioned that 80% of tests can be eliminated with a good type system, not that a good type system can entirely replace tests. IMO working with a good type system for a decent length of should absolutely make it apparent that a whole host basic but pervasive errors are practically eliminated, or at the very least heavily mitigated. Silly errors like thinking an array of numbers is just a number, or an Id is a string instead of an int. Every developer I have ever worked with makes these mistakes all the time, especially the developers that say they have never needed a type system or that type systems somehow limit the expressiveness of their code. I would really like to know if there is an efficient way to write tests that can adequately cover these silly errors without over burdening the codebase with heaps of extra test code that causes rigidity and friction for refactoring.
Jach 3y ago
It amazed me when a dev on one of my teams put up a code review, no tests (to be added "during the QE pass later" (a practice that itself has issues but in this case was at least plausible)), but from my quick inspection couldn't possibly work. So I download the patch, built it into my local app, test it.. and yeah, it doesn't work. At that point it was faster for me to actually make it work and just submit the new diff instead of do the back-and-forth dance over code review. My team did pretty good on reporting the time spent (sunk) to BS like infrastructure, flaky tests (especially those involving complex selenium automation which additionally suck for speed because they require the whole app to be running), and out-of-our-team's-hands problems in the wider company. I think it helped that the most senior devs ran into the same problems and felt fine complaining about them as a way to explain why something is taking longer than might otherwise be expected. We also had interns every summer to run into problems that hadn't been fixed and we had just grown accustomed to or found our own mitigations. Sometimes it's just a knowledge issue, being honest about it means someone might be able to help. Even new members with "lead engineer" in their title can learn new things that were taught to the last crop of interns and be more productive. There is a downside risk in that the excuses can be taken advantage of and someone can get away with doing pretty much no code work for days. The excuses can also be taken advantage of to get some various work in that doesn't nicely fit into its own work ticket, though... Some slack in orgs is important and comes in various forms. Property testing helps cover some cases you don't fully anticipate. It's worth integrating a library (https://hypothesis.works/articles/quickcheck-in-every-language/ https://hypothesis.works/articles/quickcheck-in-every-langua... is old but at least points to a bunch for different languages) into the automation infrastructure even if it's not used all that often.
lowbloodsugar 3y ago
They've written tests, but not verified that the tests work. Specifically, not verified that the tests can detect the failure mode they claim to.
- brabel 3y ago
  You must always verify the tests pick up whatever it is they're testing, you do that by intentionally asserting a wrong value, or running tests before the bug was fixed, for example... it's not hard.
kaashif 3y ago
> passes the tests but they have not actually tested Surely this is a problem with the tests then? Whatever manual steps you're referring to with "actually tested" should be automated and part of the test suite. Much easier said than done in some cases obviously, but I've seen some cases where people were manually doing things to "actually test" their code that could obviously and easily be automated.
- makeitdouble 3y ago
  I can relate to the issue: tests work with specific check conditions, and there might other side effects that are not critical to the test but still impacting the user. For instance a page that properly loads, but surprisingly slowly. If you had no requirement of speed and it still loads within the test timeouts, it will pass fine, but a manual check would have raised the issue and perhaps underlying well hidden bugs.
- masklinn 3y ago
  > Surely this is a problem with the tests then? Whatever manual steps you're referring to with "actually tested" should be automated and part of the test suite. As far as I know there is no real way to automate ensuring that the code being submitted is tested. Coverage gates are the closest, and coverage is routinely decried on this here site. The only thing that actually comes close is a very strong type system obviating most tests… but not all, and we’re back to square one. Then there’s the issue of tests testing something useful, yet not over-testing.
misternugget 3y ago
> I'm shocked by how many developers check in code that passes the tests but they have not actually tested to make sure it works. That's actually one of the other topics I wanted to write about yesterday. Chose to write the article above instead. Yes, 100%. I've seen it many times: manually testing reveals more in 1min than hours of previous discussions/reviews/test-writing.
- CuriouslyC 3y ago
  Manual testing runs into the same problems, just like coupled messy code needs to be mocked to high heaven to the point that it's hard to be completely sure you're testing the functionality, messy code is also really hard to manually test. I worked at a place that had a 100% test coverage requirement and the codebase was highly coupled so there were mocks everywhere and tests were ridiculously brittle. Manual testing was nearly a non-starter because it was a microservice architecture with a lot of moving pieces and non-deterministic docker builds failed with surprising regularity (to the point that people shipped development VMs around much to the chagrin of the project "architect"). Getting the stars to align for docker-compose to result in a working test system, then populating it with whatever specific data was needed for that test case so you could actually walk through the flow was a minor miracle.
godelski 3y ago
I'm with you in spirit but I'm not shocked given how many people say that they don't need to write docs because their code is so well written that it is self documented. Which tells me your code is any combination of 1) spaghetti 2) highly unoptimized 3) a holy piece of text. What's the saying? 30 years of improvements in programming has completely undone 30 years of improvements in hardware? Moving fast and breaking things is great to get going, but eventually someone has to come around and clean up all the mess.
- makeitdouble 3y ago
  I never heard people say their code was well written (I wouldn't personally), but many of us have stopped reading docs and comments outside of specifically added ones in surprising locations and vendor provided doc. Many orgs have mandatory docs and comments, yet the devs find nothing specific to write about (they already wrote design documents), and deeply down in their heart don't want to maintain it either. So you get bland and sometimes inane docs and comments, with a bunch of it going stale for a combination of reasons. To the point where you feel you lost your time reading that prose 9 times out of 10, when it wasn't straight misleading/factually wrong. It takes real dedication, discipline and talent to have good documentation, and I wouldn't expect any random dev org to be able to pull it off.
  - godelski 3y ago
    > I never heard people say their code was well written It's shockingly common...[0] TBH there's very few people I've worked with that document, comment, or any form of help. You can see in that thread that much of my frustration in work (including big tech) is that people hand be aliases and call them scripts or hand be hard coded spaghetti and expect me to figure it out without handing over any of the files. But I'm also a researcher, but I have worked on some projects in big tech and this happens more. > It takes real dedication, discipline and talent to have good documentation, and I wouldn't expect any random dev org to be able to pull it off. I don't disagree, but I think writing docs makes a better programmer. You have to think more about your code and methods. It's like the Feynman technique of learning: teach others. I generally write docs for myself because I need to remember. Because writing comments helps me not rush and think a bit more about what I'm doing and where I'm going or could go. As a researcher I'm changing what I'm doing all the time so the latter is quite useful, thinking what I'm going to come back and hack on and making it easier or providing myself hints to come back and rethink. You're right, it takes more time, but often it is faster too. I definitely think it has made me a substantially better programmer and why shouldn't it? A professional athlete reviews their performance, analyzes, criticizes, and rethinks their plan of attack. Are we not doing the same to better ourselves? I hope the pay we get is enough to not just check out and coast but I don't want to be average. I want to be better than I was a month ago. [0] https://news.ycombinator.com/item?id=39476290 https://news.ycombinator.com/item?id=39476290
scotty79 3y ago
> passes the tests but they have not actually tested Maybe that's where the TDD should be mandatory? Before you commit a fix you should commit a test that fails without your fix. CI could check if along the branch created for solving the issue some failing tests were added.

Gigachad 3y ago

Tests are unbelievably useful for updating libraries. Every time I update rails I see a ton of specs fail all over the app highlighting breaking changes not mentioned in the docs. Stuff that is impossible to anticipate otherwise.

bschwindHN 3y ago
This is where a nice type system and compilation checks in CI are very useful to have.
- Gigachad 3y ago
  I agree a type system does wipe out 80% of the tests you need, but I still feel like the 20% is useful. You can just write tests that verify the output looks right without having to run every single line of code to make sure there are no typos or type issues.
  - reissbaker 3y ago
    Yeah, I think that's one of the driving forces away from high code coverage / dogmatic TDD: dynamically typed languages used to be a lot more popular, and you really do need incredible amounts of testing to keep those working reliably in the long run. Now that typed languages are more popular — while some of the tricky logic parts are still worth testing — you can wipe out a lot of the rest because it simply won't compile when, say, you passed a nullable variable to a function that didn't expect it.
- 1propionyl 3y ago
  Types don't replace tests. Tests don't replace types. For either statement to be true, they would have to be entirely equivalent, at which point it would be a distinction without a difference to complain about. And last I checked no type system automatically generates and minimizes failure cases. And last I checked no test system can be used to formally prove anything about the behavior of code.
  - MrJohz 3y ago
    The two are definitely not equivalent, but they overlap. Types can replace tests - if an edge case exists but I can define it out of existence with my type system, then I no longer need tests for that case. And likewise, tests can replace types - I can use dependent type systems to define invariants in my system but at a certain point it becomes so unwieldy that it's easier to use tests to define those invariants less formally. In the end, they're just two different tools that both serve the overall aim of software correctness. Demonstrating correctness with one tool may be easier but less thorough, or more complicated but more precise, or whatever else. In this context, if you can get the type system to help you, you'll be able to get away with a lot fewer tests because you don't need to test things like "does this function call a method that actually exists?" or "what happens when this function is passed bad data?".
  - JoeyJoJoJr 3y ago
    Why does the statement need to be either true or false, and why would tests versus a type system have to be entirely equivalent? The parent comment mentioned that 80% of tests can be eliminated with a good type system, not that a good type system can entirely replace tests. IMO working with a good type system for a decent length of should absolutely make it apparent that a whole host basic but pervasive errors are practically eliminated, or at the very least heavily mitigated. Silly errors like thinking an array of numbers is just a number, or an Id is a string instead of an int. Every developer I have ever worked with makes these mistakes all the time, especially the developers that say they have never needed a type system or that type systems somehow limit the expressiveness of their code. I would really like to know if there is an efficient way to write tests that can adequately cover these silly errors without over burdening the codebase with heaps of extra test code that causes rigidity and friction for refactoring.

Jach 3y ago

It amazed me when a dev on one of my teams put up a code review, no tests (to be added "during the QE pass later" (a practice that itself has issues but in this case was at least plausible)), but from my quick inspection couldn't possibly work. So I download the patch, built it into my local app, test it.. and yeah, it doesn't work. At that point it was faster for me to actually make it work and just submit the new diff instead of do the back-and-forth dance over code review. My team did pretty good on reporting the time spent (sunk) to BS like infrastructure, flaky tests (especially those involving complex selenium automation which additionally suck for speed because they require the whole app to be running), and out-of-our-team's-hands problems in the wider company. I think it helped that the most senior devs ran into the same problems and felt fine complaining about them as a way to explain why something is taking longer than might otherwise be expected. We also had interns every summer to run into problems that hadn't been fixed and we had just grown accustomed to or found our own mitigations. Sometimes it's just a knowledge issue, being honest about it means someone might be able to help. Even new members with "lead engineer" in their title can learn new things that were taught to the last crop of interns and be more productive. There is a downside risk in that the excuses can be taken advantage of and someone can get away with doing pretty much no code work for days. The excuses can also be taken advantage of to get some various work in that doesn't nicely fit into its own work ticket, though... Some slack in orgs is important and comes in various forms. Property testing helps cover some cases you don't fully anticipate. It's worth integrating a library (https://hypothesis.works/articles/quickcheck-in-every-language/ https://hypothesis.works/articles/quickcheck-in-every-langua... is old but at least points to a bunch for different languages) into the automation infrastructure even if it's not used all that often.

lowbloodsugar 3y ago

They've written tests, but not verified that the tests work. Specifically, not verified that the tests can detect the failure mode they claim to.

brabel 3y ago
You must always verify the tests pick up whatever it is they're testing, you do that by intentionally asserting a wrong value, or running tests before the bug was fixed, for example... it's not hard.

kaashif 3y ago

> passes the tests but they have not actually tested Surely this is a problem with the tests then? Whatever manual steps you're referring to with "actually tested" should be automated and part of the test suite. Much easier said than done in some cases obviously, but I've seen some cases where people were manually doing things to "actually test" their code that could obviously and easily be automated.

makeitdouble 3y ago
I can relate to the issue: tests work with specific check conditions, and there might other side effects that are not critical to the test but still impacting the user. For instance a page that properly loads, but surprisingly slowly. If you had no requirement of speed and it still loads within the test timeouts, it will pass fine, but a manual check would have raised the issue and perhaps underlying well hidden bugs.
masklinn 3y ago
> Surely this is a problem with the tests then? Whatever manual steps you're referring to with "actually tested" should be automated and part of the test suite. As far as I know there is no real way to automate ensuring that the code being submitted is tested. Coverage gates are the closest, and coverage is routinely decried on this here site. The only thing that actually comes close is a very strong type system obviating most tests… but not all, and we’re back to square one. Then there’s the issue of tests testing something useful, yet not over-testing.

misternugget 3y ago

> I'm shocked by how many developers check in code that passes the tests but they have not actually tested to make sure it works. That's actually one of the other topics I wanted to write about yesterday. Chose to write the article above instead. Yes, 100%. I've seen it many times: manually testing reveals more in 1min than hours of previous discussions/reviews/test-writing.

CuriouslyC 3y ago
Manual testing runs into the same problems, just like coupled messy code needs to be mocked to high heaven to the point that it's hard to be completely sure you're testing the functionality, messy code is also really hard to manually test. I worked at a place that had a 100% test coverage requirement and the codebase was highly coupled so there were mocks everywhere and tests were ridiculously brittle. Manual testing was nearly a non-starter because it was a microservice architecture with a lot of moving pieces and non-deterministic docker builds failed with surprising regularity (to the point that people shipped development VMs around much to the chagrin of the project "architect"). Getting the stars to align for docker-compose to result in a working test system, then populating it with whatever specific data was needed for that test case so you could actually walk through the flow was a minor miracle.

godelski 3y ago

I'm with you in spirit but I'm not shocked given how many people say that they don't need to write docs because their code is so well written that it is self documented. Which tells me your code is any combination of 1) spaghetti 2) highly unoptimized 3) a holy piece of text. What's the saying? 30 years of improvements in programming has completely undone 30 years of improvements in hardware? Moving fast and breaking things is great to get going, but eventually someone has to come around and clean up all the mess.

makeitdouble 3y ago
I never heard people say their code was well written (I wouldn't personally), but many of us have stopped reading docs and comments outside of specifically added ones in surprising locations and vendor provided doc. Many orgs have mandatory docs and comments, yet the devs find nothing specific to write about (they already wrote design documents), and deeply down in their heart don't want to maintain it either. So you get bland and sometimes inane docs and comments, with a bunch of it going stale for a combination of reasons. To the point where you feel you lost your time reading that prose 9 times out of 10, when it wasn't straight misleading/factually wrong. It takes real dedication, discipline and talent to have good documentation, and I wouldn't expect any random dev org to be able to pull it off.
- godelski 3y ago
  > I never heard people say their code was well written It's shockingly common...[0] TBH there's very few people I've worked with that document, comment, or any form of help. You can see in that thread that much of my frustration in work (including big tech) is that people hand be aliases and call them scripts or hand be hard coded spaghetti and expect me to figure it out without handing over any of the files. But I'm also a researcher, but I have worked on some projects in big tech and this happens more. > It takes real dedication, discipline and talent to have good documentation, and I wouldn't expect any random dev org to be able to pull it off. I don't disagree, but I think writing docs makes a better programmer. You have to think more about your code and methods. It's like the Feynman technique of learning: teach others. I generally write docs for myself because I need to remember. Because writing comments helps me not rush and think a bit more about what I'm doing and where I'm going or could go. As a researcher I'm changing what I'm doing all the time so the latter is quite useful, thinking what I'm going to come back and hack on and making it easier or providing myself hints to come back and rethink. You're right, it takes more time, but often it is faster too. I definitely think it has made me a substantially better programmer and why shouldn't it? A professional athlete reviews their performance, analyzes, criticizes, and rethinks their plan of attack. Are we not doing the same to better ourselves? I hope the pay we get is enough to not just check out and coast but I don't want to be average. I want to be better than I was a month ago. [0] https://news.ycombinator.com/item?id=39476290 https://news.ycombinator.com/item?id=39476290

scotty79 3y ago

> passes the tests but they have not actually tested Maybe that's where the TDD should be mandatory? Before you commit a fix you should commit a test that fails without your fix. CI could check if along the branch created for solving the issue some failing tests were added.