4 ms·

> If none of your tests fail then does it really matter? Yes. Absolutely. You don't believe your software is correct because your tests don't fail. You believ

> If none of your tests fail then does it really matter? Yes. Absolutely. You don't believe your software is correct because your tests don't fail. You believe your software is correct because you have a mental model of your code. If your tests are not failing but your software is not behaving correctly, that your mental model of your code is broken.

taneq 2y ago
And/or so are your tests.
- alex_smart 2y ago
  No, they are not. They are a cheap way of verifying that something hasn't gone wrong, not a proof of correctness. Tests failing implies the code is incorrect. Tests not failing does not imply that the code is correct.
  - fc417fc802 2y ago
    > Tests not failing does not imply that the code is correct. I don't think that's what's being suggested. Tests not failing when your code does implies that you are missing test cases. In other words things are underspecified. Haskell is the extreme example of this. If it successfully compiles then it most likely does exactly what you intended but it might be difficult to get it to compile in the first place.
    - alex_smart 2y ago
      >Tests not failing when your code does implies that you are missing test cases. In other words things are underspecified. I am really confused. Have you guys never written any multithreaded code? You can write the most disgusting thread-unsafe code without a single lock and be perfectly green on all your tests. And who in the world can write tests to simulate all possible timing scenarios to test for race conditions? I give multithreading as just the most egregiously obvious example that this "tests can prove correctness" idea is fundamentally broken, but I think it applies more generally. >Haskell is the extreme example of this. If it successfully compiles then it most likely does exactly what you intended but it might be difficult to get it to compile in the first place. Absolutely 100% of the safety of haskell comes from the mental model (functional programming, immutable data structures etc) and none from the test cases (although their community appears to even do testing slightly better than others).
      - fc417fc802 2y ago
        My Haskell comment was regarding specification of the overall system, not tests specifically. It was a reference to the incredible type system. > this "tests can prove correctness" idea You are the only one putting forward such an idea. It's not that I think tests passing proves correctness. It's that I know from experience that I don't fully understand the system. The code that breaks is always a surprise to me because if it wasn't then I would have fixed it before it broke. So if my code breaks and tests don't catch it then I figure that before I fix it I should add a test for it. Of course there are some categories such as multithreading that you generally can't test. So you take extra care with those, and then inevitably end up sinking time into debugging them later regardless.
        alex_smart 2y ago
        >You are the only one putting forward such an idea This was the very first comment you made that started this thread: >> If none of your tests fail then does it really matter?
        fc417fc802 2y ago
        "it doesn't matter" != "code is correct" Sometimes it's just not cost effective to solve every last bit of jank. If you write software that controls safety critical systems then obviously that statement does not apply. But if you write webapps or games or ...
        alex_smart 2y ago
        >"it doesn't matter" != "code is correct" Fair. Despite the lengthy argument, I don't think our stances are all that drastically different. I am not saying that you shouldn't write tests. Just that the mental model comes first and that the tests are informed by it (in an ideal world the test are an automatically executable specification of your mental model). Still I don't understand why you insist that "there exists an automated test for it" has to be the definition of whether something matters. > But if you write webapps or games or ... In which case it might just be fine to YOLO it without tests. People have been delivering valuable software without automated tests for decades. Checkout the code of the linux kernel from circa 2005 and see how many tests there are.
  - taneq 2y ago
    I'm not saying that passing tests proves the code is correct, I'm saying that if you find a problem with the code that your tests don't pick up, then you should add a test for it.
Kiro 2y ago
I highly doubt anyone has a mental model of the all the code they're working with. You very often work with code that you kind of understand but not fully.
- alex_smart 2y ago
  I obviously meant the code that you own and are responsible for.
  - Kiro 2y ago
    Same thing there.
fc417fc802 2y ago
I agree for small systems. But as they get larger you often can't keep track of every last piece simultaneously. It can also become quite involved to figure out why a relatively obscure thing happened in a particular case. Consider something like Unreal Engine for example. It's not realistic to expect to have a full mental image of the entire system in such a case. At least in theory the tests are supposed to cover the observable behavior that matters. So I figure if the tests pass all is well. If I still find something broken then I need to add a test case for it.
- alex_smart 2y ago
  > But as they get larger you often can't keep track of every last piece simultaneously Sure, but then you divide the larger system into smaller components where each team is responsible for one or few of these individual pieces and the chief architect is responsible for making sure of how the pieces are put together. > At least in theory the tests are supposed to cover the observable behavior that matters. So I figure if the tests pass all is well. If I still find something broken then I need to add a test case for it. But you sure as hell hope that the engineer working on implementing your database has a decent mental model for the thread safety of his code and not introduces subtle concurrency bugs because his tests are still green. You also hope that he understands that he needs to call fsync to actually flush to data to disk instead of going yolo (systems never crash and disks never fail ). How are you supposed to cover the user observable behavior in this case? You cut off the power supply to your system/plug off your disk while writing to the database and assert that all the statements that got committed actually persisted? And how many times you repeat that test to really convince you that you are not leaving behind a bug that will only happen in production systems say once every three years? I am only giving database and multithreading as examples because they are the most obvious, but I think the principle applies more generally. Take the simplest piece of code everyone learns to write first thing in uni, quicksort. If you don't have a sufficient mental model for how that algorithm works, what amount of tests will you write to convince yourself that your implementation is correct?
  - fmbb 2y ago
    > then you divide the larger system into smaller components where each team is responsible for one or few of these individual pieces and the chief architect is responsible for making sure of how the pieces are put together And then you have ravioli code in the large. It is not going to make it easier to understand the bigger system, but it will make it harder to debug. https://en.wikipedia.org/wiki/Spaghetti_code#Ravioli_code https://en.wikipedia.org/wiki/Spaghetti_code#Ravioli_code If you can reproduce an error, you can fix it. Do that. If you cannot reproduce it after a day of trying and it doesn’t happen often, don’t fix it.

taneq 2y ago

And/or so are your tests.

alex_smart 2y ago
No, they are not. They are a cheap way of verifying that something hasn't gone wrong, not a proof of correctness. Tests failing implies the code is incorrect. Tests not failing does not imply that the code is correct.
- fc417fc802 2y ago
  > Tests not failing does not imply that the code is correct. I don't think that's what's being suggested. Tests not failing when your code does implies that you are missing test cases. In other words things are underspecified. Haskell is the extreme example of this. If it successfully compiles then it most likely does exactly what you intended but it might be difficult to get it to compile in the first place.
  - alex_smart 2y ago
    >Tests not failing when your code does implies that you are missing test cases. In other words things are underspecified. I am really confused. Have you guys never written any multithreaded code? You can write the most disgusting thread-unsafe code without a single lock and be perfectly green on all your tests. And who in the world can write tests to simulate all possible timing scenarios to test for race conditions? I give multithreading as just the most egregiously obvious example that this "tests can prove correctness" idea is fundamentally broken, but I think it applies more generally. >Haskell is the extreme example of this. If it successfully compiles then it most likely does exactly what you intended but it might be difficult to get it to compile in the first place. Absolutely 100% of the safety of haskell comes from the mental model (functional programming, immutable data structures etc) and none from the test cases (although their community appears to even do testing slightly better than others).
    - fc417fc802 2y ago
      My Haskell comment was regarding specification of the overall system, not tests specifically. It was a reference to the incredible type system. > this "tests can prove correctness" idea You are the only one putting forward such an idea. It's not that I think tests passing proves correctness. It's that I know from experience that I don't fully understand the system. The code that breaks is always a surprise to me because if it wasn't then I would have fixed it before it broke. So if my code breaks and tests don't catch it then I figure that before I fix it I should add a test for it. Of course there are some categories such as multithreading that you generally can't test. So you take extra care with those, and then inevitably end up sinking time into debugging them later regardless.
      - alex_smart 2y ago
        >You are the only one putting forward such an idea This was the very first comment you made that started this thread: >> If none of your tests fail then does it really matter?
        fc417fc802 2y ago
        "it doesn't matter" != "code is correct" Sometimes it's just not cost effective to solve every last bit of jank. If you write software that controls safety critical systems then obviously that statement does not apply. But if you write webapps or games or ...
        alex_smart 2y ago
        >"it doesn't matter" != "code is correct" Fair. Despite the lengthy argument, I don't think our stances are all that drastically different. I am not saying that you shouldn't write tests. Just that the mental model comes first and that the tests are informed by it (in an ideal world the test are an automatically executable specification of your mental model). Still I don't understand why you insist that "there exists an automated test for it" has to be the definition of whether something matters. > But if you write webapps or games or ... In which case it might just be fine to YOLO it without tests. People have been delivering valuable software without automated tests for decades. Checkout the code of the linux kernel from circa 2005 and see how many tests there are.
- taneq 2y ago
  I'm not saying that passing tests proves the code is correct, I'm saying that if you find a problem with the code that your tests don't pick up, then you should add a test for it.

Kiro 2y ago

I highly doubt anyone has a mental model of the all the code they're working with. You very often work with code that you kind of understand but not fully.

alex_smart 2y ago
I obviously meant the code that you own and are responsible for.
- Kiro 2y ago
  Same thing there.

fc417fc802 2y ago

I agree for small systems. But as they get larger you often can't keep track of every last piece simultaneously. It can also become quite involved to figure out why a relatively obscure thing happened in a particular case. Consider something like Unreal Engine for example. It's not realistic to expect to have a full mental image of the entire system in such a case. At least in theory the tests are supposed to cover the observable behavior that matters. So I figure if the tests pass all is well. If I still find something broken then I need to add a test case for it.

alex_smart 2y ago
> But as they get larger you often can't keep track of every last piece simultaneously Sure, but then you divide the larger system into smaller components where each team is responsible for one or few of these individual pieces and the chief architect is responsible for making sure of how the pieces are put together. > At least in theory the tests are supposed to cover the observable behavior that matters. So I figure if the tests pass all is well. If I still find something broken then I need to add a test case for it. But you sure as hell hope that the engineer working on implementing your database has a decent mental model for the thread safety of his code and not introduces subtle concurrency bugs because his tests are still green. You also hope that he understands that he needs to call fsync to actually flush to data to disk instead of going yolo (systems never crash and disks never fail ). How are you supposed to cover the user observable behavior in this case? You cut off the power supply to your system/plug off your disk while writing to the database and assert that all the statements that got committed actually persisted? And how many times you repeat that test to really convince you that you are not leaving behind a bug that will only happen in production systems say once every three years? I am only giving database and multithreading as examples because they are the most obvious, but I think the principle applies more generally. Take the simplest piece of code everyone learns to write first thing in uni, quicksort. If you don't have a sufficient mental model for how that algorithm works, what amount of tests will you write to convince yourself that your implementation is correct?
- fmbb 2y ago
  > then you divide the larger system into smaller components where each team is responsible for one or few of these individual pieces and the chief architect is responsible for making sure of how the pieces are put together And then you have ravioli code in the large. It is not going to make it easier to understand the bigger system, but it will make it harder to debug. https://en.wikipedia.org/wiki/Spaghetti_code#Ravioli_code https://en.wikipedia.org/wiki/Spaghetti_code#Ravioli_code If you can reproduce an error, you can fix it. Do that. If you cannot reproduce it after a day of trying and it doesn’t happen often, don’t fix it.