5 ms·

I think this is highly domain dependent. I currently am working on codebase that has tests for a part of it that are an incredibly useful tool at helping me ref

I think this is highly domain dependent. I currently am working on codebase that has tests for a part of it that are an incredibly useful tool at helping me refactor that particular part. Other parts are so much UI behavior that it is significantly faster to catch bugs by manual testing because the UI/workflow either changes so fast that you don’t write tests for it (knowing they’ll be useless when the workflow is redesigned in the next iteration) or so slow that that particular UI/workflow just doesn’t get touched again so refactors don’t happen to it to introduce more bugs. I have never found tests to be universally necessary or helpful (just like types). They are a tool for a job, not a holy grail. I have also never met a codebase that had good test coverage and yet was free of bugs that aren’t then found with either manual testing or usage. Somewhat hyperbolically and sarcastically: if you are good enough to write perfect tests for your code, just write perfect code. If you aren’t perfect at writing tests, how do you know the tests are complete, bug free, and actually useful? :)

klyrs 2y ago
> the UI/workflow either changes so fast that you don’t write tests for it This is my number one pet peeve in software. Every aspect of every interface is subject to change always; not to mention the bonanza of dickbars and other dark patterns. Interfaces are a minefield of "operator error" but really it's an operational error.
- niemandhier 2y ago
  People are building multimodal transformers, that try to simulate users. No matter how stupid the ai, if it can break your ai code, you have a bug.
256_ 2y ago
> Somewhat hyperbolically and sarcastically: if you are good enough to write perfect tests for your code, just write perfect code. If you aren’t perfect at writing tests, how do you know the tests are complete, bug free, and actually useful? :) Well obviously, you just write tests for the tests. :3 It's called induction.
- MikeDelta 2y ago
  Qui testet ipsos tests?
- derf_ 2y ago
  > Well obviously, you just write tests for the tests. :3 I had a friend whose first job out of school (many years ago) was part of a team at Intel assigned to write tests for their testing tools. When it matters enough economically, it will happen. As you can see from recent news, that is still not enough to guarantee a good result.
- wilgertvelinga 2y ago
  It's actually called mutation testing. And luckily it's almost fully automated.
bubblebeard 2y ago
Types are there to ensure against human error and reduce the amount of code we need to write. Tests exist to guarantee functionality and increase productivity (by ensuring intended functionality remains as we refactor/change our code/UI). There may be cases where some tests are too expensive to write, but I have never come across this myself. For example, in functional tests you would attempt to find a secure way to distinguish elements regardless of future changes to that UI. If your UI changes so much between iterations that this is impossible it sounds like you need to consider the layout a little more before building anything. I’m saying that based on experience, having been involved in several projects where this was a problem. Having said that, I’m myself horrible at writing tests for UI, an area I’m trying to improve myself, it really bothers me :)
- whstl 2y ago
  Tests can be expensive to write if they are an afterthought, and/or the code is not written in way that is easy to test. UI tests can be cheap but they require some experience in knowing how to write a testable UI. One way of achieving that is writing them as early as possible, of course. Which is not always possible :/
  - protomolecule 2y ago
    > the code is not written in way that is easy to test Which isn't devoid of downsides either
    - whstl 2y ago
      That's a good point. Sometimes more ergonomic APIs can be harder to test.
saithound 2y ago
> If you aren’t perfect at writing tests, how do you know the tests are complete, bug free, and actually useful? :) I did like the rest of the post, but this is not hyperbole. It's just a disingenuous argument, and one that looks orthogonal to your point that "tests are a tool for a job". If you aren't perfect at magnetizing iron, and you need a working compass, you better magnetize two needles and use one to test the other. The worse you are at magnetizing iron, the more important it is that you do this if you want to end up with a working compass.
- lelanthran 2y ago
  > If you aren't perfect at magnetizing iron, and you need a working compass, you better magnetize two needles and use one to test the other. The worse you are at magnetizing iron, the more important it is that you do this if you want to end up with a working compass. This is modern testing in a nutshell - it's ineffective but the author of the test can't actually tell that! Using this analogy, if you created 10 magnetised needles using the wrong process and getting the wrong result, then all 10 would agree with each other and your test passes, but your needle is still broken.
  - saithound 2y ago
    I don't think you understand how magnets work. Hint: if you think the way to test whether a needle is magnetized using another possibly magnetized needle is by building both needles into two separate compasses, you're nowhere close.
    - lelanthran 2y ago
      > Hint: if you think the way to test whether a needle is magnetized using another magnetized needle is by building both needles into two separate compasses, you're nowhere close. I thought it was clear from my post that I do not think this. I also think you are missing the point.
      - saithound 2y ago
watwut 2y ago
> if you are good enough to write perfect tests for your code, just write perfect code. I have yet to see anyone claim they write perfect tests. > If you aren’t perfect at writing tests, how do you know the tests are complete, bug free, I never claimed to produce or seen complete tests. I never claimed or seen bug free tests. > and actually useful? :) I know that whenever I fix something or refactor, test fails and I found a bug in code. I know that when we do not have have the same bag again and then again the same bug and again the same bug. I know when testers time is saved and they dont have to test repetitive basic stuff anymore and can focuse on more complicated stuff.
- lelanthran 2y ago
  watwut wrote: > I know that when we do not have have the same bag again and then again the same bug and again the same bug. Well, username checks out :-)
Cthulhu_ 2y ago
IMO if your implementation is that unstable (you mentioned the UI/workflow changes fast) it isn't worth writing a test for it, but also, I don't think it shoud be released to end-users because (and this is making a big assumption, granted), it sounds like the product is trying to figure out what it wants to be. I am a proponent of having the UI/UX design of a feature be done before development gets started on it. In an ideal XP/agile environment the designers and developers work closely together and constantly iterate, but in practice there are so many variables involved in UX design and so many parties that have an opinion, that it'll be a moving target in that case, which makes development work (and automated tests) an exercise in rework.
- rtpg 2y ago
  I think there's a great balance here in these environments: - write tests as part of figuring out the implementation (basically: automate the random clicking you're doing to test things anyways) - Make these tests really loose - Just check them in Being unprecious about test coverage means you just write a handful of "don't blow up" tests for features, that help you get the ball rolling and establish at least a baseline of functionality, without really getting in the way.
- happysadpanda2 2y ago
  Chiming in as an end-user of software: please try to minimize the amount of times I need to re-learn the user interface you put in front of me.
  - happysadpanda2 2y ago
    Aaaaaaand I replied to the wrong comment, mea culpa!
- softfalcon 2y ago
  I agree with you that the ideal is to have UI/UX work resolved before starting dev work. In my experience, this has never happened. I’ve moved around hoping that somewhere, leadership has fixed this problem and nope. It never happens. There are just too many unknowns and never enough time for design to stabilize. It’s always a mad dash to collect whatever user info you can before you slap together a final mock-up of the interface and expected behaviour.
ffsm8 2y ago
I feel like our industry kinda went the wrong way wrt UI frontend tests. It should be much less focused on unit testing and more about flow and state representation, both of which can only be tested visually. And if a flow or state representation changed, that should equate to a simple warning which automatically approves the new representation as the default. So a good testing framework would make it trivial to mock the API responses to create such a flow, and then automatically do a visual regression of the process. Cypress component tests do some of this, but it's still a lackluster developer experience, honestly This is specifically about UI frontend tests. Code that doesn't end up in the DOM are great for unit tests.
bregma 2y ago
Tests are just a way of providing evidence that your software does what it's supposed to. If you're not providing evidence, you're just saying "trust me, I'm a programmer." Think back to grade school math class and your teacher has given you a question about trains with the requirement "show your work." Now, I know a lot of kids will complain about that requirement and just give the answer because "I did it in my head" or something. They fail. Here's the fact: the teacher already knows the trains will meet in Peoria at 12:15. What they're looking for is evidence that you have learned the lesson of how to solve a certain class of problems using the method taught. If you're a professional software developer, it is often necessary to provide evidence of correctness of your code. In a world where dollars or even human lives are on the line, arrogance is rarely a successful defense in a court of law.
- randomdata 2y ago
  Not quite. Tests are just a way to document your software's behaviour, mostly so that future people (including future you) working with the software know what the software is intended to do – to not leave them to guess based on observation of how undefined behaviour plays out. That the documentation is self-validating is merely icing on the cake.
  - bregma 2y ago
    That's a beneficial side effect, not the purpose of having tests. It's like saying the purpose of eating is to fill your senses with wonderful flavours and texture. No, that's a beneficial side effect: the purpose of eating is to prevent death from starvation.
    - randomdata 2y ago
      Indeed. You could write the documentation in Word instead, but when the "herbs and spices" come for free...
scott_w 2y ago
> if you are good enough to write perfect tests for your code, just write perfect code. If you aren’t perfect at writing tests, how do you know the tests are complete, bug free, and actually useful? This sentence makes no sense. Tests are infinitely more straightforward than code. I always go back to my dad's work as a winder before he retired: After repairing a generator, they'd test it can handle the current that it's expected to take by putting it in a platform and... running electricity through it. They'd occasionally melt all the wiring on the generator and have to rewind it. By your logic, since they weren't "good enough" to fix it perfectly, how could they know their test even worked? Should they have just shipped the generator back to the customer without testing it?
- adamc 2y ago
  No, they often aren't, and UI can be complex to test.
  - scott_w 2y ago
    > if you are good enough to write perfect tests for your code, just write perfect code. If you aren’t perfect at writing tests, how do you know the tests are complete, bug free, and actually useful? This was the original quote. Just because one aspect of a system is difficult to test doesn't make this quote true. > No, they often aren't, and UI can be complex to test. It can be but I've seen different ways to get confidence through testing. Just because it's complex doesn't mean we shouldn't do things to improve our confidence that what we're building works. Go back to my example above: I guarantee that a repair has been tested, returned to the customer and failed the test after it was fitted. Why? The fault was elsewhere. But testing gave him confidence the generator wasn't the source of the problem.

klyrs 2y ago

> the UI/workflow either changes so fast that you don’t write tests for it This is my number one pet peeve in software. Every aspect of every interface is subject to change always; not to mention the bonanza of dickbars and other dark patterns. Interfaces are a minefield of "operator error" but really it's an operational error.

niemandhier 2y ago
People are building multimodal transformers, that try to simulate users. No matter how stupid the ai, if it can break your ai code, you have a bug.

256_ 2y ago

> Somewhat hyperbolically and sarcastically: if you are good enough to write perfect tests for your code, just write perfect code. If you aren’t perfect at writing tests, how do you know the tests are complete, bug free, and actually useful? :) Well obviously, you just write tests for the tests. :3 It's called induction.

MikeDelta 2y ago
Qui testet ipsos tests?
derf_ 2y ago
> Well obviously, you just write tests for the tests. :3 I had a friend whose first job out of school (many years ago) was part of a team at Intel assigned to write tests for their testing tools. When it matters enough economically, it will happen. As you can see from recent news, that is still not enough to guarantee a good result.
wilgertvelinga 2y ago
It's actually called mutation testing. And luckily it's almost fully automated.

bubblebeard 2y ago

Types are there to ensure against human error and reduce the amount of code we need to write. Tests exist to guarantee functionality and increase productivity (by ensuring intended functionality remains as we refactor/change our code/UI). There may be cases where some tests are too expensive to write, but I have never come across this myself. For example, in functional tests you would attempt to find a secure way to distinguish elements regardless of future changes to that UI. If your UI changes so much between iterations that this is impossible it sounds like you need to consider the layout a little more before building anything. I’m saying that based on experience, having been involved in several projects where this was a problem. Having said that, I’m myself horrible at writing tests for UI, an area I’m trying to improve myself, it really bothers me :)

whstl 2y ago
Tests can be expensive to write if they are an afterthought, and/or the code is not written in way that is easy to test. UI tests can be cheap but they require some experience in knowing how to write a testable UI. One way of achieving that is writing them as early as possible, of course. Which is not always possible :/
- protomolecule 2y ago
  > the code is not written in way that is easy to test Which isn't devoid of downsides either
  - whstl 2y ago
    That's a good point. Sometimes more ergonomic APIs can be harder to test.

saithound 2y ago

> If you aren’t perfect at writing tests, how do you know the tests are complete, bug free, and actually useful? :) I did like the rest of the post, but this is not hyperbole. It's just a disingenuous argument, and one that looks orthogonal to your point that "tests are a tool for a job". If you aren't perfect at magnetizing iron, and you need a working compass, you better magnetize two needles and use one to test the other. The worse you are at magnetizing iron, the more important it is that you do this if you want to end up with a working compass.

lelanthran 2y ago
> If you aren't perfect at magnetizing iron, and you need a working compass, you better magnetize two needles and use one to test the other. The worse you are at magnetizing iron, the more important it is that you do this if you want to end up with a working compass. This is modern testing in a nutshell - it's ineffective but the author of the test can't actually tell that! Using this analogy, if you created 10 magnetised needles using the wrong process and getting the wrong result, then all 10 would agree with each other and your test passes, but your needle is still broken.
- saithound 2y ago
  I don't think you understand how magnets work. Hint: if you think the way to test whether a needle is magnetized using another possibly magnetized needle is by building both needles into two separate compasses, you're nowhere close.
  - lelanthran 2y ago
    > Hint: if you think the way to test whether a needle is magnetized using another magnetized needle is by building both needles into two separate compasses, you're nowhere close. I thought it was clear from my post that I do not think this. I also think you are missing the point.
    - saithound 2y ago

watwut 2y ago

> if you are good enough to write perfect tests for your code, just write perfect code. I have yet to see anyone claim they write perfect tests. > If you aren’t perfect at writing tests, how do you know the tests are complete, bug free, I never claimed to produce or seen complete tests. I never claimed or seen bug free tests. > and actually useful? :) I know that whenever I fix something or refactor, test fails and I found a bug in code. I know that when we do not have have the same bag again and then again the same bug and again the same bug. I know when testers time is saved and they dont have to test repetitive basic stuff anymore and can focuse on more complicated stuff.

lelanthran 2y ago
watwut wrote: > I know that when we do not have have the same bag again and then again the same bug and again the same bug. Well, username checks out :-)

Cthulhu_ 2y ago

IMO if your implementation is that unstable (you mentioned the UI/workflow changes fast) it isn't worth writing a test for it, but also, I don't think it shoud be released to end-users because (and this is making a big assumption, granted), it sounds like the product is trying to figure out what it wants to be. I am a proponent of having the UI/UX design of a feature be done before development gets started on it. In an ideal XP/agile environment the designers and developers work closely together and constantly iterate, but in practice there are so many variables involved in UX design and so many parties that have an opinion, that it'll be a moving target in that case, which makes development work (and automated tests) an exercise in rework.

rtpg 2y ago
I think there's a great balance here in these environments: - write tests as part of figuring out the implementation (basically: automate the random clicking you're doing to test things anyways) - Make these tests really loose - Just check them in Being unprecious about test coverage means you just write a handful of "don't blow up" tests for features, that help you get the ball rolling and establish at least a baseline of functionality, without really getting in the way.
happysadpanda2 2y ago
Chiming in as an end-user of software: please try to minimize the amount of times I need to re-learn the user interface you put in front of me.
- happysadpanda2 2y ago
  Aaaaaaand I replied to the wrong comment, mea culpa!
softfalcon 2y ago
I agree with you that the ideal is to have UI/UX work resolved before starting dev work. In my experience, this has never happened. I’ve moved around hoping that somewhere, leadership has fixed this problem and nope. It never happens. There are just too many unknowns and never enough time for design to stabilize. It’s always a mad dash to collect whatever user info you can before you slap together a final mock-up of the interface and expected behaviour.

ffsm8 2y ago

I feel like our industry kinda went the wrong way wrt UI frontend tests. It should be much less focused on unit testing and more about flow and state representation, both of which can only be tested visually. And if a flow or state representation changed, that should equate to a simple warning which automatically approves the new representation as the default. So a good testing framework would make it trivial to mock the API responses to create such a flow, and then automatically do a visual regression of the process. Cypress component tests do some of this, but it's still a lackluster developer experience, honestly This is specifically about UI frontend tests. Code that doesn't end up in the DOM are great for unit tests.

bregma 2y ago

Tests are just a way of providing evidence that your software does what it's supposed to. If you're not providing evidence, you're just saying "trust me, I'm a programmer." Think back to grade school math class and your teacher has given you a question about trains with the requirement "show your work." Now, I know a lot of kids will complain about that requirement and just give the answer because "I did it in my head" or something. They fail. Here's the fact: the teacher already knows the trains will meet in Peoria at 12:15. What they're looking for is evidence that you have learned the lesson of how to solve a certain class of problems using the method taught. If you're a professional software developer, it is often necessary to provide evidence of correctness of your code. In a world where dollars or even human lives are on the line, arrogance is rarely a successful defense in a court of law.

randomdata 2y ago
Not quite. Tests are just a way to document your software's behaviour, mostly so that future people (including future you) working with the software know what the software is intended to do – to not leave them to guess based on observation of how undefined behaviour plays out. That the documentation is self-validating is merely icing on the cake.
- bregma 2y ago
  That's a beneficial side effect, not the purpose of having tests. It's like saying the purpose of eating is to fill your senses with wonderful flavours and texture. No, that's a beneficial side effect: the purpose of eating is to prevent death from starvation.
  - randomdata 2y ago
    Indeed. You could write the documentation in Word instead, but when the "herbs and spices" come for free...

scott_w 2y ago

> if you are good enough to write perfect tests for your code, just write perfect code. If you aren’t perfect at writing tests, how do you know the tests are complete, bug free, and actually useful? This sentence makes no sense. Tests are infinitely more straightforward than code. I always go back to my dad's work as a winder before he retired: After repairing a generator, they'd test it can handle the current that it's expected to take by putting it in a platform and... running electricity through it. They'd occasionally melt all the wiring on the generator and have to rewind it. By your logic, since they weren't "good enough" to fix it perfectly, how could they know their test even worked? Should they have just shipped the generator back to the customer without testing it?

adamc 2y ago
No, they often aren't, and UI can be complex to test.
- scott_w 2y ago
  > if you are good enough to write perfect tests for your code, just write perfect code. If you aren’t perfect at writing tests, how do you know the tests are complete, bug free, and actually useful? This was the original quote. Just because one aspect of a system is difficult to test doesn't make this quote true. > No, they often aren't, and UI can be complex to test. It can be but I've seen different ways to get confidence through testing. Just because it's complex doesn't mean we shouldn't do things to improve our confidence that what we're building works. Go back to my example above: I guarantee that a repair has been tested, returned to the customer and failed the test after it was fitted. Why? The fault was elsewhere. But testing gave him confidence the generator wasn't the source of the problem.