3 ms·

No one who claims to automate testing ever seems to have a sophisticated understanding of testing. I suspect the urge people have for automating anything comes

No one who claims to automate testing ever seems to have a sophisticated understanding of testing. I suspect the urge people have for automating anything comes not from the love and skill they possess for that activity, but rather for the imagined fruits of it. Software testing is easy to fake, which means it’s particularly easy to claim automation accomplished. Therefore, all these people thinking they are going to automate what I do when I test? No. You have no idea what I do. But what you can do it convince ignorant managers to buy fake software testing. That’s the real danger. Imagine if a company came around claiming to have revolutionized blood testing with radical tech, labeling its competitors slow and obsolete. The sober and responsible players in that industry may experience real damage before people figure out that Theranos is a fraud.

sillysaurusx 3y ago
Someone could’ve made the same arguments about automating horse-drawn carriages. Even the safety arguments would fit: going faster would be much more dangerous. I think it’s a good idea to keep an open mind and see what happens. You might be right, but if you’re wrong, you’d be better served to be ready for it. Which seems more likely: that it’s impossible to automate testing with AI, or that it might be possible?
- 99_00 3y ago
  >Someone could’ve made the same arguments about automating horse-drawn carriages. They can't make the same argument because the experience is first hand and can't be faked.
shepherdjerred 3y ago
What's the gap between a Co-pilot written test and a human-written test? Are tests written by co-pilot better than no tests? Developers generally hate writing tests, so I suspect more and more unit tests will be generated. I've worked with someone who does this and it seems insane to me, although I never got to see the generated tests they were referring to, so I can't give specifics.
- js8 3y ago
  > What's the gap between a Co-pilot written test and a human-written test? IMHO the gap is that human-written test understands the (unwritten) specification (which might be as simple as "program doesn't crash"), and tests the program against it. I am of the (pretty radical, it seems) opinion that the (especially unit) tests that just take the code as the specification, instead of having specification coming from elsewhere to test against, are useless. So you cannot build a useful test just out of the code, because you always need to test an implementation against a specification, or, two implementations against each other.
  - bbbobbb 3y ago
    I agree with you about ideal approach to tests and spec but sometimes the "specification coming from elsewhere" is pretty vague, incomplete or simply lost in time. In that case, if I have code that satisfies that specification as demonstrated by say acceptance testing or years of use, I will happily opt for test that take the code as specification. I am not using those tests to prove the code "correct" but to prevent regressions during refactor or new feature development later on. Worse case scenario they should keep the program as broken as it is now or has been for years.
    - js8 3y ago
      > I am not using those tests to prove the code "correct" but to prevent regressions during refactor or new feature development later on. Then you don't really need to have the tests written, what you need to do is to compare the output and behavior of the two code bases before and after refactoring. I mean why use a low fidelity approximation of the original (i.e. test that only works in some cases), if you already have the original at hand? (After all, how can you predict ahead of time, which features of the original version will need to be tested as preserved in the new version? Isn't writing regression tests in advance a form of premature optimization?) From the view point of testing, you're comparing two implementations. So again there needs to be some understanding which one is the correct specification - the original one or the new one? It would be more useful if the industry actually treated the two cases - i.e. testing against specification and regression testing of the refactored code - as completely separate, instead of trying to push unit tests for everything. Because in the latter case, ideally you can prove that the refactored code is doing identical thing, which is stronger than testing.
deleted 3y ago
[deleted]

sillysaurusx 3y ago

Someone could’ve made the same arguments about automating horse-drawn carriages. Even the safety arguments would fit: going faster would be much more dangerous. I think it’s a good idea to keep an open mind and see what happens. You might be right, but if you’re wrong, you’d be better served to be ready for it. Which seems more likely: that it’s impossible to automate testing with AI, or that it might be possible?

99_00 3y ago
>Someone could’ve made the same arguments about automating horse-drawn carriages. They can't make the same argument because the experience is first hand and can't be faked.

shepherdjerred 3y ago

What's the gap between a Co-pilot written test and a human-written test? Are tests written by co-pilot better than no tests? Developers generally hate writing tests, so I suspect more and more unit tests will be generated. I've worked with someone who does this and it seems insane to me, although I never got to see the generated tests they were referring to, so I can't give specifics.

js8 3y ago
> What's the gap between a Co-pilot written test and a human-written test? IMHO the gap is that human-written test understands the (unwritten) specification (which might be as simple as "program doesn't crash"), and tests the program against it. I am of the (pretty radical, it seems) opinion that the (especially unit) tests that just take the code as the specification, instead of having specification coming from elsewhere to test against, are useless. So you cannot build a useful test just out of the code, because you always need to test an implementation against a specification, or, two implementations against each other.
- bbbobbb 3y ago
  I agree with you about ideal approach to tests and spec but sometimes the "specification coming from elsewhere" is pretty vague, incomplete or simply lost in time. In that case, if I have code that satisfies that specification as demonstrated by say acceptance testing or years of use, I will happily opt for test that take the code as specification. I am not using those tests to prove the code "correct" but to prevent regressions during refactor or new feature development later on. Worse case scenario they should keep the program as broken as it is now or has been for years.
  - js8 3y ago
    > I am not using those tests to prove the code "correct" but to prevent regressions during refactor or new feature development later on. Then you don't really need to have the tests written, what you need to do is to compare the output and behavior of the two code bases before and after refactoring. I mean why use a low fidelity approximation of the original (i.e. test that only works in some cases), if you already have the original at hand? (After all, how can you predict ahead of time, which features of the original version will need to be tested as preserved in the new version? Isn't writing regression tests in advance a form of premature optimization?) From the view point of testing, you're comparing two implementations. So again there needs to be some understanding which one is the correct specification - the original one or the new one? It would be more useful if the industry actually treated the two cases - i.e. testing against specification and regression testing of the refactored code - as completely separate, instead of trying to push unit tests for everything. Because in the latter case, ideally you can prove that the refactored code is doing identical thing, which is stronger than testing.

deleted 3y ago

[deleted]