3 ms·

That’s why I make the test fail before writing the code. If the code is already written, then I break it in the minimal way to test the test, and then fix it.

by chubot 7y ago

sojournerc 7y ago
IME things never fail in the way you expect them to... You can build a fortification where you think you are weak only to find the enemy is already in the castle.
lonelappde 7y ago
That's a test for your test, so why only run it once transiently instead of running every time? "Mutant" testing helps with this. It's basically fuzzing your test code to make sure that every line is meaningful.
- chubot 7y ago
  I can see how that would be useful, but I also think it's a matter of priorities. I'm basically saying I rarely have bugs in my tests because I verify them first. In fact I can't think of a single bug in my tests over the last 4 years (or even 10 years), but I can think of dozens of bugs in my code. For example here are some pretty exhaustive tests I've written for shell, which have exposed dozens of bugs in bash and other shells (and my own shell Oil): https://www.oilshell.org/release/0.7.pre11/test/spec.wwz/osh.html https://www.oilshell.org/release/0.7.pre11/test/spec.wwz/osh... I would rather spend time using the tests to improve the code than improving the tests themselves. But I don't doubt that technique could be useful for some projects (likely very old and mature ones)
  - scaryclam 7y ago
    ITYM "I rarely have bugs in my tests that I'm aware of". The amount of tests I've seen that look like they are working properly, have been "verified" and are buggy is huge. Usually we find they were buggy because someone moves some other tests around or changes functionality that should have broken the tests, but didn't. Please, don't ever assume that your tests are beyond reproach just because you verified them. Tests are software as well and are as prone to bugs as anything.
- pvorb 7y ago
  And how do you do this in practice? I am struggling to think of a good way to keep the production code that fails the test and the production code that doesn't fail the test together. I might have my test check out an old version of the production code, compile it and test against that. But that is hard to get right.
serpix 7y ago
A green test does not equal bug free code. There may be a misimplementation / misunderstanding of the spec or your code passes beautifully under right conditions, with just this data setup.

That’s why I make the test fail before writing the code. If the code is already written, then I break it in the minimal way to test the test, and then fix it.

by chubot 7y ago

That’s why I make the test fail before writing the code. If the code is already written, then I break it in the minimal way to test the test, and then fix it.

sojournerc 7y ago

IME things never fail in the way you expect them to... You can build a fortification where you think you are weak only to find the enemy is already in the castle.

lonelappde 7y ago

That's a test for your test, so why only run it once transiently instead of running every time? "Mutant" testing helps with this. It's basically fuzzing your test code to make sure that every line is meaningful.

chubot 7y ago
I can see how that would be useful, but I also think it's a matter of priorities. I'm basically saying I rarely have bugs in my tests because I verify them first. In fact I can't think of a single bug in my tests over the last 4 years (or even 10 years), but I can think of dozens of bugs in my code. For example here are some pretty exhaustive tests I've written for shell, which have exposed dozens of bugs in bash and other shells (and my own shell Oil): https://www.oilshell.org/release/0.7.pre11/test/spec.wwz/osh.html https://www.oilshell.org/release/0.7.pre11/test/spec.wwz/osh... I would rather spend time using the tests to improve the code than improving the tests themselves. But I don't doubt that technique could be useful for some projects (likely very old and mature ones)
- scaryclam 7y ago
  ITYM "I rarely have bugs in my tests that I'm aware of". The amount of tests I've seen that look like they are working properly, have been "verified" and are buggy is huge. Usually we find they were buggy because someone moves some other tests around or changes functionality that should have broken the tests, but didn't. Please, don't ever assume that your tests are beyond reproach just because you verified them. Tests are software as well and are as prone to bugs as anything.
pvorb 7y ago
And how do you do this in practice? I am struggling to think of a good way to keep the production code that fails the test and the production code that doesn't fail the test together. I might have my test check out an old version of the production code, compile it and test against that. But that is hard to get right.

serpix 7y ago

A green test does not equal bug free code. There may be a misimplementation / misunderstanding of the spec or your code passes beautifully under right conditions, with just this data setup.