3 ms·
By "you", I think you may mean "I". If ssp is hacking the test suite to bypass a failing test, ssp has the problem, not the test suite. jerf does not have much
by jerf 16y ago
By "you", I think you may mean "I". If ssp is hacking the test suite to bypass a failing test, ssp has the problem, not the test suite. jerf does not have much trouble with that, jerf has experienced "the one failing test that turns out to reveal a major underlying problem followed by a real fix that it would have taken him multiple customer-losing hard-to-reproduce bugs in field to learn about" multiple times.
Not that this is perfect, I've got just such a bug out there right now that simply refuses to be reproduced by anyone once a developer is looking at it, but I sure have far fewer of those than I would without the tests.
- ssp 16y agoIf ssp is hacking the test suite to bypass a failing test I'm talking about hacking the application to pass the test suite, not hacking the test suite. You can often "fix" a failing test by doing if (condition that failed) whack the application state so that the test suite will pass. without understanding what the actual bug were. And it's not always obvious to yourself that this is what you are doing. ssp has the problem, not the test suite Tuning to benchmarks is not some unique character flaw of mine. When you measure some aspect of people's behavior, they will optimize to that measurement. If the measurement is a boolean PASS/FAIL from the test suite, then they will optimize their behavior to get a PASS. But the actual desired outcome is not PASS, it is "bug free program".
- lutorm 16y agoThe answer to the problem of optimization by proxy is not no optimization at all.
- deleted 16y ago[deleted]
- jerf 16y agoHack the app, hack the test suite, I meant either equally. Which I think if you read my post is pretty clear that the main dichotomy is between "hack" and "real fix" and where the hack goes hardly matters. Nevertheless, you are arguing that because people sometimes program to benchmarks, you are apparently better off without the benchmarks. I say that's nonsense. The solution is to use the benchmarks better. Are we programmers or automatons? (Or managers?) If you're going to be that defeatist about programming you're not going to be able to be a successful programmer under any circumstances, the entire field is a minefield of superficially-appealing optimization opportunities!
- ssp 16y agoNevertheless, you are arguing that because people sometimes program to benchmarks, you are apparently better off without the benchmarks. No, I am not.