3 ms·
Adding too many tests was a real issue before LLMs. You end up in situations where you when you add a feature there's 200 (out of e.g. 10k) tests that fail and
by IsTom 2mo ago
Adding too many tests was a real issue before LLMs. You end up in situations where you when you add a feature there's 200 (out of e.g. 10k) tests that fail and you have to figure out which of them should fail and you need to fix them and which are actual bugs.
- jeffreyrogers 2mo agoI haven't run into this yet. My test failures have either been real or have been triggered by an (intentional) breaking change. The later does require updating the tests, but I think catching the real bugs is worth that tradeoff. I haven't found spurious failures to be a big problem (I've had a few but rewriting the tests that have this problem has eliminated it for me). My codebase is a pretty modular rails app, which I think helps with this. I have worked on other projects where flaky tests are a problem (and generally cause developers to ignore and submit anyways), but so far I've avoided that. My experience with flaky tests is that they're typically due to poor modularity or to subsystems that other teams can modify. This project exists in a monorepo and I'm the sole developer so that's not a problem here.
- preg_match 2mo agoI disagree, I say the more tests the merrier, if you can afford the time cost. It's fairly rare to see faulty tests in my experience. Yes you can have hundreds failing tests and the code still works 100% - because those tests are testing API usage in a way the actual application never uses. But that's still a bug, it's just a future bug waiting to happen that you caught early. I wouldn't delete those tests.