16 ms·
Delete tests
- avg_dev 1y agoidk, i never thought > “it is blasphemy to delete a test”, was ever a thing. i still don't. if a test is flaky, but it covers something useful, it should be made not flaky somehow. if tests are slow, but they are useful, then they should be optimized so they run faster. if tests cover some bit of functionality that the software no longer needs to provide, the functionality should be deleted from the code and the tests. if updating a small bit of code causes many tests to need to be adjusted, and that's a pain, and it happens frequently, then the tests should be refactored or adjusted. > Confidence is the point of writing tests. yes, agreed. but tests are code, too. just maintain the tests with the code in a sensible way. if there is something worth deleting, delete it; there is no gospel that says you can't. but tests provide value just like the author describes in the "fix after revert after fix after revert" counterexample. just remember they're code like anything else is all and treat them accordingly.
- skybrian 1y agoI think the argument is that sometimes updating a flaky test is not worth the effort, so consider deleting it,
- rcktmrtn 1y ago> > “it is blasphemy to delete a test”, > was ever a thing. i still don't. I experienced this when working at a giant company where all the teams were required to report their "code coverage" metrics to middle management. We had the flaky test problem too, but I think another angle of is being shackled to test tech-debt. The "coverage goals" in practice encouraged writing a lot of low quality tests with questionable and complex fixtures (using regular expressions to yoink C++ functions/variables out of their modules and place them into test fixtures). Fiddling with tests slowed down a lot of things, but there was a general agreement that the whole projected needed to be re-architected (it was split up over a zillion different little "libraries" that pretended to be independent, but were actually highly interdependent) and while I was there I always felt like we needed to cut the Gordian knot and accept that it might decrease the sacred code coverage. Not sure if I was right or what ever happened with that project but it sure was a learning experience.
- arkis22 1y agoAdding a test is easy. Deleting a test should involve like 3-4 people who all know the codebase.
- recursivedoubts 1y agoOne of the most important things you can do is move your tests up the abstraction layers and away from unit tests. For lack of a better term, to move to integration tests. End-to-end tests are often too far from the system to easily understand what's wrong when they break, and can overwhelm a development org. Integration tests (or whatever you want to call them) are often the sweet spot: not tied to a particular implementation, able to survive fairly significant system changes, but also easy enough to debug when they break. https://grugbrain.dev/#grug-on-testing https://grugbrain.dev/#grug-on-testing
- MathMonkeyMan 1y agoIntegration tests at $DAY_JOB are often slow (sleeps, retries, inadequate synchronization, startup and shut down 8 processes that are slow to start and stop), flaky (the metrics for this rate limiter should be within 5%, this should be true within 3 seconds, the output of this shell command is the same on all platforms), undocumented, and sometimes cannot be run locally or with locally available configurations. When I run a set of integration tests associated with some code I'm modifying, I have no idea what they are, why they were written, what they do, how long they will take to run, or whether I should take failures seriously. Integration tests are closer to what you want to know, but they're also more. If I want to make sure that my state machine returns an error when it receives a message for which no state transition is defined, I could spin up a process and set up log collection and orchestrate with python and... or I could write a unit test that instantiates a state machine, gives it a message, and checks the result. My point is that we need both. Write a unit test to ensure that your component behaves to its spec, especially with respect to edge cases. Write an integration test to make sure that the feature of which your component is a part behaves as expected.
- skydhash 1y agoMy current mental model is a car. If it’s a function or some other things you’re fully confident you captured the domain, add unit tests to capture that. Just like an engine. But the most important is integration tests that couple something like the engine, the ignition system and test that when the user press the start button, the engine start and the dashboard light up. Unit tests are great for DX but only integration and above tests matter business wise.
- joeyguerra 1y agoI've always deleted tests. I've never heard anyone say not to delete tests.
- fritzo 1y agoI repeatedly, emphatically tell AI coding assistants not to delete tests.
- readthenotes1 1y agoI groaned when a co-worker deleted a test that was pointing out his code was broken. I didn't tell him not to delete tests. It wouldn't have done any good.
- cjfd 1y agoI think the word you were looking for was 'cow orker'.
- efitz 1y agoI have had a weird thought lately about testing at runtime. My thought is just to log violations of expectations- i.e. log when the test would have failed. This doesn’t prevent the bug from being introduced but can remove a huge amount of complexity for test cases that are hard to set up.
- seer 1y agoI’ve kinda of the opinion that if introducing tests, especially the useful integration tests is hard and complex, then it is a code smell. Most of the times, especially while I was learning, making your code “more testable” has always involved things that should have been done in the first place, but we were lazy/didn’t know better. Things like reducing dependencies, moving state away from the core and into the shell. Using more formal state machines etc. Once the “painful changes” were done I’ve found that it was actually beneficial in a lot of other contexts. That given, I’ve kinda almost stopped writing unit tests - with the advent of expressive types everywhere, the job of unit tests has now been shifted to the compiler. In one typescript project I’ve managed to set it up, the part that kept the state was statically typed (a database) making sure any data that went in and out was _exactly_ like the compiler expected. After typing and validating all the other user / non-user inputs into the code, it ended up in a situation where “if the code compiles, it will work” and that was glorious. We had very minimal unit tests - only around actual business logic with state machines, the rest was kinda handled by the compiler and we didn’t feel the need to do it manually. Apart from that, the integration tests had the philosophy of “don’t specify anything that the user is not seeing” so no button test ids, urls or weird expectation of the underlying code, just an explanation of “the user is on the page with this title, they see a button named this and they press it, expecting they are now in a page titled this” The concept was taken from the capybara ruby testing library way back in the day, and the tests this produced have been incredibly resilient. Any update that changes the user experience would fail the tests (as they should) and any refactor, up to the level of changing urls or even changing the underlying libraries and frameworks, would be ignored.
- runstop 1y agoSounds a bit like "design by contract", leaving the assertions enabled in production code. It would be great to have solid DbC support in mainstream languages.
- dcminter 1y agoHow about you fix the flakey tests? The tests I'd delete are the ones that just test that the code is written in a particular way instead of testing the expected behaviour of thr code.
- XorNot 1y agoThis: anything which starts doing stuff like "called API N times" is utterly worthless (looking at you whole bunch of AWS API mock tests...)
- silversmith 1y agoCame here to comment this. Most of the flakey tests are badly written, some warn you about bugs you don't yet understand. Couple years ago I helped to bring a project back on track. They had a notoriously flakey part of test suite, turned out to be caused by a race condition. And a very puzzling case of occasional data corruption - also, turns out, caused by the same race condition.
- MoreQARespect 1y agoI tend to find that those bugs are in the extreme minority. Most flakiness ends up being a bug in the test or nondeterminism exhibited by the code which users dont actually care about.
- Shank 1y ago> How about you fix the flakey tests? Often times a flakey test is not flakey because it was well-written and something else strange is failing. Often times the test reveals something about the system that is somewhat non-deterministic, but not non-deterministic in a detrimental way. When you have multiple levels of abstraction and parallelization and interdependent behavior, fixing a single test becomes a time consuming process that is difficult to work with (because it's flakey, you can't always replicate the failure). If a test fails in CI and the traceback is unclear, many people will re-run once and let it continue to flake. Obvious flakes around time and other dependencies are much easier to spot and fix, so they are. It's only the weird ones that lead to pain and regret.
- bubblebeard 1y agoThe author has a point. Obsolete tests serves no one, but deleting a test because it will randomly fail is an indication of an unstable process. Maybe there is a race condition, maybe your code has some dependency that is sporadically unavailable. Deleting such tests is just turning a blind eye to the problem. Unstable tests means you either didn’t write that test very well to begin with, or the process you are testing is itself unstable.
- jampa 1y agoI work in an app where bugs are unacceptable due to the nature of the company's reputation. We've been having a lot of success with E2E, but getting there was NOT easy. Some tips: - False negative results will make your devs hate the tests. People want to get things done and will start ignoring them if you unnecessarily break their workflow. In the CI, you should always retry on failure to avoid flaky false-negative tests. - E2E Tests can fail suddenly. To avoid breaking people's workflow, we do a megabenchmark every day at 1 AM, and the test runs multiple times - even if it passes - so that we can measure flakiness. If a test fails in the benchmark, we remove it from the CI so we don't break other developers' workflows. The next day, we either fix the test or the bug. - Claude Code SDK has been a blessing for E2E. Before, you couldn't run all the E2E in the PR's CI due to the time they all take. Now, we can send the branch to the Claude Code SDK to determine what E2E tests should run. - Also, MCPs and Claude Code now write most of my E2E. I wrote a detailed Claude.md to let it run autonomously --writing, validating, and repeating -- while I do something else. It does in 3 to 4 shots. For the price of a cup of coffee, it saves me 30-60 minutes per test.
- _caw 1y agoWould love to hear more about using Claude to determine which E2E tests to run. What context are you giving it? Is it like, "this looks like a billing feature, let me run any tests that seem relevant"?
- jampa 1y agoI feed the `git diff` of the branch (excluding large files like package-lock) and a list of the E2E files. Claude reads and compares the E2E tests against the modified content. The good thing about Claude Code is that it uses tool calls to explore the files to check which E2E tests can validate the changes.
- anon7000 1y agoHow well does this work for massive codebases and thousands of e2e tests?
- jhhh 1y agoIf you are having to refactor 150 things each time you change your codebase then maybe you need to refactor your test suite first. Direct calls in tests to constructed/mocked objects is usually something you can just stuff into a private method so you only need to change it in one place. Not quite sure I agree with the conclusion of the tiers of testing section either. If a test suite takes a long time but still covers something useful, then just deleting it because it takes too long makes no sense. Yes, if you have a 'fastTests' profile that doesn't run it that could temporarily convince you your changes are fine when they aren't. But the alternative is just never knowing your change is bad until it breaks production instead of just breaking your CI prior to that point.
- eru 1y agoA simple thing you can set up is to run short tests on every push to a feature branch, but run the long tests only when merging into master. Basically, you provisionally make the merge commit, run the expensive tests against it, and iff they pass, declare the newly created commit to be the new master.
- strogonoff 1y agoTests are code. Code has bugs. More complex code has more bugs. The more complex your tests, the more bugs in your tests. Who tests the tests? It’s one thing if you rely on functionality provided by a stable testing framework, but I bet grug no like call stacks in own test code.
- jessekv 1y ago> Who tests the tests? To me it's a bit like double entry bookkeeping. Two layers is valuable, but there's rapidly diminishing returns beyond two.
- pydry 1y agoTests get implicitly tested by being run against code. When they fail in spite of the presence of no bugs then congratulations youve found a bug in your test.
- imiric 1y agoI'm a big believer in the utility of tests, and I do think the author has a point. There is a time and place when a test is not useful, and should be deleted. However... > If the future bug occurs, fix it and write a new test that doesn’t flake. Today, delete the tests. How is this different from simply fixing the flaky test today? Tests are code, and can also incur technical debt. The reason the test is flaky is likely because nobody is willing to take the time to address it properly. Sometimes it requires a refactoring of the SUT to allow making the test more reliable. Sometimes the test itself is too convoluted and difficult to change. All of this is chore work, and is often underappreciated. Nobody got promoted or celebrated for fixing something that is an issue a random percentage of times. After all, how do we know for sure that it's permanently fixed? Only time will tell. But the flaky test might still deliver confidence and be valuable when it does run successfully. So deleting it would bring more uncertainty. That doesn't seem like a fair tradeoff for removing an annoyance. The better approach would be to deal with the annoyance. > What if your tests are written so that a one line code change means updating 150 tests? That might be a sign that the tests are too brittle, and too "whiteboxy". So fix the tests. That said, there are situations when a change does require updating many tests. These are usually large refactors or major business logic changes. This doesn't mean that the tests are and won't be useful. It's just a side effect of the change. Tests are code, so fix the tests. I've often heard negativity around unit tests, from programmers who strongly believe that more utility comes from integration tests (the inverted test pyramid, etc.). One of the primary reasons is this belief that unit tests slow you down because they need to be constantly updated. This is a harmful mentality, coming from a place of laziness. Tests are code, and require maintenance just as well. Unit tests in particular are tightly coupled to the SUT, which makes them require maintenance more frequently. There should also be more unit tests than other types, adding more maintenance burden. But none of these are reasons to not write unit tests, and codebases without them are more difficult to change, and more susceptible to regressions. > What if your tests take so long to run that you can’t run them all between merges, and you start skipping some? That is an organizational problem. Label your tests by category (unit, integration, E2E), and provide quick ways to run a subset of them. During development, you can run the quick tests for a sanity check, while the more expensive tests run in CI. There's also the problem of long test suites because the tests are inefficient. Again: *fix the tests*. > Even worse, what if your business requirements have changed, and now you have thousands of lines of tests failing because they test the wrong thing? That is a general maintenance task. Would you say the same because you had to update a library that depended on the previous business logic? Would you simply delete the library because it would take a lot of effort to update it? No? Then *fix the tests*. :)
- simianwords 1y agoA good heuristic for a test is: how many times you are having to fix it when you change real code. If every small change in the code base causes you to go back and fix the tests then your tests are bad. They should not get in the way so often. There should be a concept of “test maintenance overhead” that is weighted against the number of bugs it catches. You could also think of it as false positives vs true positives.
- throwmeaway222 1y agodelete all mocked tests imo mock exists call exists assert exists was called one time so so so useless so that you can increase your coverage. just move to integration tests
- rustystump 1y agoAt the end of the day, you need to have some kind of way to know if shit work or dont. This article feels a bit contrived to make an edgy point of “delete the test” which feels like it misses the real why behind testing.
- mirekrusin 1y agoWe add .skip and QAs are taking over in the background to address those issues.
- sitkack 1y agoWhat a poorly written article. I should delete my tests because they fail randomly. My tests don't fail randomly.
- vitonsky 1y agoNo, thanks. I already spent time to write tests while implementing a features, now I have a lot of tests that proof the feature is works fine, and I no more fear to make changes, because tests keep me safe of regression bugs. The typical problems of any code base with no tests is a regression bugs, rigid team (because they must keep in mind all cases when code may destroy everything), fear driven development (because even team with zero rotation factor don't actually remember all problems they've fixed).
- willio58 1y agoDid you read the article? What is your answer to the points the author makes around flaky tests/changing business requirements/too many tests confirming the same functionality and taking too long to run?
- CPLX 1y agoHacker News is a prominent message board where users create wide ranging conversations based on article titles.
- snovv_crash 1y agoFlaky tests: tests should be deterministic. If your tests are flakey in a 100% controlled environment, probably your real system is unreliable too. Changing business requirements: business logic should be tested separately. It is expected to change, so if all of your tests include it, then yes of course it will be hard to maintain. Too many tests for the same thing: yeah then maybe delete some of the duplicates? Taking too long: mock stuff out. Also, maybe reconsider some architectural decisions you made, if your tests take too long it's probably going to bother your customers with slow behaviour too.
- ikari_pl 1y agoI think the point of article is to delete the BAD tests. Just like you need to delete the bad code, not all the code. ;)
- rotbart 1y agoSo... clickbait title for an article that could have been called "Delete flakey tests"...but then and most of us would have just gone "yep" and not clicked.
- 4ndrewl 1y ago> If your test is creating confidence in broken code with failing tests, it would be better for it to not exist. The author never considers the other option of fixing the flaky tests. I find this odd.
- teiferer 1y agoAll seems like "fix tests" is the better advice. Flaky test? Fix it! Make it rock solid! Slow test? Fix it! Make it fast! That can be hard (if it was easy, people would have already done it), but it's vastly more useful than deleting. Even the mentioned overtesting requires a fix by focusing the tests on separate things. You could call that "deleting" but that's oversimplifying what's going on. Same with changed requirements.
- jessekv 1y agoI hate to admit it, but flaky tests almost always highlight weaknesses in my software architecture. And fixing a flaky test usually involves making the actual code more robust.
- mattlondon 1y agoDon't delete flaky tests, fix them.
- juped 1y agoI think all the listed reasons are good reasons to delete tests. I like to keep the test suite running in a single-digit number of seconds. (Sometimes a test you really need takes a while, and you can skip it by default and enable it on the CI test runner or whatever.) Another one I really agree with is "What if your tests are written so that a one line code change means updating 150 tests?". If you update a test, basically, ever, it's probably a bad test and is better off not existing than being like that. It's meant to distinguish main code with errors from main code without errors; if it must be updated in tandem with main code, it's just a detector that anything changed, which is counterproductive. Of course you're changing things, that's why you fenced them with tests.
- nine_k 1y agoUn-clickbaiting the title: "Delete useless tests". I once faced a suite of half-broken tests; so many were broken that engineers stopped caring if their changes broke another test. I suggested to separate a subset of still-working, useful tests, keep them always green, and make them passing a required check in CI/CD. Ignore the rest of the tests for CI/CD purposes. Gradually fix some of the flaky or out-of-sync tests if they are still useful, and promote them to the evergreen subset. Delete tests that are found to be beyond repair (like the article suggests). This worked.
- egeozcan 1y agoAt one of the companies I worked with when I was doing consulting, they could make the slow tests, which used to take around 3 hours, run much faster and in parallel by throwing engineering and hardware resources at the problem. First it was 30 minutes, then it was 10, then around 2-3 minutes. I think it was one of the best investments that company made. So my point is, don't delete slow tests, just make them fast.
- huflungdung 1y agoContrarian blog post trying to challenge the status quo without understanding the implications in order to look like a visionary. I say this in interviews just to look smart. And people think it’s revolutionary. Everyone loves an outspoken opposition -what do they know, where did they get this knowledge?! Only a couple have ever pushed back and those that do are the companies that I want to work with. A wild example, let’s delete a test that ensures a heart pump works at the correct duty cycle given its parameters. Now someone comes along and redefines milliseconds to microseconds for some unrelated component. The tests are all fine. Patient now has a 60000 bpm heart rate. Stupid idea.
- codeulike 1y agoBut before you delete the test, write a test that tests whether the test is deleted, and make sure that test is failing as expected. Then delete the test. Then run the other test that makes sure the test is deleted and it should now pass /s
- dgunay 1y agoI see a lot of debate about the definition and merits of unit, integration, and e2e testing here. For my workplace, we have recognized a few issues with an overreliance on unit tests. When you have important behaviors and invariants enforced by your database, you mock it out at your own peril. This literally caused a bug to slip through to prod this week. Unit tests just don't help here. We use clean architecture. There are pockets of our codebase where, through deliberate or accidental deviations from our architecture, stuff like e.g. controllers with business logic in them exists. In some cases it is easier to just integration or e2e test this code instead of do the ugly refactoring to bring it into compliance. Doing the test first will even make the refactor easier. Parts of our codebase are just big, pure functions. Arguments go in, return values come out. These are the ideal candidates for unit testing, and we do so extensively because they're cheap and fast. I think what occurs to me as I write this is that if you live in an idyllic codebase which can express every single state transition in memory, without any dependency on external systems, sure, unit tests are great. For those of us who are not so lucky, e2e tests can be a lifeline and a way to maintain control with minimal mock-induced churn in the test suite.
- BobbyTables2 1y agoEasier than deleting tests is to delete the runners that execute them. Problem solved once and for all!
- senfiaj 1y agoTests are also code, thus they also add maintenance costs. IMHO, usually about 8-15% of code (depends on the project) may need automatic testing. If the code can be quickly tested manually, maybe you should think twice before writing tests. The code that changes too quickly (because of requirements changes as mentioned in the article) is also not the best candidate to be tested automatically since what's the point of regression testing if it's almost rewritten from scratch. A better candidate that is worth to be tested in automatic way is a code that is critical (medical equipment, bank systems, airplanes, etc.) and changes slowly/gradually enough (so regression testing will make sense here).