8 ms·

When you have no tests your problems go away because you don’t see any test failures. Never have I tested anything and NOT found a bug, and most things I teste

When you have no tests your problems go away because you don’t see any test failures. Never have I tested anything and NOT found a bug, and most things I tested I thought were already OK to ship. Thus, when you delete your tests, the only person you are fooling is probably yourself :( From reading your page I get the impression you are more burnt out from variation/configuration management which is completely relatable… I am too. This is a hard problem. But user volume is required to make $$. If the problem was easy then the market would be saturated with one size fits all solutions for everything.

akkartik 2y ago
> When you have no tests your problems go away because you don’t see any test failures. The flip side of this is the quote that "tests can show the presence of bugs, but never their absence". It better fits my experience here; every few months I'd find a new bug and diligently write a test for it. But then there was a new bug in a few months, discovered by someone in the first 10 minutes of using it. I'm sure I have bugs to discover in the new version. But the data structures I chose make many of the old tests obsolete by construction. So I'm hopeful that I'm a few bugs away from something fairly stable at least for idle use. Tests are definitely invaluable for a large team constantly making changes to a codebase. But here I'm trying to build something with a frozen feature set.
- monkpit 2y ago
  If your tests break or go away when your implementation changes, aren’t those bad tests by definition?
  - akkartik 2y ago
    I have two answers: 1. Yes. To the same extent that we are all bad people by definition, made of base material and unworthy urges. I'd love to have better programmers show me how I can make my tests better. The code is out there. 2. Even if I have good tests "by definition", a radical rewrite might make old tests look like "assert(2x1 == 2), assert (2x2 == 4)". Tests exist in a context, and radically changing the context can change the tests you need. --- This is not in OP, but I do also have a problem of brittle tests in my editor. In this case I need to test a word-wrapping algorithm. This depends intimately on pixel-precise details of the font. I'd love for better programmers than me to suggest how I can write tests that are robust and also self-evidently correct without magic constants that don't communicate anything to the reader. "Failure: 'x' started at x=67 rather than x=68." Reader's thought: "Why is this a problem?" etc. Comments appreciated on https://git.sr.ht/~akkartik/lines.love/tree/main/item/text_tests.lua https://git.sr.ht/~akkartik/lines.love/tree/main/item/text_t.... The summary at https://git.sr.ht/~akkartik/lines.love/tree/main/item/text_tests https://git.sr.ht/~akkartik/lines.love/tree/main/item/text_t... might help orient readers.
    - AdieuToLogic 2y ago
      >> If your tests break or go away when your implementation changes, aren’t those bad tests by definition? > 1. Yes. To the same extent that we are all bad people by definition, made of base material and unworthy urges. Good and bad are forms of judgement, so let's eschew judgement for the purposes of this reply :-). > I'd love to have better programmers show me how I can make my tests better. Better is also a form of judgement and, so, I will not claim I am or am not. What I will claim to do is offer my perspective regarding: > This is not in OP, but I do also have a problem of brittle tests in my editor. Unfortunately, brittle tests are the result of being overly specific. This is usually due to tests enforcing implementation knowledge instead of verifying a usage contract. The example assertions above are good examples of this (consider "assert (canMultiply ...)" as a conceptual alternative). What helps mitigate this situation is use of key abstractions relevant to the problem domain along with insulating implementation logic (note that this is not the same as encapsulation, as insulation makes the implementation opaque to collaborators). In your post, you posit: > Types, abstractions, tests, versions, state machines, immutability, formal analysis, all these are tools available to us in unfamiliar terrain. I suggest they serve a purpose beyond when "in unfamiliar terrain." Specifically, these tools provide confidence in system correctness in the presence of change. They also allow people to reason about the nature of a system, including your future-self. Perhaps most relevant to "brittle tests" are the first two you enumerated - types and abstractions. Having them can allow test suites to be defined against the public contract they provide. And as you rightly point out in your post, having the wrong ones can lead to problems. The trick is, when incorrect types and/or abstractions are identified, this presents an opportunity to refine understanding of the problem domain and improve key abstractions/collaborations accordingly. Functional testing[0] is really handy to do this fairly rapidly when employed early and often. HTH 0 - https://en.wikipedia.org/wiki/Functional_testing https://en.wikipedia.org/wiki/Functional_testing
  - Jtsummers 2y ago
    A lot of tests don't survive implementation changes, that doesn't make them "bad tests by definition". It means their value came and went. Think of it like scaffolding. You need it for a time, then the time is up, and it's removed. That doesn't make it bad, it was still necessary (or at least useful) for a time. When there's an implementation change you'll likely end up discarding a fair number of unit tests and creating new ones that reflect the new implementation details. That's just natural.
    - seanmcdirmid 2y ago
      A lot of tests, especially unit tests, are just change detectors and get updated/go away when change happens, that is just WAI. It is fairly hard to write non change detection tests, it requires for you to really reason abstractly about the contract of your module, or to write integration tests that are moving a bunch of things at once.
      - zem 2y ago
        small, fine-grained black box tests can be really good for this. in my last project, a type checker, the vast majority of the test suite was code snippets and assertions about expected errors the checker needed to catch, and it was an invaluable aid when making complex changes to the implementation.
        seanmcdirmid 2y ago
        Anything that transforms or processes text, like a compiler or type checker, is pretty easy to test. You get into trouble with user interfaces, however.
      - watwut 2y ago
        If that is the case too often, I ditch them and write integration tests for that part.
    - codr7 2y ago
      Yeah, especially when you're exploring new ground. Unit tests are awesome for fleshing out APIs; but once the fundamentals are in place, the tests no longer add any value.
- creesch 2y ago
  Automated tests ideally don't entirely replace manually executed tests. What they do replace is repetitive regression tests that don't need to be executed manually. In an ideal world this opens up room for exploratory testing where someone goes "off-script" and focuses specifically on those areas that are not covered by your automated tests. The thing is that automated tests aren't really tests, even though we call them that. They are automated checks at specified points, so they only check the outcome at those point in time. So yeah, they are also completely blind from the sort of thing a human* might easily spot while using the application. *Just to be ahead of the AI bros, we are not there yet, hold your horses.
IgorPartola 2y ago
I think this is highly domain dependent. I currently am working on codebase that has tests for a part of it that are an incredibly useful tool at helping me refactor that particular part. Other parts are so much UI behavior that it is significantly faster to catch bugs by manual testing because the UI/workflow either changes so fast that you don’t write tests for it (knowing they’ll be useless when the workflow is redesigned in the next iteration) or so slow that that particular UI/workflow just doesn’t get touched again so refactors don’t happen to it to introduce more bugs. I have never found tests to be universally necessary or helpful (just like types). They are a tool for a job, not a holy grail. I have also never met a codebase that had good test coverage and yet was free of bugs that aren’t then found with either manual testing or usage. Somewhat hyperbolically and sarcastically: if you are good enough to write perfect tests for your code, just write perfect code. If you aren’t perfect at writing tests, how do you know the tests are complete, bug free, and actually useful? :)
- klyrs 2y ago
  > the UI/workflow either changes so fast that you don’t write tests for it This is my number one pet peeve in software. Every aspect of every interface is subject to change always; not to mention the bonanza of dickbars and other dark patterns. Interfaces are a minefield of "operator error" but really it's an operational error.
  - niemandhier 2y ago
    People are building multimodal transformers, that try to simulate users. No matter how stupid the ai, if it can break your ai code, you have a bug.
- 256_ 2y ago
  > Somewhat hyperbolically and sarcastically: if you are good enough to write perfect tests for your code, just write perfect code. If you aren’t perfect at writing tests, how do you know the tests are complete, bug free, and actually useful? :) Well obviously, you just write tests for the tests. :3 It's called induction.
YZF 2y ago
The focus on automated unit/integrations tests is a relatively modern thing (late 90's?). There was some pretty large and extremely reliable software shipped before that focus. Random example is that the Linux kernel didn't have much tests (I think these days there is more testing). Unix likely didn't have a lot of "tests". Compilers tended to have them. Operating systems less so. Games (e.g. I'm sure Doom) didn't tend to have tests. You need to find a balance point. I think we know that (some) automated tests (unit, integration, end to end) can help build quality software. We also know good tests aren't always easy to write, bad tests make for harder refactoring and flaky tests can suck a lot of time on large projects. At the same time it's always interesting to try different things and find out what works, especially for you if you're a solo developers.
- amluto 2y ago
  > Random example is that the Linux kernel didn't have much tests (I think these days there is more testing). As the author of many of Linux’s x86 tests: many of those tests would fail on old kernels, and a decent number of those failures are related to very severe bugs. Linux has worked well for many years, but working well didn’t mean it wasn’t buggy.
  - YZF 2y ago
    As was said in another comment, tests don't prove the lack of bugs. There is no software of enough complexity without bugs. Working is something ;) Lots of software barely does that and there is certainly plenty of software with tests that doesn't meet the no-test Linux quality bar. That said, tests certainly have their place in the world of software quality, so thanks for your work!
- galaxyLogic 2y ago
  "Unit-Testing" became popular about the time of Extreme Programming. The reason I think it became so popular was that its proponents programmed in dynamically typed languages like Smalltalk, and later JavaScript. It seems to me that synamic languages needs testing more than statically typed ones.
osigurdson 2y ago
Tests written for pure functions are great. Tests written for everything else may be helpful but might not be.
- Ma8ee 2y ago
  You need tests for all part of the functionality you care about. I write tests for making sure that what is persisted is what we get back. Just the other day I found a bug due to our database didn't care about the timezone offset for our timestamps.
  - osigurdson 2y ago
    Not suggesting that testing other things isn't useful but not as straightforward and not as obviously beneficial as pure function testing. It is easy to just dogmatically pile on tests but they may not be helpful.
    - Ma8ee 2y ago
      I’d say, as beneficial. But as you say, not as straightforward. One if the reasons functional programming is popular is because it makes it easier to test, but it’s not that other code needs less testing.
whatever1 2y ago
A test will only catch an edge case you already thought of. If you thought of it anyway why just not fix the bug instead? Tests have burned out software engineers who waste the majority of their time deriving tests that will pass anyway. And then a significant code change will render them useless, at which point they have to be rewritten from scratch. No your program will not be more correct with more tests. Deal with it.
- 7bit 2y ago
  Do you think the test is written and the bug left in? What a weird take. And then, you write the test so that future changes (small or big) that causes regressions get noticed before the regression is put into production again. Especially in complex systems, you can define the end result and test if all your cases are covered. You do this anyway manually, so why not just write a test instead?
  - whatever1 2y ago
    The time that you spent to write a test is the time you would have spent in finding another bug in the code. Your time is finite. Bugs are not.
- becquerel 2y ago
  You write the test to prevent the bug from being accidentally reintroduced in the future. I have seen showstopper bugs reintroduced into production multiple times after they were fixed.
- bubblebeard 2y ago
  For me at least, designing a test will usually let me discover problems with my code which may otherwise gone unnoticed. Leaving the tests there once written to help us in future refactoring costs nothing. Granted, in some languages tests are more complicated to write compared to others. In PHP it’s a nightmare, in Rust it’s so easy it’s hard to avoid doing. I hear what you are saying though, sometimes writing tests consume more time then is necessary.
lelanthran 2y ago
> When you have no tests your problems go away because you don’t see any test failures. > > Never have I tested anything and NOT found a bug, and most things I tested I thought were already OK to ship. It's a trade-off. Most of the business world ran on, and to some extent still runs on, Excel programs. There are no tests there, but for the non-tech types who created these monsters, spending time on writing a test suite has a very real cost - there's less time to do the actual job they were hired for! So, yeah, each test you write means one less piece of functionality you add. You gotta make the trade-off between "acceptably (in frequency and period) buggy" and "absolutely bullet-proof no matter what input is thrown at it". With Excel programs, for example, if the user sees an error in the output, they fix the input data, they don't typically fix the program. It has to be a dealbreaker bug before they will dive into their code again to fix the program. And that is acceptable to them.
- Ma8ee 2y ago
  > There are no tests there, but for the non-tech types who created these monsters, spending time on writing a test suite has a very real cost - there's less time to do the actual job they were hired for! Not spending time on writing tests has a very real cost - a lot of time is spent on figuring out why your forecast was way off, or your year end figures don't add up. Not to mention how big parts of the world are thrown into austerity, causing hundred of thousand dead, due to errors in your published research [0]. [0] https://en.wikipedia.org/wiki/Growth_in_a_Time_of_Debt#Methodological_criticism https://en.wikipedia.org/wiki/Growth_in_a_Time_of_Debt#Metho...
  - lelanthran 2y ago
    >> It's a trade-off. >> spending time on writing a test suite has a very real cost > Not spending time on writing tests has a very real cost Yes. That's what "trade-off" means.
    - Ma8ee 2y ago
      My point is that there isn't a tradeoff between getting "real work" done or writing tests. Either you write tests, or you spend the even more time mitigating the consequences of not writing tests. You can't save time by not writing tests (except for the most trivial cases).
ChrisMarshallNY 2y ago
> Never have I tested anything and NOT found a bug, and most things I tested I thought were already OK to ship. I have found that, in my own case, every time I’ve written a unit test, it has exposed bugs. I don’t usually do the TDD thing, where I write failing tests first (but I do it, occasionally), so these tests are usually against code that I already think works. That said, I generally prefer test harnesses to unit tests[0]. They still find bugs, but the workflow is less straightforward. They also cause me to do more testing, as I develop, so the bugs are fixed in situ, so to speak. [0] https://littlegreenviper.com/testing-harness-vs-unit/ https://littlegreenviper.com/testing-harness-vs-unit/
- drewcoo 2y ago
  > That said, I generally prefer test harnesses to unit tests[0]. That's a strange redefinition of harness. The larger-scoped tests are more often called integration or even system tests. And while I'm here, those are slow tests that are harder to debug and require more maintenance (often maintenance of an entire environment to run them in!). Unit tests are closer to what they test, fast, and aren't tied to an environment - they can be run on every push.
  - ChrisMarshallNY 2y ago
    Not “strange,” in my opinion. Back in The Day, we called what people insist are “test harnesses,” “unit tests.” But these days, the term “unit test” has morphed into a particular configuration.
h1fra 2y ago
I'm puzzled by people debating tests. why such hate? They catch bugs, prevent breaking changes, and ensure API stability. I have never seen tests preventing me from refactoring anything. I guess it depends on the company and the processes :thinking:
- codr7 2y ago
  There are different kinds of tests. Integration tests at the outer edges often gives you most bang for buck. Granular, mocked unit tests often add little value and will become a maintenance burden sooner or later. And some of it is unconscious; maybe having that big, comfy test suite is preventing the software from evolving in optimal directions; because it would just be too much work and risk.
- HelloNurse 2y ago
  I think there is a mostly psychological "problem": tests are not perceived as progress (unless you are mature enough to treat quality assurance as an objective) and finding them fun to write or satisfying to run is an unusual acquired taste.
- eithed 2y ago
  Tests are tools - you won't be using screwdriver for everything, even though it's a tool that useful in many things. Having said that - tests, codebase and data consistency, static types are things I'd not want to be without
- swat535 2y ago
  Because writing good tests is very hard and many engineers are simply mediocre so they write brittle tests that require a lot of time to fix and don't actually test the right things (e.g too many mocks) or simply overconfident (like some people in the thread) that their code will always work. Also the TDD cultists are partially to blame for this attitude as well. Instead of focusing on teaching people how to write valuable tests, they decided to preach dogma and that frustrated many engineers. I'm firmly in the circle of writing tests of course, I don't think a system that is not tested should ever be in production (and no, you opening your browser on a local machine to see if it works is not sufficient testing for production..).
devjab 2y ago
It depends a lot on what you work on and how you program. Virtually none of our software has actual coding errors, and when developers write new parts or change them, it’s always very obvious if something breaks. Partly because of how few abstractions we use, partly because of how short we keep our chains. Letting every function live in isolation and almost never being used by multiple parts of the software. Both the lack of abstractions and the lack of reuse is against a lot of principles, and it’s not exactly like we refuse to do either religiously, but the only real principle we have is YAGNI, and if you build and abstraction before you need it you’re never going to pass a code review. As far as code reuse goes, well, in the perfect world it’s sort of stupid to have a lot of duplicate code. In a world where a lot of code is written on a Thursday afternoon by people who are tired, their babies kept them awake, the meetings were horrible, management doesn’t do the right things and so on. Well, in that world it’s almost always better to duplicate code so that it doesn’t eventually become a complicated abstract mess. It shouldn’t, and I’m sure it doesn’t in some places, I’ve just never worked in such a place. I have worked with a lot of people who followed things like clean code religiously and the results were always unwieldy code where even small changes would take weeks to implement. Which is completely counterproductive to what the actual business needs. The benefit of YAGNI is that it mostly applies to tests as well, exactly because it’s basically impossible to make changes without knowing exactly what impact you’re having on the entire system. What isn’t easy is business logic, and here I think tests are useful. Or at least they can be. Because far too often, the business doesn’t have a clue what they want up front. Even more often the business logic will change so rapidly that tests automated tests become virtually useless since you’re going to rely on acceptance tests anyway. Like I said, I’m not religious about it. I sometimes write tests, but in my anecdotal experience things like full test-coverage is an insane waste of time over a long period.
dgb23 2y ago
I watched a video by Russ Cox that was recommended in a recent thread, Go Testing By Example: https://www.youtube.com/watch?v=X4rxi9jStLo https://www.youtube.com/watch?v=X4rxi9jStLo There's _a lot_ of useful advice in there. But what I wanted to mention specifically is this: One of the things he's saying is that you can sometimes test against a simpler (let's say brute force) implementation that is easier to verify than what you want to test. There's a deeper wisdom implied in there: The usefulness of tests is dependent on the simplicity of their implementation relative to the simplicity of the implementation of what they are testing. Or said more strongly, tests are only useful if they are simpler than what they test. No matter how many tests are written, in the end we need to reason about code. Something being a "test", doesn't necessarily imply anything useful by itself. This is why I think a lot of programmers are wary of: - Splitting up functions into pieces, which don't represent a useful interface, just so the tests are easier to write. - Testing simple/trivial functions (helpers, small queries etc.) just for coverage. The tests are not any simpler than these functions. - Dependency inversion and mocking, especially if they introduce abstractions just in order to write those tests. I don't think of those things in absolute terms though, one can have reasons for each. The point is to not lose the plot.
6510 2y ago
> When you have no tests your problems go away because you don’t see any test failures. > Never have I tested anything and NOT found a bug, and most things I tested I thought were already OK to ship. I wasn't a very fast typist, I could do about 180 strokes per minute. My teacher, a tiny 80 year old lady, talked the whole time to intentionally distract her 5-6 students. It was a hilarious experience. One time, when I had an extra slow day, the monologue was about her learning to type, the teaching diploma required 300 strokes per minute, from print, hand writing and dictation. Not on such a fancy electronic type writer! We had mechanical type writers! And no correction lint! She was not the fastest in her class by far and many had band-aids around smashed fingers. Trying to read type, not listen and not burst out in laughter I think she forced me down to 80 strokes per minute. Sometimes she had me sit next to a girl doing 450 strokes per minute. Sounded like a machine gun. They would have casual conversation with eye contact. I should not have noticed it, I was suppose to be typing. When writing code and think about those "inevitable" bugs I always think of the old lady, who had 1000 ways of saying: you only think you are trying hard enough... and: we had no correction lint.... Take a piano, there is no backspace. You are suppose to get it right without mistakes. If you have all of those fancy tools to find bugs, test code, the ability to quickly go back and forwards, of course there will be plenty mistakes. If they need to be there no one knows.
- Yossarrian22 2y ago
  World class best in the world gymnasts still fall off a balance beam from time to time. Mistakes are inevitable, it’s why whiteout and then word processors were made
  - 6510 2y ago
    Pain is a great teacher.
datavirtue 2y ago
He was basically starting over. Definitely need to delete the tests. One of the issues with enterprise development is choking the project with tests and other compliance shit as soon as people start coding. Any project should be in a workable/deployable state before you commit to tests.
jjice 2y ago
Completely agree on tests. It's much more enjoyable for me to write some automated tests (unit or integration) and be able to re-run them over and over again than it is for me to manually run some HTTP requests against the server or something. While more work up front, they stay consistent and I can feel more comfortable with my code when I release. It's also just more fun to write code (even a test) than it is to manually run some tests over and over again, at which point I eventually get lazy and skip it for that last "simple, inconsequential" commit. Coming from a place where we never wrote tests, I introduce way fewer bugs and feel way more confident every day, especially when I change code in an existing place. One trick is to not go overboard and to strike an 80/20 balance for tests.

akkartik 2y ago

> When you have no tests your problems go away because you don’t see any test failures. The flip side of this is the quote that "tests can show the presence of bugs, but never their absence". It better fits my experience here; every few months I'd find a new bug and diligently write a test for it. But then there was a new bug in a few months, discovered by someone in the first 10 minutes of using it. I'm sure I have bugs to discover in the new version. But the data structures I chose make many of the old tests obsolete by construction. So I'm hopeful that I'm a few bugs away from something fairly stable at least for idle use. Tests are definitely invaluable for a large team constantly making changes to a codebase. But here I'm trying to build something with a frozen feature set.

monkpit 2y ago
If your tests break or go away when your implementation changes, aren’t those bad tests by definition?
- akkartik 2y ago
  I have two answers: 1. Yes. To the same extent that we are all bad people by definition, made of base material and unworthy urges. I'd love to have better programmers show me how I can make my tests better. The code is out there. 2. Even if I have good tests "by definition", a radical rewrite might make old tests look like "assert(2x1 == 2), assert (2x2 == 4)". Tests exist in a context, and radically changing the context can change the tests you need. --- This is not in OP, but I do also have a problem of brittle tests in my editor. In this case I need to test a word-wrapping algorithm. This depends intimately on pixel-precise details of the font. I'd love for better programmers than me to suggest how I can write tests that are robust and also self-evidently correct without magic constants that don't communicate anything to the reader. "Failure: 'x' started at x=67 rather than x=68." Reader's thought: "Why is this a problem?" etc. Comments appreciated on https://git.sr.ht/~akkartik/lines.love/tree/main/item/text_tests.lua https://git.sr.ht/~akkartik/lines.love/tree/main/item/text_t.... The summary at https://git.sr.ht/~akkartik/lines.love/tree/main/item/text_tests https://git.sr.ht/~akkartik/lines.love/tree/main/item/text_t... might help orient readers.
  - AdieuToLogic 2y ago
    >> If your tests break or go away when your implementation changes, aren’t those bad tests by definition? > 1. Yes. To the same extent that we are all bad people by definition, made of base material and unworthy urges. Good and bad are forms of judgement, so let's eschew judgement for the purposes of this reply :-). > I'd love to have better programmers show me how I can make my tests better. Better is also a form of judgement and, so, I will not claim I am or am not. What I will claim to do is offer my perspective regarding: > This is not in OP, but I do also have a problem of brittle tests in my editor. Unfortunately, brittle tests are the result of being overly specific. This is usually due to tests enforcing implementation knowledge instead of verifying a usage contract. The example assertions above are good examples of this (consider "assert (canMultiply ...)" as a conceptual alternative). What helps mitigate this situation is use of key abstractions relevant to the problem domain along with insulating implementation logic (note that this is not the same as encapsulation, as insulation makes the implementation opaque to collaborators). In your post, you posit: > Types, abstractions, tests, versions, state machines, immutability, formal analysis, all these are tools available to us in unfamiliar terrain. I suggest they serve a purpose beyond when "in unfamiliar terrain." Specifically, these tools provide confidence in system correctness in the presence of change. They also allow people to reason about the nature of a system, including your future-self. Perhaps most relevant to "brittle tests" are the first two you enumerated - types and abstractions. Having them can allow test suites to be defined against the public contract they provide. And as you rightly point out in your post, having the wrong ones can lead to problems. The trick is, when incorrect types and/or abstractions are identified, this presents an opportunity to refine understanding of the problem domain and improve key abstractions/collaborations accordingly. Functional testing[0] is really handy to do this fairly rapidly when employed early and often. HTH 0 - https://en.wikipedia.org/wiki/Functional_testing https://en.wikipedia.org/wiki/Functional_testing
- Jtsummers 2y ago
  A lot of tests don't survive implementation changes, that doesn't make them "bad tests by definition". It means their value came and went. Think of it like scaffolding. You need it for a time, then the time is up, and it's removed. That doesn't make it bad, it was still necessary (or at least useful) for a time. When there's an implementation change you'll likely end up discarding a fair number of unit tests and creating new ones that reflect the new implementation details. That's just natural.
  - seanmcdirmid 2y ago
    A lot of tests, especially unit tests, are just change detectors and get updated/go away when change happens, that is just WAI. It is fairly hard to write non change detection tests, it requires for you to really reason abstractly about the contract of your module, or to write integration tests that are moving a bunch of things at once.
    - zem 2y ago
      small, fine-grained black box tests can be really good for this. in my last project, a type checker, the vast majority of the test suite was code snippets and assertions about expected errors the checker needed to catch, and it was an invaluable aid when making complex changes to the implementation.
      - seanmcdirmid 2y ago
        Anything that transforms or processes text, like a compiler or type checker, is pretty easy to test. You get into trouble with user interfaces, however.
    - watwut 2y ago
      If that is the case too often, I ditch them and write integration tests for that part.
  - codr7 2y ago
    Yeah, especially when you're exploring new ground. Unit tests are awesome for fleshing out APIs; but once the fundamentals are in place, the tests no longer add any value.
creesch 2y ago
Automated tests ideally don't entirely replace manually executed tests. What they do replace is repetitive regression tests that don't need to be executed manually. In an ideal world this opens up room for exploratory testing where someone goes "off-script" and focuses specifically on those areas that are not covered by your automated tests. The thing is that automated tests aren't really tests, even though we call them that. They are automated checks at specified points, so they only check the outcome at those point in time. So yeah, they are also completely blind from the sort of thing a human* might easily spot while using the application. *Just to be ahead of the AI bros, we are not there yet, hold your horses.

IgorPartola 2y ago

I think this is highly domain dependent. I currently am working on codebase that has tests for a part of it that are an incredibly useful tool at helping me refactor that particular part. Other parts are so much UI behavior that it is significantly faster to catch bugs by manual testing because the UI/workflow either changes so fast that you don’t write tests for it (knowing they’ll be useless when the workflow is redesigned in the next iteration) or so slow that that particular UI/workflow just doesn’t get touched again so refactors don’t happen to it to introduce more bugs. I have never found tests to be universally necessary or helpful (just like types). They are a tool for a job, not a holy grail. I have also never met a codebase that had good test coverage and yet was free of bugs that aren’t then found with either manual testing or usage. Somewhat hyperbolically and sarcastically: if you are good enough to write perfect tests for your code, just write perfect code. If you aren’t perfect at writing tests, how do you know the tests are complete, bug free, and actually useful? :)

klyrs 2y ago
> the UI/workflow either changes so fast that you don’t write tests for it This is my number one pet peeve in software. Every aspect of every interface is subject to change always; not to mention the bonanza of dickbars and other dark patterns. Interfaces are a minefield of "operator error" but really it's an operational error.
- niemandhier 2y ago
  People are building multimodal transformers, that try to simulate users. No matter how stupid the ai, if it can break your ai code, you have a bug.
256_ 2y ago
> Somewhat hyperbolically and sarcastically: if you are good enough to write perfect tests for your code, just write perfect code. If you aren’t perfect at writing tests, how do you know the tests are complete, bug free, and actually useful? :) Well obviously, you just write tests for the tests. :3 It's called induction.

YZF 2y ago

The focus on automated unit/integrations tests is a relatively modern thing (late 90's?). There was some pretty large and extremely reliable software shipped before that focus. Random example is that the Linux kernel didn't have much tests (I think these days there is more testing). Unix likely didn't have a lot of "tests". Compilers tended to have them. Operating systems less so. Games (e.g. I'm sure Doom) didn't tend to have tests. You need to find a balance point. I think we know that (some) automated tests (unit, integration, end to end) can help build quality software. We also know good tests aren't always easy to write, bad tests make for harder refactoring and flaky tests can suck a lot of time on large projects. At the same time it's always interesting to try different things and find out what works, especially for you if you're a solo developers.

amluto 2y ago
> Random example is that the Linux kernel didn't have much tests (I think these days there is more testing). As the author of many of Linux’s x86 tests: many of those tests would fail on old kernels, and a decent number of those failures are related to very severe bugs. Linux has worked well for many years, but working well didn’t mean it wasn’t buggy.
- YZF 2y ago
  As was said in another comment, tests don't prove the lack of bugs. There is no software of enough complexity without bugs. Working is something ;) Lots of software barely does that and there is certainly plenty of software with tests that doesn't meet the no-test Linux quality bar. That said, tests certainly have their place in the world of software quality, so thanks for your work!
galaxyLogic 2y ago
"Unit-Testing" became popular about the time of Extreme Programming. The reason I think it became so popular was that its proponents programmed in dynamically typed languages like Smalltalk, and later JavaScript. It seems to me that synamic languages needs testing more than statically typed ones.

osigurdson 2y ago

Tests written for pure functions are great. Tests written for everything else may be helpful but might not be.

Ma8ee 2y ago
You need tests for all part of the functionality you care about. I write tests for making sure that what is persisted is what we get back. Just the other day I found a bug due to our database didn't care about the timezone offset for our timestamps.
- osigurdson 2y ago
  Not suggesting that testing other things isn't useful but not as straightforward and not as obviously beneficial as pure function testing. It is easy to just dogmatically pile on tests but they may not be helpful.
  - Ma8ee 2y ago
    I’d say, as beneficial. But as you say, not as straightforward. One if the reasons functional programming is popular is because it makes it easier to test, but it’s not that other code needs less testing.

whatever1 2y ago

A test will only catch an edge case you already thought of. If you thought of it anyway why just not fix the bug instead? Tests have burned out software engineers who waste the majority of their time deriving tests that will pass anyway. And then a significant code change will render them useless, at which point they have to be rewritten from scratch. No your program will not be more correct with more tests. Deal with it.

7bit 2y ago
Do you think the test is written and the bug left in? What a weird take. And then, you write the test so that future changes (small or big) that causes regressions get noticed before the regression is put into production again. Especially in complex systems, you can define the end result and test if all your cases are covered. You do this anyway manually, so why not just write a test instead?
- whatever1 2y ago
  The time that you spent to write a test is the time you would have spent in finding another bug in the code. Your time is finite. Bugs are not.
becquerel 2y ago
You write the test to prevent the bug from being accidentally reintroduced in the future. I have seen showstopper bugs reintroduced into production multiple times after they were fixed.
bubblebeard 2y ago
For me at least, designing a test will usually let me discover problems with my code which may otherwise gone unnoticed. Leaving the tests there once written to help us in future refactoring costs nothing. Granted, in some languages tests are more complicated to write compared to others. In PHP it’s a nightmare, in Rust it’s so easy it’s hard to avoid doing. I hear what you are saying though, sometimes writing tests consume more time then is necessary.

lelanthran 2y ago

> When you have no tests your problems go away because you don’t see any test failures. > > Never have I tested anything and NOT found a bug, and most things I tested I thought were already OK to ship. It's a trade-off. Most of the business world ran on, and to some extent still runs on, Excel programs. There are no tests there, but for the non-tech types who created these monsters, spending time on writing a test suite has a very real cost - there's less time to do the actual job they were hired for! So, yeah, each test you write means one less piece of functionality you add. You gotta make the trade-off between "acceptably (in frequency and period) buggy" and "absolutely bullet-proof no matter what input is thrown at it". With Excel programs, for example, if the user sees an error in the output, they fix the input data, they don't typically fix the program. It has to be a dealbreaker bug before they will dive into their code again to fix the program. And that is acceptable to them.

Ma8ee 2y ago
> There are no tests there, but for the non-tech types who created these monsters, spending time on writing a test suite has a very real cost - there's less time to do the actual job they were hired for! Not spending time on writing tests has a very real cost - a lot of time is spent on figuring out why your forecast was way off, or your year end figures don't add up. Not to mention how big parts of the world are thrown into austerity, causing hundred of thousand dead, due to errors in your published research [0]. [0] https://en.wikipedia.org/wiki/Growth_in_a_Time_of_Debt#Methodological_criticism https://en.wikipedia.org/wiki/Growth_in_a_Time_of_Debt#Metho...
- lelanthran 2y ago
  >> It's a trade-off. >> spending time on writing a test suite has a very real cost > Not spending time on writing tests has a very real cost Yes. That's what "trade-off" means.
  - Ma8ee 2y ago
    My point is that there isn't a tradeoff between getting "real work" done or writing tests. Either you write tests, or you spend the even more time mitigating the consequences of not writing tests. You can't save time by not writing tests (except for the most trivial cases).

ChrisMarshallNY 2y ago

> Never have I tested anything and NOT found a bug, and most things I tested I thought were already OK to ship. I have found that, in my own case, every time I’ve written a unit test, it has exposed bugs. I don’t usually do the TDD thing, where I write failing tests first (but I do it, occasionally), so these tests are usually against code that I already think works. That said, I generally prefer test harnesses to unit tests[0]. They still find bugs, but the workflow is less straightforward. They also cause me to do more testing, as I develop, so the bugs are fixed in situ, so to speak. [0] https://littlegreenviper.com/testing-harness-vs-unit/ https://littlegreenviper.com/testing-harness-vs-unit/

drewcoo 2y ago
> That said, I generally prefer test harnesses to unit tests[0]. That's a strange redefinition of harness. The larger-scoped tests are more often called integration or even system tests. And while I'm here, those are slow tests that are harder to debug and require more maintenance (often maintenance of an entire environment to run them in!). Unit tests are closer to what they test, fast, and aren't tied to an environment - they can be run on every push.
- ChrisMarshallNY 2y ago
  Not “strange,” in my opinion. Back in The Day, we called what people insist are “test harnesses,” “unit tests.” But these days, the term “unit test” has morphed into a particular configuration.

h1fra 2y ago

I'm puzzled by people debating tests. why such hate? They catch bugs, prevent breaking changes, and ensure API stability. I have never seen tests preventing me from refactoring anything. I guess it depends on the company and the processes :thinking:

codr7 2y ago
There are different kinds of tests. Integration tests at the outer edges often gives you most bang for buck. Granular, mocked unit tests often add little value and will become a maintenance burden sooner or later. And some of it is unconscious; maybe having that big, comfy test suite is preventing the software from evolving in optimal directions; because it would just be too much work and risk.
HelloNurse 2y ago
I think there is a mostly psychological "problem": tests are not perceived as progress (unless you are mature enough to treat quality assurance as an objective) and finding them fun to write or satisfying to run is an unusual acquired taste.
eithed 2y ago
Tests are tools - you won't be using screwdriver for everything, even though it's a tool that useful in many things. Having said that - tests, codebase and data consistency, static types are things I'd not want to be without
swat535 2y ago
Because writing good tests is very hard and many engineers are simply mediocre so they write brittle tests that require a lot of time to fix and don't actually test the right things (e.g too many mocks) or simply overconfident (like some people in the thread) that their code will always work. Also the TDD cultists are partially to blame for this attitude as well. Instead of focusing on teaching people how to write valuable tests, they decided to preach dogma and that frustrated many engineers. I'm firmly in the circle of writing tests of course, I don't think a system that is not tested should ever be in production (and no, you opening your browser on a local machine to see if it works is not sufficient testing for production..).

devjab 2y ago

It depends a lot on what you work on and how you program. Virtually none of our software has actual coding errors, and when developers write new parts or change them, it’s always very obvious if something breaks. Partly because of how few abstractions we use, partly because of how short we keep our chains. Letting every function live in isolation and almost never being used by multiple parts of the software. Both the lack of abstractions and the lack of reuse is against a lot of principles, and it’s not exactly like we refuse to do either religiously, but the only real principle we have is YAGNI, and if you build and abstraction before you need it you’re never going to pass a code review. As far as code reuse goes, well, in the perfect world it’s sort of stupid to have a lot of duplicate code. In a world where a lot of code is written on a Thursday afternoon by people who are tired, their babies kept them awake, the meetings were horrible, management doesn’t do the right things and so on. Well, in that world it’s almost always better to duplicate code so that it doesn’t eventually become a complicated abstract mess. It shouldn’t, and I’m sure it doesn’t in some places, I’ve just never worked in such a place. I have worked with a lot of people who followed things like clean code religiously and the results were always unwieldy code where even small changes would take weeks to implement. Which is completely counterproductive to what the actual business needs. The benefit of YAGNI is that it mostly applies to tests as well, exactly because it’s basically impossible to make changes without knowing exactly what impact you’re having on the entire system. What isn’t easy is business logic, and here I think tests are useful. Or at least they can be. Because far too often, the business doesn’t have a clue what they want up front. Even more often the business logic will change so rapidly that tests automated tests become virtually useless since you’re going to rely on acceptance tests anyway. Like I said, I’m not religious about it. I sometimes write tests, but in my anecdotal experience things like full test-coverage is an insane waste of time over a long period.

dgb23 2y ago

I watched a video by Russ Cox that was recommended in a recent thread, Go Testing By Example: https://www.youtube.com/watch?v=X4rxi9jStLo https://www.youtube.com/watch?v=X4rxi9jStLo There's _a lot_ of useful advice in there. But what I wanted to mention specifically is this: One of the things he's saying is that you can sometimes test against a simpler (let's say brute force) implementation that is easier to verify than what you want to test. There's a deeper wisdom implied in there: The usefulness of tests is dependent on the simplicity of their implementation relative to the simplicity of the implementation of what they are testing. Or said more strongly, tests are only useful if they are simpler than what they test. No matter how many tests are written, in the end we need to reason about code. Something being a "test", doesn't necessarily imply anything useful by itself. This is why I think a lot of programmers are wary of: - Splitting up functions into pieces, which don't represent a useful interface, just so the tests are easier to write. - Testing simple/trivial functions (helpers, small queries etc.) just for coverage. The tests are not any simpler than these functions. - Dependency inversion and mocking, especially if they introduce abstractions just in order to write those tests. I don't think of those things in absolute terms though, one can have reasons for each. The point is to not lose the plot.

6510 2y ago

> When you have no tests your problems go away because you don’t see any test failures. > Never have I tested anything and NOT found a bug, and most things I tested I thought were already OK to ship. I wasn't a very fast typist, I could do about 180 strokes per minute. My teacher, a tiny 80 year old lady, talked the whole time to intentionally distract her 5-6 students. It was a hilarious experience. One time, when I had an extra slow day, the monologue was about her learning to type, the teaching diploma required 300 strokes per minute, from print, hand writing and dictation. Not on such a fancy electronic type writer! We had mechanical type writers! And no correction lint! She was not the fastest in her class by far and many had band-aids around smashed fingers. Trying to read type, not listen and not burst out in laughter I think she forced me down to 80 strokes per minute. Sometimes she had me sit next to a girl doing 450 strokes per minute. Sounded like a machine gun. They would have casual conversation with eye contact. I should not have noticed it, I was suppose to be typing. When writing code and think about those "inevitable" bugs I always think of the old lady, who had 1000 ways of saying: you only think you are trying hard enough... and: we had no correction lint.... Take a piano, there is no backspace. You are suppose to get it right without mistakes. If you have all of those fancy tools to find bugs, test code, the ability to quickly go back and forwards, of course there will be plenty mistakes. If they need to be there no one knows.

Yossarrian22 2y ago
World class best in the world gymnasts still fall off a balance beam from time to time. Mistakes are inevitable, it’s why whiteout and then word processors were made
- 6510 2y ago
  Pain is a great teacher.

datavirtue 2y ago

He was basically starting over. Definitely need to delete the tests. One of the issues with enterprise development is choking the project with tests and other compliance shit as soon as people start coding. Any project should be in a workable/deployable state before you commit to tests.

jjice 2y ago

Completely agree on tests. It's much more enjoyable for me to write some automated tests (unit or integration) and be able to re-run them over and over again than it is for me to manually run some HTTP requests against the server or something. While more work up front, they stay consistent and I can feel more comfortable with my code when I release. It's also just more fun to write code (even a test) than it is to manually run some tests over and over again, at which point I eventually get lazy and skip it for that last "simple, inconsequential" commit. Coming from a place where we never wrote tests, I introduce way fewer bugs and feel way more confident every day, especially when I change code in an existing place. One trick is to not go overboard and to strike an 80/20 balance for tests.