6 ms·

One of the most important things you can do is move your tests up the abstraction layers and away from unit tests. For lack of a better term, to move to integr

One of the most important things you can do is move your tests up the abstraction layers and away from unit tests. For lack of a better term, to move to integration tests. End-to-end tests are often too far from the system to easily understand what's wrong when they break, and can overwhelm a development org. Integration tests (or whatever you want to call them) are often the sweet spot: not tied to a particular implementation, able to survive fairly significant system changes, but also easy enough to debug when they break. https://grugbrain.dev/#grug-on-testing https://grugbrain.dev/#grug-on-testing

MathMonkeyMan 1y ago
Integration tests at $DAY_JOB are often slow (sleeps, retries, inadequate synchronization, startup and shut down 8 processes that are slow to start and stop), flaky (the metrics for this rate limiter should be within 5%, this should be true within 3 seconds, the output of this shell command is the same on all platforms), undocumented, and sometimes cannot be run locally or with locally available configurations. When I run a set of integration tests associated with some code I'm modifying, I have no idea what they are, why they were written, what they do, how long they will take to run, or whether I should take failures seriously. Integration tests are closer to what you want to know, but they're also more. If I want to make sure that my state machine returns an error when it receives a message for which no state transition is defined, I could spin up a process and set up log collection and orchestrate with python and... or I could write a unit test that instantiates a state machine, gives it a message, and checks the result. My point is that we need both. Write a unit test to ensure that your component behaves to its spec, especially with respect to edge cases. Write an integration test to make sure that the feature of which your component is a part behaves as expected.
- skydhash 1y ago
  My current mental model is a car. If it’s a function or some other things you’re fully confident you captured the domain, add unit tests to capture that. Just like an engine. But the most important is integration tests that couple something like the engine, the ignition system and test that when the user press the start button, the engine start and the dashboard light up. Unit tests are great for DX but only integration and above tests matter business wise.
- majormajor 1y ago
  You need to test contracts with external code without having to include full external systems. Unit tests on internal implementation details are fragile as behavior changes. Unit tests on your module's contracts give you confidence in refactoring. Passing params in instead of making external calls inside your business logic functions can help. DI can help if that's too impractical or unwieldy for whatever reason in the domain. It's hard to do right the first time - sometimes its fuzzy what's an internal detail vs what's an external contract - but you need to get there ASAP.
- lenkite 1y ago
  These are "system tests" at your $DAY_JOB, not "integration tests".
  - MathMonkeyMan 1y ago
    There is an even higher level of testing that they call "system tests." Those are for things like tracking performance regressions. I think there's always a spectrum among the terms "system," "integration," and "unit."
- MoreQARespect 1y ago
  The way some programmers treat test flakiness is weird. With other types of bug programmers want to fix it. With flakiness they either want to rerun the test until it passes or tear it down and write an entirely different type of test - as if it is in fact not a bug, but some immutable fact of life.
RHSeeger 1y ago
Integration tests and Unit tests are different tools; and each has their place and purpose. Using one "instead" of the other is a mistake.
- MrJohz 1y ago
  I've never really found this to be the case in practice. When I look at well-written unit tests and well-written integration tests, they're usually doing exactly the same sort of thing and have very similar concerns in terms of code organisation and test structure. For example, in both cases, the tests work best if I test the subject under test as a black box (i.e. interact only with its public interface) but use my knowledge of its internals to identify the weaknesses that will most require testing. In both cases, I want to structure the code so that the subject under test is as isolated as possible - i.e. no complex interactions with global state, no mocking of unrelated modules, and no complex mechanism to reset anything after the test is done. In both cases, I want the test to run fast, ideally instantaneously, so I get immediate results. The biggest difference is that it's usually harder to write good integration tests because they're interacting with external systems that are generally slower and stateful, so I've got to put extra work into getting the tests themselves to be fast and stateless. But when that works, there's really not much difference at all between a test that tests a single function, and a test that tests a service class with a database dependency.
  - RHSeeger 1y ago
    I'll go with a bank account, because that was one of the initial examples for automated testing. I would write integration/system (different, but similar, imo) to test that the black box integrations with the system work as expected. Generally closer to the "user story" end of things. I would write integration tests for smaller, targeted thing. Like making sure the sort method works in various cases, etc. Individual methods, especially ones that don't interact with data outside what is passed into them (functional methods), are good for unit testing.
    - 9rx 1y ago
      > to test that the black box integrations with the system work as expected. Generally closer to the "user story" end of things. This is what unit testing was originally described as. Which confirms my belief that unit testing and integration testing has always been the very same thing. > Individual methods, especially ones that don't interact with data outside what is passed into them (functional methods), are good for unit testing. Perhaps unit testing has come to mean this, but these kinds of tests are rarely ever worth writing, so it is questionable if it even needs a name. Sometimes it can be helpful to isolate a function like that for the sake of pinning down complex logic or edge cases, but is likely you'll want to delete this kind of test once you're done. This is where testing brittleness is born.
yeswecatan 1y ago
I find testing terminology very confusing and inconsistent. Personally, I prefer tests that cover multiple components. Is that an integration test because you test multiple components? What if you system is designed in such a way that these tests are _fast_ because the data access is abstracted away and you can use in memory repositories instead of hitting the database?
- jessekv 1y ago
  I think it's relative, right? That's how abstractions and interfaces work. I can write a module with integration tests at the module level and unit tests on its functions. I can now write an application that uses my module. From the perspective of my application, my module's integration tests look like unit tests. My module might, for example, implicitly depend on the test suite of CPython, the C compiler, the QA at the chip fab. But I don't need to run those tests any more. In your case you hope the in-memory database matches the production one enough that you can write fast isolated unit tests on your application logic. You can trust this works because something else unit-tested the in-memory database, and integration tested the db client against the various db backends.
- creesch 1y ago
  > I find testing terminology very confusing and inconsistent. That's because it is both confusing and inconsistent. In my experience, every company uses slightly different names for different types of tests. Unit tests are generally fairly well understood as testing the single unit (a method/function) but after that things get murky fast. For example, integration tests as reflected by the confused conversation in this thread already has wildly different definitions depending on who you ask. For example, someone might interpret them as "unit integration tests" where it reflects a test that tests a class, builder, etc. Basically something where a few units are combined. But, in some companies I have seen these being called "component tests". Then there is the word "functional tests" which in some companies means the same as "manual tests done by QA" but for others simply means automated front-end tests. But in yet other companies those automated tests are called end 2 end tests. What's interesting to me when viewing these online discussions is the complete lack of awareness people display about this. You will see people very confidently say that "test X should by done in such and such way" in response to someone where it is very clear they are actually talking about different types of tests.
ashishb 1y ago
What you are asking for is to write tests along the axis of least change https://ashishb.net/programming/bad-and-good-ways-to-write-automated-tests/ https://ashishb.net/programming/bad-and-good-ways-to-write-a...
3036e4 1y ago
I remember reading blogs (and Testing on the Toilet) around 2010 about how Google divided tests into Small/Medium/Large, with specific definitions, rather than trying to use more vague and overloaded terminology that no one ever agreed on. Seems like they are no longer doing that? Too bad, since I think it was a clever trick to avoid having to get into pointless discussions about things like "what is a unit?". Having experienced more than one project where a unit test was uselessly defined to "have to only run a single method, everything else must be mocked" I like the idea of not having any level of tests below "small" (that is still above a level most would call "unit"). Found this long 2011 post now that goes into some detail on the background and the reasons for introducing that ("The Testing Grouplet"?): https://mike-bland.com/2011/11/01/small-medium-large.html https://mike-bland.com/2011/11/01/small-medium-large.html But I am not sure even after reading all that if the SML terminology was still used in 2011 or if they had moved on already? Can't really find any newer sources that mention it.
BinaryIgor 1y ago
Was exactly about to point that out - if you mostly have integration (aka in-between tests) tests, you rarely need to refactor your tests. It's about testing mostly at the right abstraction level: https://binaryigor.com/unit-integration-e2e-contract-x-tests-what-should-we-focus-on.html https://binaryigor.com/unit-integration-e2e-contract-x-tests...
- RaftPeople 1y ago
  Agree. I don't think people realize the stats based on studies: Unit testing catches about 30% of bugs Visual code inspection catches about 70% End to end testing also catches about 70% All are important, but emphasis should be on the more effective methods. > We’ve had decades of thought leadership around testing I really disagree that this is the case that the industry has had thought leadership. I think we've had people pushing automated unit testing very hard when end to end is more effective. I don't think the position was based on facts but more of a few people's opinion.
deleted 1y ago
[deleted]
strogonoff 1y ago
If there is one single test-related thing you must have, that would be e2e testing. Integration tests are, in a way, worst of both worlds: they are more complicated than unit tests, they require involved setup, and yet they can’t really guarantee that things work in production. End-to-end tests, meanwhile, do show whether things work or not. If something fails with an error, error reporting should be good enough in the first place to show you what exactly is wrong. If something failed without an error but you know it failed, make it fail with an error first by writing another test case. If there was an error but error reporting somehow doesn’t capture it, you have a bigger problem than tests. At the end of the day, you want certainty that you deliver working software. If it’s too difficult to identify the failure, improve your error reporting system. Giving up that certainty because your error reporting is not good enough seems like a bad tradeoff. Incidentally, grug-friendly e2e tests absolutely exist: just take your software, exactly as it’s normally built, and run a script that uses it like it would be used in production. This gives you a good enough guarantee that it works. If there is no script, just do it yourself, go through a checklist, write a script later. It doesn’t get more grug than that.
- dsego 1y ago
  E2e tests are the hardest to maintain and take a lot of time for little benefit in my experience. I'm talking about simulating a browser to open pages and click on buttons. They are flaky and brittle, the UI is easily the component which gets updated the most often, it's also easy to manually test while developing during QA and UAT. It's hard to mock out things, so you either have to bootstrap or maintain a whole 2nd working system with all the bells and whistles, including authentication, users, real data in the database, 3rd party integrations etc. It's just too overwhelming for little benefit. It's also hard to cover all error cases to see if a thing works correctly or breaks subtly. Most commonly in e2e we test for the happy path just to see that the thing doesn't fall over.
  - strogonoff 1y ago
    The benefit is certainty that the system you are building and delivering to people works. If that benefit is little, then I don’t quite understand the point of testing. > it's also easy to manually test while developing during QA and UAT. As I said in the original comment, e2e tests can definitely be manual. Invoke your CLI, curl your API, click around in GUI. That said, comprehensively testing it that way quickly becomes infeasible as your software grows.
mstigge 1y ago
[dead]
barrkel 1y ago
Yes. When testing a component, use its public API, inject real implementations where you can, and use fakes where it's too expensive. Don't use mocks, don't test interfaces that have very complex setups to invoke, if possible.
riehwvfbk 1y ago
Yes, do this. No unit tests, integration only. Your competitors will thank you for this blunder. Why is it a blunder? Well, you just slowed down your edit-compile-run cycle by about 10x, and debugging when things go wrong (and it's when, not if) by 100-1000 times depending on the complexity of your environment. Perhaps the answer is "AI will fix it", but we aren't there yet.
- recursivedoubts 1y ago
  idk i used this approach and done ok competitively (e.g. htmx, me, one dude, vs react, facebook & vercel) and, as grugbrain.dev says, unit test tend to lock you in to a particular implementation integration tests don't need to be slow on modern hardware, are easier to debug than end-to-end if they are kept at the right level of abstraction and catch more real-world bugs than overly-specialized unit tests w/ complicated mocks, etc

MathMonkeyMan 1y ago

Integration tests at $DAY_JOB are often slow (sleeps, retries, inadequate synchronization, startup and shut down 8 processes that are slow to start and stop), flaky (the metrics for this rate limiter should be within 5%, this should be true within 3 seconds, the output of this shell command is the same on all platforms), undocumented, and sometimes cannot be run locally or with locally available configurations. When I run a set of integration tests associated with some code I'm modifying, I have no idea what they are, why they were written, what they do, how long they will take to run, or whether I should take failures seriously. Integration tests are closer to what you want to know, but they're also more. If I want to make sure that my state machine returns an error when it receives a message for which no state transition is defined, I could spin up a process and set up log collection and orchestrate with python and... or I could write a unit test that instantiates a state machine, gives it a message, and checks the result. My point is that we need both. Write a unit test to ensure that your component behaves to its spec, especially with respect to edge cases. Write an integration test to make sure that the feature of which your component is a part behaves as expected.

skydhash 1y ago
My current mental model is a car. If it’s a function or some other things you’re fully confident you captured the domain, add unit tests to capture that. Just like an engine. But the most important is integration tests that couple something like the engine, the ignition system and test that when the user press the start button, the engine start and the dashboard light up. Unit tests are great for DX but only integration and above tests matter business wise.
majormajor 1y ago
You need to test contracts with external code without having to include full external systems. Unit tests on internal implementation details are fragile as behavior changes. Unit tests on your module's contracts give you confidence in refactoring. Passing params in instead of making external calls inside your business logic functions can help. DI can help if that's too impractical or unwieldy for whatever reason in the domain. It's hard to do right the first time - sometimes its fuzzy what's an internal detail vs what's an external contract - but you need to get there ASAP.
lenkite 1y ago
These are "system tests" at your $DAY_JOB, not "integration tests".
- MathMonkeyMan 1y ago
  There is an even higher level of testing that they call "system tests." Those are for things like tracking performance regressions. I think there's always a spectrum among the terms "system," "integration," and "unit."
MoreQARespect 1y ago
The way some programmers treat test flakiness is weird. With other types of bug programmers want to fix it. With flakiness they either want to rerun the test until it passes or tear it down and write an entirely different type of test - as if it is in fact not a bug, but some immutable fact of life.

RHSeeger 1y ago

Integration tests and Unit tests are different tools; and each has their place and purpose. Using one "instead" of the other is a mistake.

MrJohz 1y ago
I've never really found this to be the case in practice. When I look at well-written unit tests and well-written integration tests, they're usually doing exactly the same sort of thing and have very similar concerns in terms of code organisation and test structure. For example, in both cases, the tests work best if I test the subject under test as a black box (i.e. interact only with its public interface) but use my knowledge of its internals to identify the weaknesses that will most require testing. In both cases, I want to structure the code so that the subject under test is as isolated as possible - i.e. no complex interactions with global state, no mocking of unrelated modules, and no complex mechanism to reset anything after the test is done. In both cases, I want the test to run fast, ideally instantaneously, so I get immediate results. The biggest difference is that it's usually harder to write good integration tests because they're interacting with external systems that are generally slower and stateful, so I've got to put extra work into getting the tests themselves to be fast and stateless. But when that works, there's really not much difference at all between a test that tests a single function, and a test that tests a service class with a database dependency.
- RHSeeger 1y ago
  I'll go with a bank account, because that was one of the initial examples for automated testing. I would write integration/system (different, but similar, imo) to test that the black box integrations with the system work as expected. Generally closer to the "user story" end of things. I would write integration tests for smaller, targeted thing. Like making sure the sort method works in various cases, etc. Individual methods, especially ones that don't interact with data outside what is passed into them (functional methods), are good for unit testing.
  - 9rx 1y ago
    > to test that the black box integrations with the system work as expected. Generally closer to the "user story" end of things. This is what unit testing was originally described as. Which confirms my belief that unit testing and integration testing has always been the very same thing. > Individual methods, especially ones that don't interact with data outside what is passed into them (functional methods), are good for unit testing. Perhaps unit testing has come to mean this, but these kinds of tests are rarely ever worth writing, so it is questionable if it even needs a name. Sometimes it can be helpful to isolate a function like that for the sake of pinning down complex logic or edge cases, but is likely you'll want to delete this kind of test once you're done. This is where testing brittleness is born.

yeswecatan 1y ago

I find testing terminology very confusing and inconsistent. Personally, I prefer tests that cover multiple components. Is that an integration test because you test multiple components? What if you system is designed in such a way that these tests are _fast_ because the data access is abstracted away and you can use in memory repositories instead of hitting the database?

jessekv 1y ago
I think it's relative, right? That's how abstractions and interfaces work. I can write a module with integration tests at the module level and unit tests on its functions. I can now write an application that uses my module. From the perspective of my application, my module's integration tests look like unit tests. My module might, for example, implicitly depend on the test suite of CPython, the C compiler, the QA at the chip fab. But I don't need to run those tests any more. In your case you hope the in-memory database matches the production one enough that you can write fast isolated unit tests on your application logic. You can trust this works because something else unit-tested the in-memory database, and integration tested the db client against the various db backends.
creesch 1y ago
> I find testing terminology very confusing and inconsistent. That's because it is both confusing and inconsistent. In my experience, every company uses slightly different names for different types of tests. Unit tests are generally fairly well understood as testing the single unit (a method/function) but after that things get murky fast. For example, integration tests as reflected by the confused conversation in this thread already has wildly different definitions depending on who you ask. For example, someone might interpret them as "unit integration tests" where it reflects a test that tests a class, builder, etc. Basically something where a few units are combined. But, in some companies I have seen these being called "component tests". Then there is the word "functional tests" which in some companies means the same as "manual tests done by QA" but for others simply means automated front-end tests. But in yet other companies those automated tests are called end 2 end tests. What's interesting to me when viewing these online discussions is the complete lack of awareness people display about this. You will see people very confidently say that "test X should by done in such and such way" in response to someone where it is very clear they are actually talking about different types of tests.

ashishb 1y ago

What you are asking for is to write tests along the axis of least change https://ashishb.net/programming/bad-and-good-ways-to-write-automated-tests/ https://ashishb.net/programming/bad-and-good-ways-to-write-a...

3036e4 1y ago

I remember reading blogs (and Testing on the Toilet) around 2010 about how Google divided tests into Small/Medium/Large, with specific definitions, rather than trying to use more vague and overloaded terminology that no one ever agreed on. Seems like they are no longer doing that? Too bad, since I think it was a clever trick to avoid having to get into pointless discussions about things like "what is a unit?". Having experienced more than one project where a unit test was uselessly defined to "have to only run a single method, everything else must be mocked" I like the idea of not having any level of tests below "small" (that is still above a level most would call "unit"). Found this long 2011 post now that goes into some detail on the background and the reasons for introducing that ("The Testing Grouplet"?): https://mike-bland.com/2011/11/01/small-medium-large.html https://mike-bland.com/2011/11/01/small-medium-large.html But I am not sure even after reading all that if the SML terminology was still used in 2011 or if they had moved on already? Can't really find any newer sources that mention it.

BinaryIgor 1y ago

Was exactly about to point that out - if you mostly have integration (aka in-between tests) tests, you rarely need to refactor your tests. It's about testing mostly at the right abstraction level: https://binaryigor.com/unit-integration-e2e-contract-x-tests-what-should-we-focus-on.html https://binaryigor.com/unit-integration-e2e-contract-x-tests...

RaftPeople 1y ago
Agree. I don't think people realize the stats based on studies: Unit testing catches about 30% of bugs Visual code inspection catches about 70% End to end testing also catches about 70% All are important, but emphasis should be on the more effective methods. > We’ve had decades of thought leadership around testing I really disagree that this is the case that the industry has had thought leadership. I think we've had people pushing automated unit testing very hard when end to end is more effective. I don't think the position was based on facts but more of a few people's opinion.

deleted 1y ago

[deleted]

strogonoff 1y ago

If there is one single test-related thing you must have, that would be e2e testing. Integration tests are, in a way, worst of both worlds: they are more complicated than unit tests, they require involved setup, and yet they can’t really guarantee that things work in production. End-to-end tests, meanwhile, do show whether things work or not. If something fails with an error, error reporting should be good enough in the first place to show you what exactly is wrong. If something failed without an error but you know it failed, make it fail with an error first by writing another test case. If there was an error but error reporting somehow doesn’t capture it, you have a bigger problem than tests. At the end of the day, you want certainty that you deliver working software. If it’s too difficult to identify the failure, improve your error reporting system. Giving up that certainty because your error reporting is not good enough seems like a bad tradeoff. Incidentally, grug-friendly e2e tests absolutely exist: just take your software, exactly as it’s normally built, and run a script that uses it like it would be used in production. This gives you a good enough guarantee that it works. If there is no script, just do it yourself, go through a checklist, write a script later. It doesn’t get more grug than that.

dsego 1y ago
E2e tests are the hardest to maintain and take a lot of time for little benefit in my experience. I'm talking about simulating a browser to open pages and click on buttons. They are flaky and brittle, the UI is easily the component which gets updated the most often, it's also easy to manually test while developing during QA and UAT. It's hard to mock out things, so you either have to bootstrap or maintain a whole 2nd working system with all the bells and whistles, including authentication, users, real data in the database, 3rd party integrations etc. It's just too overwhelming for little benefit. It's also hard to cover all error cases to see if a thing works correctly or breaks subtly. Most commonly in e2e we test for the happy path just to see that the thing doesn't fall over.
- strogonoff 1y ago
  The benefit is certainty that the system you are building and delivering to people works. If that benefit is little, then I don’t quite understand the point of testing. > it's also easy to manually test while developing during QA and UAT. As I said in the original comment, e2e tests can definitely be manual. Invoke your CLI, curl your API, click around in GUI. That said, comprehensively testing it that way quickly becomes infeasible as your software grows.

mstigge 1y ago

[dead]

barrkel 1y ago

Yes. When testing a component, use its public API, inject real implementations where you can, and use fakes where it's too expensive. Don't use mocks, don't test interfaces that have very complex setups to invoke, if possible.

riehwvfbk 1y ago

Yes, do this. No unit tests, integration only. Your competitors will thank you for this blunder. Why is it a blunder? Well, you just slowed down your edit-compile-run cycle by about 10x, and debugging when things go wrong (and it's when, not if) by 100-1000 times depending on the complexity of your environment. Perhaps the answer is "AI will fix it", but we aren't there yet.

recursivedoubts 1y ago
idk i used this approach and done ok competitively (e.g. htmx, me, one dude, vs react, facebook & vercel) and, as grugbrain.dev says, unit test tend to lock you in to a particular implementation integration tests don't need to be slow on modern hardware, are easier to debug than end-to-end if they are kept at the right level of abstraction and catch more real-world bugs than overly-specialized unit tests w/ complicated mocks, etc