6 ms·

> No more need for mocks or complicated environment configurations. Define your test dependencies as code, then simply run your tests and containers will be cre

> No more need for mocks or complicated environment configurations. Define your test dependencies as code, then simply run your tests and containers will be created and then deleted. Wait what? They think you don't need unit tests because you can run integration tests with containers? It's trivial to set up a docker container with one of your dependencies, but starting containers is painful and slow.

tomnipotent 3y ago
No mocks doesn't mean no tests. It means running tests against the full code path which includes requests to running instances of the services you might otherwise mock. For many apps and use cases, the overhead in managing container state is worth it.
- ants_everywhere 3y ago
  > It means running tests against the full code path which includes requests to running instances of the services you might otherwise mock. Yeah, those are called end to end tests and you run them after integration tests which you run after unit tests. It sounds to me like they're saying just skip to the end to end tests. > For many apps and use cases, the overhead in managing container state is worth it. Yeah, and typically you'd run them after you run unit and integration tests. If I have 10 libraries to test that have database access, I have to run 10 database containers simultaneously every few minutes as part of the development process? That's overkill.
  - deleted 3y ago
    [deleted]
  - lmm 3y ago
    > Yeah, and typically you'd run them after you run unit and integration tests. If I have 10 libraries to test that have database access, I have to run 10 database containers simultaneously every few minutes as part of the development process? That's overkill. If it's actually causing you problems, then by all means replace some of them with more lightweight tests, at the cost of some test environment faithfulness. But don't optimise prematurely.
sverhagen 3y ago
1) At least in the Java world, the term "unit testing" is often confused by "things you do in JUnit", which runs both "pure" unit tests and project-level integration tests, i.e. spinning up an application context (like Spring) and testing against real REST endpoints etc. 2) While unit tests are cheaper and quicker than (project-level) integration tests, they also in many cases don't provide results as good a result and level of confidence, because a lot of run-time aspects (serialization, HTTP responses, database responses, etc.) are not as straightforward to mock. There's been some noise about The Testing Trophy, instead of the Testing Pyramid where, in short, there are still unit tests where it makes sense, but a lot of testing has moved to the (project-level) integration tests. These are slower, but only by so much that the trade-off is often worth it. Whether it's worth it, depends heavily on what you're testing. If it's a CRUD API: I use integration tests. If it's something algorithmic, or string manipulation, etc.: I use unit tests. When I saw the Testing Trophy presented, it came with the asterisk that (project-level) integration testing has gotten easier and cheaper over time, thus allowing a shift in trade-off. Testcontainers is one of the primary reasons why this shift has happened. (And... I respect that it's not for everyone.) Some references: https://kentcdodds.com/blog/the-testing-trophy-and-testing-classifications https://kentcdodds.com/blog/the-testing-trophy-and-testing-c... https://www.youtube.com/watch?v=t-HmXomntCA https://www.youtube.com/watch?v=t-HmXomntCA
- ants_everywhere 3y ago
  Yeah, certainly I see the value of those kinds of tests. And clearly as you say the simpler tests don't provide as realistic a simulation as the more expensive tests. But on the test philosophy angle, my take on what's happening is just that developers traditionally look for any reason to skip tests. I've seen this in a few different forms. - right now containers make it trivial to run all of your dependencies. That's much easier than creating a mock or a fake, so we do that and don't bother creating a mock/fake. - compiler folks have created great static analysis tools. That's easier than writing a bunch of tests, so we'll just assume static analysis will catch our bugs for us. - <my language>'s types system does a bunch of work type checking, so I don't need tests. Or maybe I just need randomly generated property tests. - no tests can sufficiently emulate our production environment, so tests are noise and we'll work out issues in dev and prod. What I've noticed, though, is that looking across a wide number of software projects is there's a clear difference in quality between projects that have a strong testing discipline and those that convince themselves they don't need tests because of <containers, or types, or whatever else>. Sure it's possible that tests don't cause the quality difference (maybe there's a third factor for example that causes both). And of course if you have limited resources you have to make a decision about which quality assurance steps to cut. But personally I respect a project more if they just say they don't have the bandwidth to test properly so they're just skipping to the integration stage (or whatever) rather than convince themselves that those tests weren't important any way. Because I've seen so many projects that would have been much better with even a small number of unit tests where they only had integration tests.
zer00eyz 3y ago
I have been doing E2E testing exclusively for close to a decade on several apps and it works great. Note, not integration, E2E. I can go from bare vm to fully tested system in under 15 minutes. I can re run that test in 1-5 (depending on project) ... Im creating 100's of records in that time, and fuzzing a lot of data entry. I could get it to go "even faster" if I went in and removed some of the stepwise testing... A->B->C->D could be broken out to a->b, a->c, a->d. Because my tests are external, they would be durable across a system re-write (if I need to change language, platform etc). They can also be re-used/tweeked to test system perf under load (something unit tests could never do).

tomnipotent 3y ago

No mocks doesn't mean no tests. It means running tests against the full code path which includes requests to running instances of the services you might otherwise mock. For many apps and use cases, the overhead in managing container state is worth it.

ants_everywhere 3y ago
> It means running tests against the full code path which includes requests to running instances of the services you might otherwise mock. Yeah, those are called end to end tests and you run them after integration tests which you run after unit tests. It sounds to me like they're saying just skip to the end to end tests. > For many apps and use cases, the overhead in managing container state is worth it. Yeah, and typically you'd run them after you run unit and integration tests. If I have 10 libraries to test that have database access, I have to run 10 database containers simultaneously every few minutes as part of the development process? That's overkill.
- deleted 3y ago
  [deleted]
- lmm 3y ago
  > Yeah, and typically you'd run them after you run unit and integration tests. If I have 10 libraries to test that have database access, I have to run 10 database containers simultaneously every few minutes as part of the development process? That's overkill. If it's actually causing you problems, then by all means replace some of them with more lightweight tests, at the cost of some test environment faithfulness. But don't optimise prematurely.

sverhagen 3y ago

1) At least in the Java world, the term "unit testing" is often confused by "things you do in JUnit", which runs both "pure" unit tests and project-level integration tests, i.e. spinning up an application context (like Spring) and testing against real REST endpoints etc. 2) While unit tests are cheaper and quicker than (project-level) integration tests, they also in many cases don't provide results as good a result and level of confidence, because a lot of run-time aspects (serialization, HTTP responses, database responses, etc.) are not as straightforward to mock. There's been some noise about The Testing Trophy, instead of the Testing Pyramid where, in short, there are still unit tests where it makes sense, but a lot of testing has moved to the (project-level) integration tests. These are slower, but only by so much that the trade-off is often worth it. Whether it's worth it, depends heavily on what you're testing. If it's a CRUD API: I use integration tests. If it's something algorithmic, or string manipulation, etc.: I use unit tests. When I saw the Testing Trophy presented, it came with the asterisk that (project-level) integration testing has gotten easier and cheaper over time, thus allowing a shift in trade-off. Testcontainers is one of the primary reasons why this shift has happened. (And... I respect that it's not for everyone.) Some references: https://kentcdodds.com/blog/the-testing-trophy-and-testing-classifications https://kentcdodds.com/blog/the-testing-trophy-and-testing-c... https://www.youtube.com/watch?v=t-HmXomntCA https://www.youtube.com/watch?v=t-HmXomntCA

ants_everywhere 3y ago
Yeah, certainly I see the value of those kinds of tests. And clearly as you say the simpler tests don't provide as realistic a simulation as the more expensive tests. But on the test philosophy angle, my take on what's happening is just that developers traditionally look for any reason to skip tests. I've seen this in a few different forms. - right now containers make it trivial to run all of your dependencies. That's much easier than creating a mock or a fake, so we do that and don't bother creating a mock/fake. - compiler folks have created great static analysis tools. That's easier than writing a bunch of tests, so we'll just assume static analysis will catch our bugs for us. - <my language>'s types system does a bunch of work type checking, so I don't need tests. Or maybe I just need randomly generated property tests. - no tests can sufficiently emulate our production environment, so tests are noise and we'll work out issues in dev and prod. What I've noticed, though, is that looking across a wide number of software projects is there's a clear difference in quality between projects that have a strong testing discipline and those that convince themselves they don't need tests because of <containers, or types, or whatever else>. Sure it's possible that tests don't cause the quality difference (maybe there's a third factor for example that causes both). And of course if you have limited resources you have to make a decision about which quality assurance steps to cut. But personally I respect a project more if they just say they don't have the bandwidth to test properly so they're just skipping to the integration stage (or whatever) rather than convince themselves that those tests weren't important any way. Because I've seen so many projects that would have been much better with even a small number of unit tests where they only had integration tests.

zer00eyz 3y ago

I have been doing E2E testing exclusively for close to a decade on several apps and it works great. Note, not integration, E2E. I can go from bare vm to fully tested system in under 15 minutes. I can re run that test in 1-5 (depending on project) ... Im creating 100's of records in that time, and fuzzing a lot of data entry. I could get it to go "even faster" if I went in and removed some of the stepwise testing... A->B->C->D could be broken out to a->b, a->c, a->d. Because my tests are external, they would be durable across a system re-write (if I need to change language, platform etc). They can also be re-used/tweeked to test system perf under load (something unit tests could never do).