3 ms·

> it really isn't worth it in the long term to take on a new external dependency for a bit of code that (1) is a critical part of the team's development and rel

by jchw 3y ago

> it really isn't worth it in the long term to take on a new external dependency for a bit of code that (1) is a critical part of the team's development and release process and (2) can be written in-house in maybe an hour. This seems to be quite a contradiction. If it's so easy to just write from scratch, then why would it be scary to depend on? Of course, it's not that easy to write from scratch. You could make a proof-of-concept in maybe an hour... Maybe. But they already took the proof of concept to a complete phase. Made it work with Podman. Added tons of integration code to make it easy to use with many common services. Ported it to several different languages. And, built a community around it. If you do this from scratch, you have to go through most of the effort and problems they already did, except if you write your own solution, you have to maintain it right from the git-go, whereas if you choose Testcontainers, you'll only wind up having to maintain it if the project is left for dead and starts to bitrot. The Docker API is pretty stable though, so honestly, this doesn't seem likely to be a huge issue. Testcontainers is exactly the sort of thing open source is great for; it's something where everyone gets to benefit from the wisdom and battle-testing of everyone else. For most of the problems you might run into, there is a pretty decent chance someone already did, so there's a pretty decent chance it's already been fixed. Most people have GitHub and Dockerhub dependencies in their critical dependency path for builds and deployment. Services go down, change their policies, deprecate APIs, and go under, but code continues to work if you replicate the environment it originally worked in. The biggest risk with code dependencies (for non-production code like test code) is usually that it blocks you from updating some other software. The biggest risk with services is that they completely disappear and you are completely blocked until you fully remove the dependency. I think people depending on Testcontainers are fine and doing very well with their risk analysis.

TheDong 3y ago
> This seems to be quite a contradiction. If it's so easy to just write from scratch, then why would it be scary to depend on? It's easy to write an implementation specific to your existing project's development environment and workflow correctly. It's hard to write a generic version that works in any environment. It's hard to write a version that lets you build a company and make money. It's scary to depend on this generic version because it's too generic, and it's built by a for-profit company now who wants to upsell you to some "testcontainer cloud" crap, which doesn't exactly incentivize them to make the OSS version perfect. For example, we were already using bazel, so writing a version using bazel to create a correct bare rootfs for a dependency + using runc to execute it in tests resulted in something with a roughly 3ms warm startup time that fulfilled all our needs, and cached correctly (since bazel has good caching). A junior engineer, starry-eyed at test-containers, switched some code over to it, and the warm startup time went up by 1000x from 3ms to 3s, as did the flake-rate, and docker's caching is far worse than bazel's so the several minute cold-starts also happened even more often. > Testcontainers is exactly the sort of thing open source is great for; it's something where everyone gets to benefit from the wisdom and battle-testing of everyone else. You get to have a mismatched mess of solutions to everyone's problems, including solution's to problems you don't have. You get code of the quality of the average OSS programmer which, while higher than the average programmer, is still pretty crap. Free Software is great when it's run by a small group of smart opinionated people. As soon as you try to build a company around that OSS and, god forbid, hire some enterprise sales people and product managers, it quickly becomes worse at the actual small developer problem than what even a mediocre developer could hack out in an hour. Depending on testcontainers is fine, but if you know what you're doing, writing something purpose-built is fine too, and probably gives you something much nicer in the end.
- jchw 3y ago
  To be honest, it doesn't sound like you really had a good use case for Testcontainers anyways. Where it excels the most is in just pulling in some external containers, especially databases, e.g. PostgreSQL and Redis, directly in your test harness. In those cases, it works well.
  - TheDong 3y ago
    Our use-case was redis servers, one of your supposedly good use-cases for testcontainers. I didn't mention, but the testcontainer code also preferred to add that network io to the actual test runtime, which made measuring the test's performance harder, and meant we couldn't as safely cache the test's results. The bazel version made it easy to build the test's dependency as part of the test compilation process (as it should be) so the test runtime didn't have to do external network IO. "Excels" is also a stretch; we had a few hundred tests launching dedicated redis servers, and with hand-rolled code, that worked fine with zero flakes. With testcontainers, it regularly flaked with some opaque docker network error or sometimes just plain a timeout because apparently launching 100s of containers in parallel is a hard problem for docker or something. I'm sure it works well for some people, but if those people wanted to build out their own version without docker and testcontainers, specific to their development tooling and environment, it would probably work better in most cases I think.
    - jchw 3y ago
      Yeah, I couldn't tell you: I didn't have serious issues with performance or reliability when I used this. It didn't really seem like it was doing anything special, it seemed like it is just starting a Docker container like you'd expect, at least on Linux. I sincerely doubt I would've had any better experience handrolling it since my solution to handroll it would also be to use the Docker API; the reason for this is for portability.
Too 3y ago
When writing something like this for yourself you would only support one database, one programming language, one unit testing framework, much less documentation and none of the edge cases that someone else requires. The effort would not be equal to replicating the whole project. It wouldn’t be the same quality of course. Question is do you need all of it or not.
- jchw 3y ago
  Yeah, but that's kind of the thing, you get less battle testing most likely for also less surface area. If you pull in a project like this, it makes it vastly easier when you need more, because if it happens to already have built-in support, you don't need to spend additional engineer time on it. To me it seems straightforwardly a win to start with Testcontainer and move to something else when it proves insufficient.
paxys 3y ago
This thought process taken to the extreme is what results in half the internet depending on leftPad and isOdd to function. Open source is great, but there is a difference between providing a utility vs adding needless layers of abstraction. It is the responsibility of the developer to figure out which is which rather than reach for the package manager for every task in front of them. In this case, like I said earlier Docker already has a RESTful API. You don't need to replicate the entire feature set of Testcontainers, just make a couple of HTTP calls in your language of choice to achieve the exact same outcome.
- jchw 3y ago
  That's just a slippery slope argument. I am not arguing for people to depend on leftPad, I'm arguing that depending on a library like this which is probably tens of thousands of lines of tested code that does roughly what you want already is a damn good idea. Again, I don't want to handroll a solution and have to test it out across Windows, Mac, Linux and then check again to make sure it works with both Docker and Podman, and then again to check AArch64 Windows, AArch64 Mac and AArch64 Linux, and so on. I'd prefer to use a library. And hell, even if any of those things don't work, I can report it to the maintainers and possibly contribute a fix, and then everyone else gets to benefit too.

by jchw 3y ago

TheDong 3y ago

> This seems to be quite a contradiction. If it's so easy to just write from scratch, then why would it be scary to depend on? It's easy to write an implementation specific to your existing project's development environment and workflow correctly. It's hard to write a generic version that works in any environment. It's hard to write a version that lets you build a company and make money. It's scary to depend on this generic version because it's too generic, and it's built by a for-profit company now who wants to upsell you to some "testcontainer cloud" crap, which doesn't exactly incentivize them to make the OSS version perfect. For example, we were already using bazel, so writing a version using bazel to create a correct bare rootfs for a dependency + using runc to execute it in tests resulted in something with a roughly 3ms warm startup time that fulfilled all our needs, and cached correctly (since bazel has good caching). A junior engineer, starry-eyed at test-containers, switched some code over to it, and the warm startup time went up by 1000x from 3ms to 3s, as did the flake-rate, and docker's caching is far worse than bazel's so the several minute cold-starts also happened even more often. > Testcontainers is exactly the sort of thing open source is great for; it's something where everyone gets to benefit from the wisdom and battle-testing of everyone else. You get to have a mismatched mess of solutions to everyone's problems, including solution's to problems you don't have. You get code of the quality of the average OSS programmer which, while higher than the average programmer, is still pretty crap. Free Software is great when it's run by a small group of smart opinionated people. As soon as you try to build a company around that OSS and, god forbid, hire some enterprise sales people and product managers, it quickly becomes worse at the actual small developer problem than what even a mediocre developer could hack out in an hour. Depending on testcontainers is fine, but if you know what you're doing, writing something purpose-built is fine too, and probably gives you something much nicer in the end.

jchw 3y ago
To be honest, it doesn't sound like you really had a good use case for Testcontainers anyways. Where it excels the most is in just pulling in some external containers, especially databases, e.g. PostgreSQL and Redis, directly in your test harness. In those cases, it works well.
- TheDong 3y ago
  Our use-case was redis servers, one of your supposedly good use-cases for testcontainers. I didn't mention, but the testcontainer code also preferred to add that network io to the actual test runtime, which made measuring the test's performance harder, and meant we couldn't as safely cache the test's results. The bazel version made it easy to build the test's dependency as part of the test compilation process (as it should be) so the test runtime didn't have to do external network IO. "Excels" is also a stretch; we had a few hundred tests launching dedicated redis servers, and with hand-rolled code, that worked fine with zero flakes. With testcontainers, it regularly flaked with some opaque docker network error or sometimes just plain a timeout because apparently launching 100s of containers in parallel is a hard problem for docker or something. I'm sure it works well for some people, but if those people wanted to build out their own version without docker and testcontainers, specific to their development tooling and environment, it would probably work better in most cases I think.
  - jchw 3y ago
    Yeah, I couldn't tell you: I didn't have serious issues with performance or reliability when I used this. It didn't really seem like it was doing anything special, it seemed like it is just starting a Docker container like you'd expect, at least on Linux. I sincerely doubt I would've had any better experience handrolling it since my solution to handroll it would also be to use the Docker API; the reason for this is for portability.

Too 3y ago

When writing something like this for yourself you would only support one database, one programming language, one unit testing framework, much less documentation and none of the edge cases that someone else requires. The effort would not be equal to replicating the whole project. It wouldn’t be the same quality of course. Question is do you need all of it or not.

jchw 3y ago
Yeah, but that's kind of the thing, you get less battle testing most likely for also less surface area. If you pull in a project like this, it makes it vastly easier when you need more, because if it happens to already have built-in support, you don't need to spend additional engineer time on it. To me it seems straightforwardly a win to start with Testcontainer and move to something else when it proves insufficient.

paxys 3y ago

This thought process taken to the extreme is what results in half the internet depending on leftPad and isOdd to function. Open source is great, but there is a difference between providing a utility vs adding needless layers of abstraction. It is the responsibility of the developer to figure out which is which rather than reach for the package manager for every task in front of them. In this case, like I said earlier Docker already has a RESTful API. You don't need to replicate the entire feature set of Testcontainers, just make a couple of HTTP calls in your language of choice to achieve the exact same outcome.

jchw 3y ago
That's just a slippery slope argument. I am not arguing for people to depend on leftPad, I'm arguing that depending on a library like this which is probably tens of thousands of lines of tested code that does roughly what you want already is a damn good idea. Again, I don't want to handroll a solution and have to test it out across Windows, Mac, Linux and then check again to make sure it works with both Docker and Podman, and then again to check AArch64 Windows, AArch64 Mac and AArch64 Linux, and so on. I'd prefer to use a library. And hell, even if any of those things don't work, I can report it to the maintainers and possibly contribute a fix, and then everyone else gets to benefit too.