3 ms·

Why did you build this? Any particular motivating experience? At $WORK, we use Jest to run each test suite in its own process. The places we struggle with isola

by jitl 2y ago

Why did you build this? Any particular motivating experience? At $WORK, we use Jest to run each test suite in its own process. The places we struggle with isolation are for integration tests that read/write from the DB, so I’m curious what kinds of issues motivated container-per-test. Are you going to sell it somehow? Does it work nested inside Docker? I already have a CI setup that stripes batches of tests across worker containers.

nfachan 2y ago
At my previous company, we had a lot of tests. They were a mix of C and Python. Running all of them on a single machine took on the order of an hour or more. Even just limiting the tests run to those that could theoretically be affected by your change could take minutes or even tens of minutes. We ended up building a shared cluster of ~1000 cores that was available to all developers, and that was used by CI. This changed our developers' workflows quite a bit. It was now possible to run large amounts of tests regularly: like every few minutes instead of a once or twice a day. This in turn encouraged developers to write more tests and do more test-driven development. On top of that, having the cluster available provided other benefits. If a test was flakey, it was easy to run it tens or even hundreds of thousands of times, making it easy to reproduce and identify the bug. We also occasionally did Monte Carlo simulations, and it was really handy to have a lot of cores available for general developer use. I got used to working that way and I've missed it since I left that company. So this project is an attempt to make a more general-purpose implementation of that system. I hope others will find similar workflows that make them more productive using this system or something like it. Regarding the container-per-test idea. It really comes about because it's the obvious way to package up jobs to submit them to a cluster. Plus, it makes tests reproducible for all developers in a project, and between developer machines and CI. Using Linux namespaces, the overhead of running tests in individual containers isn't much more than running tests in individual processes.
nfachan 2y ago
I forgot to answer your two other questions. Maelstrom is open source and we plan to keep it that way. We may look at ways of selling access to a hosted cluster as service. Test running is very elastic, and could benefit from having an elastic service to support it. Maelstrom is completely root-less, so it'll work inside of Docker just fine. We regularly test Maelstrom within Maelstrom.

by jitl 2y ago

nfachan 2y ago

At my previous company, we had a lot of tests. They were a mix of C and Python. Running all of them on a single machine took on the order of an hour or more. Even just limiting the tests run to those that could theoretically be affected by your change could take minutes or even tens of minutes. We ended up building a shared cluster of ~1000 cores that was available to all developers, and that was used by CI. This changed our developers' workflows quite a bit. It was now possible to run large amounts of tests regularly: like every few minutes instead of a once or twice a day. This in turn encouraged developers to write more tests and do more test-driven development. On top of that, having the cluster available provided other benefits. If a test was flakey, it was easy to run it tens or even hundreds of thousands of times, making it easy to reproduce and identify the bug. We also occasionally did Monte Carlo simulations, and it was really handy to have a lot of cores available for general developer use. I got used to working that way and I've missed it since I left that company. So this project is an attempt to make a more general-purpose implementation of that system. I hope others will find similar workflows that make them more productive using this system or something like it. Regarding the container-per-test idea. It really comes about because it's the obvious way to package up jobs to submit them to a cluster. Plus, it makes tests reproducible for all developers in a project, and between developer machines and CI. Using Linux namespaces, the overhead of running tests in individual containers isn't much more than running tests in individual processes.

nfachan 2y ago

I forgot to answer your two other questions. Maelstrom is open source and we plan to keep it that way. We may look at ways of selling access to a hosted cluster as service. Test running is very elastic, and could benefit from having an elastic service to support it. Maelstrom is completely root-less, so it'll work inside of Docker just fine. We regularly test Maelstrom within Maelstrom.