6 ms·

Our Rails test suite runs in 1 minute on Buildkite

57 points by mschoening 5y ago

blue_box 5y ago
Would be nice if they shared how many tests they have :)
- pmontra 5y ago
  Indeed. The only hint is "Our whole test suite locally takes around 12 minutes running serially on a MacBook Pro" so it should be a real world case. They used 64 cores on Buildkite.
- byroot 5y ago
  In the grand scheme of things it doesn't matter that much. Once you starts parallelizing, CI time is: setup_time + (test_run_time / workers), so assuming money isn't a problem you can add more workers as you keep adding more tests. What really matter is how fast you can setup your test workers and how slow individual tests are.
  - Ataraxic 5y ago
    I think your last statement captures something that not often emphasized, which is that startup time of workers and app are variable costs which scale with the amount of parallelization and so if you have a very large test suite there may be a limit where setup eats up so much time that parallelizing more is going to be pretty "wasteful" and I guess that's where the money comes in. Secondly if you're actually trying to startup 100+ test workers per build and so on there's going to be some time distribution for how long it takes for each worker to startup and that adds a bit more time for all workers to complete. This distribution probably isn't _that_ wide timewise but if you really start to push your test suite runtime down it may pop up. If you're running things in docker sometimes a node doesn't have the image in it's docker cache... Unsure if CI services like buildkite have really made this that much faster but it seems like they are using a single box with 64 cores.
    - byroot 5y ago
      > Unsure if CI services like buildkite have really made this that much faster Buildkite doesn't directly help with it, but since you bring your own hardware and that it's highly customizable, it does allow you to invest in improving setup time quite dramatically. It's a great product.
iqanq 5y ago
- exdsq 5y ago
  If you're not programming in Rust and Julia, are you really programming? /s
  - bnt 5y ago
    I thought HN crowd was more into Nim or Zig?
    - devoutsalsa 5y ago
      Nah, it's all about MUMPS.
      - aaronbrethorst 5y ago
        I hear Verona is beautiful this time of year.
        bnt 5y ago
        Canada is indeed beautiful
        aaronbrethorst 5y ago
        (Epic Systems is probably the biggest user of MUMPS in the world, and is headquartered in Verona, Wisconsin.)
    - ksec 5y ago
      Hell No. Zig is still a relatively new. But if you search Rust on HN, you still see it wins by a very wide margin.
    - exdsq 5y ago
      Nim is 14 years old, why don’t you just recommend FORTRAN you graybeard?
inetknght 5y ago
One minute? That's terrible. What kinds of tests and how many are you running? > We never run all of our application’s tests in local development. It’s not a good use of time and will never be as fast as running them on CI. WTF this is bad practice. Developers should always get in the habit of running tests locally first. No test should require running in the cloud. And, even more, it's actually rare that CI is more performant than your developers' machines. You should invest in your developers. > Our whole test suite locally takes around 12 minutes running serially on a MacBook Pro. That's really really really awful. Most projects' tests, even when they're poorly written, still run in 3 or 4 minutes. My own tests run on the order of single-digit seconds. > We haven’t put much effort here because it’s not something our engineers ever run. Well of course not. They can't iterate fast if they're slowed down by tests. That's why tests have to be fast! > This gave us some speed gains, but we wanted it really fast. Our infrastructure team set us up with some 64 core machines Holy schnapple if I had 64 core machines, a complete test suite would finish on the order of tens of milliseconds! I'm glad ya'll have gotten your tests down from 12 minutes to 1 minutes. There's a ton more improving you can do though.
- hdjjhhvvhga 5y ago
  How can you tell that if you don't know the complexity of their projects?
  - pskinner 5y ago
    This is because RoR projects do not do unit testing, they do integration testing with a real DB. This approach is terrible and causes the poor performance. There are whole sets of testing "tools" for RoR projects that promote this mechanism of testing - if you're testing an API you need to use an external tool to properly hit that API. If you're writing unit tests, write unit tests with mocked dependencies.
    - pmontra 5y ago
      It's hard to really test business logic without hitting the database or other components, no matter the language and the framework. You really have to trust your mocks and sometimes you discover only in production that one of them didn't do what it should do. It happened to me. The teams I work with tend to write unit tests to demonstrate that a class or module or whatever work, then integration tests to demonstrate that that part of the application works from the frontend code down to the database.
renewiltord 5y ago
Nice simple article describing a technique to parallelize tests and a quick fix to an issue blocking it. It is interesting to observe the time that things take in various environments. I worked in ad tech previously with lots of offline big-data jobs. Test completion took a few minutes even fully parallelized. Now I work in HFT and our test suite completes on an M1 Max in 21 s and it's got higher coverage. The situations aren't really comparable. But it's interesting to see the different baselines.
reillyse 5y ago
"We never run all of our application’s tests in local development. It’s not a good use of time and will never be as fast as running them on CI. " Actually I've just released something for that. It's a way of running from your local environment extremely quickly (spoiler alert we use non-local servers). You can check it out at https://brisktest.com/ https://brisktest.com/ we have a demo running Jest at the moment but I'm recording a rails demo right now.
- moltar 5y ago
  How do I sign up?
rubyist5eva 5y ago
Our Rails test suite runs in about 20 minutes (mostly integration tests) but parallelized on CircleCI with 3 workers it's down to ~6. If I run our entire suite locally on my 8-core macbook I get about 3 minutes but I can't do much with my laptop during that time because my system is monopolized, a colleague of mine runs it on a threadripper and it's sub-2 minutes. So running on Circle is well worth it, I can go top off my coffee, make a snack, answer emails, etc...

57 points by mschoening 5y ago

blue_box 5y ago

Would be nice if they shared how many tests they have :)

pmontra 5y ago
Indeed. The only hint is "Our whole test suite locally takes around 12 minutes running serially on a MacBook Pro" so it should be a real world case. They used 64 cores on Buildkite.
byroot 5y ago
In the grand scheme of things it doesn't matter that much. Once you starts parallelizing, CI time is: setup_time + (test_run_time / workers), so assuming money isn't a problem you can add more workers as you keep adding more tests. What really matter is how fast you can setup your test workers and how slow individual tests are.
- Ataraxic 5y ago
  I think your last statement captures something that not often emphasized, which is that startup time of workers and app are variable costs which scale with the amount of parallelization and so if you have a very large test suite there may be a limit where setup eats up so much time that parallelizing more is going to be pretty "wasteful" and I guess that's where the money comes in. Secondly if you're actually trying to startup 100+ test workers per build and so on there's going to be some time distribution for how long it takes for each worker to startup and that adds a bit more time for all workers to complete. This distribution probably isn't _that_ wide timewise but if you really start to push your test suite runtime down it may pop up. If you're running things in docker sometimes a node doesn't have the image in it's docker cache... Unsure if CI services like buildkite have really made this that much faster but it seems like they are using a single box with 64 cores.
  - byroot 5y ago
    > Unsure if CI services like buildkite have really made this that much faster Buildkite doesn't directly help with it, but since you bring your own hardware and that it's highly customizable, it does allow you to invest in improving setup time quite dramatically. It's a great product.

iqanq 5y ago

exdsq 5y ago
If you're not programming in Rust and Julia, are you really programming? /s
- bnt 5y ago
  I thought HN crowd was more into Nim or Zig?
  - devoutsalsa 5y ago
    Nah, it's all about MUMPS.
    - aaronbrethorst 5y ago
      I hear Verona is beautiful this time of year.
      - bnt 5y ago
        Canada is indeed beautiful
        aaronbrethorst 5y ago
        (Epic Systems is probably the biggest user of MUMPS in the world, and is headquartered in Verona, Wisconsin.)
  - ksec 5y ago
    Hell No. Zig is still a relatively new. But if you search Rust on HN, you still see it wins by a very wide margin.
  - exdsq 5y ago
    Nim is 14 years old, why don’t you just recommend FORTRAN you graybeard?

inetknght 5y ago

One minute? That's terrible. What kinds of tests and how many are you running? > We never run all of our application’s tests in local development. It’s not a good use of time and will never be as fast as running them on CI. WTF this is bad practice. Developers should always get in the habit of running tests locally first. No test should require running in the cloud. And, even more, it's actually rare that CI is more performant than your developers' machines. You should invest in your developers. > Our whole test suite locally takes around 12 minutes running serially on a MacBook Pro. That's really really really awful. Most projects' tests, even when they're poorly written, still run in 3 or 4 minutes. My own tests run on the order of single-digit seconds. > We haven’t put much effort here because it’s not something our engineers ever run. Well of course not. They can't iterate fast if they're slowed down by tests. That's why tests have to be fast! > This gave us some speed gains, but we wanted it really fast. Our infrastructure team set us up with some 64 core machines Holy schnapple if I had 64 core machines, a complete test suite would finish on the order of tens of milliseconds! I'm glad ya'll have gotten your tests down from 12 minutes to 1 minutes. There's a ton more improving you can do though.

hdjjhhvvhga 5y ago
How can you tell that if you don't know the complexity of their projects?
- pskinner 5y ago
  This is because RoR projects do not do unit testing, they do integration testing with a real DB. This approach is terrible and causes the poor performance. There are whole sets of testing "tools" for RoR projects that promote this mechanism of testing - if you're testing an API you need to use an external tool to properly hit that API. If you're writing unit tests, write unit tests with mocked dependencies.
  - pmontra 5y ago
    It's hard to really test business logic without hitting the database or other components, no matter the language and the framework. You really have to trust your mocks and sometimes you discover only in production that one of them didn't do what it should do. It happened to me. The teams I work with tend to write unit tests to demonstrate that a class or module or whatever work, then integration tests to demonstrate that that part of the application works from the frontend code down to the database.

renewiltord 5y ago

Nice simple article describing a technique to parallelize tests and a quick fix to an issue blocking it. It is interesting to observe the time that things take in various environments. I worked in ad tech previously with lots of offline big-data jobs. Test completion took a few minutes even fully parallelized. Now I work in HFT and our test suite completes on an M1 Max in 21 s and it's got higher coverage. The situations aren't really comparable. But it's interesting to see the different baselines.

reillyse 5y ago

"We never run all of our application’s tests in local development. It’s not a good use of time and will never be as fast as running them on CI. " Actually I've just released something for that. It's a way of running from your local environment extremely quickly (spoiler alert we use non-local servers). You can check it out at https://brisktest.com/ https://brisktest.com/ we have a demo running Jest at the moment but I'm recording a rails demo right now.

moltar 5y ago
How do I sign up?

rubyist5eva 5y ago

Our Rails test suite runs in about 20 minutes (mostly integration tests) but parallelized on CircleCI with 3 workers it's down to ~6. If I run our entire suite locally on my 8-core macbook I get about 3 minutes but I can't do much with my laptop during that time because my system is monopolized, a colleague of mine runs it on a threadripper and it's sub-2 minutes. So running on Circle is well worth it, I can go top off my coffee, make a snack, answer emails, etc...