20 ms·

Spark joy by running fewer tests

147 points by caution 6y ago

erulabs 6y ago
As a long time DevOps engineer and now founder - my perspective on tests has really gone thru a rollercoaster. In the past life, I’d regularly be the guy rejecting deployments and asking for additional tests - barking at developers who ignored failures, lecturing on about the sanity saving features of a good integration test. These days? Well the headlong rush to release features and fixes is a real thing. Ditching some tests in favor of manual verification is a good example of YC’s advice “do things that don’t scale”. I add tests when the existential fear notches up - but not a whole lot before that. Like with almost all topics in software development - the longer I’m in the field - the less intense my opinions become. The right answer: “eh, tests would be nice here! Let’s get to that once the customer is happy!”
- commandlinefan 6y ago
  > Ditching some tests in favor of manual verification I think that's the biggest problem with (unit) testing: there's too much "either test absolutely everything, or don't (unit) test anything at all" thinking. You can save yourself a lot of hassle if you just stick to writing tests for the things that are relatively straightforward to test: stay away from anything that requires an external system or anything that runs in parallel (multithreading, for example), and leave that stuff for manual testing. A big benefit of having reasonable code coverage that I don't often see discussed is that if you have a unit test for each function in the system, you can also run any function without recompiling and redeploying the entire app: when you've isolated the source of a problem, this capability can be a godsend in allowing you to iterate quickly to narrow down the cause. I invariably find myself stuck working with systems for which unit tests were never developed, so there's no way to test a change besides recompiling the whole thing, redeploying a whole environment, waiting for the whole thing to start up, firing up the UI, navigating to the problem spot, and finally testing what I was after in the first place.
  - cellularmitosis 6y ago
    > "You can save yourself a lot of hassle if you just stick to writing tests for the things that are relatively straightforward to test" Yup. Focus most of your effort on testing your pure functions. If you've followed functional core / imperative shell, typically what's left in the shell isn't very interesting from a testing perspective. And if the majority of your codebase is still in the imperative shell, well, time to get to work :) https://www.destroyallsoftware.com/talks/boundaries https://www.destroyallsoftware.com/talks/boundaries
- tick_tock_tick 6y ago
  Some people view testing as a goal in and of itself rather than serving to make the product better. I think a lot of people start off there career in the similar place to you testing is needed because testing is good and slowly move towards a more realistic or big picture view of the role of testing.
bigmanwalter 6y ago
When it comes to testing, I now follow the advice of Gary Bernhardt's presentation, Functional Core, Imperative Shell: https://www.destroyallsoftware.com/screencasts/catalog/functional-core-imperative-shell https://www.destroyallsoftware.com/screencasts/catalog/funct... The idea is to move all the logic of your app into pure functions that can be tested on the order of milliseconds. When you refactor your code to allow for this, everything just makes more sense. Tests can be run more often and you are far more confident about the behaviour of your code.
- mceachen 6y ago
  It's excellent advice, but you still need to test that the wiring of all those functions works as well. As an example, PhotoStructure has over 4,000 tests which test isolated functions: classical "unit" tests, which run under a minute. There are only a few hundred integration tests, but those take a couple minutes. All of these tests are run on every commit across all supported OSes using GitLab's CI and self-hosted runners.
- 8192kjshad09- 6y ago
  Maybe I just don't get it, but this design seems impractical for systems that use the network alot. I work on an app whose basic structure is 1. Receive data from a webhook (not controlled by me) 2. Make some HTTP requests based on data from 1 3. Send data to another HTTP API The core of the logic is in step 2. How can I possibly have a functional core here? Step 2s behavior completely depends on the HTTP responses.
  - bigmanwalter 6y ago
    Pardon my pseudo-JavaScript, but given an event handler like this: const handler = ( message, data, req=superagent.post ) => { if (message === “a”) return req(“/api/a”, data) if (message === “b”) return {label:”b”, value: req(“/api/b”)} } By supplying the http client using the default argument on line 4, the client can be replaced when testing with a mock handler: test(“handler requests /api/a”, t=> t.assertDeepEqual( handler(“a”, {b:”c”}, (url, body)=>[url, body]) [“/api/a”, {b:”c”}] ) ) test(“handler requests /api/b”, t=> t.assertDeepEqual( handler(“b”, {b:”c”}, (url, body)=>[url, body]) {label:”b”, value:[“/api/b”, {b:”c”}]} ) ) You could also write test cases that handle multiple steps, complex transformations, and so on. It’s really nice being able to specify plainly the inputs and expected outputs of your functions and testing against that! It may not guarantee that everything behaves as the real system but it will catch 100% of your logic errors.
hospadar 6y ago
We just did pretty much the exact same thing with a large in-house ETL application. We can do great static analysis of the dependencies of different jobs and the coverage of various tests. Most PRs are now running a tiny fraction of the test suite (thousands of tests total) in minutes instead of an hour. Run all test cases before deployment of master just in case we missed something.
- yawaramin 6y ago
  Was there internal resistance to this, and if so what was it?
jwalton 6y ago
I wonder if they considered using something like ptrace to track which .rb files a given test suite loads? This would probably be orders of magnitude faster than Rotoscope or Tracepoint. You'd get a much "coarser" data set, since you wouldn't know which individual tests called which modules (unless you run each test case in it's own Ruby runtime). On the upside, you'd be able to watch JSON and YAML files.
- byroot 6y ago
  > I wonder if they considered using something like ptrace to track which .rb files a given test suite loads? It's not really possible as we eager load the entire application on CI. And even if we didn't to trace depedencies like this we'd need to boot the application in lazy load mode for each test suite, that alone would be slower than running the entire test suite.
mdoms 6y ago
> Unfortunately, one can’t fully eradicate intermittently failing tests, Oh boy do I disagree with this. I have a zero tolerance policy for flaky tests. In code bases where I have the authority, I immediately remove flaky tests and notify the relevant teams. If you let flaky tests fester - or worse, hack together test re-runners and flaky reporters - they will erode trust in your test suite. "Just re-run the build" will become a common refrain, and hours are wasted performing re-runs instead of tracking down the actual problems causing the red tests.
- tick_tock_tick 6y ago
  The biggest thing is anything time based the effort to mock out of fake time is normally way larger than just dealing with a test that flakes from time to time. That being said I make a concentrated effort to take in time rather then randomly calling now() in the middle of my code but most code bases aren't green fields.
  - TallGuyShort 6y ago
    It is my personal opinion that in that first statement you are severely underestimating the time and product quality that is eventually lost because of a flaky test that isn't dealt with quickly.
- quicklime 6y ago
  I generally agree, but there's one thing that bothers me about the practice: Before disabling the test, does anyone even consider the possibility that maybe it's not the test that's flaky, but that the software under test is itself buggy in a non-deterministic way? I don't have a good answer for how to deal with this. It would be an enormous amount of effort to go through every flaky result and reason about whether the problem is with the test or the code under test, especially in larger codebases. Maybe the heuristic of assuming that it's the test's fault is good enough, and this is what I've done in every team I've worked in. But it does bother me.
  - wyldfire 6y ago
mehrdadn 6y ago
Does anyone have thoughts on whether test suites suffer from Goodhart's law? Sometimes I feel like they only work well if people assume they don't exist and commit accordingly.
- jwalton 6y ago
  My dad was a huge proponent of this back when he was in software; don't tell you developers what tests failed, only what percentage of tests failed. A software test is, in many ways, like an exam you might write in university; just like an exam can't possibly cover 100% of what you're supposed to know, a test (especially an integration test for a large and complex system) can't possibly cover 100% of the conditions the system is supposed to operate under. An exam is a good way to measure if you know a subject though, and similarly a test suite is a good way to check that the quality of a system meets a certain bar. Once you write the exam, if I go back and say "Here are all the questions you got wrong. Go study up and write the same exam tomorrow," though, it very much ceases to be a good measure of whether or not you know the subject matter. You can now "cheat the system" by studying only the parts of the subject that the exam covers. Similarly, once your integration tests are failing, if someone tells you which tests are failing and how, you're going to go back and fix only what you need to get the tests passing. At this point, the tests stop being a good indication of code quality - 100% of the tests are passing, but you can't say that 100% of the defects have been removed, so the tests are, in a sense, now kind of worthless. They might stop a limited number of future defects getting in, but they're not doing an arguably much more important job of telling you what your overall quality level is. If instead, when you submit a commit, I say "5% of the tests are now failing" and nothing else, you have to go look for defects in your code, and you're probably going to find a lot of defects on your own before you even get to the 5% that the tests are complaining about.
  - mdoms 6y ago
    This sounds like a fun game to play with a team of developers who have no time constraints. In every organisation I've worked with you would get a very stern talking to for behaving like this. My tests are specifically designed to show you where the defect is, so you can solve the immediate problem and get back to work. I don't expect every developer who triggered a failing test to perform a full analysis of the code base and resolve every other defect. That would be nice, if we had the time.
wwright 6y ago
Interesting that they don't talk about Bazel. Isn't skipping tests like this one of its biggest selling points, particularly for monorepo users?
- spankalee 6y ago
  Exactly. The first thing they should dispense with is the idea that they should run _all_ the tests in the monorepo. That's what doesn't scale. Run only the effected tests and the overhead is now more proportional the potential impact of the change.
  - wwright 6y ago
    That seemed to be what they were describing, but with dynamic dependency detection via introspection.
- yawaramin 6y ago
  Does Shopify use Bazel? I would have guessed not. In any case, the more natural fit for them would be Nix, since it should be good at eliminiating redundant builds, and they recently went all-in on it.
  - wwright 6y ago
    AFAIK Bazel and Nix are orthogonal and actually go well together. Nix is great for packaging and environments, but is anemic at actual build tooling (a big one: no incremental builds within a package). Bazel does very little to manage the environment or packages, but it has very granular and efficient build support.
    - yawaramin 6y ago
      True, but they both require a rather large complexity and learning and complexity budget. It would be difficult to introduce both into a shop at the same time.
umaar 6y ago
When it comes to browser automation tests, everywhere I've worked at suffered from intermittently failing tests. When I was at the Ministry of Justice (UK), I configured CircleCI to run tests hundreds of times [1] (over a number of days) through cron jobs. This allowed me to reflect on all test results, and find out what failed most often and eventually solve those root causes. This strategy worked well. Interestingly enough, just today I posted a GitHub thread [2] and asked the community to 'thumbs up' the video course they'd like for me to create. "Learn Browser Automation" is currently the highest voted. If it's the one I end up making, a huge focus will absolutely be on: How to reduce test flakiness with headless browsers. Words of advice, avoiding sleep() and other brittle bits of code will help. But in addition, run your tests frequently to catch out the flakiness early. Invest in tooling which helps you diagnose failing tests (screenshots, devtools trace dumps). Configure something like VNC so you can remotely connect to the machine running the test. [1] https://github.com/ministryofjustice/fb-automated-tests/blob/7c9cee58db902419abf5449aaef6e91e575502d9/.circleci/config.yml#L52 https://github.com/ministryofjustice/fb-automated-tests/blob... [2] https://github.com/umaar/dev-tips-tracker/issues/33 https://github.com/umaar/dev-tips-tracker/issues/33
hideo 6y ago
I wonder if Marie Kondo-ing your tests is a good idea in general. This article reminded me a lot of "Write tests. Not too many. Mostly integration" https://kentcdodds.com/blog/write-tests/ https://kentcdodds.com/blog/write-tests/ and other such articles for limiting testing, like this test diamond article http://toddlittleweb.com/wordpress/2014/06/23/the-testing-diamond-and-the-pyramid-2/ http://toddlittleweb.com/wordpress/2014/06/23/the-testing-di... I've seen far too many unit tests at this point that just assumed too much to be meaningful, and conversations about unit testing quickly devolve into "no true scotsman" style arguments about what is truly a unit and what isn't.
- yawaramin 6y ago
  A unit test is a test that fails in a way that immediately points to the failure cause.
- sidlls 6y ago
  Kent's article isn't so much about limiting the test coverage so much as preferring one kind (integration) over another (unit). He's right. Tons of unit testing in application codebases is a huge, stinking code smell in my experience. Usually it means the code is overly abstract (interfaces everywhere for "mockability") or unnecessarily complex (e.g. conditional expressions factored out into functions called in one site just so they can be tested separately) or both.
lazyant 6y ago
> Before the feature was rolled out, many developers were skeptical that it would work. Frankly, I was one of them. Kudos for the Skepticism section. Every "how we fixed problem X at company Y" should have this section, and esp. written by someone who opposed the solution. The challenging and battle stories tend to be the most interesting part, at least for me.
flukus 6y ago
And here's a makefile implementation with a statically typed language: #run test and output to a .testresult file when the test is modified %.testresult: %.test $< > $@ 2>&1 #depends on a .testresult file for each test test: $(ALL_TEST_RESULTS) cat $(ALL_TEST_RESULTS) Sure the dynamic typing introduces many problems, but surely you could have something a bit more half way like "%.testresult: %.test $(DEPENENDENCIES) $(METAPROGRAMMING_MAGIC_DEPENDENCIES)", then only hopefully rare changes to the core files require the full test suite to be rerun. Seems like dynamic typing is the core of their problem though and this is a crazy complicated solution to try and work around that.
- jolmg 6y ago
  > $< 2&1> $@ I'm not really well versed in Makefiles, but that seems to pass the argument "2" to the testing program, launching it in the background with "&", and then doing a redirection for the stdout of an empty command to the target file with "1>". I think you meant $< > $@ 2>&1 which could also be $< &> $@ I can't see how your example relates to static or dynamic typing, though.
  - flukus 6y ago
    Fixed, the former is correct, the latter I think is a bashism that may not work. Unfortunately I plucked that from a project in a very broken state. > I can't see how your example relates to static or dynamic typing, though. Because with static typing you know the object/file graph at compile time and only affected tests need to be run. In that example the .test files will only be built when their dependencies change and .testresult will only be created if .test is built, everything is incremental and "make test" won't execute any tests if there are no changes that could affect them.
    - joshuamorton 6y ago
      You know this in dynamic languages too, based on imports. So if you have the build graph represented for your dynamic language, you get the same result. This is just a poor (poor, poor) buggy implementation of Bazel.
overgard 6y ago
I wonder if the tests are even serving a purpose at that point? If they can't reliably answer the question "did I accidentally break something" in a reasonable period of time what's the point?
m12k 6y ago
I think this shows how counting on horizontal scaling to handle inefficiency only works for a while, and will eventually introduce its own set of complexities - and dealing with those might be more trouble than coding things more efficiently in the first place. Then again, maybe it's worth it for a huge company like Shopify, because they save work for the feature devs by adding extra load on the test infrastructure devs, effectively increasing developer parallelism. Finally I wonder if they could have done more to speed up their tests? I'm maintaining a Rails codebase too, and I cut test time down by two thirds by rewriting tests with efficiency in mind - e.g. by ignoring dogma like 'each test needs to run completely independently of previous ones' (if I verify the state between tests, do I really need to pay the performance penalty of a database wipe?) The test-prof gem has a great tool, let-it-be, that allows you to designate db state that should not get wiped between tests. That and focusing on more unit tests and fewer integration test has really gone a long way toward speeding things up again for me.
simon_000666 6y ago
> has over 150,000 tests > takes about 30-40 min to run on hundreds of docker containers in parallel This seems like it might be a signal that now could be the time to start splitting services out in an SOA fashion and having their own test suites and some contract driven tests? Having to run that many tests on each commit is definitely a smell that something architecturally fundamental is wrong...
Seb-C 6y ago
My experience is that most of the time, randomly failing tests are actually failing because they were made to be random. Sure, adding Faker to your test/mock data may help you to find rare edge cases. The problem is that those edge cases will be triggered in the future, in another context and by another developer. So not only it probably won't be fixed, but it will be a waste of time and annoyance for someone. Same thing for some option like the `random` setting in Jasmine (runs your unit tests in a different random order each time). So now I only have tests with static, predictable data. Having tests that depends on the state of the previous one (or are affected by it) is quite common too. And not always easy to fix properly. Once this is removed, the remaining random failures are usually authentic and helpful.
davewritescode 6y ago
The approach they use to apply this to a large Ruby project is interesting but this type of strategy has been in use since forever and at least to me, seems fairly obvious. Running all the tests with every build is always a bad idea. A better approach that doesn't require fancy dynamic analysis is to organize tests in a way that it's clear what's likely to break and to make sure you're constantly running your test suite in QA environments. Making a change to a module should force you to run that module's test suite. Interactions between modules can be tested all day in a loop and monitored before deploying to production.
darksaints 6y ago
This sort of thing is why I always scoff at the idea of slow compile times for static strongly typed languages (eg. Scala, Haskell, OCaml, Rust, etc). Sure, it definitely compiles slower than some other compiled language, and there is no compilation for an interpreted language. But if you factor in testing, the type systems of those languages can easily remove a massive amount of testing that other less rigorous languages would either a) not test at all, or b) test at a significant cost. I'd be willing to bet that if shopify were using a strongly typed language, 50-90% of their testing would be completely redundant, because the type system already takes care of it. That isn't to say that there are not reasons to use dynamically typed languages...just that if you are building production systems in strongly typed languages, compile time is almost completely irrelevant as a factor in productivity, regardless of how much slower they compile in comparison to an alternative.

147 points by caution 6y ago

As a long time DevOps engineer and now founder - my perspective on tests has really gone thru a rollercoaster. In the past life, I’d regularly be the guy rejecting deployments and asking for additional tests - barking at developers who ignored failures, lecturing on about the sanity saving features of a good integration test. These days? Well the headlong rush to release features and fixes is a real thing. Ditching some tests in favor of manual verification is a good example of YC’s advice “do things that don’t scale”. I add tests when the existential fear notches up - but not a whole lot before that. Like with almost all topics in software development - the longer I’m in the field - the less intense my opinions become. The right answer: “eh, tests would be nice here! Let’s get to that once the customer is happy!”

commandlinefan 6y ago
> Ditching some tests in favor of manual verification I think that's the biggest problem with (unit) testing: there's too much "either test absolutely everything, or don't (unit) test anything at all" thinking. You can save yourself a lot of hassle if you just stick to writing tests for the things that are relatively straightforward to test: stay away from anything that requires an external system or anything that runs in parallel (multithreading, for example), and leave that stuff for manual testing. A big benefit of having reasonable code coverage that I don't often see discussed is that if you have a unit test for each function in the system, you can also run any function without recompiling and redeploying the entire app: when you've isolated the source of a problem, this capability can be a godsend in allowing you to iterate quickly to narrow down the cause. I invariably find myself stuck working with systems for which unit tests were never developed, so there's no way to test a change besides recompiling the whole thing, redeploying a whole environment, waiting for the whole thing to start up, firing up the UI, navigating to the problem spot, and finally testing what I was after in the first place.
- cellularmitosis 6y ago
  > "You can save yourself a lot of hassle if you just stick to writing tests for the things that are relatively straightforward to test" Yup. Focus most of your effort on testing your pure functions. If you've followed functional core / imperative shell, typically what's left in the shell isn't very interesting from a testing perspective. And if the majority of your codebase is still in the imperative shell, well, time to get to work :) https://www.destroyallsoftware.com/talks/boundaries https://www.destroyallsoftware.com/talks/boundaries
tick_tock_tick 6y ago
Some people view testing as a goal in and of itself rather than serving to make the product better. I think a lot of people start off there career in the similar place to you testing is needed because testing is good and slowly move towards a more realistic or big picture view of the role of testing.

bigmanwalter 6y ago

When it comes to testing, I now follow the advice of Gary Bernhardt's presentation, Functional Core, Imperative Shell: https://www.destroyallsoftware.com/screencasts/catalog/functional-core-imperative-shell https://www.destroyallsoftware.com/screencasts/catalog/funct... The idea is to move all the logic of your app into pure functions that can be tested on the order of milliseconds. When you refactor your code to allow for this, everything just makes more sense. Tests can be run more often and you are far more confident about the behaviour of your code.

mceachen 6y ago
It's excellent advice, but you still need to test that the wiring of all those functions works as well. As an example, PhotoStructure has over 4,000 tests which test isolated functions: classical "unit" tests, which run under a minute. There are only a few hundred integration tests, but those take a couple minutes. All of these tests are run on every commit across all supported OSes using GitLab's CI and self-hosted runners.
8192kjshad09- 6y ago
Maybe I just don't get it, but this design seems impractical for systems that use the network alot. I work on an app whose basic structure is 1. Receive data from a webhook (not controlled by me) 2. Make some HTTP requests based on data from 1 3. Send data to another HTTP API The core of the logic is in step 2. How can I possibly have a functional core here? Step 2s behavior completely depends on the HTTP responses.
- bigmanwalter 6y ago
  Pardon my pseudo-JavaScript, but given an event handler like this: const handler = ( message, data, req=superagent.post ) => { if (message === “a”) return req(“/api/a”, data) if (message === “b”) return {label:”b”, value: req(“/api/b”)} } By supplying the http client using the default argument on line 4, the client can be replaced when testing with a mock handler: test(“handler requests /api/a”, t=> t.assertDeepEqual( handler(“a”, {b:”c”}, (url, body)=>[url, body]) [“/api/a”, {b:”c”}] ) ) test(“handler requests /api/b”, t=> t.assertDeepEqual( handler(“b”, {b:”c”}, (url, body)=>[url, body]) {label:”b”, value:[“/api/b”, {b:”c”}]} ) ) You could also write test cases that handle multiple steps, complex transformations, and so on. It’s really nice being able to specify plainly the inputs and expected outputs of your functions and testing against that! It may not guarantee that everything behaves as the real system but it will catch 100% of your logic errors.

hospadar 6y ago

We just did pretty much the exact same thing with a large in-house ETL application. We can do great static analysis of the dependencies of different jobs and the coverage of various tests. Most PRs are now running a tiny fraction of the test suite (thousands of tests total) in minutes instead of an hour. Run all test cases before deployment of master just in case we missed something.

yawaramin 6y ago
Was there internal resistance to this, and if so what was it?

jwalton 6y ago

I wonder if they considered using something like ptrace to track which .rb files a given test suite loads? This would probably be orders of magnitude faster than Rotoscope or Tracepoint. You'd get a much "coarser" data set, since you wouldn't know which individual tests called which modules (unless you run each test case in it's own Ruby runtime). On the upside, you'd be able to watch JSON and YAML files.

byroot 6y ago
> I wonder if they considered using something like ptrace to track which .rb files a given test suite loads? It's not really possible as we eager load the entire application on CI. And even if we didn't to trace depedencies like this we'd need to boot the application in lazy load mode for each test suite, that alone would be slower than running the entire test suite.

mdoms 6y ago

> Unfortunately, one can’t fully eradicate intermittently failing tests, Oh boy do I disagree with this. I have a zero tolerance policy for flaky tests. In code bases where I have the authority, I immediately remove flaky tests and notify the relevant teams. If you let flaky tests fester - or worse, hack together test re-runners and flaky reporters - they will erode trust in your test suite. "Just re-run the build" will become a common refrain, and hours are wasted performing re-runs instead of tracking down the actual problems causing the red tests.

tick_tock_tick 6y ago
The biggest thing is anything time based the effort to mock out of fake time is normally way larger than just dealing with a test that flakes from time to time. That being said I make a concentrated effort to take in time rather then randomly calling now() in the middle of my code but most code bases aren't green fields.
- TallGuyShort 6y ago
  It is my personal opinion that in that first statement you are severely underestimating the time and product quality that is eventually lost because of a flaky test that isn't dealt with quickly.
quicklime 6y ago
I generally agree, but there's one thing that bothers me about the practice: Before disabling the test, does anyone even consider the possibility that maybe it's not the test that's flaky, but that the software under test is itself buggy in a non-deterministic way? I don't have a good answer for how to deal with this. It would be an enormous amount of effort to go through every flaky result and reason about whether the problem is with the test or the code under test, especially in larger codebases. Maybe the heuristic of assuming that it's the test's fault is good enough, and this is what I've done in every team I've worked in. But it does bother me.
- wyldfire 6y ago

mehrdadn 6y ago

Does anyone have thoughts on whether test suites suffer from Goodhart's law? Sometimes I feel like they only work well if people assume they don't exist and commit accordingly.

jwalton 6y ago
My dad was a huge proponent of this back when he was in software; don't tell you developers what tests failed, only what percentage of tests failed. A software test is, in many ways, like an exam you might write in university; just like an exam can't possibly cover 100% of what you're supposed to know, a test (especially an integration test for a large and complex system) can't possibly cover 100% of the conditions the system is supposed to operate under. An exam is a good way to measure if you know a subject though, and similarly a test suite is a good way to check that the quality of a system meets a certain bar. Once you write the exam, if I go back and say "Here are all the questions you got wrong. Go study up and write the same exam tomorrow," though, it very much ceases to be a good measure of whether or not you know the subject matter. You can now "cheat the system" by studying only the parts of the subject that the exam covers. Similarly, once your integration tests are failing, if someone tells you which tests are failing and how, you're going to go back and fix only what you need to get the tests passing. At this point, the tests stop being a good indication of code quality - 100% of the tests are passing, but you can't say that 100% of the defects have been removed, so the tests are, in a sense, now kind of worthless. They might stop a limited number of future defects getting in, but they're not doing an arguably much more important job of telling you what your overall quality level is. If instead, when you submit a commit, I say "5% of the tests are now failing" and nothing else, you have to go look for defects in your code, and you're probably going to find a lot of defects on your own before you even get to the 5% that the tests are complaining about.
- mdoms 6y ago
  This sounds like a fun game to play with a team of developers who have no time constraints. In every organisation I've worked with you would get a very stern talking to for behaving like this. My tests are specifically designed to show you where the defect is, so you can solve the immediate problem and get back to work. I don't expect every developer who triggered a failing test to perform a full analysis of the code base and resolve every other defect. That would be nice, if we had the time.

wwright 6y ago

Interesting that they don't talk about Bazel. Isn't skipping tests like this one of its biggest selling points, particularly for monorepo users?

spankalee 6y ago
Exactly. The first thing they should dispense with is the idea that they should run _all_ the tests in the monorepo. That's what doesn't scale. Run only the effected tests and the overhead is now more proportional the potential impact of the change.
- wwright 6y ago
  That seemed to be what they were describing, but with dynamic dependency detection via introspection.
yawaramin 6y ago
Does Shopify use Bazel? I would have guessed not. In any case, the more natural fit for them would be Nix, since it should be good at eliminiating redundant builds, and they recently went all-in on it.
- wwright 6y ago
  AFAIK Bazel and Nix are orthogonal and actually go well together. Nix is great for packaging and environments, but is anemic at actual build tooling (a big one: no incremental builds within a package). Bazel does very little to manage the environment or packages, but it has very granular and efficient build support.
  - yawaramin 6y ago
    True, but they both require a rather large complexity and learning and complexity budget. It would be difficult to introduce both into a shop at the same time.

umaar 6y ago

When it comes to browser automation tests, everywhere I've worked at suffered from intermittently failing tests. When I was at the Ministry of Justice (UK), I configured CircleCI to run tests hundreds of times [1] (over a number of days) through cron jobs. This allowed me to reflect on all test results, and find out what failed most often and eventually solve those root causes. This strategy worked well. Interestingly enough, just today I posted a GitHub thread [2] and asked the community to 'thumbs up' the video course they'd like for me to create. "Learn Browser Automation" is currently the highest voted. If it's the one I end up making, a huge focus will absolutely be on: How to reduce test flakiness with headless browsers. Words of advice, avoiding sleep() and other brittle bits of code will help. But in addition, run your tests frequently to catch out the flakiness early. Invest in tooling which helps you diagnose failing tests (screenshots, devtools trace dumps). Configure something like VNC so you can remotely connect to the machine running the test. [1] https://github.com/ministryofjustice/fb-automated-tests/blob/7c9cee58db902419abf5449aaef6e91e575502d9/.circleci/config.yml#L52 https://github.com/ministryofjustice/fb-automated-tests/blob... [2] https://github.com/umaar/dev-tips-tracker/issues/33 https://github.com/umaar/dev-tips-tracker/issues/33

hideo 6y ago

I wonder if Marie Kondo-ing your tests is a good idea in general. This article reminded me a lot of "Write tests. Not too many. Mostly integration" https://kentcdodds.com/blog/write-tests/ https://kentcdodds.com/blog/write-tests/ and other such articles for limiting testing, like this test diamond article http://toddlittleweb.com/wordpress/2014/06/23/the-testing-diamond-and-the-pyramid-2/ http://toddlittleweb.com/wordpress/2014/06/23/the-testing-di... I've seen far too many unit tests at this point that just assumed too much to be meaningful, and conversations about unit testing quickly devolve into "no true scotsman" style arguments about what is truly a unit and what isn't.

yawaramin 6y ago
A unit test is a test that fails in a way that immediately points to the failure cause.
sidlls 6y ago
Kent's article isn't so much about limiting the test coverage so much as preferring one kind (integration) over another (unit). He's right. Tons of unit testing in application codebases is a huge, stinking code smell in my experience. Usually it means the code is overly abstract (interfaces everywhere for "mockability") or unnecessarily complex (e.g. conditional expressions factored out into functions called in one site just so they can be tested separately) or both.

lazyant 6y ago

> Before the feature was rolled out, many developers were skeptical that it would work. Frankly, I was one of them. Kudos for the Skepticism section. Every "how we fixed problem X at company Y" should have this section, and esp. written by someone who opposed the solution. The challenging and battle stories tend to be the most interesting part, at least for me.

flukus 6y ago

And here's a makefile implementation with a statically typed language: #run test and output to a .testresult file when the test is modified %.testresult: %.test $< > $@ 2>&1 #depends on a .testresult file for each test test: $(ALL_TEST_RESULTS) cat $(ALL_TEST_RESULTS) Sure the dynamic typing introduces many problems, but surely you could have something a bit more half way like "%.testresult: %.test $(DEPENENDENCIES) $(METAPROGRAMMING_MAGIC_DEPENDENCIES)", then only hopefully rare changes to the core files require the full test suite to be rerun. Seems like dynamic typing is the core of their problem though and this is a crazy complicated solution to try and work around that.

jolmg 6y ago
> $< 2&1> $@ I'm not really well versed in Makefiles, but that seems to pass the argument "2" to the testing program, launching it in the background with "&", and then doing a redirection for the stdout of an empty command to the target file with "1>". I think you meant $< > $@ 2>&1 which could also be $< &> $@ I can't see how your example relates to static or dynamic typing, though.
- flukus 6y ago
  Fixed, the former is correct, the latter I think is a bashism that may not work. Unfortunately I plucked that from a project in a very broken state. > I can't see how your example relates to static or dynamic typing, though. Because with static typing you know the object/file graph at compile time and only affected tests need to be run. In that example the .test files will only be built when their dependencies change and .testresult will only be created if .test is built, everything is incremental and "make test" won't execute any tests if there are no changes that could affect them.
  - joshuamorton 6y ago
    You know this in dynamic languages too, based on imports. So if you have the build graph represented for your dynamic language, you get the same result. This is just a poor (poor, poor) buggy implementation of Bazel.

overgard 6y ago

I wonder if the tests are even serving a purpose at that point? If they can't reliably answer the question "did I accidentally break something" in a reasonable period of time what's the point?

m12k 6y ago

I think this shows how counting on horizontal scaling to handle inefficiency only works for a while, and will eventually introduce its own set of complexities - and dealing with those might be more trouble than coding things more efficiently in the first place. Then again, maybe it's worth it for a huge company like Shopify, because they save work for the feature devs by adding extra load on the test infrastructure devs, effectively increasing developer parallelism. Finally I wonder if they could have done more to speed up their tests? I'm maintaining a Rails codebase too, and I cut test time down by two thirds by rewriting tests with efficiency in mind - e.g. by ignoring dogma like 'each test needs to run completely independently of previous ones' (if I verify the state between tests, do I really need to pay the performance penalty of a database wipe?) The test-prof gem has a great tool, let-it-be, that allows you to designate db state that should not get wiped between tests. That and focusing on more unit tests and fewer integration test has really gone a long way toward speeding things up again for me.

simon_000666 6y ago

> has over 150,000 tests > takes about 30-40 min to run on hundreds of docker containers in parallel This seems like it might be a signal that now could be the time to start splitting services out in an SOA fashion and having their own test suites and some contract driven tests? Having to run that many tests on each commit is definitely a smell that something architecturally fundamental is wrong...

Seb-C 6y ago

My experience is that most of the time, randomly failing tests are actually failing because they were made to be random. Sure, adding Faker to your test/mock data may help you to find rare edge cases. The problem is that those edge cases will be triggered in the future, in another context and by another developer. So not only it probably won't be fixed, but it will be a waste of time and annoyance for someone. Same thing for some option like the `random` setting in Jasmine (runs your unit tests in a different random order each time). So now I only have tests with static, predictable data. Having tests that depends on the state of the previous one (or are affected by it) is quite common too. And not always easy to fix properly. Once this is removed, the remaining random failures are usually authentic and helpful.

davewritescode 6y ago

The approach they use to apply this to a large Ruby project is interesting but this type of strategy has been in use since forever and at least to me, seems fairly obvious. Running all the tests with every build is always a bad idea. A better approach that doesn't require fancy dynamic analysis is to organize tests in a way that it's clear what's likely to break and to make sure you're constantly running your test suite in QA environments. Making a change to a module should force you to run that module's test suite. Interactions between modules can be tested all day in a loop and monitored before deploying to production.

darksaints 6y ago

This sort of thing is why I always scoff at the idea of slow compile times for static strongly typed languages (eg. Scala, Haskell, OCaml, Rust, etc). Sure, it definitely compiles slower than some other compiled language, and there is no compilation for an interpreted language. But if you factor in testing, the type systems of those languages can easily remove a massive amount of testing that other less rigorous languages would either a) not test at all, or b) test at a significant cost. I'd be willing to bet that if shopify were using a strongly typed language, 50-90% of their testing would be completely redundant, because the type system already takes care of it. That isn't to say that there are not reasons to use dynamically typed languages...just that if you are building production systems in strongly typed languages, compile time is almost completely irrelevant as a factor in productivity, regardless of how much slower they compile in comparison to an alternative.