6 ms·

Hi, I have personal experience working with the some of the best software engineers on the planet for the last decade and despite everyone's best efforts, when

Hi, I have personal experience working with the some of the best software engineers on the planet for the last decade and despite everyone's best efforts, when a unit test suite grows to thousands and thousands of tests, there will always be some that are flaky. Perhaps they call into a code path which was unwittingly refactored to access a local database. Perhaps they read the current time for some time-dependent string formatting. Perhaps they mutate some global state and forget to clean it up. Perhaps they call into some code that opens a TCP connection to some third-party service. Every large test suite I've seen in every language at every company I've worked with has intermittent or flaky tests. Maybe each test on average is 99.9% reliable. But across the entire suite, you will have some low-level degree of intermittent test failures. Then you need a team of people whose job it is to triage, investigate, and either fix or ignore those failures. This is a complete waste of human capacity, and it's not a very career-enriching task either. Haskell's ability to define restricted subsets of generalize IO is extremely powerful and sidesteps this entire problem. The power of the technique in the article is that your tests are _guaranteed_ reliable by the type system, because it is no longer possible to accidentally call into some general IO action. Code under test must go through the World API, which is fully and deterministically mocked in tests.

peteretep 11y ago
I have worked with huge test suites. If your test suite contains tests that fail for the reasons given, and the result is a shrug, your test suite becomes useless very quickly, and then triaging which tests are allowed to fail, which are real problems, and so on becomes a much bigger waste of time and treasure. Making a big deal out of it when tests are added that mutate global state or call 3rd party APIs is an investment that pays back very quickly. Shrugging and saying "98% of the tests passed, that's probably good enough" means your test suite itself is a waste of time and resources and you could be getting the same result by just doing a quick happy path minimal set.
- chadaustin 11y ago
  Agreed with that. In a world where you can't statically enforce that tests are deterministic, you absolutely must be diligent. However, it still happens. An example of "innocent intermittence" recently came up at my current employer: there was a test that invoked a web service twice, verifying idempotence. It then asserted the responses were identical. This worked almost all of the time, but when the two service invocations straddled a minute boundary, then some output text that included a rendered timestamp (rounded to the minute), which would cause the test to fail. It's extremely hard to get an engineering organization to just inherently know not to call, say, datetime.now() in their code, especially when the tests pass most of the time.
  - crdoconnor 11y ago
    It sounds like your problem was with the test, not necessarily the developers calling datetime.now().
    - implicit 11y ago
      The problem is that developers do not know that they mustn't call datetime.now() and there will always be a developer who doesn't know this, either because people have a tough time knowing everything, or just because you're hiring new people. Wouldn't it be nice if you could just obstruct such patterns within the first millisecond of their first interaction with the compiler?
      - crdoconnor 11y ago
        I don't have a problem with it. I send a mocked time to the application under test using libfaketime using this: http://hitchtest.com http://hitchtest.com Et voila, no more time issues.
      - peteretep 11y ago
        > developers do not know that they mustn't call > datetime.now() Plenty of languages where it's easy to mock the equivalent to that call.
  - 11y ago
pron 11y ago
> Haskell's ability to define restricted subsets of generalize IO is extremely powerful and sidesteps this entire problem. Except it doesn't, precisely where that problem is most annoying. For example, suppose you want to test a scheduler, or any mechanism that places a timeout on an operation. You could abstract away the entire timing mechanism, but that is exactly what you're trying to test: maybe the developer used some system function wrong, and is now waiting 5 seconds instead of 5 milliseconds? This approach (which is the same as the mocking approach -- there's nothing special about Haskell here) doesn't do too well when you want to test timing, and those tests are usually the most brittle (hosted continuous integration services often introduce really long pauses). Of course, you could weave virtual time services throughout your code, but then you have the third-party library problem again. Mocking (or IO isolating as you call it in Haskell) ends up producing really good unit-tests, but often the more important tests span more than one small unit, and making those non-flaky is very hard. In theory, you could replace all references to time in a JVM language -- even in libraries -- using an agent or a bootstrap classloader in order to make larger tests run predictably, but I'm not aware of people going to such great lengths (maybe Google does it; I know, for example, that when it comes to accessing the filesystem, they replace the JVM's filesystem provider with an in-memory file-system[1] when they don't mock, but I don't know how the deal with clocks). EDIT: Apparently somebody has done just that as a research project[2]. It's actually given me an idea for a nice afternoon-project. [1]: https://github.com/google/jimfs https://github.com/google/jimfs [2]: http://www3.imperial.ac.uk/pls/portallive/docs/1/55125696.PDF http://www3.imperial.ac.uk/pls/portallive/docs/1/55125696.PD...
- chriswarbo 11y ago
  > For example, suppose you want to test a scheduler, or any mechanism that places a timeout on an operation. You could abstract away the entire timing mechanism, but that is exactly what you're trying to test: maybe the developer used some system function wrong, and is now waiting 5 seconds instead of 5 milliseconds? Time is an IO effect, so it's true that mocking it away removes that test. I'd treat that kind of "do I understand the API?" issue as an integration test, and split the problem up a little: - "Mock" the timeout calls, to allow as many tests as possible to run outside IO - Write tests where the operation always succeeds (eg. "Just x") - Write tests where the operation always times out (eg. "Nothing") - Write tests which do a combination of success/timeout and a bunch of different interleavings (QuickCheck would probably be useful here) - Write some small integration tests, eg. give a timeout of "5" to a process that sleeps for 5 seconds, it should time out; give a timeout of "5" to a process that sleeps for 4 seconds, it should succeed; and so on. > Mocking (or IO isolating as you call it in Haskell) ends up producing really good unit-tests, but often the more important tests span more than one small unit, and making those non-flaky is very hard. Haskell's IO isolation causes the vast majority of a program to be defined outside IO. You're right that the trickiest parts are often in IO, but at least those are pretty much separate from any complex calculation/logic.
rtpg 11y ago
The trickiness comes when you have to write these mocks, and whether the mocks are based off of correct assumptions. Granted, Haskell's type system goes a long way to helping with that (often a function of type A -> B only has a handful of possible definitions given sufficiently complex types A and B), but it's still worth using actual production-use code paths when feasible. EDIT: Where I work, we have a large amount of tests, but we have pretty much never deployed an instance of our application without having every test pass. The idea of having 100% of your tests pass is a lot easier when it's the default, and when you're doing more continuous integration. If you're using a language like Haskell, your business logic will already be pretty separate from the ickiness of the IO layer, so most of your tests will be deterministic and nice by default. If you're using a messier language... well there's a lot of test suites out there. If you're using something that's mainly Django/Rails-y stuff, you don't have many excuses not to be 100% passing by default. Most major frameworks/languages offer extensive mocking tools and the like to avoid non-determinent behavior in tests.

peteretep 11y ago

I have worked with huge test suites. If your test suite contains tests that fail for the reasons given, and the result is a shrug, your test suite becomes useless very quickly, and then triaging which tests are allowed to fail, which are real problems, and so on becomes a much bigger waste of time and treasure. Making a big deal out of it when tests are added that mutate global state or call 3rd party APIs is an investment that pays back very quickly. Shrugging and saying "98% of the tests passed, that's probably good enough" means your test suite itself is a waste of time and resources and you could be getting the same result by just doing a quick happy path minimal set.

chadaustin 11y ago
Agreed with that. In a world where you can't statically enforce that tests are deterministic, you absolutely must be diligent. However, it still happens. An example of "innocent intermittence" recently came up at my current employer: there was a test that invoked a web service twice, verifying idempotence. It then asserted the responses were identical. This worked almost all of the time, but when the two service invocations straddled a minute boundary, then some output text that included a rendered timestamp (rounded to the minute), which would cause the test to fail. It's extremely hard to get an engineering organization to just inherently know not to call, say, datetime.now() in their code, especially when the tests pass most of the time.
- crdoconnor 11y ago
  It sounds like your problem was with the test, not necessarily the developers calling datetime.now().
  - implicit 11y ago
    The problem is that developers do not know that they mustn't call datetime.now() and there will always be a developer who doesn't know this, either because people have a tough time knowing everything, or just because you're hiring new people. Wouldn't it be nice if you could just obstruct such patterns within the first millisecond of their first interaction with the compiler?
    - crdoconnor 11y ago
      I don't have a problem with it. I send a mocked time to the application under test using libfaketime using this: http://hitchtest.com http://hitchtest.com Et voila, no more time issues.
    - peteretep 11y ago
      > developers do not know that they mustn't call > datetime.now() Plenty of languages where it's easy to mock the equivalent to that call.
- 11y ago

pron 11y ago

> Haskell's ability to define restricted subsets of generalize IO is extremely powerful and sidesteps this entire problem. Except it doesn't, precisely where that problem is most annoying. For example, suppose you want to test a scheduler, or any mechanism that places a timeout on an operation. You could abstract away the entire timing mechanism, but that is exactly what you're trying to test: maybe the developer used some system function wrong, and is now waiting 5 seconds instead of 5 milliseconds? This approach (which is the same as the mocking approach -- there's nothing special about Haskell here) doesn't do too well when you want to test timing, and those tests are usually the most brittle (hosted continuous integration services often introduce really long pauses). Of course, you could weave virtual time services throughout your code, but then you have the third-party library problem again. Mocking (or IO isolating as you call it in Haskell) ends up producing really good unit-tests, but often the more important tests span more than one small unit, and making those non-flaky is very hard. In theory, you could replace all references to time in a JVM language -- even in libraries -- using an agent or a bootstrap classloader in order to make larger tests run predictably, but I'm not aware of people going to such great lengths (maybe Google does it; I know, for example, that when it comes to accessing the filesystem, they replace the JVM's filesystem provider with an in-memory file-system[1] when they don't mock, but I don't know how the deal with clocks). EDIT: Apparently somebody has done just that as a research project[2]. It's actually given me an idea for a nice afternoon-project. [1]: https://github.com/google/jimfs https://github.com/google/jimfs [2]: http://www3.imperial.ac.uk/pls/portallive/docs/1/55125696.PDF http://www3.imperial.ac.uk/pls/portallive/docs/1/55125696.PD...

chriswarbo 11y ago
> For example, suppose you want to test a scheduler, or any mechanism that places a timeout on an operation. You could abstract away the entire timing mechanism, but that is exactly what you're trying to test: maybe the developer used some system function wrong, and is now waiting 5 seconds instead of 5 milliseconds? Time is an IO effect, so it's true that mocking it away removes that test. I'd treat that kind of "do I understand the API?" issue as an integration test, and split the problem up a little: - "Mock" the timeout calls, to allow as many tests as possible to run outside IO - Write tests where the operation always succeeds (eg. "Just x") - Write tests where the operation always times out (eg. "Nothing") - Write tests which do a combination of success/timeout and a bunch of different interleavings (QuickCheck would probably be useful here) - Write some small integration tests, eg. give a timeout of "5" to a process that sleeps for 5 seconds, it should time out; give a timeout of "5" to a process that sleeps for 4 seconds, it should succeed; and so on. > Mocking (or IO isolating as you call it in Haskell) ends up producing really good unit-tests, but often the more important tests span more than one small unit, and making those non-flaky is very hard. Haskell's IO isolation causes the vast majority of a program to be defined outside IO. You're right that the trickiest parts are often in IO, but at least those are pretty much separate from any complex calculation/logic.

rtpg 11y ago

The trickiness comes when you have to write these mocks, and whether the mocks are based off of correct assumptions. Granted, Haskell's type system goes a long way to helping with that (often a function of type A -> B only has a handful of possible definitions given sufficiently complex types A and B), but it's still worth using actual production-use code paths when feasible. EDIT: Where I work, we have a large amount of tests, but we have pretty much never deployed an instance of our application without having every test pass. The idea of having 100% of your tests pass is a lot easier when it's the default, and when you're doing more continuous integration. If you're using a language like Haskell, your business logic will already be pretty separate from the ickiness of the IO layer, so most of your tests will be deterministic and nice by default. If you're using a messier language... well there's a lot of test suites out there. If you're using something that's mainly Django/Rails-y stuff, you don't have many excuses not to be 100% passing by default. Most major frameworks/languages offer extensive mocking tools and the like to avoid non-determinent behavior in tests.