13 ms·

Testable IO in Haskell

76 points by kilimchoi 11y ago

pron 11y ago
To those interested, the exact same technique is achievable in imperative languages[1] -- only with less effort and in a less intrusive way -- with mocks. [1]: That support at least some notion of meta-programming/reflection.
- chadaustin 11y ago
  I think you've misunderstood the implications of the technique described. It's superficially similar to mocks, but the real power is that it defines _restricted_ effects, so that it is _impossible_ for the code under test to access, say, the current system time or to print to stdout. It is _only_ allowed to access APIs which are fully replaced in tests. The real benefit is that the type system guarantees these tests are not flaky or intermittent - in unit tests they are guaranteed pure and deterministic. This technique isn't applicable to languages like Python or C++ or JavaScript where any computation is free to have arbitrary side effects. In Haskell you can restrict computations to subsets of side effects, which is an enormously powerful technique, and is, for example, why Haskell's STM implementation is so great (and simple) compared to languages with unrestricted effects.
  - pron 11y ago
    First, let's separate two kinds of effects: memory effects and IO. While PFP treats memory effects as side-effects separate from program logic, imperative languages do not (either approach has its pros and cons). As the post talks about IO, let's restrict ourselves to that. If you believe that the main contribution of this approach is by absolutely preventing any kind of uncaptured IO (I think it is extremely valuable even without this language-supervised restriction), then this too, is trivially possible in, say, Java (or all other JVM languages). Just install a security manager in your tests, and it will make sure you don't accidentally access IO by bypassing mocks. This would still have the advantage of not writing your program in any special way to accommodate this technique and it would apply to third-party libraries, too.
    - chadaustin 11y ago
      I didn't know you could use Java SecurityManager to implement a similar system. That's cool. Do you know anyone who does that in their test suite? I'd love to chat with them. The imvujs test framework for JavaScript tries to achieve the same effect by disabling known common sources of test intermittency: https://github.com/imvu/imvujs/blob/master/src/imvujstest/testglobals.js#L24 https://github.com/imvu/imvujs/blob/master/src/imvujstest/te... Sadly, this technique, like any blacklist, doesn't work for types of IO that can't be prevented, like mutations to document.location. The technique described in the article is effectively a named whitelist of IO operations (called World) that, say, all HTTP request handlers are restricted to.
      - Peaker 11y ago
        Also, if tests take a while to run, it's nice to get the feedback from the compile step, rather than the run step. Also, perhaps some non determinism is allowed for the tests. Then runtime safety isn't equivalent to compile time safety.
        pron 11y ago
        > Also, if tests take a while to run, it's nice to get the feedback from the compile step, rather than the run step. True. We do that in Java, too. The idea is that you don't want to let methods use any "global" IO access (like System.out or new FileOutputStream) -- only IO streams that are passed in as arguments (which can then be mocked) -- so you use an annotation processor[1] to prohibit any such use; it's automatically picked up, loaded and used by all IDEs to mark any use of "prohibited operations" as errors while you type. Square/Google's Dagger DI library uses the same approach to make sure -- at compile time, with IDE support -- that all dependencies are satisfied. Again, this works without requiring any use of new/non-standard APIs. [1]: It's a class that gets run during one of the compilation phases, like a poor-man's pattern macros.
      - pron 11y ago
        > Do you know anyone who does that in their test suite? I'd love to chat with them. I do. Not often, as I don't have a lot of IO in my tests, but I've found the security manager useful for that purpose from time to time. I also use it to help our users enforce global contracts, such as prohibit IO or any blocking code in fork-join computations.
- implicit 11y ago
  You can, but in Haskell you can enlist the compiler's help to prove that everything is completely airtight. This property continues to hold, even as the application and the tests are modified by developers who do not even fully know what sort of effects they should otherwise be concerned with. For instance, most programming platforms give you pretty ready access to a pseudo-random number generator. This is great until some well-intentioned new hire checks in code someplace that uses it to instrument something 1 out of 1000 runs. We can't have this problem in Haskell and we don't have to spend any ongoing code review hours on it. It's gone.
  - pron 11y ago
    Yep, it's also great that we can enlist the runtime's help to do the same thing in Java (or any other JVM language) and support use of third-party libraries (the Haskell approach places a pretty severe burden on the code) and make it even more airtight (IIANM you can't prevent unsafePerformIO): We just disable IO by enabling the security manager with a "no-IO" policy[1]. So it's less intrusive, more widely applicable and more airtight than the Haskell approach. True, the security manager does not limit access to the random-number generator, but this, too, can be done with an extra bit of one-time effort, by injecting a security check into the random number generator's seed generation method (we still want to allow fixed seeds like StdGen). [1]: Or a finer-grained policy of "no IO except for logging".
    - chongli 11y ago
      An application which does "no IO" does nothing other than make the computer get hot. Why even run it?
      - pron 11y ago
        Why, for testing. The calls to IO are replaced with calls to mocks which are then used to verify IO behavior. It's like replacing the IO monad with a type that doesn't actually do IO, only it doesn't require the code to change, so it works even for third-party code. You can then use the security manager during the test to verify that all IO calls are indeed mocked.
peteretep 11y ago
If your tests fail for any of the reasons the author leads with, and don't get fixed straight away with some soul searching on the part of your developers, you need to start hiring some engineers.
- chadaustin 11y ago
  Hi, I have personal experience working with the some of the best software engineers on the planet for the last decade and despite everyone's best efforts, when a unit test suite grows to thousands and thousands of tests, there will always be some that are flaky. Perhaps they call into a code path which was unwittingly refactored to access a local database. Perhaps they read the current time for some time-dependent string formatting. Perhaps they mutate some global state and forget to clean it up. Perhaps they call into some code that opens a TCP connection to some third-party service. Every large test suite I've seen in every language at every company I've worked with has intermittent or flaky tests. Maybe each test on average is 99.9% reliable. But across the entire suite, you will have some low-level degree of intermittent test failures. Then you need a team of people whose job it is to triage, investigate, and either fix or ignore those failures. This is a complete waste of human capacity, and it's not a very career-enriching task either. Haskell's ability to define restricted subsets of generalize IO is extremely powerful and sidesteps this entire problem. The power of the technique in the article is that your tests are _guaranteed_ reliable by the type system, because it is no longer possible to accidentally call into some general IO action. Code under test must go through the World API, which is fully and deterministically mocked in tests.
  - peteretep 11y ago
    I have worked with huge test suites. If your test suite contains tests that fail for the reasons given, and the result is a shrug, your test suite becomes useless very quickly, and then triaging which tests are allowed to fail, which are real problems, and so on becomes a much bigger waste of time and treasure. Making a big deal out of it when tests are added that mutate global state or call 3rd party APIs is an investment that pays back very quickly. Shrugging and saying "98% of the tests passed, that's probably good enough" means your test suite itself is a waste of time and resources and you could be getting the same result by just doing a quick happy path minimal set.
deepnet 11y ago
Can State not be considered as an immutable infinite list, indexed by time ? Does Haskell have a forgettable concept ? State could then be a forgettable infinite list rather than a side/effect.
- wz1000 11y ago
  State is modeled in Haskell as the type newtype State s a = State (s -> (a, s)) Where s is the type of your state and a is the type of your result. It's basically a function that takes the current state and gives you a result as well as a new state.
tinco 11y ago
So let me try to see if I get it. You change the signature of your I/O functions to return a self defined monad called World. To make sure this works in production you make World an instance of I/O so it can be ran by your regular main. Then for the test suite, you make an instance of World that keeps a state monad and mock implementations of the I/O functions that return/change that state instead of having undeterministic side effects. The whole process seems logical, but the Haskell is rather ugly. Surely someone has written a DSL or library to make working like this a little easier? Especially defining the FakeState monad with S.modify calls looks like a pain.
- mralston 11y ago
  The thing is that while the implementation of the fake can get ugly, that's effectively library code that's written once - the actual application code is incredibly clean since all you have to do to it is tag it as being World instead of IO. Likewise, for tests, the actual interface in practice is clean - we have a few fake-World-only setup functions, and when testing things like the database we can just use the regular insert interface to do our setup.
zimbatm 11y ago
The nice thing about this approach is that it's then possible to use QuickCheck to generate FakeIO data.
clarus 11y ago
We run into the same kind of challenges to test programs with IO in Coq.io. To stay pure, we keep the IOs uninterpreted until the compilation (to an OCaml program). We define the tests on the program with uninterpreted IOs. For clarity of the tests, we use an interactive debugger (reusing the tactical mode of Coq) to step through the IO operations. The main advantage of using Coq is that the tests can be made symbolic, thus covering a larger number of cases (if not all the cases). A simple example is explained here: http://coq.io/getting_started.html#use-cases http://coq.io/getting_started.html#use-cases (coincidentally, this is the same example as in this blog post).

76 points by kilimchoi 11y ago

pron 11y ago

To those interested, the exact same technique is achievable in imperative languages[1] -- only with less effort and in a less intrusive way -- with mocks. [1]: That support at least some notion of meta-programming/reflection.

chadaustin 11y ago
I think you've misunderstood the implications of the technique described. It's superficially similar to mocks, but the real power is that it defines _restricted_ effects, so that it is _impossible_ for the code under test to access, say, the current system time or to print to stdout. It is _only_ allowed to access APIs which are fully replaced in tests. The real benefit is that the type system guarantees these tests are not flaky or intermittent - in unit tests they are guaranteed pure and deterministic. This technique isn't applicable to languages like Python or C++ or JavaScript where any computation is free to have arbitrary side effects. In Haskell you can restrict computations to subsets of side effects, which is an enormously powerful technique, and is, for example, why Haskell's STM implementation is so great (and simple) compared to languages with unrestricted effects.
- pron 11y ago
  First, let's separate two kinds of effects: memory effects and IO. While PFP treats memory effects as side-effects separate from program logic, imperative languages do not (either approach has its pros and cons). As the post talks about IO, let's restrict ourselves to that. If you believe that the main contribution of this approach is by absolutely preventing any kind of uncaptured IO (I think it is extremely valuable even without this language-supervised restriction), then this too, is trivially possible in, say, Java (or all other JVM languages). Just install a security manager in your tests, and it will make sure you don't accidentally access IO by bypassing mocks. This would still have the advantage of not writing your program in any special way to accommodate this technique and it would apply to third-party libraries, too.
  - chadaustin 11y ago
    I didn't know you could use Java SecurityManager to implement a similar system. That's cool. Do you know anyone who does that in their test suite? I'd love to chat with them. The imvujs test framework for JavaScript tries to achieve the same effect by disabling known common sources of test intermittency: https://github.com/imvu/imvujs/blob/master/src/imvujstest/testglobals.js#L24 https://github.com/imvu/imvujs/blob/master/src/imvujstest/te... Sadly, this technique, like any blacklist, doesn't work for types of IO that can't be prevented, like mutations to document.location. The technique described in the article is effectively a named whitelist of IO operations (called World) that, say, all HTTP request handlers are restricted to.
    - Peaker 11y ago
      Also, if tests take a while to run, it's nice to get the feedback from the compile step, rather than the run step. Also, perhaps some non determinism is allowed for the tests. Then runtime safety isn't equivalent to compile time safety.
      - pron 11y ago
        > Also, if tests take a while to run, it's nice to get the feedback from the compile step, rather than the run step. True. We do that in Java, too. The idea is that you don't want to let methods use any "global" IO access (like System.out or new FileOutputStream) -- only IO streams that are passed in as arguments (which can then be mocked) -- so you use an annotation processor[1] to prohibit any such use; it's automatically picked up, loaded and used by all IDEs to mark any use of "prohibited operations" as errors while you type. Square/Google's Dagger DI library uses the same approach to make sure -- at compile time, with IDE support -- that all dependencies are satisfied. Again, this works without requiring any use of new/non-standard APIs. [1]: It's a class that gets run during one of the compilation phases, like a poor-man's pattern macros.
    - pron 11y ago
      > Do you know anyone who does that in their test suite? I'd love to chat with them. I do. Not often, as I don't have a lot of IO in my tests, but I've found the security manager useful for that purpose from time to time. I also use it to help our users enforce global contracts, such as prohibit IO or any blocking code in fork-join computations.
implicit 11y ago
You can, but in Haskell you can enlist the compiler's help to prove that everything is completely airtight. This property continues to hold, even as the application and the tests are modified by developers who do not even fully know what sort of effects they should otherwise be concerned with. For instance, most programming platforms give you pretty ready access to a pseudo-random number generator. This is great until some well-intentioned new hire checks in code someplace that uses it to instrument something 1 out of 1000 runs. We can't have this problem in Haskell and we don't have to spend any ongoing code review hours on it. It's gone.
- pron 11y ago
  Yep, it's also great that we can enlist the runtime's help to do the same thing in Java (or any other JVM language) and support use of third-party libraries (the Haskell approach places a pretty severe burden on the code) and make it even more airtight (IIANM you can't prevent unsafePerformIO): We just disable IO by enabling the security manager with a "no-IO" policy[1]. So it's less intrusive, more widely applicable and more airtight than the Haskell approach. True, the security manager does not limit access to the random-number generator, but this, too, can be done with an extra bit of one-time effort, by injecting a security check into the random number generator's seed generation method (we still want to allow fixed seeds like StdGen). [1]: Or a finer-grained policy of "no IO except for logging".
  - chongli 11y ago
    An application which does "no IO" does nothing other than make the computer get hot. Why even run it?
    - pron 11y ago
      Why, for testing. The calls to IO are replaced with calls to mocks which are then used to verify IO behavior. It's like replacing the IO monad with a type that doesn't actually do IO, only it doesn't require the code to change, so it works even for third-party code. You can then use the security manager during the test to verify that all IO calls are indeed mocked.

peteretep 11y ago

If your tests fail for any of the reasons the author leads with, and don't get fixed straight away with some soul searching on the part of your developers, you need to start hiring some engineers.

chadaustin 11y ago
Hi, I have personal experience working with the some of the best software engineers on the planet for the last decade and despite everyone's best efforts, when a unit test suite grows to thousands and thousands of tests, there will always be some that are flaky. Perhaps they call into a code path which was unwittingly refactored to access a local database. Perhaps they read the current time for some time-dependent string formatting. Perhaps they mutate some global state and forget to clean it up. Perhaps they call into some code that opens a TCP connection to some third-party service. Every large test suite I've seen in every language at every company I've worked with has intermittent or flaky tests. Maybe each test on average is 99.9% reliable. But across the entire suite, you will have some low-level degree of intermittent test failures. Then you need a team of people whose job it is to triage, investigate, and either fix or ignore those failures. This is a complete waste of human capacity, and it's not a very career-enriching task either. Haskell's ability to define restricted subsets of generalize IO is extremely powerful and sidesteps this entire problem. The power of the technique in the article is that your tests are _guaranteed_ reliable by the type system, because it is no longer possible to accidentally call into some general IO action. Code under test must go through the World API, which is fully and deterministically mocked in tests.
- peteretep 11y ago
  I have worked with huge test suites. If your test suite contains tests that fail for the reasons given, and the result is a shrug, your test suite becomes useless very quickly, and then triaging which tests are allowed to fail, which are real problems, and so on becomes a much bigger waste of time and treasure. Making a big deal out of it when tests are added that mutate global state or call 3rd party APIs is an investment that pays back very quickly. Shrugging and saying "98% of the tests passed, that's probably good enough" means your test suite itself is a waste of time and resources and you could be getting the same result by just doing a quick happy path minimal set.

deepnet 11y ago

Can State not be considered as an immutable infinite list, indexed by time ? Does Haskell have a forgettable concept ? State could then be a forgettable infinite list rather than a side/effect.

wz1000 11y ago
State is modeled in Haskell as the type newtype State s a = State (s -> (a, s)) Where s is the type of your state and a is the type of your result. It's basically a function that takes the current state and gives you a result as well as a new state.

tinco 11y ago

So let me try to see if I get it. You change the signature of your I/O functions to return a self defined monad called World. To make sure this works in production you make World an instance of I/O so it can be ran by your regular main. Then for the test suite, you make an instance of World that keeps a state monad and mock implementations of the I/O functions that return/change that state instead of having undeterministic side effects. The whole process seems logical, but the Haskell is rather ugly. Surely someone has written a DSL or library to make working like this a little easier? Especially defining the FakeState monad with S.modify calls looks like a pain.

mralston 11y ago
The thing is that while the implementation of the fake can get ugly, that's effectively library code that's written once - the actual application code is incredibly clean since all you have to do to it is tag it as being World instead of IO. Likewise, for tests, the actual interface in practice is clean - we have a few fake-World-only setup functions, and when testing things like the database we can just use the regular insert interface to do our setup.

zimbatm 11y ago

The nice thing about this approach is that it's then possible to use QuickCheck to generate FakeIO data.

clarus 11y ago

We run into the same kind of challenges to test programs with IO in Coq.io. To stay pure, we keep the IOs uninterpreted until the compilation (to an OCaml program). We define the tests on the program with uninterpreted IOs. For clarity of the tests, we use an interactive debugger (reusing the tactical mode of Coq) to step through the IO operations. The main advantage of using Coq is that the tests can be made symbolic, thus covering a larger number of cases (if not all the cases). A simple example is explained here: http://coq.io/getting_started.html#use-cases http://coq.io/getting_started.html#use-cases (coincidentally, this is the same example as in this blog post).