3 ms·

I think you've misunderstood the implications of the technique described. It's superficially similar to mocks, but the real power is that it defines _restricte

I think you've misunderstood the implications of the technique described. It's superficially similar to mocks, but the real power is that it defines _restricted_ effects, so that it is _impossible_ for the code under test to access, say, the current system time or to print to stdout. It is _only_ allowed to access APIs which are fully replaced in tests. The real benefit is that the type system guarantees these tests are not flaky or intermittent - in unit tests they are guaranteed pure and deterministic. This technique isn't applicable to languages like Python or C++ or JavaScript where any computation is free to have arbitrary side effects. In Haskell you can restrict computations to subsets of side effects, which is an enormously powerful technique, and is, for example, why Haskell's STM implementation is so great (and simple) compared to languages with unrestricted effects.

pron 11y ago
First, let's separate two kinds of effects: memory effects and IO. While PFP treats memory effects as side-effects separate from program logic, imperative languages do not (either approach has its pros and cons). As the post talks about IO, let's restrict ourselves to that. If you believe that the main contribution of this approach is by absolutely preventing any kind of uncaptured IO (I think it is extremely valuable even without this language-supervised restriction), then this too, is trivially possible in, say, Java (or all other JVM languages). Just install a security manager in your tests, and it will make sure you don't accidentally access IO by bypassing mocks. This would still have the advantage of not writing your program in any special way to accommodate this technique and it would apply to third-party libraries, too.
- chadaustin 11y ago
  I didn't know you could use Java SecurityManager to implement a similar system. That's cool. Do you know anyone who does that in their test suite? I'd love to chat with them. The imvujs test framework for JavaScript tries to achieve the same effect by disabling known common sources of test intermittency: https://github.com/imvu/imvujs/blob/master/src/imvujstest/testglobals.js#L24 https://github.com/imvu/imvujs/blob/master/src/imvujstest/te... Sadly, this technique, like any blacklist, doesn't work for types of IO that can't be prevented, like mutations to document.location. The technique described in the article is effectively a named whitelist of IO operations (called World) that, say, all HTTP request handlers are restricted to.
  - Peaker 11y ago
    Also, if tests take a while to run, it's nice to get the feedback from the compile step, rather than the run step. Also, perhaps some non determinism is allowed for the tests. Then runtime safety isn't equivalent to compile time safety.
    - pron 11y ago
      > Also, if tests take a while to run, it's nice to get the feedback from the compile step, rather than the run step. True. We do that in Java, too. The idea is that you don't want to let methods use any "global" IO access (like System.out or new FileOutputStream) -- only IO streams that are passed in as arguments (which can then be mocked) -- so you use an annotation processor[1] to prohibit any such use; it's automatically picked up, loaded and used by all IDEs to mark any use of "prohibited operations" as errors while you type. Square/Google's Dagger DI library uses the same approach to make sure -- at compile time, with IDE support -- that all dependencies are satisfied. Again, this works without requiring any use of new/non-standard APIs. [1]: It's a class that gets run during one of the compilation phases, like a poor-man's pattern macros.
  - pron 11y ago
    > Do you know anyone who does that in their test suite? I'd love to chat with them. I do. Not often, as I don't have a lot of IO in my tests, but I've found the security manager useful for that purpose from time to time. I also use it to help our users enforce global contracts, such as prohibit IO or any blocking code in fork-join computations.
    - chadaustin 11y ago
      That's cool. I chatted with a friend about this too and they said they've used SecurityManager in this way. But one thing they said is that SecurityManager can't be used to restrict things like accessing the current time. Do you know if that's true or not?
      - pron 11y ago

pron 11y ago

First, let's separate two kinds of effects: memory effects and IO. While PFP treats memory effects as side-effects separate from program logic, imperative languages do not (either approach has its pros and cons). As the post talks about IO, let's restrict ourselves to that. If you believe that the main contribution of this approach is by absolutely preventing any kind of uncaptured IO (I think it is extremely valuable even without this language-supervised restriction), then this too, is trivially possible in, say, Java (or all other JVM languages). Just install a security manager in your tests, and it will make sure you don't accidentally access IO by bypassing mocks. This would still have the advantage of not writing your program in any special way to accommodate this technique and it would apply to third-party libraries, too.

chadaustin 11y ago
I didn't know you could use Java SecurityManager to implement a similar system. That's cool. Do you know anyone who does that in their test suite? I'd love to chat with them. The imvujs test framework for JavaScript tries to achieve the same effect by disabling known common sources of test intermittency: https://github.com/imvu/imvujs/blob/master/src/imvujstest/testglobals.js#L24 https://github.com/imvu/imvujs/blob/master/src/imvujstest/te... Sadly, this technique, like any blacklist, doesn't work for types of IO that can't be prevented, like mutations to document.location. The technique described in the article is effectively a named whitelist of IO operations (called World) that, say, all HTTP request handlers are restricted to.
- Peaker 11y ago
  Also, if tests take a while to run, it's nice to get the feedback from the compile step, rather than the run step. Also, perhaps some non determinism is allowed for the tests. Then runtime safety isn't equivalent to compile time safety.
  - pron 11y ago
    > Also, if tests take a while to run, it's nice to get the feedback from the compile step, rather than the run step. True. We do that in Java, too. The idea is that you don't want to let methods use any "global" IO access (like System.out or new FileOutputStream) -- only IO streams that are passed in as arguments (which can then be mocked) -- so you use an annotation processor[1] to prohibit any such use; it's automatically picked up, loaded and used by all IDEs to mark any use of "prohibited operations" as errors while you type. Square/Google's Dagger DI library uses the same approach to make sure -- at compile time, with IDE support -- that all dependencies are satisfied. Again, this works without requiring any use of new/non-standard APIs. [1]: It's a class that gets run during one of the compilation phases, like a poor-man's pattern macros.
- pron 11y ago
  > Do you know anyone who does that in their test suite? I'd love to chat with them. I do. Not often, as I don't have a lot of IO in my tests, but I've found the security manager useful for that purpose from time to time. I also use it to help our users enforce global contracts, such as prohibit IO or any blocking code in fork-join computations.
  - chadaustin 11y ago
    That's cool. I chatted with a friend about this too and they said they've used SecurityManager in this way. But one thing they said is that SecurityManager can't be used to restrict things like accessing the current time. Do you know if that's true or not?
    - pron 11y ago