3 ms·

> For example, suppose you want to test a scheduler, or any mechanism that places a timeout on an operation. You could abstract away the entire timing mechanism

> For example, suppose you want to test a scheduler, or any mechanism that places a timeout on an operation. You could abstract away the entire timing mechanism, but that is exactly what you're trying to test: maybe the developer used some system function wrong, and is now waiting 5 seconds instead of 5 milliseconds? Time is an IO effect, so it's true that mocking it away removes that test. I'd treat that kind of "do I understand the API?" issue as an integration test, and split the problem up a little: - "Mock" the timeout calls, to allow as many tests as possible to run outside IO - Write tests where the operation always succeeds (eg. "Just x") - Write tests where the operation always times out (eg. "Nothing") - Write tests which do a combination of success/timeout and a bunch of different interleavings (QuickCheck would probably be useful here) - Write some small integration tests, eg. give a timeout of "5" to a process that sleeps for 5 seconds, it should time out; give a timeout of "5" to a process that sleeps for 4 seconds, it should succeed; and so on. > Mocking (or IO isolating as you call it in Haskell) ends up producing really good unit-tests, but often the more important tests span more than one small unit, and making those non-flaky is very hard. Haskell's IO isolation causes the vast majority of a program to be defined outside IO. You're right that the trickiest parts are often in IO, but at least those are pretty much separate from any complex calculation/logic.

pron 11y ago
> but at least those are pretty much separate from any complex calculation/logic. Yes, but in imperative languages (with meta-programming) you can disentangle mixed IO and logic even without changing the code (or if it's in a third-party library). The advantage is that it allows you (not always, but sometimes) to support even larger tests in this way. Getting those tiny, one-function unit-tests to run predictably is not hard in any language; you just make them smaller and smaller and mock everything else. The tricky bit is making larger tests run predictably, and a flexible, hackable runtime has the advantage there. That's precisely where meta-programming shines.
- chriswarbo 11y ago
  > in imperative languages (with meta-programming) you can disentangle mixed IO and logic even without changing the code I don't understand what you mean by "mixed", "disentangle" and "change". I would say "mixed IO and logic" is when one block of code describes both how to calculate some value and which side-effects to perform. "Disentangling" these would mean separating the description of the calculation from the description of the side-effects. Doing that "without changing the code" doesn't make sense to me.
  - pron 11y ago
    What I mean is that if your code (or a library you call) mixes logic with IO, meta-programming lets you capture the IO calls in a mock (like turning it into an IO monad), which you can then query and verify, without changing the code and the API's it's using. E.g. on the JVM you do that with Mockito (similar mocking libraries exist for other languages/runtimes). Say you have a (silly) method that does computation and IO: void printDouble(int x, PrintStream out) { out.println("Result: " + (x * 2)); } You can then do: @Test public void testPrintDouble() { PrintStream out = mock(PrintStream.class); printDouble(3, out); verify(out).println("Result: 6"); } No output is actually done. The call to println (which may be deep in some library printDouble calls rather than directly in the tested code). We've essentially replaced any object that does direct IO with something akin to an IO monad. We can query the mock in more sophisticated ways, too. Here's a very contrived example: void printDoubleWithLog(int x, PrintStream out, PrintStream log) { log.println("Computing..."); out.println("Result: " + (x * 2)); log.println("Done."); } @Test public void testPrintDoubleWithLog() { PrintStream out = mock(PrintStream.class); PrintStream log = mock(PrintStream.class); printDouble(3, out, log); InOrder ordered = inOrder(out, log); ordered.verify(log).println("Computing..."); ordered.verify(out).println("Result: 6"); ordered.verify(log).println("Done."); } This contrived example also verifies the relative ordering of IO done on two separate streams. Again, no API change is necessary, and the actual IO calls may be deep in some library.
    - chriswarbo 11y ago

pron 11y ago

> but at least those are pretty much separate from any complex calculation/logic. Yes, but in imperative languages (with meta-programming) you can disentangle mixed IO and logic even without changing the code (or if it's in a third-party library). The advantage is that it allows you (not always, but sometimes) to support even larger tests in this way. Getting those tiny, one-function unit-tests to run predictably is not hard in any language; you just make them smaller and smaller and mock everything else. The tricky bit is making larger tests run predictably, and a flexible, hackable runtime has the advantage there. That's precisely where meta-programming shines.

chriswarbo 11y ago
> in imperative languages (with meta-programming) you can disentangle mixed IO and logic even without changing the code I don't understand what you mean by "mixed", "disentangle" and "change". I would say "mixed IO and logic" is when one block of code describes both how to calculate some value and which side-effects to perform. "Disentangling" these would mean separating the description of the calculation from the description of the side-effects. Doing that "without changing the code" doesn't make sense to me.
- pron 11y ago
  What I mean is that if your code (or a library you call) mixes logic with IO, meta-programming lets you capture the IO calls in a mock (like turning it into an IO monad), which you can then query and verify, without changing the code and the API's it's using. E.g. on the JVM you do that with Mockito (similar mocking libraries exist for other languages/runtimes). Say you have a (silly) method that does computation and IO: void printDouble(int x, PrintStream out) { out.println("Result: " + (x * 2)); } You can then do: @Test public void testPrintDouble() { PrintStream out = mock(PrintStream.class); printDouble(3, out); verify(out).println("Result: 6"); } No output is actually done. The call to println (which may be deep in some library printDouble calls rather than directly in the tested code). We've essentially replaced any object that does direct IO with something akin to an IO monad. We can query the mock in more sophisticated ways, too. Here's a very contrived example: void printDoubleWithLog(int x, PrintStream out, PrintStream log) { log.println("Computing..."); out.println("Result: " + (x * 2)); log.println("Done."); } @Test public void testPrintDoubleWithLog() { PrintStream out = mock(PrintStream.class); PrintStream log = mock(PrintStream.class); printDouble(3, out, log); InOrder ordered = inOrder(out, log); ordered.verify(log).println("Computing..."); ordered.verify(out).println("Result: 6"); ordered.verify(log).println("Done."); } This contrived example also verifies the relative ordering of IO done on two separate streams. Again, no API change is necessary, and the actual IO calls may be deep in some library.
  - chriswarbo 11y ago