4 ms·
> Haskell's ability to define restricted subsets of generalize IO is extremely powerful and sidesteps this entire problem. Except it doesn't, precisely where t
by pron 11y ago
> Haskell's ability to define restricted subsets of generalize IO is extremely powerful and sidesteps this entire problem.
Except it doesn't, precisely where that problem is most annoying. For example, suppose you want to test a scheduler, or any mechanism that places a timeout on an operation. You could abstract away the entire timing mechanism, but that is exactly what you're trying to test: maybe the developer used some system function wrong, and is now waiting 5 seconds instead of 5 milliseconds?
This approach (which is the same as the mocking approach -- there's nothing special about Haskell here) doesn't do too well when you want to test timing, and those tests are usually the most brittle (hosted continuous integration services often introduce really long pauses). Of course, you could weave virtual time services throughout your code, but then you have the third-party library problem again.
Mocking (or IO isolating as you call it in Haskell) ends up producing really good unit-tests, but often the more important tests span more than one small unit, and making those non-flaky is very hard.
In theory, you could replace all references to time in a JVM language -- even in libraries -- using an agent or a bootstrap classloader in order to make larger tests run predictably, but I'm not aware of people going to such great lengths (maybe Google does it; I know, for example, that when it comes to accessing the filesystem, they replace the JVM's filesystem provider with an in-memory file-system[1] when they don't mock, but I don't know how the deal with clocks).
EDIT: Apparently somebody has done just that as a research project[2]. It's actually given me an idea for a nice afternoon-project.
[1]: https://github.com/google/jimfs https://github.com/google/jimfs
[2]: http://www3.imperial.ac.uk/pls/portallive/docs/1/55125696.PDF http://www3.imperial.ac.uk/pls/portallive/docs/1/55125696.PD...
- chriswarbo 11y ago> For example, suppose you want to test a scheduler, or any mechanism that places a timeout on an operation. You could abstract away the entire timing mechanism, but that is exactly what you're trying to test: maybe the developer used some system function wrong, and is now waiting 5 seconds instead of 5 milliseconds? Time is an IO effect, so it's true that mocking it away removes that test. I'd treat that kind of "do I understand the API?" issue as an integration test, and split the problem up a little: - "Mock" the timeout calls, to allow as many tests as possible to run outside IO - Write tests where the operation always succeeds (eg. "Just x") - Write tests where the operation always times out (eg. "Nothing") - Write tests which do a combination of success/timeout and a bunch of different interleavings (QuickCheck would probably be useful here) - Write some small integration tests, eg. give a timeout of "5" to a process that sleeps for 5 seconds, it should time out; give a timeout of "5" to a process that sleeps for 4 seconds, it should succeed; and so on. > Mocking (or IO isolating as you call it in Haskell) ends up producing really good unit-tests, but often the more important tests span more than one small unit, and making those non-flaky is very hard. Haskell's IO isolation causes the vast majority of a program to be defined outside IO. You're right that the trickiest parts are often in IO, but at least those are pretty much separate from any complex calculation/logic.
- pron 11y ago> but at least those are pretty much separate from any complex calculation/logic. Yes, but in imperative languages (with meta-programming) you can disentangle mixed IO and logic even without changing the code (or if it's in a third-party library). The advantage is that it allows you (not always, but sometimes) to support even larger tests in this way. Getting those tiny, one-function unit-tests to run predictably is not hard in any language; you just make them smaller and smaller and mock everything else. The tricky bit is making larger tests run predictably, and a flexible, hackable runtime has the advantage there. That's precisely where meta-programming shines.
- chriswarbo 11y ago> in imperative languages (with meta-programming) you can disentangle mixed IO and logic even without changing the code I don't understand what you mean by "mixed", "disentangle" and "change". I would say "mixed IO and logic" is when one block of code describes both how to calculate some value and which side-effects to perform. "Disentangling" these would mean separating the description of the calculation from the description of the side-effects. Doing that "without changing the code" doesn't make sense to me.
- pron 11y agoWhat I mean is that if your code (or a library you call) mixes logic with IO, meta-programming lets you capture the IO calls in a mock (like turning it into an IO monad), which you can then query and verify, without changing the code and the API's it's using. E.g. on the JVM you do that with Mockito (similar mocking libraries exist for other languages/runtimes). Say you have a (silly) method that does computation and IO: void printDouble(int x, PrintStream out) { out.println("Result: " + (x * 2)); } You can then do: @Test public void testPrintDouble() { PrintStream out = mock(PrintStream.class); printDouble(3, out); verify(out).println("Result: 6"); } No output is actually done. The call to println (which may be deep in some library printDouble calls rather than directly in the tested code). We've essentially replaced any object that does direct IO with something akin to an IO monad. We can query the mock in more sophisticated ways, too. Here's a very contrived example: void printDoubleWithLog(int x, PrintStream out, PrintStream log) { log.println("Computing..."); out.println("Result: " + (x * 2)); log.println("Done."); } @Test public void testPrintDoubleWithLog() { PrintStream out = mock(PrintStream.class); PrintStream log = mock(PrintStream.class); printDouble(3, out, log); InOrder ordered = inOrder(out, log); ordered.verify(log).println("Computing..."); ordered.verify(out).println("Result: 6"); ordered.verify(log).println("Done."); } This contrived example also verifies the relative ordering of IO done on two separate streams. Again, no API change is necessary, and the actual IO calls may be deep in some library.
- chadaustin 11y agoHi pron, I get the feeling you're spouting very strong opinions without ever having actually written real production code in Haskell. Because what you're saying is nonsense. Anything you can do in an imperative language you can do in Haskell - the only difference is that Haskell allows you to define subsets of IO that are enforced by the type system. Mocking time is crucial for testing timeouts. You don't want your test to actually take five seconds. You want to rapidly test the code in the case that a timeout occurs before completion of the action and vice versa. You want those tests to be 100% deterministic and fast. With the World approach described you can get that. Also, there is nothing about the technique described that limits you to unit tests. IMVU tests entire web services with this approach and it works well. "World" in that case includes APIs for accessing Redis, MySQL, the customer's session, time, concurrency, and so on. They're all perfectly faked out in tests, and the tests are 100% reliable.
- pron 11y agoOf course I haven't written production code in Haskell, and I'm not saying that there's stuff you can't do in Haskell. So far, I've said two things. One is that the technique described in the post, which "uses the power of Haskell", is actually trivially done (and has been used by countless projects for years) in almost all imperative languages, and less intrusively (it doesn't require introducing new types and works with third-party code). I don't know much about Ruby and Python, but the JVM also allows you to define subsets of IO (even restricted by actual files) that are enforced by the runtime, or subsets of IO operations that will be enforced by the compiler. This technique has been used in imperative languages for a very long time. That's not an opinion but a fact. The second thing I've said is an opinion, and is unrelated to a specific programming language. I said that a lot of very small, fully-faked unit-tests only get you so far, and it's fairly easy to get those tests to run 100% predictable. The more troublesome tests are those that aren't faked, and they are crucial to expose many bugs. That's it. I don't think I've said anoything too controversial.
- chadaustin 11y agoThanks for the clarification. But statements like this are why I think you're spouting crap: "is actually trivially done (and has been used by countless projects for years) in almost all imperative languages, and less intrusively" Is simply not true. In a JavaScript test framework, drawing from top-of-mind, recent examples, you cannot prevent the code under test from replacing document.location. In Java, as you gave in another examples in this thread, you can't statically prevent access to the RNG or perhaps the current time. In Python, there's nothing stopping any code from calling into a C module that has arbitrary effects. Haskell is unique among any languages I've ever used in production that allows deterministic testability as a static, enforced-by-the-compiler guarantee. I partially agree with your second comment, and I do think other forms of testing (such as fuzz testing, integration tests with third parties, acceptance tests) are valuable, but the technique described can be used in much larger tests than simple unit tests. As I mentioned, IMVU uses it for testing entire web service request handlers. I get the feeling some people think that World is onerous and expensive to set up, but in practice you just write all of your code in World and it's not boilerplatey or complicated at all. The framework around World is set up once and never touched again. * Disclaimer: I no longer work at IMVU, but I super super super miss Haskell in production.