5 ms·

> On a more serious note, the author is describing a scenario where mocks are generally not useful, ML or not: never mock the code that is under your control if

> On a more serious note, the author is describing a scenario where mocks are generally not useful, ML or not: never mock the code that is under your control if you can help it. When I'm deciding what to mock, I don't use "code under my control" as the determining factor. For me the question is, "is this the code I'm trying to test with this test?". If not, it's often a good idea to mock it even if it's your code, because you want your test failures to indicate where the error is. Of course, you probably shouldn't be testing code that isn't yours, so that does mean you can mock code that isn't under your control. For example, let's say you have two components, A and B, where tests of A end up triggering calls to B. In an ideal world, you have tests of B in isolation, so when writing tests of A, you don't need to test B again. If you don't mock B when you test A, then if something fails in B, you get failures in both A's tests and B's tests, and that makes it harder to figure out what's wrong. Instead, if you mock out B in A's tests, when something fails in B, you only get failures in B, and that pinpoints where the issue is. There are caveats galore. The first caveat that comes to mind: If you're on a project with poor test coverage, B might not have any tests, and it may be worthwhile to write tests against A that don't mock B, so that you get some testing of B "for free". You can mock out B in A's tests once you have tests against B, but that may not be worth the effort. Yes, that's sort of blurring the lines between "unit test" and "integration test", but whether or not the code is tested at all is far more important than whether it's unit tested or integration tested. [...] LLMs are a prime candidate for mocking, because they're non-deterministic. Non-deterministic code it terrible for unit tests, because it will fail randomly, and over time that slowly normalizes ignoring test failures. You can make probablistic assertions on non-deterministic code, but there are two problems with this: 1. The more "wiggle room" you allow in your probablistic assertions, the less useful the assertion is--3rd-standard-deviation results are sometimes bugs, not just unusual results, and you don't want to ignore them, and... 2. Probablistic never equals deterministic; if you're running your tests often (and you should) then you'll get test failures on probablistic tests eventually. Broadening probablistic assertions only decreases the chance of failure, it doesn't remove it. Probablistic tests create an impossible problem: if you tighten your assertions you struggle with meaningless failing tests and normalize ignoring test failures, while if you loosen your assertions your tests aren't really asserting anything any more. And there isn't a happy balance: if you go somewhere in the middle, you end up having both problems. Mocking the LLM removes the non-determinism problem when the LLM is not the unit under test. That is to say, if you're testing code that calls the LLM, it may be a good idea to mock the LLM in those tests. Obviously, don't mock the LLM if your intent is to test the LLM. Another possibility for removing non-determinism, which is probably more appropriate when you are testing the LLM, is to make it non-deterministic. You can do this by abstracting the RNG out of the LLM. Then when you want to test the LLM, you pass in an RNG that is seeded with a constant or even mock the RNG (gasp! You can do that!!?). There are no silver bullets, and this approach has its downsides, because non-breaking changes to the LLM will likely result in changes to the now-deterministic outputs, meaning that a non-breaking change to the LLM code will still result in a bunch of breaking tests. This is why I would prefer mocking the LLM for tests that aren't intended for testing the LLM.

jeremyloy_wt 3y ago
Counterpoint when using mocks - if Bs behavior changes, one may not remember to update A’s test which would be falsely passing. This problem is exacerbated if B is a popular object used by many components. IMO if you own A and B, never use mock. Possibly write a Fake B if B is non deterministic or slow. Then write a parity test for B and Fake B
- kerkeslager 3y ago
  > Counterpoint when using mocks - if Bs behavior changes, one may not remember to update A’s test which would be falsely passing. > This problem is exacerbated if B is a popular object used by many components. This is a great addition to the caveats I mentioned. > IMO if you own A and B, never use mock. I'm going to take issue with your use of the word "never" here. If you said something like, "If you own A and B, don't use mocks the vast majority of the time", I'd even agree--but there are important exceptions (and I'd argue that non-deterministic B is one such exception). > Possibly write a Fake B if B is non deterministic or slow. Then write a parity test for B and Fake B Would you mind explaining what you mean here?
  - jeremyloy_wt 3y ago
    > I’m going to take issue with to you’re use of the word never here Perfectly fair. I shouldn’t have stated it as an absolute. If I could still edit it, I would change my statement to “don’t default to a mock” > Would you mind explaining what you mean here Gladly. Martin Fowler provides really clear definitions for the different types of test doubles. https://martinfowler.com/articles/mocksArentStubs.html https://martinfowler.com/articles/mocksArentStubs.html Maintaining that your Fakes are correct takes work. An easy way that I’ve found to do that is to run the tests against a “real” component and the Fake component with the exact same set of assertions and set up. If that test breaks, then you know that consuming code should also break
    - kerkeslager 3y ago
      Okay, I think I understand. In Fowler's terminology it appears I've been using stubs and mocks and calling them both mocks (shrug). I haven't used fakes so take these critiques with a grain of salt, but I have two concerns with fakes: 1. The process you're describing to verify fakes does take work, and sounds suspiciously like it veers into testing your tests. That's probably worth it for applications like the space shuttle or an X-ray machine that need an extreme degree of reliability, but seems pretty overkill for the kinds of applications that make up most of the work in the software industry. 2. More often than not, I can't think of a useful fake for most things I'd use mocks for. The in-memory database mentioned by Fowler is an exception: you're skipping writing to disk, which saves a lot of time. But in most cases, the fastest implementation of a unit is... the unit, because if it wasn't the fastest you'd use the fake instead of the unit. And all this sort of sets aside the larger issue which is that there are plenty of cases where the unit you want to test isn't the interface between A and B, so having a real B in that test just adds unwanted dependencies. The ideal here is that if you break the interface between A and B, one test fails--the test of that feature of the interface between A and B--which tells you exactly where the bug is. With an LLM, for example, this could look like you calling the real LLM and just testing that the call doesn't call any errors and returns syntactically valid output (even if the output is unpredictable, you can verify the syntax). If you're using fakes everywhere, then a failure of the interface between A and B is going to cause failures in a bunch of A tests, which makes debugging harder, not easier. The case of the in-memory database makes sense because a general purpose fake (the in-memory database) can be written by a third-party and act as an effective mock for all the places you're using the DB--it's easier to use the in-memory database than to mock all the places where you call the DB (and notably, I've never seen anyone write a bunch of tests to test the in-memory database as you describe being a good idea for fakes). And in most cases, I'm using mocks because it's not easier to use the real thing.

jeremyloy_wt 3y ago

Counterpoint when using mocks - if Bs behavior changes, one may not remember to update A’s test which would be falsely passing. This problem is exacerbated if B is a popular object used by many components. IMO if you own A and B, never use mock. Possibly write a Fake B if B is non deterministic or slow. Then write a parity test for B and Fake B

kerkeslager 3y ago
> Counterpoint when using mocks - if Bs behavior changes, one may not remember to update A’s test which would be falsely passing. > This problem is exacerbated if B is a popular object used by many components. This is a great addition to the caveats I mentioned. > IMO if you own A and B, never use mock. I'm going to take issue with your use of the word "never" here. If you said something like, "If you own A and B, don't use mocks the vast majority of the time", I'd even agree--but there are important exceptions (and I'd argue that non-deterministic B is one such exception). > Possibly write a Fake B if B is non deterministic or slow. Then write a parity test for B and Fake B Would you mind explaining what you mean here?
- jeremyloy_wt 3y ago
  > I’m going to take issue with to you’re use of the word never here Perfectly fair. I shouldn’t have stated it as an absolute. If I could still edit it, I would change my statement to “don’t default to a mock” > Would you mind explaining what you mean here Gladly. Martin Fowler provides really clear definitions for the different types of test doubles. https://martinfowler.com/articles/mocksArentStubs.html https://martinfowler.com/articles/mocksArentStubs.html Maintaining that your Fakes are correct takes work. An easy way that I’ve found to do that is to run the tests against a “real” component and the Fake component with the exact same set of assertions and set up. If that test breaks, then you know that consuming code should also break
  - kerkeslager 3y ago
    Okay, I think I understand. In Fowler's terminology it appears I've been using stubs and mocks and calling them both mocks (shrug). I haven't used fakes so take these critiques with a grain of salt, but I have two concerns with fakes: 1. The process you're describing to verify fakes does take work, and sounds suspiciously like it veers into testing your tests. That's probably worth it for applications like the space shuttle or an X-ray machine that need an extreme degree of reliability, but seems pretty overkill for the kinds of applications that make up most of the work in the software industry. 2. More often than not, I can't think of a useful fake for most things I'd use mocks for. The in-memory database mentioned by Fowler is an exception: you're skipping writing to disk, which saves a lot of time. But in most cases, the fastest implementation of a unit is... the unit, because if it wasn't the fastest you'd use the fake instead of the unit. And all this sort of sets aside the larger issue which is that there are plenty of cases where the unit you want to test isn't the interface between A and B, so having a real B in that test just adds unwanted dependencies. The ideal here is that if you break the interface between A and B, one test fails--the test of that feature of the interface between A and B--which tells you exactly where the bug is. With an LLM, for example, this could look like you calling the real LLM and just testing that the call doesn't call any errors and returns syntactically valid output (even if the output is unpredictable, you can verify the syntax). If you're using fakes everywhere, then a failure of the interface between A and B is going to cause failures in a bunch of A tests, which makes debugging harder, not easier. The case of the in-memory database makes sense because a general purpose fake (the in-memory database) can be written by a third-party and act as an effective mock for all the places you're using the DB--it's easier to use the in-memory database than to mock all the places where you call the DB (and notably, I've never seen anyone write a bunch of tests to test the in-memory database as you describe being a good idea for fakes). And in most cases, I'm using mocks because it's not easier to use the real thing.