4 ms·

On a more serious note, the author is describing a scenario where mocks are generally not useful, ML or not: never mock the code that is under your control if y

On a more serious note, the author is describing a scenario where mocks are generally not useful, ML or not: never mock the code that is under your control if you can help it. Also, any test that calls out to another "function" (not necessarily a programming language function) is more than a unit test, and is usually considered an "integration" test (it tests that the code that calls out to something else is written properly). In general, an integration point is sufficiently well covered if the logic for the integration is tested. If you properly apply DI (Dependency Inversion/Injection), replacing external function with a fake/mock/stub implementation allows the integration point to be sufficiently tested, depending on the quality of fake/mock/stub. If you really want to test unpredictable output (this also applies to eg. performance testing), you want to introduce acceptable range (error deltas), and limit the test to exactly the point that's unpredictable by structuring the code appropriatelly. All the other code and tests should be able to trust that this bit of unpredictable behaviour is tested elsewhere and be able to test different outputs.

deleted 3y ago
[deleted]
bluGill 3y ago
I consider the distinction between unit and integration test useless. The important part is tests give me confidence that my system is working, or if not working I can quickly find the problem. I don't care if the bug is in my code or in some third party code, I care that there is a bug. More than once the bug was because the third party code works different from what I expected, but mocks only tell me that if the code works as I expect then my code works - which isn't very good. Fast tests are important. However I find that most integration tests run very fast if I select the right data. I can write unit tests that run very slot if I select the wrong data. I find that the local filesystem is very fast and so I don't worry about testing it (I place my data in a temporary directory - the problems with temporary directories are about security which I don't worry about in unit tests). Likewise I find that testing with a real database is fast enough (my DB is sqlite which supports in-memory databases - not everyone can work with a real database in that way). Which is to say you should challenge your assumptions about what makes a test slow and see if you can work around them.
- necovek 3y ago
  It's funny you bring up speed — people have different ideas of fast: if you need to run 50k tests, filesystem or database through ORM are usually not fast (you won't have those tests complete in sub-9s if that's the not-lose-focus time). Having 50k tests complete in that time when backed by memory on a modern CPU is trivial (easily achievable with proper unit and integration tests). I've been on one too many projects where tests were exactly slow because developers didn't care about testing just-enough when unit- or integration-testing — not to mention becoming parallyzed by any tiny change breaking a ton of unrelated tests. What you are talking about are tests that cover even more than just integration points, and are usually described as end-to-end or system tests (depending on the scope and context of the code under test). I suggest having as few of those, but in general people structure their code badly so sometimes it's hard to avoid it (most web frameworks don't help, for instance).
  - bluGill 3y ago
    I have tools like make (not actually make, but I expect you to know what make is unlike my tools) that ensure while I in theory have more than 100k tests and so running them all would take 10s of minutes, in practice I'm never running more than a few hundred at a time and so the total time is a couple seconds and I don't have to worry. There is more than one way to solve the problem you suggest one alternative. I have found other answers that let me not need to worry so much.
- xandrius 3y ago
  You call both tests (which they both are) but if one specifically uses the word "unit test" then they should try using it appropriately, since there is a reason why the extra word is there.
  - Ma8ee 3y ago
    The problem is that “unit” isn’t well defined. Anything from a method to a service can be considered the unit under test.
kerkeslager 3y ago
> On a more serious note, the author is describing a scenario where mocks are generally not useful, ML or not: never mock the code that is under your control if you can help it. When I'm deciding what to mock, I don't use "code under my control" as the determining factor. For me the question is, "is this the code I'm trying to test with this test?". If not, it's often a good idea to mock it even if it's your code, because you want your test failures to indicate where the error is. Of course, you probably shouldn't be testing code that isn't yours, so that does mean you can mock code that isn't under your control. For example, let's say you have two components, A and B, where tests of A end up triggering calls to B. In an ideal world, you have tests of B in isolation, so when writing tests of A, you don't need to test B again. If you don't mock B when you test A, then if something fails in B, you get failures in both A's tests and B's tests, and that makes it harder to figure out what's wrong. Instead, if you mock out B in A's tests, when something fails in B, you only get failures in B, and that pinpoints where the issue is. There are caveats galore. The first caveat that comes to mind: If you're on a project with poor test coverage, B might not have any tests, and it may be worthwhile to write tests against A that don't mock B, so that you get some testing of B "for free". You can mock out B in A's tests once you have tests against B, but that may not be worth the effort. Yes, that's sort of blurring the lines between "unit test" and "integration test", but whether or not the code is tested at all is far more important than whether it's unit tested or integration tested. [...] LLMs are a prime candidate for mocking, because they're non-deterministic. Non-deterministic code it terrible for unit tests, because it will fail randomly, and over time that slowly normalizes ignoring test failures. You can make probablistic assertions on non-deterministic code, but there are two problems with this: 1. The more "wiggle room" you allow in your probablistic assertions, the less useful the assertion is--3rd-standard-deviation results are sometimes bugs, not just unusual results, and you don't want to ignore them, and... 2. Probablistic never equals deterministic; if you're running your tests often (and you should) then you'll get test failures on probablistic tests eventually. Broadening probablistic assertions only decreases the chance of failure, it doesn't remove it. Probablistic tests create an impossible problem: if you tighten your assertions you struggle with meaningless failing tests and normalize ignoring test failures, while if you loosen your assertions your tests aren't really asserting anything any more. And there isn't a happy balance: if you go somewhere in the middle, you end up having both problems. Mocking the LLM removes the non-determinism problem when the LLM is not the unit under test. That is to say, if you're testing code that calls the LLM, it may be a good idea to mock the LLM in those tests. Obviously, don't mock the LLM if your intent is to test the LLM. Another possibility for removing non-determinism, which is probably more appropriate when you are testing the LLM, is to make it non-deterministic. You can do this by abstracting the RNG out of the LLM. Then when you want to test the LLM, you pass in an RNG that is seeded with a constant or even mock the RNG (gasp! You can do that!!?). There are no silver bullets, and this approach has its downsides, because non-breaking changes to the LLM will likely result in changes to the now-deterministic outputs, meaning that a non-breaking change to the LLM code will still result in a bunch of breaking tests. This is why I would prefer mocking the LLM for tests that aren't intended for testing the LLM.

deleted 3y ago

[deleted]

bluGill 3y ago

I consider the distinction between unit and integration test useless. The important part is tests give me confidence that my system is working, or if not working I can quickly find the problem. I don't care if the bug is in my code or in some third party code, I care that there is a bug. More than once the bug was because the third party code works different from what I expected, but mocks only tell me that if the code works as I expect then my code works - which isn't very good. Fast tests are important. However I find that most integration tests run very fast if I select the right data. I can write unit tests that run very slot if I select the wrong data. I find that the local filesystem is very fast and so I don't worry about testing it (I place my data in a temporary directory - the problems with temporary directories are about security which I don't worry about in unit tests). Likewise I find that testing with a real database is fast enough (my DB is sqlite which supports in-memory databases - not everyone can work with a real database in that way). Which is to say you should challenge your assumptions about what makes a test slow and see if you can work around them.

necovek 3y ago
It's funny you bring up speed — people have different ideas of fast: if you need to run 50k tests, filesystem or database through ORM are usually not fast (you won't have those tests complete in sub-9s if that's the not-lose-focus time). Having 50k tests complete in that time when backed by memory on a modern CPU is trivial (easily achievable with proper unit and integration tests). I've been on one too many projects where tests were exactly slow because developers didn't care about testing just-enough when unit- or integration-testing — not to mention becoming parallyzed by any tiny change breaking a ton of unrelated tests. What you are talking about are tests that cover even more than just integration points, and are usually described as end-to-end or system tests (depending on the scope and context of the code under test). I suggest having as few of those, but in general people structure their code badly so sometimes it's hard to avoid it (most web frameworks don't help, for instance).
- bluGill 3y ago
  I have tools like make (not actually make, but I expect you to know what make is unlike my tools) that ensure while I in theory have more than 100k tests and so running them all would take 10s of minutes, in practice I'm never running more than a few hundred at a time and so the total time is a couple seconds and I don't have to worry. There is more than one way to solve the problem you suggest one alternative. I have found other answers that let me not need to worry so much.
xandrius 3y ago
You call both tests (which they both are) but if one specifically uses the word "unit test" then they should try using it appropriately, since there is a reason why the extra word is there.
- Ma8ee 3y ago
  The problem is that “unit” isn’t well defined. Anything from a method to a service can be considered the unit under test.

kerkeslager 3y ago

> On a more serious note, the author is describing a scenario where mocks are generally not useful, ML or not: never mock the code that is under your control if you can help it. When I'm deciding what to mock, I don't use "code under my control" as the determining factor. For me the question is, "is this the code I'm trying to test with this test?". If not, it's often a good idea to mock it even if it's your code, because you want your test failures to indicate where the error is. Of course, you probably shouldn't be testing code that isn't yours, so that does mean you can mock code that isn't under your control. For example, let's say you have two components, A and B, where tests of A end up triggering calls to B. In an ideal world, you have tests of B in isolation, so when writing tests of A, you don't need to test B again. If you don't mock B when you test A, then if something fails in B, you get failures in both A's tests and B's tests, and that makes it harder to figure out what's wrong. Instead, if you mock out B in A's tests, when something fails in B, you only get failures in B, and that pinpoints where the issue is. There are caveats galore. The first caveat that comes to mind: If you're on a project with poor test coverage, B might not have any tests, and it may be worthwhile to write tests against A that don't mock B, so that you get some testing of B "for free". You can mock out B in A's tests once you have tests against B, but that may not be worth the effort. Yes, that's sort of blurring the lines between "unit test" and "integration test", but whether or not the code is tested at all is far more important than whether it's unit tested or integration tested. [...] LLMs are a prime candidate for mocking, because they're non-deterministic. Non-deterministic code it terrible for unit tests, because it will fail randomly, and over time that slowly normalizes ignoring test failures. You can make probablistic assertions on non-deterministic code, but there are two problems with this: 1. The more "wiggle room" you allow in your probablistic assertions, the less useful the assertion is--3rd-standard-deviation results are sometimes bugs, not just unusual results, and you don't want to ignore them, and... 2. Probablistic never equals deterministic; if you're running your tests often (and you should) then you'll get test failures on probablistic tests eventually. Broadening probablistic assertions only decreases the chance of failure, it doesn't remove it. Probablistic tests create an impossible problem: if you tighten your assertions you struggle with meaningless failing tests and normalize ignoring test failures, while if you loosen your assertions your tests aren't really asserting anything any more. And there isn't a happy balance: if you go somewhere in the middle, you end up having both problems. Mocking the LLM removes the non-determinism problem when the LLM is not the unit under test. That is to say, if you're testing code that calls the LLM, it may be a good idea to mock the LLM in those tests. Obviously, don't mock the LLM if your intent is to test the LLM. Another possibility for removing non-determinism, which is probably more appropriate when you are testing the LLM, is to make it non-deterministic. You can do this by abstracting the RNG out of the LLM. Then when you want to test the LLM, you pass in an RNG that is seeded with a constant or even mock the RNG (gasp! You can do that!!?). There are no silver bullets, and this approach has its downsides, because non-breaking changes to the LLM will likely result in changes to the now-deterministic outputs, meaning that a non-breaking change to the LLM code will still result in a bunch of breaking tests. This is why I would prefer mocking the LLM for tests that aren't intended for testing the LLM.