5 ms·

Simple: use mocker.patch to patch out random.randrange and have it return a fixed sequence of results that would imply a known result. Your test should not dep

by mlthoughts2018 6y ago

Simple: use mocker.patch to patch out random.randrange and have it return a fixed sequence of results that would imply a known result. Your test should not depend on the internal workings of randrange or anything relying on unfixed random state. Your test is only checking if, given correct results from randrange (or any other external world source of random draws) that the rest of your function correctly produces the two-digit bias number. Otherwise you are just testing randrange itself. If you are asking how to test a pseudorandom number generator, you have a few choices. You can either fix the random seed and test the algorithm’s precise implementation on a large number of known results. Or you can define tests statistically with margins of error, for example testing the entropy in a series of uniformly generated bits or the resulting distribution in a series of random draws from a fixed list, and decide what level of precision is tolerable before considering a test failed.

abdullahkhalids 6y ago
Thanks for your answer. With mocking, what happens if someone writes their if condition as: (if randrange(1000) < 1000*p) or worse, (if randrange(1000) > (1000-1000p). I am trying to write tests to autograde some HW assignments, and so I really have to think adversarially and imagine my students writing the weirdest yet correctly working functions. I really haven't come up with a better answer than checking the statistics of the returned value by running the function a whole bunch of times, like you suggest.
- mlthoughts2018 6y ago
  “How do I test all possible equivalent definitions of this function” is probably not practically answerable for this. What if they wrote this? def biasedcoin(n, p): for answer in answer_key[n][p]: yield answer Or this: def biasedcoin(n, p): while True: mine_bitcoin_on_teachers_computer()
- madsbuch 6y ago
  Sure, this is simple! You provide them with a definition written in Coq and make them implement a proof that their implementation satisfies that definition. Use probabilistic couplings to make sure the output of the biased coin function adheres to specified distribution. You can easily implement a grader that checks the the proof.
nanis 6y ago
> Simple: use mocker.patch to patch out random.randrange and have it return a fixed sequence of results that would imply a known result. Thank you for this. Another way to put this is "your tests for code that uses a 3rd party API should not amount to an uptime test for the 3rd party API." Some time ago, I observed that tests in the Boost.Accumulators library were similarly confounding tests for the code one wrote vs tests of statistical hypotheses: https://www.nu42.com/2016/12/cpp-boost-median-test.html https://www.nu42.com/2016/12/cpp-boost-median-test.html
inertiatic 6y ago
Mocks should be used when the call you depend on is hard to get to behave in the ways you need for your test. "Randrange" isn't hard to get to behave in the way you want. In fact, the way you want it to behave here is all it does! I'd argue if what you want is to do engineering, mocking this isn't what you'd want to do. You want to ensure your code works. And your test can tell you that it works, even if "randrange" works differently while maintaining the same signature after you update the module it came from. If you're doing software engineering for aviation for example, I doubt "well, my tests using mocks passed, it's just that the dependency broke it's contract" is a good enough excuse for a catastrophe.
- mlthoughts2018 6y ago
  No, mocks should be used for any kind of external dependency, whether it is the external OS system calls, built in library functions that interface with sockets, databases, etc. It’s not about whether the resource you are patching is easy / hard to deal with, for example the pytest built in fixture tmpdir abstracts this patching for temporary directories and filesystem ops even though that stuff is “easy” and behaves. Anytime you rely on something outside the runtime environment (even if through a stdlib) you should patch it. You should have a completely different set of tests (integration tests / end to end tests) that exercises important unmocked validation points. And your test runner should allow you to seamlessly switch between the two sets or combine them. For example, use @pytest.mark.integration to distinguish all tests needed unmocked dependencies, and have some “integration-tests.ini” config for that. Then “pytest” runs all the tests with mocks (runs fast, tests logical correctness with tight feedback) and “pytest -c integration-tests.ini” runs all tests or runs the subset requiring real third party resource access. It can run slower, sometimes fail for flaky reasons like network blip, etc.
chriswarbo 6y ago
> Simple: use mocker.patch to patch out random.randrange Why not just pass in 'randrange' as a function argument? def findBias(randrange): heads = tails = 0 for i in range(n): if randrange(100) < 100*p: heads += 1 else: tails += 1 return heads/tails import random bias = findBias(random.randrange) def myTest(): result = findBias(lambda: generateInSomeWay) assert checkInSomeWay(result) It's amazing to me how many complicated and convoluted hacks people can come up with, to avoid calling a function with an argument.
- mlthoughts2018 6y ago
  Putting randrange as an argument creates deeply worse complexity issues and externalizes a needless burden on your user who must carry around their rand function and must understand what type of return value format the passed-in function has to give in order to be usable in the internals of findBias. It’s shocking to me you think this is somehow simpler or that the testability is somehow better. This is how nasty convoluted API get created. Now other functions up the call stack will also have to have a randrange param if they have to pass it forward. At best I could say in some unique use cases there could be API trade offs that make this ok, but definitely not at all for passing around pseudorandom sampler functions. Your change is strictly more confusing, has strictly worse coupling and has a strictly worse API and even after all that, the test is not simpler and isn’t even less code than a one-liner mocker.patch in pytest.
  - chriswarbo 6y ago
    > Putting randrange as an argument creates deeply worse complexity issues and externalizes a needless burden on your user who must carry around their rand function and must understand what type of return value format the passed-in function has to give in order to be usable in the internals of findBias. It’s shocking to me you think this is somehow simpler or that the testability is somehow better. It gives users the option to do that, but doesn't burden them. This can be an implementation detail, whilst the public API provides a default like `bias = lambda: findBias(random.randrange)`. Since it's Python we could also use a default argument, e.g. import random def findBias(randrange=random.randrange): heads = tails = 0 for i in range(n): if randrange(100) < 100*p: heads += 1 else: tails += 1 return heads/tails > This is how nasty convoluted API get created. Now other functions up the call stack will also have to have a randrange param if they have to pass it forward. This is overly-simplistic and disingenuous: If other functions don't care about this parameter, then they don't need to use it, since it's just an implementation detail; they'll call the public API, which will use the default random.randrange, and these callers don't even have to know that the parameter exists. This scenario is strictly an improvement to what you describe; heck, we could even mock 'randrange' like you describe and it would work in exactly the same way! (except we don't need to, since we can just do normal dependency injection via function arguments instead) If other functions do care about this parameter, e.g. because they want to pass along a parameter from their own callers, then that's solving a different problem. In this case it's even better to be passing around parameters: mocking is a last-resort crutch when we're unable to refactor legacy code, but even in those cases it should only be used during testing. It's a very bad idea to be monkey-patching the standard library during the normal course of a program run; especially when it's just to avoid adding an extra parameter to a function we control. > Your change is strictly more confusing, has strictly worse coupling and has a strictly worse API and even after all that, the test is not simpler and isn’t even less code than a one-liner mocker.patch in pytest. I know a lot of this is subjective, but I'd like to point out that the mocking solution essentially works using mutable global variables. If we override the name "random.randrange", we don't actually know what else we might be affecting; for all we know, our bias-calculator could invoke a bunch of helper libraries which happen to rely on certain behaviour from random.randrange, which mocking will break in unexpected ways. In contrast, changing which value we pass as a function argument will not break arbitrary other code far-away which just-so-happens to be using the previous value. That's the fundamental problem with mocking, and why it should be avoided except as a last resort: it forces us to make big, inappropriate assumptions about not only the code we're testing, but also everything else it might ever interact with. The way pytest implements that mocking (by mutating globals like 'random.randint') just makes this even worse.

abdullahkhalids 6y ago

Thanks for your answer. With mocking, what happens if someone writes their if condition as: (if randrange(1000) < 1000*p) or worse, (if randrange(1000) > (1000-1000p). I am trying to write tests to autograde some HW assignments, and so I really have to think adversarially and imagine my students writing the weirdest yet correctly working functions. I really haven't come up with a better answer than checking the statistics of the returned value by running the function a whole bunch of times, like you suggest.

mlthoughts2018 6y ago
“How do I test all possible equivalent definitions of this function” is probably not practically answerable for this. What if they wrote this? def biasedcoin(n, p): for answer in answer_key[n][p]: yield answer Or this: def biasedcoin(n, p): while True: mine_bitcoin_on_teachers_computer()
madsbuch 6y ago
Sure, this is simple! You provide them with a definition written in Coq and make them implement a proof that their implementation satisfies that definition. Use probabilistic couplings to make sure the output of the biased coin function adheres to specified distribution. You can easily implement a grader that checks the the proof.

nanis 6y ago

> Simple: use mocker.patch to patch out random.randrange and have it return a fixed sequence of results that would imply a known result. Thank you for this. Another way to put this is "your tests for code that uses a 3rd party API should not amount to an uptime test for the 3rd party API." Some time ago, I observed that tests in the Boost.Accumulators library were similarly confounding tests for the code one wrote vs tests of statistical hypotheses: https://www.nu42.com/2016/12/cpp-boost-median-test.html https://www.nu42.com/2016/12/cpp-boost-median-test.html

inertiatic 6y ago

Mocks should be used when the call you depend on is hard to get to behave in the ways you need for your test. "Randrange" isn't hard to get to behave in the way you want. In fact, the way you want it to behave here is all it does! I'd argue if what you want is to do engineering, mocking this isn't what you'd want to do. You want to ensure your code works. And your test can tell you that it works, even if "randrange" works differently while maintaining the same signature after you update the module it came from. If you're doing software engineering for aviation for example, I doubt "well, my tests using mocks passed, it's just that the dependency broke it's contract" is a good enough excuse for a catastrophe.

mlthoughts2018 6y ago
No, mocks should be used for any kind of external dependency, whether it is the external OS system calls, built in library functions that interface with sockets, databases, etc. It’s not about whether the resource you are patching is easy / hard to deal with, for example the pytest built in fixture tmpdir abstracts this patching for temporary directories and filesystem ops even though that stuff is “easy” and behaves. Anytime you rely on something outside the runtime environment (even if through a stdlib) you should patch it. You should have a completely different set of tests (integration tests / end to end tests) that exercises important unmocked validation points. And your test runner should allow you to seamlessly switch between the two sets or combine them. For example, use @pytest.mark.integration to distinguish all tests needed unmocked dependencies, and have some “integration-tests.ini” config for that. Then “pytest” runs all the tests with mocks (runs fast, tests logical correctness with tight feedback) and “pytest -c integration-tests.ini” runs all tests or runs the subset requiring real third party resource access. It can run slower, sometimes fail for flaky reasons like network blip, etc.

chriswarbo 6y ago

> Simple: use mocker.patch to patch out random.randrange Why not just pass in 'randrange' as a function argument? def findBias(randrange): heads = tails = 0 for i in range(n): if randrange(100) < 100*p: heads += 1 else: tails += 1 return heads/tails import random bias = findBias(random.randrange) def myTest(): result = findBias(lambda: generateInSomeWay) assert checkInSomeWay(result) It's amazing to me how many complicated and convoluted hacks people can come up with, to avoid calling a function with an argument.

mlthoughts2018 6y ago
Putting randrange as an argument creates deeply worse complexity issues and externalizes a needless burden on your user who must carry around their rand function and must understand what type of return value format the passed-in function has to give in order to be usable in the internals of findBias. It’s shocking to me you think this is somehow simpler or that the testability is somehow better. This is how nasty convoluted API get created. Now other functions up the call stack will also have to have a randrange param if they have to pass it forward. At best I could say in some unique use cases there could be API trade offs that make this ok, but definitely not at all for passing around pseudorandom sampler functions. Your change is strictly more confusing, has strictly worse coupling and has a strictly worse API and even after all that, the test is not simpler and isn’t even less code than a one-liner mocker.patch in pytest.
- chriswarbo 6y ago
  > Putting randrange as an argument creates deeply worse complexity issues and externalizes a needless burden on your user who must carry around their rand function and must understand what type of return value format the passed-in function has to give in order to be usable in the internals of findBias. It’s shocking to me you think this is somehow simpler or that the testability is somehow better. It gives users the option to do that, but doesn't burden them. This can be an implementation detail, whilst the public API provides a default like `bias = lambda: findBias(random.randrange)`. Since it's Python we could also use a default argument, e.g. import random def findBias(randrange=random.randrange): heads = tails = 0 for i in range(n): if randrange(100) < 100*p: heads += 1 else: tails += 1 return heads/tails > This is how nasty convoluted API get created. Now other functions up the call stack will also have to have a randrange param if they have to pass it forward. This is overly-simplistic and disingenuous: If other functions don't care about this parameter, then they don't need to use it, since it's just an implementation detail; they'll call the public API, which will use the default random.randrange, and these callers don't even have to know that the parameter exists. This scenario is strictly an improvement to what you describe; heck, we could even mock 'randrange' like you describe and it would work in exactly the same way! (except we don't need to, since we can just do normal dependency injection via function arguments instead) If other functions do care about this parameter, e.g. because they want to pass along a parameter from their own callers, then that's solving a different problem. In this case it's even better to be passing around parameters: mocking is a last-resort crutch when we're unable to refactor legacy code, but even in those cases it should only be used during testing. It's a very bad idea to be monkey-patching the standard library during the normal course of a program run; especially when it's just to avoid adding an extra parameter to a function we control. > Your change is strictly more confusing, has strictly worse coupling and has a strictly worse API and even after all that, the test is not simpler and isn’t even less code than a one-liner mocker.patch in pytest. I know a lot of this is subjective, but I'd like to point out that the mocking solution essentially works using mutable global variables. If we override the name "random.randrange", we don't actually know what else we might be affecting; for all we know, our bias-calculator could invoke a bunch of helper libraries which happen to rely on certain behaviour from random.randrange, which mocking will break in unexpected ways. In contrast, changing which value we pass as a function argument will not break arbitrary other code far-away which just-so-happens to be using the previous value. That's the fundamental problem with mocking, and why it should be avoided except as a last resort: it forces us to make big, inappropriate assumptions about not only the code we're testing, but also everything else it might ever interact with. The way pytest implements that mocking (by mutating globals like 'random.randint') just makes this even worse.