7 ms·
Solving Algorithmic Problems in Python with Pytest (2019)
- barefeg 6y agoIf pytest is a bat mobile, I wonder what the author will think of Jest
- makapuf 6y agoI see the point of pytest and the greatness of it. I like pytest but I like minimalism more. I dont find it a good showcase here: couldn't you have kept only the asserts and just run the script?
- jaimebuelta 6y agoIn that case, only the first fail will be reported and exit. Which is fair enough in some cases, but may be confusing for long sequences of tests. (There’s the option to get that behaviour in pytest as well with a parameter. Also, pytest may run the tests in parallel, which may speed up the results)
- makapuf 6y agoWell sure but a simple function if a!=b print(f"{a} is not {b}") could do the job fine.
- 0x008 6y agoTest driven development is just such a good habit to get into. You trade just a little bit of ramp-up speed at the beginning of an implementation for a massive reduction in cognitive overhead.
- Yajirobe 6y agoI'd love to see how test driven development looks like for ML systems. You write the test for your non-existent model, then write the model and try to train the model to 'pass' the test? Do you also train it on the test case or on other data only?
- ploika 6y agoIt's doable once you're out of pure experimentation and into the development phase at which test driven development can help. Test that this ETL function expects a DataFrame with a given schema and returns one with a different (but also known) schema, even with all these edge cases in the filters and group-bys. Test that the "train_classifier" method/function rejects negative penalisation parameters, returns an object of type X (a trained sklearn object say, or dictionary of weights that can be deserialised), fails loudly if you don't have enough samples from category Y etc. Test that the predict method returns a probability as a float, a predicted class as an int, a DataFrame with metadata and headers, etc etc.
- barefeg 6y agoThese type tests are not really model tests but of the infrastructure surrounding the model. MLOps tools are tested like normal software development
- Yajirobe 6y agoAnd how do you test that your model has sufficient accuracy? How do you make sure that your model does not deteriorate over time?
- jononor 6y agoassert test_performance > threshold ? This would only be a sanity check for big deviations though. For more subtle changes one may need to do a proper statistical test. I believe the jury is still out of how to do that properly for deep ML models - challenge is the lack of independence in CV folds and generally the compute time it takes to evaluate. However, I would probably not do performance check inside a unit-testing framework. Instead treat this as quality indicators like performance benchmarks, code coverage etc. It may be a "gate", that needs to pass to allow a new model into production. To evaluate performance over time, one would preferably want labeled datasets for test gathered at different points in time. Which requires a (reliable) continuous labeling process. One can also gather customer feedback about performance, track those as metrics. These things are probably more in the "monitoring" part of a system, rather than unit-testing time though.
- eidorb 6y agoI like using doctest in these simple cases - the tests go in the docstrings! (Also, it’s part of the standard library.) https://docs.python.org/3/library/doctest.html https://docs.python.org/3/library/doctest.html
- rational_indian 6y agoWhy do you need the minimum == 0 test in the following chunk of code ... if i > 0 and (minimum == 0 or i < minimum): minimum = i
- mlthoughts2018 6y agoThe stated problem requires returning the smallest integer greater than 0 in the list. 0 will only be the running minimum if only non-positive numbers have been seen so far, and since i is greater than zero in the condition, if minimum == 0 it means i is the first positive integer we’ve seen, so even though i > minimum (i > 0) we need to swap i as the minimum at that point.
- rational_indian 6y agoThanks, that explains it.
- abdullahkhalids 6y agoHow would you write tests for the following problem with random output? Write a function biasedcoin(n,p) that takes the number of coin flips, n, and the probability of heads, p. It flips a biased coin n times, and returns the ratio of number of heads/number of tails. p is guaranteed to have two significant numbers eg. p = 0.60 or p = 0.74. You should use the random.randrange function to generate random numbers. Example implementation def biasedcoin(n,p): import randrange from random heads = tails = 0 for i in range(n): if randrange(100) < 100*p: heads += 1 else: tails += 1 return heads/tails
- mlthoughts2018 6y agoSimple: use mocker.patch to patch out random.randrange and have it return a fixed sequence of results that would imply a known result. Your test should not depend on the internal workings of randrange or anything relying on unfixed random state. Your test is only checking if, given correct results from randrange (or any other external world source of random draws) that the rest of your function correctly produces the two-digit bias number. Otherwise you are just testing randrange itself. If you are asking how to test a pseudorandom number generator, you have a few choices. You can either fix the random seed and test the algorithm’s precise implementation on a large number of known results. Or you can define tests statistically with margins of error, for example testing the entropy in a series of uniformly generated bits or the resulting distribution in a series of random draws from a fixed list, and decide what level of precision is tolerable before considering a test failed.
- abdullahkhalids 6y agoThanks for your answer. With mocking, what happens if someone writes their if condition as: (if randrange(1000) < 1000*p) or worse, (if randrange(1000) > (1000-1000p). I am trying to write tests to autograde some HW assignments, and so I really have to think adversarially and imagine my students writing the weirdest yet correctly working functions. I really haven't come up with a better answer than checking the statistics of the returned value by running the function a whole bunch of times, like you suggest.
- morelandjs 6y agoSide remark, but I really wish interviewers gave me the option of a terminal with vim since that’s pretty much standard on any Unix machine I might ssh into these days. All too often someone puts me in some weird IDE and I feel like a cat with boots on.
- Fiveplus 6y agoSomeone on our school's group chat shared they had a technical interview where she was asked to write code on a shared google doc. I really hope that was an outlier.
- mrfox321 6y agoGoogle did this during the pandemic Are other companies copying them to the Tee?
- konjin 6y agoI had someone do that to me a few weeks back. I told them flat out no, gave them my ip, made an account on my local machine and started a shared tmux session over ssh. Since that's my usual ide I didn't lose any capability.
- O_H_E 6y agoNice power move. But I have to have my ip publicly accessable (permission from isp?) beforehand right?
- konjin 6y agoDepends on the ISP, my mobile dongle has a dynamic public ip, I have a little twitter bot that sends out a tweet every time its ip changes. On my wired connection I have 1tb symmetric fiber with 16 static ipv4 ips and an arbitrarily large number of ipv6 addresses, I just need to register more if I need them (which I really don't). That said I didn't get the job, so caveat emptor.
- tmoertel 6y agoWhen testing solutions to algorithmic problems, it's often useful to use randomized property checking to verify that the solution's expected properties hold for approximately all inputs. The QuickCheck family of testing tools is probably the best known application of this approach. It's also pretty easy to roll your own. Some hand-rolled examples in Python: https://github.com/tmoertel/practice/blob/master/dailycodingproblem_com/find_non_triplicate_int.py#L266 https://github.com/tmoertel/practice/blob/master/dailycoding... https://github.com/tmoertel/practice/blob/master/dailycodingproblem_com/p18_max_subarray_values.py#L135 https://github.com/tmoertel/practice/blob/master/dailycoding... https://github.com/tmoertel/practice/blob/master/dailycodingproblem_com/p273_find_fixed_point_in_sorted_array.py#L88 https://github.com/tmoertel/practice/blob/master/dailycoding...
- henrikeh 6y ago> unittest is an old horse and cart, while pytest is the batmobile. I really wish more was said than just this. Why is it that pytest is often preferred and unittest is considered a bad choice? What is the impact on the quality of the tests?
- andreareina 6y agoPytest: - lets you use regular assert statements, rather than assertEqual, assertTrue, etc. - does not impose the use of classes - the way it does fixtures just feels nicer to use than unittest Basically, it's more ergonomic and IME more flexible.