5 ms·
Making PyPI's test suite faster
- ustad 2y agoThe article uses pytest - does anyone have similar tips when using pythons builtin unittest?
- masklinn 2y agoThe sys.monitoring and import optimisation suggestions apply as-is. If you use standard unittest discovery the third item might apply as well, though probably not to the same degree. I don’t think unittest has any support for distribution so the xdist stuff is a no. On the other hand you could use unit test as the API with Pytest as your test runner. Then you can also use xdist. And eventually migrate to the Pytest test api because it’s so much better.
- kinow 2y agoI wwsn't familiar with this sys.monitoring option for coverage. Going to give it a try in my test suite. At the moment with docker testcontainers, gh actions test matrix for multiple python versions, and unit + regression + integration tests it is taking about 3-5 minutes.
- darkamaul 2y agoWarning, there is a change in coverage 7.7.0 that disables sysmon support for coverage if using branch coverage _and_ a version of Python before 3.14alpha6. [0]: https://coverage.readthedocs.io/en/7.8.0/changes.html#version-7-7-0-2025-03-16 https://coverage.readthedocs.io/en/7.8.0/changes.html#versio...
- kinow 2y agoAh, thank you! I think you just saved me some time!
- anticodon 2y agoI profiled a huge legacy tests collection using cProfile, and found lots of low hanging fruits. Like some tests were creating 4000x3000 Pillow image in memory just to test how image saving code works (checkign that filename and extension are correct). And hundreds of tests had created this huge image for every test (in the setUp method) because of unittest reliance on inheritance. Reducing size image to 10x5 made the test suite faster for like 5-7% (it was long time ago, so I don't remember exact statistics). So, I'd run the tests under cProfile first.
- dmurray 2y agoBut the changes in TFA were of the other of 75% improvement for "dumb" changes that were agnostic to the details of the tests being run. Saying you got a 5-7% improvement from a single change, discovered using the profiler, that took understanding of the test suite and the domain to establish it was OK, and that actually changed the functionality under test - that's all an argument for doing exactly the opposite of what you recommend.
- anticodon 2y ago> that actually changed the functionality under test - that's all an argument for doing exactly the opposite of what you recommend. It was an old functionality. Someone wrote a super class that for the need of testing filesystem functionality created extremely large images. Not only there was no need to test with such large images, other developers eventually inherited more testcases from that setup code (because there were other utility methods), and now setUp code was needlessly creating images that no test used. Generating a huge 4k image takes a significant time using Pillow.
- cocoflunchy 2y agoI don't understand why pytest's collection is so slow. On our test suite (big django app) it takes about 15s to collect tests. So much that we added a util using ripgrep to find the file and pass it as an argument to pytest when using `pytest -k <testname>`.
- piokoch 2y agoEhhh, those pesky Python people, complaining and complaining, average Spring Boot application takes 15s to start even looking if the code compiled ;)
- thom 2y agoLest we start to malign the JVM as a whole, my Clojure test suite, which includes functional tests running headless browsers against a full app hitting real Postgres databases, runs end to end in 20s.
- ffsm8 2y agoThe spring tests are generally quicker then the equivalent python test, so ime - the jvm is mostly to blame. How much time actually goes by after you click "run test" (or run the equivalent cli command) until the test finished running? Any projects using the jvm I've ever worked on (none of which were clojure, admittedly) have always taken at least 10-15s until the pre-phases were finished and the actual test setup began
- thom 2y agoIf I completely clear all cached packages maybe, but I never do that locally or in CI/CD, and that's true of Python too (but no doubting UV is faster than Maven). Clojure/JVM startup time is less than half a second, obviously that's still infinitely more than Python or a systems language but tolerable to me. First test runs after about 2s? And obviously day to day these things run instantly because they're already loaded in a REPL/IPython. Maybe unfair to compare an interpreted language to a compiled one: building an uberjar would add 10 seconds but I'd never do that during development, which is part of the selling point I guess. Either way, I don't think the JVM startup time is really a massive issue in 2025, and I feel like whatever ecosystem you're in, you can always attack these slow test suites and improve your quality of life.
- throwme_123 2y agoIs Trail of Bits transitioning out of "crypto"? Imho, they are one of the best auditors out there for smart contracts. Wouldn't be surprising to see some of these talented teams find bigger markets.
- woodruffw 2y agoNo; Trail of Bits has always had multiple internal groups, including an OSS engineering group that does security and performance engineering. We still do plenty of audits as a company; you can see recent work on that front here[1] :-). Source: I run the group that produced this work. [1]: https://github.com/trailofbits/publications https://github.com/trailofbits/publications
- frogsRnice 2y agoYou all do amazing work, hope I can boast the same someday - or even 50% of it ;) Seriously, you are my heroes!
- frogsRnice 2y agoImo its not just crypto- a lot of their reports are enlightening to read
- bsamuels 2y agoIn addition to what Will posted, published reports for blockchain projects tend to be skewed compared to our other groups. Blockchain clients tend to want to publish the report, but that isn't true for our business lines/projects/clients that are more interesting to HN's audience.
- bgwalter 2y agoI get that pytest has features that unittest does not, but how is scanning for test files in a directory considered appropriate for what is called a high security application in the article? For high security applications the test suite should be boring and straightforward. pytest is full of magic, which makes it so slow. Python in general has become so complex, informally specified and bug ridden that it only survives because of AI while silencing critics in their bubble. The complexity includes PSF development processes, which lead to: https://www.schneier.com/blog/archives/2024/08/leaked-github-python-token.html https://www.schneier.com/blog/archives/2024/08/leaked-github...
- williamdclt 2y ago> it only survives because of AI I don't disagree that it's "complex, informally specified" (idk about bug ridden or silencing critics), but it's just silly to say it only survives because of AI. It was a top-used language before AI got big for web development, data science and all sorts of scientific analysis, and these haven't gone away: I don't expect Python lost much ground in these fields, if any.
- bgwalter 2y agoDropbox moved parts from Python to Golang already in 2014. Google fired the Python team last year and I hear that it does not use Python for new code. Instagram is kept afloat by gigantic hacks. The scientific ecosystem was always there, but relied on heavy marketing to academics, who (sadly) in turn indoctrinate new students to use Python as a first language. I did forget about sysadmin use cases in Linux distributions, but they could be easily replaced by even Perl, as leaner BSD distributions already do.
- guappa 2y agoYou'd be right if go wasn't an awful language designed by someone who clearly failed their compiler class at university.
- deleted 2y ago
- NeutralForest 2y agoPretty good article, it's really a challenge to properly isolate DB operations during testing so having a difference instance per worker is nice. I remember trying to use different schemas (not instances) but I had a hard time to isolate roles as well.
- lyu07282 2y agoIt's more work, but that's one benefit of clean architecture that abstracts the persistence layer. (You can replace it with an in-memory variant.)
- NeutralForest 2y agoI was using https://www.postgresql.org/docs/current/ddl-rowsecurity.html https://www.postgresql.org/docs/current/ddl-rowsecurity.html and needed to check that some complex policies were working correctly so I couldn't just replace with say, SQLite.
- boyd 2y agoThrowing cores at the problem with `pytest-xdist` is typically the lowest hanging fruit, but you still hit all the paper cuts the authors mention -- collection, DB fixtures, import time, etc. And, further optimization is really hard when the CI plumbing starts to dominate. For example, the last Warehouse `test` job I checked has 43s of Github Actions overhead for 51s of pytest execution time (half the test action time and approaching 100% overhead). Disclosure: Have been tinkering on a side project trying to provide 90% of these pytest optimizations automatically, but also get "time-to-first-test-failure" down to ~10 seconds (via warm runners, container snapshotting, etc.). Email in profile if anyone would like to swap notes.
- nine_k 2y agoOne thing not mentioned here is putting your test database on a RAM disk, aka tmpfs. This significantly speeds up all DB-related tests that use transactions, fixture loading, and migrations. In most distros, /tmp is mounted as tmpfs, but YMMV.
- qznc 2y agoI generally try to avoid mocking completely. However, speeding up tests is an appropriate use. If someone changes the implementation the mock usually simply doesn't apply and the test still works as intended. For example, a great speed optimization in our tests recently was to mock time.sleep. Why do we have so many sleeps? This is testing a test framework for embedded devices where there is plenty of fiddling-then-wait-for-the-hardware. I also mocked some file system accesses. Unit testing is about our application logic and not about Linux kernel behavior anyways.