4 ms·
Performance hacks for faster Python code
- tyingq 11mo agoMaybe also knowing when not to use python, or finding a solution in python that uses C/rust/etc underneath.
- knowitnone3 11mo agomaybe you can skip C and just use assembly
- guhcampos 11mo agoOr fab your own chip.
- postexitus 11mo agoOr wire wrap your own transistors.
- NoteyComplexity 11mo agoOr build yourself a set of vacuum tubes from sketch, don’t forget to make the glass from sand too.
- tyingq 11mo agoIf it's a module doing cryptography, maybe...Not suggesting C or rust in places where it doesn't make sense. Python itself make choices about whether the things it ships with are pure python or not. They often are not.
- lunias 11mo agoIt's kinda funny how uv is written in Rust and many Python libraries where performance is expected to matter (NumPy, Pandas, PyTorch, re, etc.) are implemented in C. Even if you call into fast code from Python you still have to contend with the GIL which I find very limiting for anything resembling performance.
- woodruffw 11mo agoPython's strong native story has always been one of its biggest draws: people find it ironic that so much of the Python ecosystem is native code, but it plays to Python's strength (native code where performance matters, Python for developer joy/ergonomics/velocity). > Even if you call into fast code from Python you still have to contend with the GIL which I find very limiting for anything resembling performance. It depends. A lot of native extension code can run without the GIL; the normal trick is to "detach" from the GIL for critical sections and only reconnect to it once Python needs to see your work. PyO3 has a nice collection of APIs for holding/releasing the GIL and for detaching from it entirely[1]. [1]: https://docs.rs/pyo3/0.27.1/pyo3/marker/struct.Python.html#method.detach https://docs.rs/pyo3/0.27.1/pyo3/marker/struct.Python.html#m...
- SAI_Peregrinus 11mo agoPython is the ultimate (for now) glue language. I'd much rather write a Python script to glue together a CLI utility & a C library with a remote database than try to do that all in C or Rust or BASH.
- lunias 11mo agoYeah, it's great for stuff like that, but I find myself using Node more in that area.
- lunias 11mo agoI didn't know about detaching from the GIL... I'll look into that. > native code where performance matters, Python for developer joy/ergonomics/velocity Makes sense, but I guess I just feel like you can eat your cake and have it too by using another language. Maybe in the past there was a serious argument to be made about the productivity benefits of Python, but I feel like that is becoming less and less the case. People may slow down (a lot) writing Rust for the first time, but I think that writing JavaScript or Groovy or something should be just as simple, but more performant, do multi-threading out of the box, and generally not require you to use other languages to implement performance critical sections as much. The primary advantage that Python has in my eyes is: there are a lot of libraries. The reason why there are a lot of libraries written in Python? I think it's because Python is the number 1 language taught to people that aren't specifically pursuing computer science / engineering or something in a closely related field.
- fainpul 11mo agoPeople are making fun of this statement here / are being sarcastic. But it's a totally legit suggestion. If you know in advance, you are going to make something where performance matters, strongly consider using something other than one of the slowest languages of them all.
- woodruffw 11mo ago(2) surprised me a little. Not because of the performance consequences, but because I almost never see explicit calls to `copy()` in Python (and I read a lot of Python). I think maybe a more realistic example there would be people using splatting without realizing/internalizing that it performs a full copy, e.g. xs = [1, *ys] Another one that stood out was (3). Slots are great, but >95% of the time I'd expect people would want to use `slots=True` with dataclasses instead of manually writing `__slots__` and a constructor like that. `slots=True` has worked since Python 3.10, so every non-EOL version of Python supports it.
- sgarland 11mo agoYou can use __slots__ for normal classes; it’s not limited to only dataclasses.
- woodruffw 11mo agoI know that; that's why I said "I'd expect" not "you can't."
- jessetemp 11mo agoI didn't find 2 surprising either, but I'm a little surprised you never see it. If you want to treat the args to a function as immutable, what can you do besides copy, modify, and return a new object?
- zahlman 11mo ago> what can you do besides copy, modify, and return a new object? You can directly produce a modified copy, rather than using a mutating operation to implement the modifications. It should be noted that "return a modified copy" algorithms can be much more efficient than "mutate the existing data" ones. For example, consider the case of removing multiple elements from a list, specified by a predicate. The version of this code that treats the input as immutable, producing a modified copy, can perform a single pass: def without(source, predicate): return [e for e in source if not predicate(e)] whereas mutating code can easily end up with quadratic runtime — and also be difficult to get right: def remove_which(source, predicate): i = 0 while i < len(source): if predicate(source[i]): # Each deletion requires O(n) elements to shift position. del source[i] else: # The index increment must be conditional, # since removing an element shifts the next one # and that shifted element must also be considered. i += 1
- automatoney 11mo agoThere's some genuinely interesting tips in here, but #10 is for sure just padding so they could call the article "10 Hacks" haha. Everything else is at least somewhat Python specific, but "Hack 10: Avoid repeated function calls in loops" is just applicable to anything.
- knowitnone3 11mo agoI agree. I can't imagine anybody would call a function that returns the same result in a loop like that. There are plenty more optimizations they could come up with. In fact, there are a couple of books https://www.oreilly.com/library/view/high-performance-python/9781492055013/ https://www.oreilly.com/library/view/high-performance-python... for instance. Didn't want to link the one on Amazon
- FuckButtons 11mo agoYeah, 10 felt like it was written by ai.
- zahlman 11mo agoA lot of it felt that way, to me.
- persedes 11mo agoSome helpful guidelines, but it's 2025 and people still use time.time and no stats with their benchmarks :( In general I feel like these kind of benchmarks might change for each python version, so some caveats might apply.
- LigmaBaulls 11mo agoa smart hack for performance is don't use python
- rurban 11mo agoExactly. Consider rewriting the hot paths to a C extension, or using cython or nim. Pypy or the jit rarely
- pansa2 11mo ago> Copying large objects like lists […] can be costly in both time and memory. > modify[ing] objects in place […] improves performance by avoiding the overhead of allocating and populating new structures. AFAIK the poor performance of list copies (demonstrated in the article by a million-element list taking 10ms) doesn’t come from memory allocation nor from copying the contents of the list itself (in this case, a million pointers). Rather it comes from the need to chase all of those pointers, accessing a million disparate memory locations, in order to increment each element’s reference count.
- trollbridge 11mo agoYeah. A more nuanced approach is that you should copy things that don't consist of lots and lots of references to other things, and you should mutate things that are mostly references to other structures. Which means, eventually, designing your data structures so you generally have two types of structures: one which isn't full of pointers, and one which mostly is.
- lunias 11mo agoWhat about using PyPy? You'll probably see a significant improvement in these benchmarks. You should also give it a shot in Node which I expect to be about on par with PyPy, but without the GIL.
- danudey 11mo agoIf anyone wants to be surprised by optimization, a great way to do it is to look at all the cases where, even though Python is slower than C, the Python interpreter written in Python is faster than the Python interpreter written in C. Also, if we're going to suggest 'write it in another language' approaches, rewrite it in Golang. I detest writing in Golang but once you get the hang of things you can get to the point where your code only takes twice the time to write and 2% of the time (and memory) to run.
- lunias 11mo ago> rewrite it in Golang Totally, I'm a big fan of statically typed, compiled languages; especially when the codebase is large and/or there are a lot of contributors. I chose the Node example because I feel like it offers the same "ease-of-use" that draws people to Python. > get to the point where your code only takes twice the time to write and 2% of the time (and memory) to run. 100%. Sometimes this matters, sometimes it doesn't, but if we're talking about "smart performance hacks" this is definitely a top contender. I work on a Python project and I really wish that it supported multi-threading. If I rewrote it, I would prioritize that feature in the target language.
- ecshafer 11mo ago1,2,5 are just kind of standard computer science knowledge things I would expect any CS2 student to know, and are pretty universal across most languages. Some of these are pretty nice python tricks though.
- qsort 11mo agoI'm sure this is plenty useful for less experienced people, but the "smart" hacks read a bit like: Hack 1: Don't Use The Obviously Wrong Data Structure For Your Problem! Hack 2: Don't Have The Computer Do Useless Stuff! Hack 3: Don't Allocate Memory When You Don't Need To! And now, a word from our sponsor: AI! Use AI to help AI build AI with AI, now with 15% more AI! Only with AI! Ask your doctor if AI is right for you! It's worth pointing out that a few of them are Python-specific. Compilers can inline code, there's usually no need to manually inline functions in most languages, that's Python being Python. Which scope the function is from being important is quintessentially Python being Python. The major gains in Python come from... not using Python. Essentially you have to rewrite your code around the fact that numpy and pandas are the ones really doing the work behind the curtain (e.g. aggressively vectorize, use algorithms that can use vectorization well rather than "normal" ones). Number 8 of the list hints at that.
- sevensor 11mo ago“Hacks” 4-10 could easily be replaced with “use numpy.” Performance gains from doing math better in pure Python are minimal compared with numpy. It’s not unusual for the numpy version of something to end up taking 0.01x as long to run.
- fluorinerocket 11mo agoYes I didn't realize anyone actually did anything numerical without numpy. I don't think I've ever imported python's math module once. Who in their right mind is making a 1e6 long python list
- qsort 11mo agoMath is a lost cause, yeah, just use numpy (with the important caveat that you need to know what you're doing, it's easy to fumble badly). But Python has a few interesting features that can easily get you big wins, like generators, e.g. https://www.dabeaz.com/generators/Generators.pdf https://www.dabeaz.com/generators/Generators.pdf
- 11mo ago
- adammarples 11mo agoI remember learning not to use dot access in performance critical loops
- nodesocket 11mo agoWhile obvious, a huge performance improvement bump can had using cachetools on functions. Cachetools is much more feature rich than lru_cache with support for TTL.
- zahlman 11mo agoA previous submission of this (https://news.ycombinator.com/item?id=45937910 https://news.ycombinator.com/item?id=45937910) didn't take off; I'm posting this to cite my comment there (https://news.ycombinator.com/item?id=45940360 https://news.ycombinator.com/item?id=45940360) rather than copying and pasting it.
- yegle 11mo agoFWIW: the `x in foo` test calls the `foo.__contains__` magic method. For a list, the only way to implement it is by iterate through it (see the `list_contains` function in the CPython code). But for the special `range` object, it can implement the `__contains__` efficiently by looking at the start/stop/step (see the `range_contains` source code). Although the Hack 1 is for demonstration purpose, in most cases you can just do `999999 in range(1000000)`. In my test, the same `999999 in foo` is 59.1ns for the range object, 27.7ns for the set, 6.7ms for the list. The set is the fastest, except converting the range object to the set takes 21ms.
- cluckindan 11mo ago6 runs two division operations in the ”unoptimized” version and one in the ”optimized” version. Gee, I wonder why it’s exactly twice faster.
- bjoli 11mo agoIsn't this pretty obvious if you spent more than three weeks writing python? There are lots tricks, but these listed here seem relevant to all languages. There is an almost an incountable amount of ways to write bad and slow python code that are not as obvious. I remember the day I realized how much I dislike python. I have never had it click for me, despite writing it in and off since the python 2.0. There are always some new arbitrary places for it to bite you, and it always feels a little yucky. And then one day I saw something like this: # b is not defined here if blahblah(): b = gronk() # might raise an exception do_stuff(b) That tingles in all the wrong places. Ruby has other issues, but the core is still feels elegant. I still prefer scheme, though. Edit: and maybe someone can explain to me why anyone would make it so that the simplest way to iterate through a collection is not the fastest? This is the case for most languages, but it still feels dumb. Just allow a slightly more obtuse syntax like for a in b<list> and make that case just as fast as doing it with a while loop. The iterator protocol is imho for when we frankly don't know/care what we are iterating over, or when we elwant to be generic, or when the data structure can't expose the most efficient way of doing it (like a tree). There is no reason why for a in b: should be slower than while when b is a list or string.
- givemeethekeys 11mo agoHack #2 seems counter to the functional way of doing things. The hack for using Math instead of operators seems dumb.
- rewgs 11mo agoTry as I might, I can't get that to raise an exception. That said, agreed that this is weird. Python makes some really unintuitive choices regarding scope.
- utbabya 11mo agoMost of the time I find the pros of not mutating variables out weight any potential memory / performance gain, of course it depends on what you're doing, but I find it rare other than perhaps scientific related code.
- driggs 11mo agoFunny that Hack #1 compares list versus set value lookup, but the timer doesn't include the time to copy the list into a set. Hack #2 warns against unnecessary copying, and the time for copying the list is almost the same as the performance gain in Hack #1. In fact, creating a set takes longer than copying a list since it requires hash insertion, so it's actually much faster to do the opposite of what they suggest for #1 (in the case of a single lookup, for this test case). Here's the results with `big_set = set(big_list)` inside the timing block for the set case: List lookup: 0.013985s Set lookup: 0.052468s
- godelski 11mo agoI'm surprised that this was not the first complaint. It's pretty clear "Hack" #1 isn't going to work. import random import time def timeit(func, _list, n=1000): start = time.time() for _ in range(n): func(_list=_list ) end = time.time() print(f"Took {end-start} s") return def lstsearch(_list): sf = random.randint(0,len(_list)) if sf in _list: return return def setsearch(_list): sf = random.randint(0, len(_list)) if sf in set(_list): return return mylist = list(range(100000)) timeit(lstsearch, mylist) timeit(setsearch, mylist) ---- Took 0.23349690437316895 s Took 0.8901607990264893 s
- mont_tag 11mo agoAll good suggestions. However, the last one should have also mentioned functools.cache and functools.lru_cache.