5 ms·

While it points at interesting questions about Python's internals, I hope people writing Python realize that optimizing it is pointless, except for cases where

• pyuser583 3y ago
  It’s also pointless because Python internals have no guarantees. Changing version or even platform can change the absolute efficiency.
  • H8crilA 3y ago
    (I generally agree that everyone should almost always choose the more readable version) In this case the speed difference will likely always be there, since `dict()` can be overriden (monkey-patched) - hence the interpreter needs to resolve it at runtime.
    • pyuser583 3y ago
      Isn’t some logic necessary to determine if you’re dealing with a dictionary or set? Or dictionary/set comprehension? You could probably override it using Python’s extension system. Forgive me, I know Python very well, but not C/C++. I’ve learned to make no assumptions.
      • H8crilA 3y ago
        Check out the Python bytecode in the article, and remember that bytecode is only generated once from the text of the program (but executed many times). "{}" is translated into a simple "make a dict" instruction.
• Spivak 3y ago
  This logic makes no sense, you're saying that trying to boost performance by small increments 0-10% isn't worthwhile on a Civic because it'll never be a Bugatti? That's the least helpful advice when you have a real-life application written in Python and want to get some easy wins on your tight loops. Also Python internals do have some guarantees but in this case it's a semantic guarantee. Because builtins can be shadowed you'll always have to pay the performance cost of looking them up which isn't true for {}.
  • geysersam 3y ago
    It's unlikely to make an important difference. That's why it's a bad idea to spend time on it. It's much more likely there are other more impactful changes you can do to improve performance, changing the algorithm or using another better tool for the job.
  • attractivechaos 3y ago
    I think the analogy is more like: why open a can with a better screwer when you can open it with a can opener. Use the right tool for the right task.
    • vacuity 3y ago
      Considering Python's library ecosystem and the general expanse of Python code, some might say it is the right tool. It's far from ideal, though. A slew of mediocre decisions just begets more, I suppose.
• willseth 3y ago
  You're off by at least an order of magnitude, and a very common class of performance problems that exists in any language is copying and allocating memory. Python is no different, and there are plenty of facilities to address that and other normal performance issues. If you're talking strictly about CPU bound performance problems, that's a bit of a red herring considering there is an entire ecosystem of Python tools to write performant CPU bound code.
  • t8sr 3y ago
    A little late to reply, but you're of course right, there's no direct way to compare the performance. Python serves (among others) a niche, where it's basically the glue between highly optimized C libraries like numpy. If you're writing that kind of CPU bound code, Python just passes blocks of memory around and it's fine for that. I think this is what you're talking about. I think outside of that niche, though, there are places where people are writing heavily CPU bound code in Python, because it is so easy to become CPU bound. Case in point: I recently sped up an ML ingestion pipeline by multiple thousands of percent by switching from a pure-python PDF library to one that wraps a .so written in C. So my point, restated: if you're CPU bound in pure Python code and you have time to try and optimize it, just rewrite the critical section in C and use the FFI. This is how 90% of "Python" libraries get implemented anyway. Compared to this, trying to make Python code more CPU efficient is a waste of time. By the way, I have done work on CPU performance optimizations in Go, Rust and C, and the things you'd typically do are not possible in Python anyway. You're basically left with randomly tweaking the code until the benchmark gives a thumbs up, because it hit on some cpython idiosyncrasy that will completely change around a few versions later.
• einpoklum 3y ago
  If optimization were pointless I'd be out of a job tomorrow. (My current day job is doing GPU optimization on iterative medical image reconstruction computation. But of course, I don't do that in Python.)
  • t8sr 3y ago
    I don't mean optimization is pointless, I mean trying to optimize Python code for CPU performance is pointless. If you're going to spend the time to do that, just rewrite the critical section in a reasonable language, then wrap it in an FFI and use that. Optimize the C code if you need to.
• shiandow 3y ago
  I know hasty optimisation is the root of all evil, but even so this has got to be a bit too far in the other direction. A 2x speed improvement is very significant if it happens to be in the critical loop of your code. You want a 5000% difference? Just find 6 tweaks like these and you might just get there. Of course in most cases whatever you're doing to build the dict is more likely taking up most of the time, not building the dict itself, but understanding why one option is 2x as fast is still important. Though one can only hope that it will soon be irrelevant when JITed python becomes a thing.
  • pi-e-sigma 3y ago
    > A 2x speed improvement is very significant if it happens to be in the critical loop of your code. You want a 5000% difference? Just find 6 tweaks like these and you might just get there. Optimizations don't compound like this.
• remus 3y ago
  Optimisation isn't necessarily handwritten assembly, sometimes you just need to squeeze a little extra juice out of your current setup whether that happens to be a python application or some c code.
  • t8sr 3y ago
    Sure, but if it's a python application, you have two options: 1) Tweak the code without understanding* of how it'll affect performance, because every single line of code hides behind it such complexity that any improvement you find is almost certainly overfitted to your version of Python. Eventually a benchmark will spit out a nice number. If successful, make a modest improvement, maybe 30%. 2) Rewrite the critical section in C. If you need to optimize further, you are now able to draw on 50 years of know-how in a well-studied field. The improvement will be on the order of thousands of percent. They both take about the same amount of time. Why should you do (1)? * There is a difference between random tips like "replace {} with dict()" and fundamentals like how the CPU cache works, or the branch predictor. The former is almost certainly a quirk of the current Python version, the latter has been the same for the past 20-30 years. If you do work on software performance, you rely on your knowledge of the fundamentals and a tool like `perf` to make educated guesses about where you can save some cycles. These fundamentals are basically irrelevant to Python code and so you have to make effectively random attempts.
• uses 3y ago
  Understanding how python's data structures work is actually really important and can make the difference between an algorithm completing in microseconds vs "basically never". Performance problems in python are going to come down to these crucial factors of when to use a set vs list vs dict or a dataframe or something else. The fact that it's interpreted is just not that significant compared to how data structures are set up and operated on. People are using python to explore and manipulate very large data - it's the most popular language for doing so.
• deleted 3y ago
  [deleted]