5 ms·

I think the OP's point was that it forces you to do all the number crunching via those extensions, and if what you want to do doesn't match up with their APIs i

I think the OP's point was that it forces you to do all the number crunching via those extensions, and if what you want to do doesn't match up with their APIs it's a bit of a pain. Granted, they have fairly complete and flexible APIs but I can still imagine it being very annoying sometimes.

xscott 5y ago
> I think the OP's point was that it forces you to do all the number crunching via those extensions [...] Almost. If you're using C extensions for performance, those extensions can release the GIL, and then you get parallelism in CPython. Although if you've got a bunch of small steps, Amdahl's law is going to bite you as each step needs to re-acquire the the GIL to proceed. Note that for Numpy, I don't know which functions/methods actually bother to release the GIL, but it's possible in theory.
- qsort 5y ago
  Yes, that's exactly what's frustrating about Python. It's not the performance per se, that doesn't really matter for Python's use cases, it's that the performance charateristics of your program are so erratic it doesn't feel like the pieces are fitting together. Denotationally equivalent operations having drastically diverging operational interpretations means that changing anything is a nightmare. If you have an API call that does what you want, Python is basically C++. If you don't, you're paying a 100x penalty. Most of the times performance doesn't matter, and when it doesn't, life is great. When it does matter, the situation feels worse than it actually is because optimizing Python is such a fight.
  - mdomans 5y ago
    In general optimising Python is: * understanding that pure Python is not a language to write CPU-heavy computational code in it and was never intended as such * understanding that Python lets you write 3-4 solutions to a problem easy and 1 will be slow, two will be OK and one will be optimal - often that involves using generators, collections, sets and is a bit nuaced * I'd say that optimising Python, compared to something like C or C++ is far easier - I can set up a decent testing and profiling environment for most of my problems easy with pip and ipython which is borderline simple compared to valgrind
    - qsort 5y ago
      I'm not talking about technical difficulties, the Python dev experience is top notch and I have no complaints in that area. What I mean is that it's hard to form a mental model of what's going on because it's as though you were using two different languages: the "fast" API calls and the "slow" munging of the results. The frustrating part is that the language doesn't seem to "compose" any longer, it doesn't feel uniform: you can't really do things the simple way without paying a hefty peformance penalty. With that said, the #1 reason why any code in any language is slow is "you're using the wrong algorithm", either explicitly or implicitly (wrong data structure, wrong SQL query, wrong caching policy, etc.), and in that sense Python definitely helps you because it makes it easier (or at least less tedious) to use the correct algorithm.
      - kristjansson 5y ago
        I think that complaint applies to the (vast, essential) ecosystem, not to the language and standard library. AFAIK, there aren't serious performance potholes to hit in the language and standard library any more. You're not paying a huge performance penalty for writing things the simple way, but things are uniformly pretty slow. It's when you pull in fast libraries that composition breaks down e.g. the many-fold difference between numpy's sum method and summing a numpy array with a python for-loop and an accumulator.
        qsort 5y ago
        I agree, the standard library doesn't have those problems. As you said though, the ecosystem is kind of the point of using Python in the first place. Again, I'm not really a fan of Python as a general purpose language for large projects, but it really, really excels at being a kind of "universal glue", and it's why I use it again and again. In that role, "uniformly slow" is better than "performance rollercoaster".

xscott 5y ago

> I think the OP's point was that it forces you to do all the number crunching via those extensions [...] Almost. If you're using C extensions for performance, those extensions can release the GIL, and then you get parallelism in CPython. Although if you've got a bunch of small steps, Amdahl's law is going to bite you as each step needs to re-acquire the the GIL to proceed. Note that for Numpy, I don't know which functions/methods actually bother to release the GIL, but it's possible in theory.

qsort 5y ago
Yes, that's exactly what's frustrating about Python. It's not the performance per se, that doesn't really matter for Python's use cases, it's that the performance charateristics of your program are so erratic it doesn't feel like the pieces are fitting together. Denotationally equivalent operations having drastically diverging operational interpretations means that changing anything is a nightmare. If you have an API call that does what you want, Python is basically C++. If you don't, you're paying a 100x penalty. Most of the times performance doesn't matter, and when it doesn't, life is great. When it does matter, the situation feels worse than it actually is because optimizing Python is such a fight.
- mdomans 5y ago
  In general optimising Python is: * understanding that pure Python is not a language to write CPU-heavy computational code in it and was never intended as such * understanding that Python lets you write 3-4 solutions to a problem easy and 1 will be slow, two will be OK and one will be optimal - often that involves using generators, collections, sets and is a bit nuaced * I'd say that optimising Python, compared to something like C or C++ is far easier - I can set up a decent testing and profiling environment for most of my problems easy with pip and ipython which is borderline simple compared to valgrind
  - qsort 5y ago
    I'm not talking about technical difficulties, the Python dev experience is top notch and I have no complaints in that area. What I mean is that it's hard to form a mental model of what's going on because it's as though you were using two different languages: the "fast" API calls and the "slow" munging of the results. The frustrating part is that the language doesn't seem to "compose" any longer, it doesn't feel uniform: you can't really do things the simple way without paying a hefty peformance penalty. With that said, the #1 reason why any code in any language is slow is "you're using the wrong algorithm", either explicitly or implicitly (wrong data structure, wrong SQL query, wrong caching policy, etc.), and in that sense Python definitely helps you because it makes it easier (or at least less tedious) to use the correct algorithm.
    - kristjansson 5y ago
      I think that complaint applies to the (vast, essential) ecosystem, not to the language and standard library. AFAIK, there aren't serious performance potholes to hit in the language and standard library any more. You're not paying a huge performance penalty for writing things the simple way, but things are uniformly pretty slow. It's when you pull in fast libraries that composition breaks down e.g. the many-fold difference between numpy's sum method and summing a numpy array with a python for-loop and an accumulator.
      - qsort 5y ago
        I agree, the standard library doesn't have those problems. As you said though, the ecosystem is kind of the point of using Python in the first place. Again, I'm not really a fan of Python as a general purpose language for large projects, but it really, really excels at being a kind of "universal glue", and it's why I use it again and again. In that role, "uniformly slow" is better than "performance rollercoaster".