6 ms·

> Python can be ahead in the performance race. We just need to get real. I don't know if this is the case. Python's entire value proposition is that it is "ex

by skohan 6y ago

> Python can be ahead in the performance race. We just need to get real. I don't know if this is the case. Python's entire value proposition is that it is "executable pseudocode" with an extremely low barrier to entry - so somebody like a scientist or business analyst can solve problems without a deep understanding of computer science. Those goals are always going to run counter to performance, and python would have to re-invent itself at a foundational level to actually compete on performance with languages which were optimized for it from the ground up. Python's already winning in a lot of super relevant ways from playing to its strengths, and I don't think it makes sense to compromise those strengths to reach middle-of-the-pack performance. If you ask me, if the python community really wanted to advance their interests, the thing to focus on would be dependency management and project encapsulation. If I could have a ux like npm or cargo for python, I would surely be tempted to use it more outside of jupyter. But it would not be because of performance ;)

akullpp 6y ago
FYI: Poetry (https://python-poetry.org https://python-poetry.org) is actually close to something like npm.
- bigfudge 6y ago
  Except like pipenv it is dog slow and totally inscrutable when dependencies clash.
  - stuaxo 6y ago
    It seems a lot faster than pipenv. From the little I've tried, it seems a lot more sane - but I'm biased - I hated pipenv and never saw the point of it.
  - korijn 6y ago
    So have you considered the possibility that it may not be the fault of these package managers, but that there may be a different problem underneath it all that a package manager cannot fix for you?
  - __alexs 6y ago
    It used to be incredibly slow but these days I find it's not so bad. It even has parallel pip installs now which while occasionally triggering some issues in projects with over complicated setup.pys is usually a lot faster than a good old pip install -r requirements.txt.
- skohan 6y ago
  So in my mind, the main things you would need to compete with npm would be: 1. near-universal adoption 2. "just works" experience: i.e. I can clone any random git repo, and run `npm start|build|whatever` and it's always going to work, without having to know anything about my environment or fiddling with configurations. Does poetry do that?
  - hobofan 6y ago
    You'll only really get 2 when 1 is fulfilled. From my experience, with repos that have Poetry set up, I do get a "it just works" experience that I've previously been lacking when working with python. AFAIK there is no "scripts" section (for e.g. "poetry start" like "npm start") in the config file, or at least no one is using them. You'll have to take a quick peek into the README, but the same is true for npm projects, as the script verbs aren't really standardized.
    - dragonwriter 6y ago
      > AFAIK there is no "scripts" section (for e.g. "poetry start" like "npm start") in the config file, or at least no one is using them. There is a scripts section in pyproject.toml, which is leveraged by “poetry run”. This is different than for npm, though, since these aren’t dev-environment scripts but the executable scripts that are installed with the project
    - RBerenguel 6y ago
      You can use a "scripts" section in pyproject.toml (it's what poetry reads, and increasingly more Python tools are leveraging it, which is good), but in general each repository/project offers what it wants to offer as commands, nothing in comparison with the "standards in use" for npm and similar. Personally I have used it to give easy access to the "main thing I want to execute" for my own weird projects. I'm pretty happy using Poetry, and agree with your initial point, partially with the second. There can be some edges with packages involving "anything" binary (to be fair, that is to be expected), and a big issue (for me) is locking the resolution of some libraries, like boto (since botocloud has hundreds of patch releases and the resolver can get pretty crazy unless you play a bit of manual bisection). But this only hits you on poetry lock or poetry add when developing, and only in some cases, I think it's a fair price to pay for a reproducible build.
true_religion 6y ago
I think you and the first poster are talking about different things. They are talking about readability of the cpython compiler and you are talking about readability of the codebase. Python has historically refused to implement many optimizations because they reduce compiler readability.
enriquto 6y ago
> Python's entire value proposition is that it is "executable pseudocode" (...) so somebody like a scientist (...) LOL'd hard at that. What kind of scientist would write "import numpy as np" on their pseudocode? Or multiply matrices with the "@" operator? From the point of view of a scientist, Matlab/Octave, or even Fortran, is executable pseudocode. Numerical stuff in Python seems like an ugly kludge.
- kwertzzz 6y ago
  I don't know why you got downvoted, but I agree that matrix manipulation in numpy look quite foreign to to somebody new to python/numpy. I had heard this "complain" from several colleagues (mostly former matlab-users). Several colleagues have migrated to Julia, which does a quite good job in producing easy readable numerical code. For instance, the dot-product can be written with the Unicode dot symbol (typed as \dot<tab>)
  - enriquto 6y ago
    > matrix manipulation in numpy look quite foreign to to somebody new to python/numpy Not only to newcomers. I've been using python+numpy for more than 10 years, nearly every day, and it despairs me as much as the first day, if not more. I relish the few moments that I get to write numeric algorithms in any other language.
- throwaway287391 6y ago
  As a scientist who had to use MATLAB up until about 2013 because that's what everyone else used, it was such a relief to move to Python. It's true that you can implement a linear algebra routine in a couple fewer characters in MATLAB, but unless that's literally all you're doing, Python is much nicer to work with. The data structures in MATLAB are just a nightmare for general purpose programming, which makes things like loading and parsing data -- things that scientists often need to do -- just terrible. The fact that the notion of the "matrix" (i.e. a 2D array rather than a general ND array) is so deeply baked into everything is a huge headache (a scalar is a 1x1 matrix in MATLAB!). I'm also surprised anyone is particularly bothered by the @ operator. The asterisk for matrix multiplication seems roughly equally unheard of in nicely typeset / handwritten math (you would just write e.g. Ab for a matrix-vector multiplication with no inset operator).
throwaway894345 6y ago
You and the parent are mistaken about the cause of Python’s performance problems. Python isn’t slow because it’s readable—it could be quite a lot faster without compromising readability. Python is slow because it exposes the entire CPython interpreter as the C-extension API, which means they can’t change much about the interpreter without breaking compatibility with some extensions (and they are unwilling to do so). Since performance is so bad, the community leans hard on C-extensions, which worsens the problem. I suspect one of the reasons packaging is so bad is because for each node on the dependency tree, you need to download and execute setup.py in order to discover the node’s direct dependencies, and to get the transitive dependencies you must download and execute the direct dependencies’ setup.py filles. Since these files aren’t idempotent, we can’t reliably cache the results (these scripts could return different sets of dependencies based on the environment or even the current time or anything else). This at least seems to be the reason all reproducible package managers are slow (by which I mean 30+ minutes to resolve dependency versions for a small non-toy project even if the resolved dependencies at their property versions are already cache to disk).
- mikepurvis 6y ago
  Whl archives ship extracted, cacheable metadata, not the setup script.
  - throwaway894345 6y ago
    Not everything is whl though.
- sergiomattei 6y ago
  To clear some confusion: both of you misunderstood my point. I don't think Python's language readability is the problem. The problem is CPython's core team shooting down any performance improvement for the sake of keeping the compiler source code readable.
  - hitekker 6y ago
    The last two discussions I recall on HN: https://news.ycombinator.com/item?id=24922522 https://news.ycombinator.com/item?id=24922522 https://news.ycombinator.com/item?id=11125769 https://news.ycombinator.com/item?id=11125769
conistonwater 6y ago
> Those goals are always going to run counter to performance These days Julia is a very clear counterexample to this sort of claim, I don't think this is true at all.
- skohan 6y ago
  Julia has a tradeoff w.r.t. startup performance doesn't it? Also doens't Julia make pretty heavy use of dynamic dispatch, or am I mistaken about that?
  - eigenspace 6y ago
    Julia's startup time is not short, but it's getting significantly shorter with every release. Right now, the startup overhead is only 0.13 seconds for me. Some packages still take a while to load, e.g. for me to load the plotting pacakge and produce the first plot takes about 8 seconds (all subsequent plots in that session are fast). This is down from like 30 seconds last year. One can also bundle heft packages like Plots.jl into their julia system image so they don't have to recompile all that machinery every time they restart julia.
  - conistonwater 6y ago
    The startup performance issue is just a regular difficult issue, as far as I'm aware. I don't think it has much to do with the goal of "being readable pseudocode" or something. It's getting better with time too. The "readable pseudocode" kind of code is exactly the sort of code in Julia that you almost always expect to be compiled down to native code quite efficiently. The kind of pseudocode I usually see is either straightforward loops iterating over something where the compiler can infer all the necessary information, or calls to library functions where somebody else has already made sure it's good. I use this a lot in my own code, and, like I said, I don't think there is a tradeoff.
Banana699 6y ago
It's a very common misconception that readability is a tradeoff against performance. The true underpinnings of readability is language malleability, nothing more and nothing less. The whole "dynamic languages are automatically more readable" impression is a misleading consequence of the much more general claim "Malleable languages are more readable", with a malleability = dynamicism substitution. Malleability is key because readability is really just languages being as close as possible to the problems they are used to solve. This means one of two things 1)Either the language come preequipped with the concepts and semantics of the problem domain built into their fabric, this is the DSL approach (this will always fail if you aspire to be a "general purpose language", you simply can't hope to match the sheer number of contexts people want to use your language in. One radical conclusion is to abandon the "general purpose language" myth absolutely, and just make all application development language-building, and focus on building the tools to make language building easy.), 2)Or you make the language malleable and stretchy enough that a tiny handful of the rare programmer-domainExperts breed can construct the whole domain inside the langauge, _with_ _the_ _language_ (no transpiler, preprocessor, etc..., this would require the rare programmer-expert to be an even rarer programmer-expert-languageHacker), then cover up all the low level machinery with syntactic elements that mirror the domain vocabulary. That's it. Any langauge that allows you to do the above is a malleable language. Any malleable language is a candidate for being a readable langauge (and a horribly unreadable langauge, if you gave in to irresponsible abstractions). Dynamic languages are malleable because they give you extremely powerful hooks into their semantics, they are very... well, dynamic. The details differ, the two poster childs are python and ruby, they have a grab bag of features that range from operator overloading in python, free form syntax in ruby, and extreme dynamic dispatch and resolution rules in both. The last feature is a common theme in all dynamic languages, and it happens to be a performance killer. But you absolutely don't have to be dynamic to be malleable, lisps have been doing it since forever, and though lisps are traditionally dynamic, it's macros that make them stretchy, not dynamicism. And there are loads of other mechanims that can make a language malleable without making it unpredictable. It can be as simple as scala allowing unicode in it's source, and as complicated as haskell making lazy evaluation the default and now control structures are just a library. Any abstraction with a "meta" flavor, with hooks extending into the language environment and doing various things depending on how you wield them, makes the language more malleable. Malleablity is allowing the language's source to take different shapes and semantics according to the whimes of the programmer, performance comes from massaging the source till it fits comfortably into machine semantics without much runtime shenanigans. They are absolutely not in tension, making the code do everything at runtime is just the easy way out.

by skohan 6y ago

akullpp 6y ago

FYI: Poetry (https://python-poetry.org https://python-poetry.org) is actually close to something like npm.

bigfudge 6y ago
Except like pipenv it is dog slow and totally inscrutable when dependencies clash.
- stuaxo 6y ago
  It seems a lot faster than pipenv. From the little I've tried, it seems a lot more sane - but I'm biased - I hated pipenv and never saw the point of it.
- korijn 6y ago
  So have you considered the possibility that it may not be the fault of these package managers, but that there may be a different problem underneath it all that a package manager cannot fix for you?
- __alexs 6y ago
  It used to be incredibly slow but these days I find it's not so bad. It even has parallel pip installs now which while occasionally triggering some issues in projects with over complicated setup.pys is usually a lot faster than a good old pip install -r requirements.txt.
skohan 6y ago
So in my mind, the main things you would need to compete with npm would be: 1. near-universal adoption 2. "just works" experience: i.e. I can clone any random git repo, and run `npm start|build|whatever` and it's always going to work, without having to know anything about my environment or fiddling with configurations. Does poetry do that?
- hobofan 6y ago
  You'll only really get 2 when 1 is fulfilled. From my experience, with repos that have Poetry set up, I do get a "it just works" experience that I've previously been lacking when working with python. AFAIK there is no "scripts" section (for e.g. "poetry start" like "npm start") in the config file, or at least no one is using them. You'll have to take a quick peek into the README, but the same is true for npm projects, as the script verbs aren't really standardized.
  - dragonwriter 6y ago
    > AFAIK there is no "scripts" section (for e.g. "poetry start" like "npm start") in the config file, or at least no one is using them. There is a scripts section in pyproject.toml, which is leveraged by “poetry run”. This is different than for npm, though, since these aren’t dev-environment scripts but the executable scripts that are installed with the project
  - RBerenguel 6y ago
    You can use a "scripts" section in pyproject.toml (it's what poetry reads, and increasingly more Python tools are leveraging it, which is good), but in general each repository/project offers what it wants to offer as commands, nothing in comparison with the "standards in use" for npm and similar. Personally I have used it to give easy access to the "main thing I want to execute" for my own weird projects. I'm pretty happy using Poetry, and agree with your initial point, partially with the second. There can be some edges with packages involving "anything" binary (to be fair, that is to be expected), and a big issue (for me) is locking the resolution of some libraries, like boto (since botocloud has hundreds of patch releases and the resolver can get pretty crazy unless you play a bit of manual bisection). But this only hits you on poetry lock or poetry add when developing, and only in some cases, I think it's a fair price to pay for a reproducible build.

true_religion 6y ago

I think you and the first poster are talking about different things. They are talking about readability of the cpython compiler and you are talking about readability of the codebase. Python has historically refused to implement many optimizations because they reduce compiler readability.

enriquto 6y ago

> Python's entire value proposition is that it is "executable pseudocode" (...) so somebody like a scientist (...) LOL'd hard at that. What kind of scientist would write "import numpy as np" on their pseudocode? Or multiply matrices with the "@" operator? From the point of view of a scientist, Matlab/Octave, or even Fortran, is executable pseudocode. Numerical stuff in Python seems like an ugly kludge.

kwertzzz 6y ago
I don't know why you got downvoted, but I agree that matrix manipulation in numpy look quite foreign to to somebody new to python/numpy. I had heard this "complain" from several colleagues (mostly former matlab-users). Several colleagues have migrated to Julia, which does a quite good job in producing easy readable numerical code. For instance, the dot-product can be written with the Unicode dot symbol (typed as \dot<tab>)
- enriquto 6y ago
  > matrix manipulation in numpy look quite foreign to to somebody new to python/numpy Not only to newcomers. I've been using python+numpy for more than 10 years, nearly every day, and it despairs me as much as the first day, if not more. I relish the few moments that I get to write numeric algorithms in any other language.
throwaway287391 6y ago
As a scientist who had to use MATLAB up until about 2013 because that's what everyone else used, it was such a relief to move to Python. It's true that you can implement a linear algebra routine in a couple fewer characters in MATLAB, but unless that's literally all you're doing, Python is much nicer to work with. The data structures in MATLAB are just a nightmare for general purpose programming, which makes things like loading and parsing data -- things that scientists often need to do -- just terrible. The fact that the notion of the "matrix" (i.e. a 2D array rather than a general ND array) is so deeply baked into everything is a huge headache (a scalar is a 1x1 matrix in MATLAB!). I'm also surprised anyone is particularly bothered by the @ operator. The asterisk for matrix multiplication seems roughly equally unheard of in nicely typeset / handwritten math (you would just write e.g. Ab for a matrix-vector multiplication with no inset operator).

throwaway894345 6y ago

You and the parent are mistaken about the cause of Python’s performance problems. Python isn’t slow because it’s readable—it could be quite a lot faster without compromising readability. Python is slow because it exposes the entire CPython interpreter as the C-extension API, which means they can’t change much about the interpreter without breaking compatibility with some extensions (and they are unwilling to do so). Since performance is so bad, the community leans hard on C-extensions, which worsens the problem. I suspect one of the reasons packaging is so bad is because for each node on the dependency tree, you need to download and execute setup.py in order to discover the node’s direct dependencies, and to get the transitive dependencies you must download and execute the direct dependencies’ setup.py filles. Since these files aren’t idempotent, we can’t reliably cache the results (these scripts could return different sets of dependencies based on the environment or even the current time or anything else). This at least seems to be the reason all reproducible package managers are slow (by which I mean 30+ minutes to resolve dependency versions for a small non-toy project even if the resolved dependencies at their property versions are already cache to disk).

mikepurvis 6y ago
Whl archives ship extracted, cacheable metadata, not the setup script.
- throwaway894345 6y ago
  Not everything is whl though.
sergiomattei 6y ago
To clear some confusion: both of you misunderstood my point. I don't think Python's language readability is the problem. The problem is CPython's core team shooting down any performance improvement for the sake of keeping the compiler source code readable.
- hitekker 6y ago
  The last two discussions I recall on HN: https://news.ycombinator.com/item?id=24922522 https://news.ycombinator.com/item?id=24922522 https://news.ycombinator.com/item?id=11125769 https://news.ycombinator.com/item?id=11125769

conistonwater 6y ago

> Those goals are always going to run counter to performance These days Julia is a very clear counterexample to this sort of claim, I don't think this is true at all.

skohan 6y ago
Julia has a tradeoff w.r.t. startup performance doesn't it? Also doens't Julia make pretty heavy use of dynamic dispatch, or am I mistaken about that?
- eigenspace 6y ago
  Julia's startup time is not short, but it's getting significantly shorter with every release. Right now, the startup overhead is only 0.13 seconds for me. Some packages still take a while to load, e.g. for me to load the plotting pacakge and produce the first plot takes about 8 seconds (all subsequent plots in that session are fast). This is down from like 30 seconds last year. One can also bundle heft packages like Plots.jl into their julia system image so they don't have to recompile all that machinery every time they restart julia.
- conistonwater 6y ago
  The startup performance issue is just a regular difficult issue, as far as I'm aware. I don't think it has much to do with the goal of "being readable pseudocode" or something. It's getting better with time too. The "readable pseudocode" kind of code is exactly the sort of code in Julia that you almost always expect to be compiled down to native code quite efficiently. The kind of pseudocode I usually see is either straightforward loops iterating over something where the compiler can infer all the necessary information, or calls to library functions where somebody else has already made sure it's good. I use this a lot in my own code, and, like I said, I don't think there is a tradeoff.

Banana699 6y ago

It's a very common misconception that readability is a tradeoff against performance. The true underpinnings of readability is language malleability, nothing more and nothing less. The whole "dynamic languages are automatically more readable" impression is a misleading consequence of the much more general claim "Malleable languages are more readable", with a malleability = dynamicism substitution. Malleability is key because readability is really just languages being as close as possible to the problems they are used to solve. This means one of two things 1)Either the language come preequipped with the concepts and semantics of the problem domain built into their fabric, this is the DSL approach (this will always fail if you aspire to be a "general purpose language", you simply can't hope to match the sheer number of contexts people want to use your language in. One radical conclusion is to abandon the "general purpose language" myth absolutely, and just make all application development language-building, and focus on building the tools to make language building easy.), 2)Or you make the language malleable and stretchy enough that a tiny handful of the rare programmer-domainExperts breed can construct the whole domain inside the langauge, _with_ _the_ _language_ (no transpiler, preprocessor, etc..., this would require the rare programmer-expert to be an even rarer programmer-expert-languageHacker), then cover up all the low level machinery with syntactic elements that mirror the domain vocabulary. That's it. Any langauge that allows you to do the above is a malleable language. Any malleable language is a candidate for being a readable langauge (and a horribly unreadable langauge, if you gave in to irresponsible abstractions). Dynamic languages are malleable because they give you extremely powerful hooks into their semantics, they are very... well, dynamic. The details differ, the two poster childs are python and ruby, they have a grab bag of features that range from operator overloading in python, free form syntax in ruby, and extreme dynamic dispatch and resolution rules in both. The last feature is a common theme in all dynamic languages, and it happens to be a performance killer. But you absolutely don't have to be dynamic to be malleable, lisps have been doing it since forever, and though lisps are traditionally dynamic, it's macros that make them stretchy, not dynamicism. And there are loads of other mechanims that can make a language malleable without making it unpredictable. It can be as simple as scala allowing unicode in it's source, and as complicated as haskell making lazy evaluation the default and now control structures are just a library. Any abstraction with a "meta" flavor, with hooks extending into the language environment and doing various things depending on how you wield them, makes the language more malleable. Malleablity is allowing the language's source to take different shapes and semantics according to the whimes of the programmer, performance comes from massaging the source till it fits comfortably into machine semantics without much runtime shenanigans. They are absolutely not in tension, making the code do everything at runtime is just the easy way out.