22 ms·

Faster Python with Guido van Rossum

226 points by WithinReason 5y ago

high_byte 5y ago
how about getting rid of GIL?
- gautamdivgi 5y ago
  I agree. At least an annotation based method of disabling the GIL. If I'm running multi-processing code and its a thread per process then I really don't need the GIL - I should be able to disable it.
- fastball 5y ago
  If you get rid of the GIL you have to replace one big lock with a bunch of very small locks, which actually hurts performance for programs which only need 1 core, which is most programs.
  - throwaway894345 5y ago
    > If you get rid of the GIL you have to replace one big lock with a bunch of very small locks, which actually hurts performance for programs which only need 1 core, which is most programs. Python could do what most other languages do and allow programmers to explicitly add locks to the code that needs to be locked, rather than locking everything, all the time.
    - qbasic_forever 5y ago
      Python has a robust and very capable set of locks and multiprocessing primitives. The issue is that the GIL has been giving you atomic/serialized access to a lot of primitives for free since the inception of the language, so no one has really felt the need to do things like lock access to their basic dicts or lists. You can't just take that code and remove the GIL locking without causing chaos. And as we saw with the decade plus transition from Python 2->3 you can't just tell people to go fix all their old code.
      - throwaway894345 5y ago
        Most code doesn't actually run in a shared memory parallel context precisely because of the GIL, so I think the "chaos" would be minimal. It will still be painful, and would require the ecosystem to adapt (and therefore will never happen because the Python ecosystem lacks the necessary leadership), but it's technically feasible and it would be politically feasible for other language communities.
  - prabhu-yu 5y ago
    Ok, can we do this? Python interpreter can understand if it is running on 1 core processor (assume 1 core = 1 HW thread) or multi core processor. Then use GIL taht suits. ie, if the code uses multiple SW threads, use GIL that suits needs... something like possible? Any reason why experts are not doing this? -p
    - overgard 5y ago
      It's been done; it just was never accepted into the core. And it never will be.
- deleted 5y ago
  [deleted]
- dekhn 5y ago
  it's never going to happen in mainline cpython.
- qbasic_forever 5y ago
  a lot of python code has subtle race conditions where they assume some operations like dictionary access, manipulation, etc. are atomic and the GIL makes it all fine. remove the GIL and there can be a huge explosion of subtle and difficult to find bugs in even simple code. people have been working on trying to remove the GIL for a decade or more now and I don't expect any magic fix soon.
pantsforbirds 5y ago
I'm excited to listen to the full podcast. The highlights look really solid. I think I would like to get into some of the optimization work on Python. I've contributed to some of the DS/ML/NLP libraries in the past, but the core language would be super interesting. Anyone have good resources on getting started on that sort of thing?
amelius 5y ago
Python needs to go multicore, like OCaml. And add a modern concurrent garbage collector. One of the difficulties is of course to not break existing C extensions.
- Qem 5y ago
  Given Python has the multiprocessing module, I always get confused when people talk about Python lack of support for multicore. What are the shortcomings of the multiprocessing module, that cause people to disregard it?
  - amelius 5y ago
    Python's multiprocessing module forces you to serialize all your data, or use ctypes. This means it's either going to be slow, or it's like writing code in a hybrid of Pythonic and C-style code. Which of course goes against the idea of using the simplest tool for the job.
    - fulafel 5y ago
      You can use shared memory without ctypes: https://docs.python.org/3/library/multiprocessing.shared_memory.html https://docs.python.org/3/library/multiprocessing.shared_mem... But you still do have to jump through some hoops then. But like the example in the docs show there are shareable primitives, you can share numpy arrays, etc.
  - stonemetal12 5y ago
    The multiprocessing module is a hack. It has so much overhead that it is not worth the effort in all but the most embarrassingly parallel situations.
  - nicolaslem 5y ago
    One thing that is often overlooked regarding Python multiprocessing is that depending on the version of Python and the OS, the processes are not launched the same way. For example MacOS cannot use fork while it was (is?) the default on Linux.
ctur 5y ago
I have become increasingly convinced Python as a language is a "trap" for any use of notable scale, be the scale about number of developers, codebase size, or performance requirements. It's a great 0->1 language and great at simple glue, but eventually you hit a wall and have to keep investing larger and larger amounts of people or computing resources to get continued returns... all due to fundamental design decisions in the language from two decades ago that are quite difficult to fix. Unless you go all-in on typing -- which is difficult today with any meaningfully sized existing codebase -- maintenance is largely a "hope manual testing and unit tests catch anything resembling type errors" which is a major challenge for, say, structural change to a code base like refactoring. Plus typing is still young, the tooling somewhat immature, and can lead to false senses of security if you aren't very careful and opt into the strictest modes. This makes large number of developers and codebase size a major stumbling block. The interpreter performance and GIL are fundamental issues as well. Multiprocessing and hacks around the GIL are quite painful if you even glance at any native threading code (say in a C++ library) and even when you stick to pure Python, you have a debugging mess when anything goes wrong. But if someone can improve performance, that'd be great, and has massive impact potential. It is incremental though and doesn't solve fundamental issues with the language. I'm also skeptical of the "5x" plan referenced in the blog, and very skeptical we can ever see meaningful removal of the GIL due to library and baked in design decisions in existing code. This means performance will fall further and further behind compiled languages. (I write all of this having been a part of supporting Python at massive scale for over two decades, including at two FAANG companies who invest heavily in it. I've seen the curve and the pain it's caused, and would never use it for any code that needs to be performant or actively developed on the multi-month or year timescale).
- _wldu 5y ago
  This is my experience as well (but not at a FAANG). You can do things quickly in Python and it is a joy to write, but down the road (as you grow), you will regret it and find yourself re-writing it.
- SoylentOrange 5y ago
  What do you use nowadays?
bborud 5y ago
I'm militantly disinterested in the performance of Python until the deployment style of most Python projects is no longer "hunter-gatherer-style installs".
- aazaa 5y ago
  What are "hunter-gatherer-style installs?"
  - taeric 5y ago
    I take that as "keep running pip install on yet another discovered dep until the script works." And heaven help you if you accidentally get conflicting versions installed in the same venv. (By conflicting, I mean two libraries that both depend on different versions of the same underlying library. Odds of this increase dramatically the more you use.)
    - deleted 5y ago
      [deleted]
    - orf 5y ago
      Pip now checks this, but you usually use something like Poetry to give you a concrete lockfile with the correct transient dependencies locked in.
      - taeric 5y ago
        My "heaven help you" was more that, if you are in that situation, you don't really have an exit strategy in python. Do you?
    - singularity2001 5y ago
      yeah people are often lazy about adding a requirements.txt
    - joconde 5y ago
      Neither of these problems have happened to me since pip introduced their new dependency resolver (I think sometime during 2020). All dependencies are installed, and packages in the default repo usually have consistent dependency lists. It's another story for conda, which flat-out refused to install my list of around 10 packages (that seems to stems from the fact that most packages are in conda-forge, and requirements are often inconsistent with the default repo).
gzer0 5y ago
Python is a very easy language to learn and allows the user to build complex systems without having to worry about the syntax. However, this ease of use has led developers into using Python for various purposes that are not right for the language. Python has a fundamental design decision that is difficult, if not impossible, to fix - its dynamic typing. This means it allows users to run code without having to declare what type of data their objects are receiving or giving up any typesafety guarantees with their data. This can lead users into building applications that are prone to errors in the future due to unforeseen changes in the application's structure.
- musingsole 5y ago
  If you fed yourself, it was a perfectly apt use of the language. Purity is just navel gazing.
stinos 5y ago
Best quote at the end: I wish I knew what went on in modern CPUs when it comes to branch prediction and inline caching because that is absolute magic. Seems spot on, and I really wonder as well. Always had the feeling that some decades ago it was still possible to outsmart the CPU (well, and the compiler/optimizer) and get performance improvements by thinking like a CPU, but these days this seems to have become impossible, for this reason I guess?
- WithinReason 5y ago
  Still works with GPUs though
- penteract 5y ago
  It seems possible that intel/AMD have used the Python interpreter in some of their benchmarks, with the result that they either add or tune hardware optimizations in ways that help it. This could mean that people working on the Python interpreter encounter more problems with CPU magic than most people do, since changes they make move it away from what the manufacturers optimized for.
m_ke 5y ago
I'd be happy with just an optimized strict subset of python that uses type annotations to compile performance critical functions like cython. Mypyc sounds like an attempt at that but doesn't seem to be constraining the language enough to allow it to shed the dynamic nature of the language. Another similar effort was EPython, but it doesn't look like anyone is working on it. I guess Numba would be the closest thing to this at the moment.
- overgard 5y ago
  PyPy's RPython is this, although it's probably weirder/more specialized than what you want.
- tekknolagi 5y ago
  Cinder (https://github.com/facebookincubator/cinder https://github.com/facebookincubator/cinder) has Static Python bundled with it, which is pretty useful for replacing Cython. [disclaimer: I work on Cinder]
Decabytes 5y ago
Python will always be important to me. It's the first programming language that I've learned, and it allows me to support my family. It's awesome that almost any problem I have, there is a library or wrapper ready made for me to use. I can google just about any error and get a solution right away. But I've found that I really dislike maintaining larger Python codebases. Whenever I've tried to develop packages or libraries, or create executables it always felt really clunky. And when it comes to personal growth as a programmer, I find it's just too easy to stack ready made libraries on top of libraries, without ever coding things from scratch. But I'm glad to hear that there has been more focus on improving CPython's performance. I don't disagree with the things that the Python Devs have chosen to focus on, and I don't think anyone can really say it hasn't been successful for them. I use Python for work, so I won't forget it anytime soon, but for programming at home, I'll use other languages that align more with my opinions of programming that I've developed after having used Python for an extended period of time.
- SoylentOrange 5y ago
  What do you use nowadays?
- abdabab 5y ago
  > And when it comes to personal growth as a programmer, I find it's just too easy to stack ready made libraries on top of libraries, without ever coding things from scratch. This is working as intended. This is how everything should be.
  - Decabytes 5y ago
    I think it is as well. Python is a batteries included language. But just like I have to hide my phone sometimes to get myself to use it less, I like to drop down into a language that is more niche so I'm less likely to find YouTube videos and Stack Overflow answers to my problems. I've found being forced to read the language reference, and just allowing myself to be stuck on a problem until I've thought my way out of it, helpful for my own personal growth. I don't have the discipline to do this in Python
streamofdigits 5y ago
I wonder now that python made it to the big league, so to speak, if there shouldn't be more high level review of what exactly it should aim to be as an language and ecosystem going forward. Is the speed optimisation problem even well defined otherwise? E.g. tackling performant numerical computations via numpy and other such libraries seems to be a workable pattern. Even with compiled languages like C/C++ and fortran these types of computations are best handled with tuned libraries. For other uses, e.g., API's fastapi/starlette claims to be as fast as go / javascript. Not to say that there isnt a performance issue, but who exactly is experiencing it as a generic issue? E.g., I know for a fact that some python based desktop GUI applications have performance issues (...) but it could be the implementation and in any case GUI use is a bit of a niche (and python on mobile which is the dominant user device these days is very very niche).
- at_a_remove 5y ago
  I agree, it needed everyone to step back and make choices as to direction. Instead, we got the "stone soup" where everyone wanted their favorite feature from some other language added in. You'll see this in most languages, where people want something from another language they are more comfortable with. Privately, I refer to this as the California problem: people move away from California (or anywhere else really, I am picking on California) but then bring all of the baggage and voting habits that made California unpalatable to them eventually. Python is especially vulnerable to this because it conflicts with "There should be one -– and preferably only one –- obvious way to do it." The more features added to the language, the more ways there are to do something, and we then must invent new idioms and lean on "convention."
- BiteCode_dev 5y ago
  It does a lot of basic things right: - iterators everywhere - context managers - generators - namespaces - powerful function arguments - a good stdlib - unicode handling - string formatting It's easy to get started, and then to get going. Turns out a lot of people value that.
czardoz 5y ago
One minor clarification here: Cinder[1] is actually 100% compatible with C extensions (by virtue of being a fork of CPython). [1]: https://github.com/facebookincubator/cinder https://github.com/facebookincubator/cinder
- tekknolagi 5y ago
  Guido appears to have meant to refer to Skybison (https://github.com/facebookexperimental/skybison https://github.com/facebookexperimental/skybison), which is not 100% compatible.
overgard 5y ago
Ugh. Python could be fast, but the python core team / GvR put so many ridiculous constraints on what can change that maybe at best it gets like 10% faster, some day. Anything meaningful means breaking compatibility with extensions, and they absolutely won't do that. I feel like GvR got burned in the python 2->3 transition, but he learned the wrong lesson. The lesson he seemed to learn was never break compatibility but what he should have learned is don't break compatibility for stupid things nobody cares about. Making the interpreter much faster is something people would care about, but they'll never modify it in any meaningful way nor will they ever endorse something like PyPy.
- Joker_vD 5y ago
  The interpreter that breaks the backward compatibility makes that code run infinitely slower (i.e. it does not run at all) instead of running faster. And it doesn't matter if the compatibility was broken for some thing you actually care about: if your code can't run at all until the broken third-party dependency is fixed by its maintainer, you simply won't upgrade, will you?
  - overgard 5y ago
    I work in the Node ecosystem, and while they break compatibility every once in a while I can't say I've ever even really noticed. There's a way to do it without it being a catastrophy.
- joconde 5y ago
  There is no way I'm doing anything serious in a language that decides to make us rewrite part of our work regularly. We have actual work to do... I think you underestimate how important stability is.
  - overgard 5y ago
    Not really suggesting any amount of rewriting of python code, just that extensions might need to be recompiled and/or not depend on reaching into internals they shouldn't touch. To users it should mostly be invisible.
_wldu 5y ago
I got faster Python by switching to Go (a long time ago). I seldom ever use Python now.
CraigJPerry 5y ago
A question i’ve often wondered about is how much slower could you make python before users would leave it behind? Python is often 1-2 orders of magnitude slower than java. That really is a lot yet Python and Java are about comparable in adoption.
- BiteCode_dev 5y ago
  That's because: - speeds matters of that level matters only for a limited number of use cases. Most programming use cases are fine with python speed. They were fine, in fact, with it 20 years ago, before we got that hardware speed. - people are starting to complain about speed in python only because of all the data science going up. Suddenly here speed matters. Before, you had to get to google level of infra to find use cases where python was too slow. - dev are expensive. Hardware is cheap. If you have a slow website, paying $200 dollar more on your server is not a big deal. But taking 3 more month dev that features with your $300k pro is another thing entirely.
- mixmastamyk 5y ago
  I’ve never had an issue with a Python performance, not in twenty years. Possibly because of using it for what it’s good at? Every year it gets faster as well. I did have an issue with the performance of Java twenty years ago, but that was a long time ago of course. Computers at an order of magnitude faster now and java startup improved.
nothrowaways 5y ago
There seem to be big appreciation for go language. How difficult is it to develop a language with (mostly) python syntax while keeping the performance of go? I guess most people who use python use it because of its aesthetics and might even never heard of the GIL issue in python, so i guess among all languages the python syntax is the most liked one.
- rich_sasha 5y ago
  Nim is basically this.
  - daypay 5y ago
    Yep, Nim is a pleasure to work with too.
    - nothrowaways 5y ago
      Looks cool! I like the elegance, i will definitely consider it.
uptownfunk 5y ago
Ok- the million dollar question- what to use instead of python..?

226 points by WithinReason 5y ago

high_byte 5y ago

how about getting rid of GIL?

gautamdivgi 5y ago
I agree. At least an annotation based method of disabling the GIL. If I'm running multi-processing code and its a thread per process then I really don't need the GIL - I should be able to disable it.
fastball 5y ago
If you get rid of the GIL you have to replace one big lock with a bunch of very small locks, which actually hurts performance for programs which only need 1 core, which is most programs.
- throwaway894345 5y ago
  > If you get rid of the GIL you have to replace one big lock with a bunch of very small locks, which actually hurts performance for programs which only need 1 core, which is most programs. Python could do what most other languages do and allow programmers to explicitly add locks to the code that needs to be locked, rather than locking everything, all the time.
  - qbasic_forever 5y ago
    Python has a robust and very capable set of locks and multiprocessing primitives. The issue is that the GIL has been giving you atomic/serialized access to a lot of primitives for free since the inception of the language, so no one has really felt the need to do things like lock access to their basic dicts or lists. You can't just take that code and remove the GIL locking without causing chaos. And as we saw with the decade plus transition from Python 2->3 you can't just tell people to go fix all their old code.
    - throwaway894345 5y ago
      Most code doesn't actually run in a shared memory parallel context precisely because of the GIL, so I think the "chaos" would be minimal. It will still be painful, and would require the ecosystem to adapt (and therefore will never happen because the Python ecosystem lacks the necessary leadership), but it's technically feasible and it would be politically feasible for other language communities.
- prabhu-yu 5y ago
  Ok, can we do this? Python interpreter can understand if it is running on 1 core processor (assume 1 core = 1 HW thread) or multi core processor. Then use GIL taht suits. ie, if the code uses multiple SW threads, use GIL that suits needs... something like possible? Any reason why experts are not doing this? -p
  - overgard 5y ago
    It's been done; it just was never accepted into the core. And it never will be.
deleted 5y ago
[deleted]
dekhn 5y ago
it's never going to happen in mainline cpython.
qbasic_forever 5y ago
a lot of python code has subtle race conditions where they assume some operations like dictionary access, manipulation, etc. are atomic and the GIL makes it all fine. remove the GIL and there can be a huge explosion of subtle and difficult to find bugs in even simple code. people have been working on trying to remove the GIL for a decade or more now and I don't expect any magic fix soon.

pantsforbirds 5y ago

I'm excited to listen to the full podcast. The highlights look really solid. I think I would like to get into some of the optimization work on Python. I've contributed to some of the DS/ML/NLP libraries in the past, but the core language would be super interesting. Anyone have good resources on getting started on that sort of thing?

amelius 5y ago

Python needs to go multicore, like OCaml. And add a modern concurrent garbage collector. One of the difficulties is of course to not break existing C extensions.

Qem 5y ago
Given Python has the multiprocessing module, I always get confused when people talk about Python lack of support for multicore. What are the shortcomings of the multiprocessing module, that cause people to disregard it?
- amelius 5y ago
  Python's multiprocessing module forces you to serialize all your data, or use ctypes. This means it's either going to be slow, or it's like writing code in a hybrid of Pythonic and C-style code. Which of course goes against the idea of using the simplest tool for the job.
  - fulafel 5y ago
    You can use shared memory without ctypes: https://docs.python.org/3/library/multiprocessing.shared_memory.html https://docs.python.org/3/library/multiprocessing.shared_mem... But you still do have to jump through some hoops then. But like the example in the docs show there are shareable primitives, you can share numpy arrays, etc.
- stonemetal12 5y ago
  The multiprocessing module is a hack. It has so much overhead that it is not worth the effort in all but the most embarrassingly parallel situations.
- nicolaslem 5y ago
  One thing that is often overlooked regarding Python multiprocessing is that depending on the version of Python and the OS, the processes are not launched the same way. For example MacOS cannot use fork while it was (is?) the default on Linux.

ctur 5y ago

I have become increasingly convinced Python as a language is a "trap" for any use of notable scale, be the scale about number of developers, codebase size, or performance requirements. It's a great 0->1 language and great at simple glue, but eventually you hit a wall and have to keep investing larger and larger amounts of people or computing resources to get continued returns... all due to fundamental design decisions in the language from two decades ago that are quite difficult to fix. Unless you go all-in on typing -- which is difficult today with any meaningfully sized existing codebase -- maintenance is largely a "hope manual testing and unit tests catch anything resembling type errors" which is a major challenge for, say, structural change to a code base like refactoring. Plus typing is still young, the tooling somewhat immature, and can lead to false senses of security if you aren't very careful and opt into the strictest modes. This makes large number of developers and codebase size a major stumbling block. The interpreter performance and GIL are fundamental issues as well. Multiprocessing and hacks around the GIL are quite painful if you even glance at any native threading code (say in a C++ library) and even when you stick to pure Python, you have a debugging mess when anything goes wrong. But if someone can improve performance, that'd be great, and has massive impact potential. It is incremental though and doesn't solve fundamental issues with the language. I'm also skeptical of the "5x" plan referenced in the blog, and very skeptical we can ever see meaningful removal of the GIL due to library and baked in design decisions in existing code. This means performance will fall further and further behind compiled languages. (I write all of this having been a part of supporting Python at massive scale for over two decades, including at two FAANG companies who invest heavily in it. I've seen the curve and the pain it's caused, and would never use it for any code that needs to be performant or actively developed on the multi-month or year timescale).

_wldu 5y ago
This is my experience as well (but not at a FAANG). You can do things quickly in Python and it is a joy to write, but down the road (as you grow), you will regret it and find yourself re-writing it.
SoylentOrange 5y ago
What do you use nowadays?

bborud 5y ago

I'm militantly disinterested in the performance of Python until the deployment style of most Python projects is no longer "hunter-gatherer-style installs".

aazaa 5y ago
What are "hunter-gatherer-style installs?"
- taeric 5y ago
  I take that as "keep running pip install on yet another discovered dep until the script works." And heaven help you if you accidentally get conflicting versions installed in the same venv. (By conflicting, I mean two libraries that both depend on different versions of the same underlying library. Odds of this increase dramatically the more you use.)
  - deleted 5y ago
    [deleted]
  - orf 5y ago
    Pip now checks this, but you usually use something like Poetry to give you a concrete lockfile with the correct transient dependencies locked in.
    - taeric 5y ago
      My "heaven help you" was more that, if you are in that situation, you don't really have an exit strategy in python. Do you?
  - singularity2001 5y ago
    yeah people are often lazy about adding a requirements.txt
  - joconde 5y ago
    Neither of these problems have happened to me since pip introduced their new dependency resolver (I think sometime during 2020). All dependencies are installed, and packages in the default repo usually have consistent dependency lists. It's another story for conda, which flat-out refused to install my list of around 10 packages (that seems to stems from the fact that most packages are in conda-forge, and requirements are often inconsistent with the default repo).

gzer0 5y ago

Python is a very easy language to learn and allows the user to build complex systems without having to worry about the syntax. However, this ease of use has led developers into using Python for various purposes that are not right for the language. Python has a fundamental design decision that is difficult, if not impossible, to fix - its dynamic typing. This means it allows users to run code without having to declare what type of data their objects are receiving or giving up any typesafety guarantees with their data. This can lead users into building applications that are prone to errors in the future due to unforeseen changes in the application's structure.

musingsole 5y ago
If you fed yourself, it was a perfectly apt use of the language. Purity is just navel gazing.

stinos 5y ago

Best quote at the end: I wish I knew what went on in modern CPUs when it comes to branch prediction and inline caching because that is absolute magic. Seems spot on, and I really wonder as well. Always had the feeling that some decades ago it was still possible to outsmart the CPU (well, and the compiler/optimizer) and get performance improvements by thinking like a CPU, but these days this seems to have become impossible, for this reason I guess?

WithinReason 5y ago
Still works with GPUs though
penteract 5y ago
It seems possible that intel/AMD have used the Python interpreter in some of their benchmarks, with the result that they either add or tune hardware optimizations in ways that help it. This could mean that people working on the Python interpreter encounter more problems with CPU magic than most people do, since changes they make move it away from what the manufacturers optimized for.

m_ke 5y ago

I'd be happy with just an optimized strict subset of python that uses type annotations to compile performance critical functions like cython. Mypyc sounds like an attempt at that but doesn't seem to be constraining the language enough to allow it to shed the dynamic nature of the language. Another similar effort was EPython, but it doesn't look like anyone is working on it. I guess Numba would be the closest thing to this at the moment.

overgard 5y ago
PyPy's RPython is this, although it's probably weirder/more specialized than what you want.
tekknolagi 5y ago
Cinder (https://github.com/facebookincubator/cinder https://github.com/facebookincubator/cinder) has Static Python bundled with it, which is pretty useful for replacing Cython. [disclaimer: I work on Cinder]

Decabytes 5y ago

Python will always be important to me. It's the first programming language that I've learned, and it allows me to support my family. It's awesome that almost any problem I have, there is a library or wrapper ready made for me to use. I can google just about any error and get a solution right away. But I've found that I really dislike maintaining larger Python codebases. Whenever I've tried to develop packages or libraries, or create executables it always felt really clunky. And when it comes to personal growth as a programmer, I find it's just too easy to stack ready made libraries on top of libraries, without ever coding things from scratch. But I'm glad to hear that there has been more focus on improving CPython's performance. I don't disagree with the things that the Python Devs have chosen to focus on, and I don't think anyone can really say it hasn't been successful for them. I use Python for work, so I won't forget it anytime soon, but for programming at home, I'll use other languages that align more with my opinions of programming that I've developed after having used Python for an extended period of time.

SoylentOrange 5y ago
What do you use nowadays?
abdabab 5y ago
> And when it comes to personal growth as a programmer, I find it's just too easy to stack ready made libraries on top of libraries, without ever coding things from scratch. This is working as intended. This is how everything should be.
- Decabytes 5y ago
  I think it is as well. Python is a batteries included language. But just like I have to hide my phone sometimes to get myself to use it less, I like to drop down into a language that is more niche so I'm less likely to find YouTube videos and Stack Overflow answers to my problems. I've found being forced to read the language reference, and just allowing myself to be stuck on a problem until I've thought my way out of it, helpful for my own personal growth. I don't have the discipline to do this in Python

streamofdigits 5y ago

I wonder now that python made it to the big league, so to speak, if there shouldn't be more high level review of what exactly it should aim to be as an language and ecosystem going forward. Is the speed optimisation problem even well defined otherwise? E.g. tackling performant numerical computations via numpy and other such libraries seems to be a workable pattern. Even with compiled languages like C/C++ and fortran these types of computations are best handled with tuned libraries. For other uses, e.g., API's fastapi/starlette claims to be as fast as go / javascript. Not to say that there isnt a performance issue, but who exactly is experiencing it as a generic issue? E.g., I know for a fact that some python based desktop GUI applications have performance issues (...) but it could be the implementation and in any case GUI use is a bit of a niche (and python on mobile which is the dominant user device these days is very very niche).

at_a_remove 5y ago
I agree, it needed everyone to step back and make choices as to direction. Instead, we got the "stone soup" where everyone wanted their favorite feature from some other language added in. You'll see this in most languages, where people want something from another language they are more comfortable with. Privately, I refer to this as the California problem: people move away from California (or anywhere else really, I am picking on California) but then bring all of the baggage and voting habits that made California unpalatable to them eventually. Python is especially vulnerable to this because it conflicts with "There should be one -– and preferably only one –- obvious way to do it." The more features added to the language, the more ways there are to do something, and we then must invent new idioms and lean on "convention."
BiteCode_dev 5y ago
It does a lot of basic things right: - iterators everywhere - context managers - generators - namespaces - powerful function arguments - a good stdlib - unicode handling - string formatting It's easy to get started, and then to get going. Turns out a lot of people value that.

czardoz 5y ago

One minor clarification here: Cinder[1] is actually 100% compatible with C extensions (by virtue of being a fork of CPython). [1]: https://github.com/facebookincubator/cinder https://github.com/facebookincubator/cinder

tekknolagi 5y ago
Guido appears to have meant to refer to Skybison (https://github.com/facebookexperimental/skybison https://github.com/facebookexperimental/skybison), which is not 100% compatible.

overgard 5y ago

Ugh. Python could be fast, but the python core team / GvR put so many ridiculous constraints on what can change that maybe at best it gets like 10% faster, some day. Anything meaningful means breaking compatibility with extensions, and they absolutely won't do that. I feel like GvR got burned in the python 2->3 transition, but he learned the wrong lesson. The lesson he seemed to learn was never break compatibility but what he should have learned is don't break compatibility for stupid things nobody cares about. Making the interpreter much faster is something people would care about, but they'll never modify it in any meaningful way nor will they ever endorse something like PyPy.

Joker_vD 5y ago
The interpreter that breaks the backward compatibility makes that code run infinitely slower (i.e. it does not run at all) instead of running faster. And it doesn't matter if the compatibility was broken for some thing you actually care about: if your code can't run at all until the broken third-party dependency is fixed by its maintainer, you simply won't upgrade, will you?
- overgard 5y ago
  I work in the Node ecosystem, and while they break compatibility every once in a while I can't say I've ever even really noticed. There's a way to do it without it being a catastrophy.
joconde 5y ago
There is no way I'm doing anything serious in a language that decides to make us rewrite part of our work regularly. We have actual work to do... I think you underestimate how important stability is.
- overgard 5y ago
  Not really suggesting any amount of rewriting of python code, just that extensions might need to be recompiled and/or not depend on reaching into internals they shouldn't touch. To users it should mostly be invisible.

_wldu 5y ago

I got faster Python by switching to Go (a long time ago). I seldom ever use Python now.

CraigJPerry 5y ago

A question i’ve often wondered about is how much slower could you make python before users would leave it behind? Python is often 1-2 orders of magnitude slower than java. That really is a lot yet Python and Java are about comparable in adoption.

BiteCode_dev 5y ago
That's because: - speeds matters of that level matters only for a limited number of use cases. Most programming use cases are fine with python speed. They were fine, in fact, with it 20 years ago, before we got that hardware speed. - people are starting to complain about speed in python only because of all the data science going up. Suddenly here speed matters. Before, you had to get to google level of infra to find use cases where python was too slow. - dev are expensive. Hardware is cheap. If you have a slow website, paying $200 dollar more on your server is not a big deal. But taking 3 more month dev that features with your $300k pro is another thing entirely.
mixmastamyk 5y ago
I’ve never had an issue with a Python performance, not in twenty years. Possibly because of using it for what it’s good at? Every year it gets faster as well. I did have an issue with the performance of Java twenty years ago, but that was a long time ago of course. Computers at an order of magnitude faster now and java startup improved.

nothrowaways 5y ago

There seem to be big appreciation for go language. How difficult is it to develop a language with (mostly) python syntax while keeping the performance of go? I guess most people who use python use it because of its aesthetics and might even never heard of the GIL issue in python, so i guess among all languages the python syntax is the most liked one.

rich_sasha 5y ago
Nim is basically this.
- daypay 5y ago
  Yep, Nim is a pleasure to work with too.
  - nothrowaways 5y ago
    Looks cool! I like the elegance, i will definitely consider it.

uptownfunk 5y ago

Ok- the million dollar question- what to use instead of python..?