5 ms·

The slow startup combined with the general lack of interest of the Python ecosystem to try to find a solution for distributing self contained applications was t

The slow startup combined with the general lack of interest of the Python ecosystem to try to find a solution for distributing self contained applications was the biggest reason we ended up writing out CLI tool in something else even though we are a Python shop. I'm really curious why there hasn't been much of a desire to change this and it even got worse as time progressed which is odd.

stevekemp 9y ago
Indeed this is a long-standing issue with Python. LWN gave some excellent coverage late last year, in this piece: https://lwn.net/Articles/730915/ https://lwn.net/Articles/730915/
fulafel 9y ago
Sure there has been desire to change this. It's a hard problem, and there are tradeoffs.
- the_mitsuhiko 9y ago
  It’s only a hard problem if there is no desire. The slowdowns for the interpreter startup did not happen because they are necessary but because site.py and friends just do more stuff now and a lot of important internal tooling became unnecessarily complex.
  - zbxxx 9y ago
    Yeah, this whole "importlib in Python" thing continues to mystify me.
    - pas 9y ago
      It would be okay, if Python would support a low level faster subPython within itself. RPython for example.
sametmax 9y ago
> it even got worse as time progressed which is odd. Quite the contrary, as I stated in my other comment, we now have nuikta.
- coldtea 9y ago
  Parents means got worse in official Python releases. Not what some fringe tool can or cannot do.
chubot 9y ago
This is disappointing to me too, but I think there are some problems baked in to the language that make it hard. - Imports can't be parsed statically. - Startup time has two major components: crawling the file system for imports, and running all the init() functions of every module, which happens before you get to main(). The first is only fixable through breaking changes, and the second is hard to fix without drastically changing the language. The import code in CPython was a mess, which was apparently cleaned up by importlib in Python 3, through tremendous effort. But unfortunately I think importlib made things slower? I recall a PyCon talk where as of 3.6, essentially everything about Python 3 is now faster than Python 2, EXCEPT startup time! This is a shame, because I would have switched to Python 3 for startup time ALONE. (As of now, most of my code and that of my former employer is Python 2.) That would have been the perfect time to address startup time, because getting a 2x-10x improvement (which is what's needed) requires breaking changes. I don't think there's a lack of interest in the broader Python community, but there might be a lack of interest/manpower in the core team, which leads to the situation wonderfully summarized in the recent xkcd: https://xkcd.com/1987/ https://xkcd.com/1987/ FWIW I was the one who sent a patch to let Python run a .zip file back in 2007 or so, for Python 2.6 I think. This was roughly based on what we did at Google for self-contained applications. A core team member did a cleaner version of my patch, although this meant it was undocumented until Python 3.5 or so: https://docs.python.org/3/library/zipapp.html https://docs.python.org/3/library/zipapp.html The .zip support at runtime was a start, but it's really the tooling that's a problem. And it's really the language that inhibits tooling. Also, even if you distributed self-contained applications, the startup time is not great. It's improved a bit because you're "statting" a zip file rather than making syscalls, but it's still not great. In other words, I have wondered about this "failure" for over a decade myself, and even tried to do something about it. I think the problem is that there are multiple parts to the solution, the responsibility for these parts is distributed. I hate to throw everything on the core team, but module systems and packaging are definitely a case where "distributed innovation" doesn't work. There has to be a central team setting standards that everyone else follows. Also, it's not a trivial problem. Go is a static language and is doing better in this regard, but still people complain about packaging. (vgo is coming out after nearly a decade, etc.) I should also add that while I think Python packaging is in the category of "barely works", I would say the same is true of Debian. And Debian is arguably the most popular Linux package manager. They're cases of "failure by success".
blattimwind 9y ago
I've written a whole bunch of CLI tools over the years and maintained some I didn't author originally, I always found it annoying how slow these are. A CLI tool for some larger project can easily take a second just to display --help, or a command line parse error. Tests running against it can be made to progress faster (no forking, no separate interpreter, in-process capture etc. which brings a lot of complexity and subtle differences and error persistence/"tainting" of the execution environment), but still you might only get a few CLI invocations per second per core. Theses experience are a major turn-down from Python for me.
cdavid 9y ago
One "simple" thing that could be done is to make it easier to build python statically, and improve the freezing toolchain. When I used to care about the problem, I looked into it (https://cournape.wordpress.com/2015/09/28/distributing-python-programs-ala-go-as-a-single-file/ https://cournape.wordpress.com/2015/09/28/distributing-pytho...) and got somewhere. It improves somewhat startup times, and allow distribution of a single binary. Lots of libraries are terribly slow to import (e.g. requests), but right now there is little incentive to fix it as there is no toolchain to build good, self-contained python CLI apps.

stevekemp 9y ago

Indeed this is a long-standing issue with Python. LWN gave some excellent coverage late last year, in this piece: https://lwn.net/Articles/730915/ https://lwn.net/Articles/730915/

fulafel 9y ago

Sure there has been desire to change this. It's a hard problem, and there are tradeoffs.

the_mitsuhiko 9y ago
It’s only a hard problem if there is no desire. The slowdowns for the interpreter startup did not happen because they are necessary but because site.py and friends just do more stuff now and a lot of important internal tooling became unnecessarily complex.
- zbxxx 9y ago
  Yeah, this whole "importlib in Python" thing continues to mystify me.
  - pas 9y ago
    It would be okay, if Python would support a low level faster subPython within itself. RPython for example.

sametmax 9y ago

> it even got worse as time progressed which is odd. Quite the contrary, as I stated in my other comment, we now have nuikta.

coldtea 9y ago
Parents means got worse in official Python releases. Not what some fringe tool can or cannot do.

chubot 9y ago

This is disappointing to me too, but I think there are some problems baked in to the language that make it hard. - Imports can't be parsed statically. - Startup time has two major components: crawling the file system for imports, and running all the init() functions of every module, which happens before you get to main(). The first is only fixable through breaking changes, and the second is hard to fix without drastically changing the language. The import code in CPython was a mess, which was apparently cleaned up by importlib in Python 3, through tremendous effort. But unfortunately I think importlib made things slower? I recall a PyCon talk where as of 3.6, essentially everything about Python 3 is now faster than Python 2, EXCEPT startup time! This is a shame, because I would have switched to Python 3 for startup time ALONE. (As of now, most of my code and that of my former employer is Python 2.) That would have been the perfect time to address startup time, because getting a 2x-10x improvement (which is what's needed) requires breaking changes. I don't think there's a lack of interest in the broader Python community, but there might be a lack of interest/manpower in the core team, which leads to the situation wonderfully summarized in the recent xkcd: https://xkcd.com/1987/ https://xkcd.com/1987/ FWIW I was the one who sent a patch to let Python run a .zip file back in 2007 or so, for Python 2.6 I think. This was roughly based on what we did at Google for self-contained applications. A core team member did a cleaner version of my patch, although this meant it was undocumented until Python 3.5 or so: https://docs.python.org/3/library/zipapp.html https://docs.python.org/3/library/zipapp.html The .zip support at runtime was a start, but it's really the tooling that's a problem. And it's really the language that inhibits tooling. Also, even if you distributed self-contained applications, the startup time is not great. It's improved a bit because you're "statting" a zip file rather than making syscalls, but it's still not great. In other words, I have wondered about this "failure" for over a decade myself, and even tried to do something about it. I think the problem is that there are multiple parts to the solution, the responsibility for these parts is distributed. I hate to throw everything on the core team, but module systems and packaging are definitely a case where "distributed innovation" doesn't work. There has to be a central team setting standards that everyone else follows. Also, it's not a trivial problem. Go is a static language and is doing better in this regard, but still people complain about packaging. (vgo is coming out after nearly a decade, etc.) I should also add that while I think Python packaging is in the category of "barely works", I would say the same is true of Debian. And Debian is arguably the most popular Linux package manager. They're cases of "failure by success".

blattimwind 9y ago

I've written a whole bunch of CLI tools over the years and maintained some I didn't author originally, I always found it annoying how slow these are. A CLI tool for some larger project can easily take a second just to display --help, or a command line parse error. Tests running against it can be made to progress faster (no forking, no separate interpreter, in-process capture etc. which brings a lot of complexity and subtle differences and error persistence/"tainting" of the execution environment), but still you might only get a few CLI invocations per second per core. Theses experience are a major turn-down from Python for me.

cdavid 9y ago

One "simple" thing that could be done is to make it easier to build python statically, and improve the freezing toolchain. When I used to care about the problem, I looked into it (https://cournape.wordpress.com/2015/09/28/distributing-python-programs-ala-go-as-a-single-file/ https://cournape.wordpress.com/2015/09/28/distributing-pytho...) and got somewhere. It improves somewhat startup times, and allow distribution of a single binary. Lots of libraries are terribly slow to import (e.g. requests), but right now there is little incentive to fix it as there is no toolchain to build good, self-contained python CLI apps.