4 ms·

I agree with optimizing Python's startup time, and I agree that namedtuple is weird and should be changed. But I doubt that the namedtuple change will noticeab

by chubot 9y ago

I agree with optimizing Python's startup time, and I agree that namedtuple is weird and should be changed. But I doubt that the namedtuple change will noticeably decrease startup time for most applications (after having experience with this problem for 10+ years). From a comment over 3 years ago: [Python interpreter startup does] random access I/O (the slowest thing your computer can do) proportional to (large constant factor) x (num imports in program) x (length of PYTHONPATH). https://news.ycombinator.com/item?id=7842873 https://news.ycombinator.com/item?id=7842873 I don't think that instantiating a Python parser 100 times for exec() is going to compare to that. I guess the difference is that I'm thinking about the cold cache case -- namedtuple might be noticeable in the warm cache case. And there are many many command line tools that start slow because they're written in Python, not just Mercurial. Mercurial is actually one of the best because they care about it (demandimport) and they don't have too many dependencies. IIRC some of the Google Cloud tools take 500-1000+ ms to start because they are crawling tons of dependencies with long PYTHONPATH.

sametmax 9y ago
Does this mean the import mechanism caches loaded module, but not where to find non imported things ? I would expect that it crawl dirs from the PYTHON PATH once and then get data from a cache. I know you can extend the PYTHON PATH dynamically, but you could intercept that in dunder methods to update said cache. Also: for the same interpreter, the same "*.pyc" files and the same PYTHON PATH, we could dump a cache file and share it across interpreter starts if no dirs in PYTHON PATH have been touched. If it's not done, then it might be interesting to bring that up to the Python ideas mailing list.
- chubot 9y ago
  I'm not sure exactly what you mean, but I think one problem is that the dirs in PYTHONPATH are only the roots. You have to consider changes to all the subdirectories of entries in PYTHONPATH too. The import mechanism is quite complicated, so while I think you could do some more aggressive .pyc-like caching, it would also be pretty hard to get right. Personally what I do is avoid Python packaging tools like setuptools, pip, virtualenv, because they tend to create long PYTHONPATH (they also are hacks piled on top of hacks IMO.) I just write a shell script that downloads dependencies and installs them with distutils / setup.py, and then I wrap the main program in a shell script that sets PYTHONPATH. Yes I know this is unorthodox but it works well for me :)
  - bennofs 9y ago
    Just don't do this for things you intend to make public - it makes installing and packaging it really difficult
  - sametmax 9y ago
    I don't mean that. I mean that you could setup a watcher on the PYTHON PATH DIRS. Then crawl all the PYTHON PATH dirs for available resources, then list that and dump in a in cache file. Then unless you have watchers telling you something changed, you pull out data from the cache, and don't have to crawl the whole thing every time you load some entry point or pkg_resource.
eesmith 9y ago
If you are really interested in improving the startup cost then the first thing is to put everything, including the Python standard library, into .zip files, so it can be zip imported. That removes a lot of the directory/stat overhead. The comment you pointed to did not do that optimization. For example, I supported a 10 year old set of CGI scripts running on a machine with Lustre filesystem where every filesystem metadata lookup was painfully slow. (See ehiggs' comment at lwn.) I spent time to trim away every import to the bare minimum. Once you do that, other factors become more significant. (Remove your biggest problem and something else becomes your biggest problem.) If the view is that a few milliseconds here and a few milliseconds there isn't a problem, well, those milliseconds add up. 'import scipy' (which depends on numpy) takes 3x longer to import than Python itself does. I have replaced dependencies on scipy when the overhead of importing the one function I needed took longer than my program took to run. There's an old story along these lines, recounted at https://www.folklore.org/StoryView.py?story=Saving_Lives.txt https://www.folklore.org/StoryView.py?story=Saving_Lives.txt : > One of the things that bothered Steve Jobs the most was the time that it took to boot when the Mac was first powered on. ... One afternoon, Steve came up with an original way to motivate us to make it faster. ... > "You know, I've been thinking about it. How many people are going to be using the Macintosh? A million? No, more than that. In a few years, I bet five million people will be booting up their Macintoshes at least once a day." > "Well, let's say you can shave 10 seconds off of the boot time. Multiply that by five million users and thats 50 million seconds, every single day. Over a year, that's probably dozens of lifetimes. So if you make it boot ten seconds faster, you've saved a dozen lives. That's really worth it, don't you think?" How many people use Python? How many times is it started per day?
- batbomb 9y ago
  The other way around this is containers or caching filesystems.

by chubot 9y ago

sametmax 9y ago

Does this mean the import mechanism caches loaded module, but not where to find non imported things ? I would expect that it crawl dirs from the PYTHON PATH once and then get data from a cache. I know you can extend the PYTHON PATH dynamically, but you could intercept that in dunder methods to update said cache. Also: for the same interpreter, the same "*.pyc" files and the same PYTHON PATH, we could dump a cache file and share it across interpreter starts if no dirs in PYTHON PATH have been touched. If it's not done, then it might be interesting to bring that up to the Python ideas mailing list.

chubot 9y ago
I'm not sure exactly what you mean, but I think one problem is that the dirs in PYTHONPATH are only the roots. You have to consider changes to all the subdirectories of entries in PYTHONPATH too. The import mechanism is quite complicated, so while I think you could do some more aggressive .pyc-like caching, it would also be pretty hard to get right. Personally what I do is avoid Python packaging tools like setuptools, pip, virtualenv, because they tend to create long PYTHONPATH (they also are hacks piled on top of hacks IMO.) I just write a shell script that downloads dependencies and installs them with distutils / setup.py, and then I wrap the main program in a shell script that sets PYTHONPATH. Yes I know this is unorthodox but it works well for me :)
- bennofs 9y ago
  Just don't do this for things you intend to make public - it makes installing and packaging it really difficult
- sametmax 9y ago
  I don't mean that. I mean that you could setup a watcher on the PYTHON PATH DIRS. Then crawl all the PYTHON PATH dirs for available resources, then list that and dump in a in cache file. Then unless you have watchers telling you something changed, you pull out data from the cache, and don't have to crawl the whole thing every time you load some entry point or pkg_resource.

eesmith 9y ago

If you are really interested in improving the startup cost then the first thing is to put everything, including the Python standard library, into .zip files, so it can be zip imported. That removes a lot of the directory/stat overhead. The comment you pointed to did not do that optimization. For example, I supported a 10 year old set of CGI scripts running on a machine with Lustre filesystem where every filesystem metadata lookup was painfully slow. (See ehiggs' comment at lwn.) I spent time to trim away every import to the bare minimum. Once you do that, other factors become more significant. (Remove your biggest problem and something else becomes your biggest problem.) If the view is that a few milliseconds here and a few milliseconds there isn't a problem, well, those milliseconds add up. 'import scipy' (which depends on numpy) takes 3x longer to import than Python itself does. I have replaced dependencies on scipy when the overhead of importing the one function I needed took longer than my program took to run. There's an old story along these lines, recounted at https://www.folklore.org/StoryView.py?story=Saving_Lives.txt https://www.folklore.org/StoryView.py?story=Saving_Lives.txt : > One of the things that bothered Steve Jobs the most was the time that it took to boot when the Mac was first powered on. ... One afternoon, Steve came up with an original way to motivate us to make it faster. ... > "You know, I've been thinking about it. How many people are going to be using the Macintosh? A million? No, more than that. In a few years, I bet five million people will be booting up their Macintoshes at least once a day." > "Well, let's say you can shave 10 seconds off of the boot time. Multiply that by five million users and thats 50 million seconds, every single day. Over a year, that's probably dozens of lifetimes. So if you make it boot ten seconds faster, you've saved a dozen lives. That's really worth it, don't you think?" How many people use Python? How many times is it started per day?

batbomb 9y ago
The other way around this is containers or caching filesystems.