3 ms·

Does this mean the import mechanism caches loaded module, but not where to find non imported things ? I would expect that it crawl dirs from the PYTHON PATH onc

Does this mean the import mechanism caches loaded module, but not where to find non imported things ? I would expect that it crawl dirs from the PYTHON PATH once and then get data from a cache. I know you can extend the PYTHON PATH dynamically, but you could intercept that in dunder methods to update said cache. Also: for the same interpreter, the same "*.pyc" files and the same PYTHON PATH, we could dump a cache file and share it across interpreter starts if no dirs in PYTHON PATH have been touched. If it's not done, then it might be interesting to bring that up to the Python ideas mailing list.

chubot 9y ago
I'm not sure exactly what you mean, but I think one problem is that the dirs in PYTHONPATH are only the roots. You have to consider changes to all the subdirectories of entries in PYTHONPATH too. The import mechanism is quite complicated, so while I think you could do some more aggressive .pyc-like caching, it would also be pretty hard to get right. Personally what I do is avoid Python packaging tools like setuptools, pip, virtualenv, because they tend to create long PYTHONPATH (they also are hacks piled on top of hacks IMO.) I just write a shell script that downloads dependencies and installs them with distutils / setup.py, and then I wrap the main program in a shell script that sets PYTHONPATH. Yes I know this is unorthodox but it works well for me :)
- bennofs 9y ago
  Just don't do this for things you intend to make public - it makes installing and packaging it really difficult
- sametmax 9y ago
  I don't mean that. I mean that you could setup a watcher on the PYTHON PATH DIRS. Then crawl all the PYTHON PATH dirs for available resources, then list that and dump in a in cache file. Then unless you have watchers telling you something changed, you pull out data from the cache, and don't have to crawl the whole thing every time you load some entry point or pkg_resource.

chubot 9y ago

I'm not sure exactly what you mean, but I think one problem is that the dirs in PYTHONPATH are only the roots. You have to consider changes to all the subdirectories of entries in PYTHONPATH too. The import mechanism is quite complicated, so while I think you could do some more aggressive .pyc-like caching, it would also be pretty hard to get right. Personally what I do is avoid Python packaging tools like setuptools, pip, virtualenv, because they tend to create long PYTHONPATH (they also are hacks piled on top of hacks IMO.) I just write a shell script that downloads dependencies and installs them with distutils / setup.py, and then I wrap the main program in a shell script that sets PYTHONPATH. Yes I know this is unorthodox but it works well for me :)

bennofs 9y ago
Just don't do this for things you intend to make public - it makes installing and packaging it really difficult
sametmax 9y ago
I don't mean that. I mean that you could setup a watcher on the PYTHON PATH DIRS. Then crawl all the PYTHON PATH dirs for available resources, then list that and dump in a in cache file. Then unless you have watchers telling you something changed, you pull out data from the cache, and don't have to crawl the whole thing every time you load some entry point or pkg_resource.