3 ms·

Python imports and slow startup speed hurts Python developers on moderate and large projects. It's the downside of nothing having more static module exports. I

Python imports and slow startup speed hurts Python developers on moderate and large projects. It's the downside of nothing having more static module exports. I believe JS/TypeScript folks try to avoid this trap. In Python, to know what things module exports you need to run the module, there is no static analysis way to find out it otherwise. Making imports fast for data classes may solve some of the problems. Old big projects like Plone/Zope have solved this problem by making more generic lazy import system that it extensively being used. Use zope.deferredimport package for this: https://zopedeferredimport.readthedocs.io/en/latest/narrative.html https://zopedeferredimport.readthedocs.io/en/latest/narrativ... Though I am not sure if zope.deferredimport has been updated to play nicely with the modern typing tools like editors and MyPy.

fnord123 7y ago
>Python imports and slow startup speed hurts Python developers on moderate and large projects What kloc is moderate and large? I have not run into a project where import took that long. >I believe JS/TypeScript folks try to avoid this trap. Have you ever had to run a node project? I haven't benched it but I think it would be faster to compile a rust project and start that up than use node. But I can't web so maybe there's special ways to make node start quickly that I just don't know about. (I think there's a law where you can get answers more quickly by declaring something impossible to trigger people into flooding you with great suggestions :-) ).
- fnord123 7y ago
  Ah, maybe it's use of pkg_resources that you find to be a problem with imports. That's indeed a horror story especially with NFS mounted python files.
- twa927 7y ago
  The problem I experienced multiple times in 50-200 KLOC projects is not the time needed to import the modules, but the memory consumption caused by the imports. Moving some imports from top-level module statements to inner functions' code could improve the memory consumption several times, e.g. from 250MB per process to 80MB per process. One tool I used was https://github.com/mnmelo/lazy_import https://github.com/mnmelo/lazy_import but I'm not sure it's updated for Python 3.7/3.8.
anentropic 7y ago
Another approach to speeding up python imports I just found out about yesterday: https://pyoxidizer.readthedocs.io/en/oxidized_importer-0.1/oxidized_importer.html https://pyoxidizer.readthedocs.io/en/oxidized_importer-0.1/o...

fnord123 7y ago

>Python imports and slow startup speed hurts Python developers on moderate and large projects What kloc is moderate and large? I have not run into a project where import took that long. >I believe JS/TypeScript folks try to avoid this trap. Have you ever had to run a node project? I haven't benched it but I think it would be faster to compile a rust project and start that up than use node. But I can't web so maybe there's special ways to make node start quickly that I just don't know about. (I think there's a law where you can get answers more quickly by declaring something impossible to trigger people into flooding you with great suggestions :-) ).

fnord123 7y ago
Ah, maybe it's use of pkg_resources that you find to be a problem with imports. That's indeed a horror story especially with NFS mounted python files.
twa927 7y ago
The problem I experienced multiple times in 50-200 KLOC projects is not the time needed to import the modules, but the memory consumption caused by the imports. Moving some imports from top-level module statements to inner functions' code could improve the memory consumption several times, e.g. from 250MB per process to 80MB per process. One tool I used was https://github.com/mnmelo/lazy_import https://github.com/mnmelo/lazy_import but I'm not sure it's updated for Python 3.7/3.8.

anentropic 7y ago

Another approach to speeding up python imports I just found out about yesterday: https://pyoxidizer.readthedocs.io/en/oxidized_importer-0.1/oxidized_importer.html https://pyoxidizer.readthedocs.io/en/oxidized_importer-0.1/o...