4 ms·

Lazy imports don't really seem that useful. The only time I've found them useful (in a Ruby project) was for unit tests/local development where only a small sub

by nijave 4y ago

Lazy imports don't really seem that useful. The only time I've found them useful (in a Ruby project) was for unit tests/local development where only a small subset of the application is loaded at a time. Anything long running you generally want the predictability of loading everything up front. For command line utilities, it seems like you're going to need to load the module at some point or another regardless (if you're actually using it) so I'm not sure how you'd see a gain unless there's some async/multi thread hacks.

klyrs 4y ago
In really large projects (e.g. SciPy as mentioned in the article), lazy imports make sense. Especially with the popularity of decorators, importing a file without any apparent module-level code will actually need to run a nontrivial amount of code. Multiply that by a few thousand files in a library with a tree of "import * from ..." and you're looking at perhaps seconds of startup time. Lazy importing can short-circuit that, but still make symbols available for ease of use.
- kelsolaar 4y ago
  Numpy and Matplotlib were quite slow to import also, I haven’t timed them recently though.
dalke 4y ago
Some package developers don't want their users to have the two step process of "import" than "use." NumPy imports 137 modules with "import numpy", of which 94 are specifically in the NumPy hierarchy: >>> import sys >>> len(sys.modules) 83 >>> import numpy as np >>> len(sys.modules) 220 >>> sum(1 for k in sys.modules if "numpy" in k) 94 so people can write a one-liner like: >>> np.polynomial.chebyshev.Chebyshev([0,1,3])(np.linspace(-1.0, 1.0, 5)) array([ 2., -2., -3., -1., 4.]) without having to import np.polynomial.chebyshev.Chebyshev first. This API design requires importing most of NumPy at startup, which has a cost they didn't consider so important because their users are primarily doing long-term computing and notebook-style development, where startup cost is relatively small. I've complained about this because I live in the short-lived program world, where it's annoying to have a 0.1 second import overhead if I only need one function from NumPy: py310% time python -c 'pass' 0.025u 0.006s 0:00.03 66.6% 0+0k 0+0io 0pf+0w py310% time python -c 'import numpy' 0.142u 0.292s 0:00.14 307.1% 0+0k 0+0io 0pf+0w As I understand it, SciPy wants a similar API design goal, but has a lot more packages. They've developed lazy imports to try to have the best of both worlds. > For command line utilities, it seems like you're going to need to load the module at some point or another regardless (if you're actually using it) Thing is, you might not actually use it. If the command-line tool uses subcommands, each different subcommand might need only a subset of the full set of packages. Perhaps only one of the subcommands uses NumPy, while for 95% of uses, NuPy isn't used at all. As the discussion for this feature points out, this can be addressed by only importing when needed. (One of the reasons I've started using click over argparse is click does more of this separation for me.) However, it's somewhat fragile, in that it's easy to add an rarely-needed expensive import at top-level without noticing it, and requires some non-standard tooling to detect issues, like the non-predictability you mentioned. I personally want something like the lazy-/auto- importer in my package, so I can reduce the two step process. My last package released used module-level getattr functions, which gets me mostly there, except for notebook auto-completion of the lazy wrappers. (It works in the command-line shell though.) I can't import everything on startup because parts of my package depend on third-party packages, which might not be installed. I instead want to raise an ImportError when those lazy objects are accessed. Plus, one of the third-party packages is through a Python/Java bridge, which has its own startup costs that I want to avoid.
- slaymaker1907 4y ago
  You could do the numpy style API lazily. They would just need to each API as an object that does the imports dynamically.
  - dalke 4y ago
    ... yes. The PEP describes existing solutions, followed by a proposal for "a more general and comprehensive solution for lazy imports that can encompass all of the above use cases and does not impose detectable overhead in real-world use", to quote the PEP at https://peps.python.org/pep-0690/ https://peps.python.org/pep-0690/ .
deleted 4y ago
[deleted]

by nijave 4y ago

klyrs 4y ago

In really large projects (e.g. SciPy as mentioned in the article), lazy imports make sense. Especially with the popularity of decorators, importing a file without any apparent module-level code will actually need to run a nontrivial amount of code. Multiply that by a few thousand files in a library with a tree of "import * from ..." and you're looking at perhaps seconds of startup time. Lazy importing can short-circuit that, but still make symbols available for ease of use.

kelsolaar 4y ago
Numpy and Matplotlib were quite slow to import also, I haven’t timed them recently though.

dalke 4y ago

Some package developers don't want their users to have the two step process of "import" than "use." NumPy imports 137 modules with "import numpy", of which 94 are specifically in the NumPy hierarchy: >>> import sys >>> len(sys.modules) 83 >>> import numpy as np >>> len(sys.modules) 220 >>> sum(1 for k in sys.modules if "numpy" in k) 94 so people can write a one-liner like: >>> np.polynomial.chebyshev.Chebyshev([0,1,3])(np.linspace(-1.0, 1.0, 5)) array([ 2., -2., -3., -1., 4.]) without having to import np.polynomial.chebyshev.Chebyshev first. This API design requires importing most of NumPy at startup, which has a cost they didn't consider so important because their users are primarily doing long-term computing and notebook-style development, where startup cost is relatively small. I've complained about this because I live in the short-lived program world, where it's annoying to have a 0.1 second import overhead if I only need one function from NumPy: py310% time python -c 'pass' 0.025u 0.006s 0:00.03 66.6% 0+0k 0+0io 0pf+0w py310% time python -c 'import numpy' 0.142u 0.292s 0:00.14 307.1% 0+0k 0+0io 0pf+0w As I understand it, SciPy wants a similar API design goal, but has a lot more packages. They've developed lazy imports to try to have the best of both worlds. > For command line utilities, it seems like you're going to need to load the module at some point or another regardless (if you're actually using it) Thing is, you might not actually use it. If the command-line tool uses subcommands, each different subcommand might need only a subset of the full set of packages. Perhaps only one of the subcommands uses NumPy, while for 95% of uses, NuPy isn't used at all. As the discussion for this feature points out, this can be addressed by only importing when needed. (One of the reasons I've started using click over argparse is click does more of this separation for me.) However, it's somewhat fragile, in that it's easy to add an rarely-needed expensive import at top-level without noticing it, and requires some non-standard tooling to detect issues, like the non-predictability you mentioned. I personally want something like the lazy-/auto- importer in my package, so I can reduce the two step process. My last package released used module-level getattr functions, which gets me mostly there, except for notebook auto-completion of the lazy wrappers. (It works in the command-line shell though.) I can't import everything on startup because parts of my package depend on third-party packages, which might not be installed. I instead want to raise an ImportError when those lazy objects are accessed. Plus, one of the third-party packages is through a Python/Java bridge, which has its own startup costs that I want to avoid.

slaymaker1907 4y ago
You could do the numpy style API lazily. They would just need to each API as an object that does the imports dynamically.
- dalke 4y ago
  ... yes. The PEP describes existing solutions, followed by a proposal for "a more general and comprehensive solution for lazy imports that can encompass all of the above use cases and does not impose detectable overhead in real-world use", to quote the PEP at https://peps.python.org/pep-0690/ https://peps.python.org/pep-0690/ .

deleted 4y ago

[deleted]