8 ms·
The return of lazy imports for Python
- rtzuul 4y agoThe import code in Python is a mess, probably made slower since the advent of importlib. You need developers who care about fast, clean code to fix the issue. Those kind of developers usually don't fare well in the Python swamp, so it won't happen.
- code_runner 4y agoDoubtful lazy imports would’ve helped at all… I joined a project where almost every import statement has side effects, some of which took multiple minutes to read things into memory etc. Tried some poor-man’s debugging and never hit a breakpoint on the first significant line of code… took a while to figure out as it was my first Python project. It almost feels like Python needs a scripting and non-scripting mode, or some kind of warning logging “you did everything wrong”
- deepsun 4y agoPython is a very good scripting language, so good that people sometimes mistake it for application language.
- KerrAvon 4y agoAll of Python, Perl (!), Ruby, Tcl have been application languages at one time or another. Python evangelists, reel it in a bit, please.
- solarkraft 4y agoI have yet to understand how to clearly separate scripts from applications.
- melony 4y agoFor me it is AOT compilation and packaging. An application language should be able to pull in most of its dependencies and runtime and be packaged into a reasonably sized and performant binary. The reason why scripting languages (that are often dynamic) struggle with this is that too many of their dependencies have hotpaths written in third party languages like C which makes it hard to do easy cross compilation and linking. Some programming languages come from a culture that eschews AOT compilation. It is difficult to compile large dynamic languages with Lisp heritages (CL, Julia, Ruby etc., Scheme is an exception to this however). These languages always resort to using a stateful VM/interpreter snapshot in lieu of true AOT packaging (or even Java/.NET style lowered bytecode binaries). The existence of eval makes many things trivial but at the same time complicates deployment tremendously. Personally I am betting on JavaScript being the first dynamic and JITted (or incrementally compiled) language that can attain the AOT compilation without massively bloated binaries.
- int_19h 4y agoSo wait, then C# wasn't an application language before there was a way to package AOT binaries to deploy directly to users?
- melony 4y agoIt is, because the runtime is bundled with the OS (similar its counterpart the JVM is also extremely widely deployed) and its output is significantly lowered. C.f. > (or even Java/.NET style lowered bytecode binaries).
- deepsun 4y agoRe. AOT, as I said in my sibling comment, there's a counterexample -- Julia. It's AOT compiled, but dynamically typed. And it falls in scripting camp, IMHO.
- melony 4y agoJulia only does partial AOT, it still bundles the sysimage (~100MB+ of runtime, what I refer to in my above comment as the VM snapshot) c.f. https://docs.juliahub.com/PackageCompiler/MMV8C/1.2.7/devdocs/sysimages_part_1.html https://docs.juliahub.com/PackageCompiler/MMV8C/1.2.7/devdoc... Traditional AOT like you see in C/C++/Rust/Go/Zig would be able to treeshake and eliminate redundant codepaths, the binaries are all fairly small with minimal startup overhead.
- deepsun 4y agoMostly it's original focus of the language and ecosystem developers, IMO. Although I found one feature to very strongly correlate -- static typing. For example, Julia is AOT compiled, but still dynamically typed. And coincidentally, it was designed for scientists, who need scripting much more. Another observation is how easy it is to write unit-tests (for canonically written code). Overloading "import" statement doesn't help there at all.
- striking 4y agoI added lazy loading by default to my workplace's monolith in Node.js. We relied on a couple of imports with side effects, which no longer worked because no one was accessing the modules that ran the side effects, and that was the trigger for actually loading the module. Our fix for this was to relegate stateful imports to files with "bootstrap" in the name, which the lazy loader stub would allow to be eagerly imported. Moreover, any imports listed in such "bootstrap" files would then be eagerly imported. But that's it (at least wrt the code belonging to our codebase); no one else, not even the children of the bootstrapping modules, were exempted from lazy loading. This allowed for a centralization of all the stateful import effects. If you tried to write stateful code outside of bootstrapping, it simply wouldn't run. (You could hypothetically hack around this. But it would be rather obvious, and I'm perfectly ready to revert pull requests that try.) Maybe you folks could try some variation of this in Python.
- sdenton4 4y agoImport side effects are the devil and should be killed with fire wherever they appear.
- Spivak 4y agoThe problem is that side effects are really broad and include things that are idiomatic Python. # jobs.py @scheduler.register_job(when=“daily”) def batch_job(): … This adds the job to the global scheduler. # models.py class Model(DBModel): def on_change(): # do stuff This adds the model to the ORM’s list of modules via a metaclasss and registers its hooks. # plugin.py class SomePlugin(PluginBase): … Same thing. The code to load plugins like this is just importing the module and letting the metaclass do the work.
- jonnycomputer 4y agoAmen
- tersers 4y agoI get an impression that regardless of topic, it’s difficult for any decisions to be made for the future of Python? The discussion seems to always revolve around “what if?”s in not the most collaborative fashion. I wonder if what most languages need are less experts in computer science or language theory or anything technical, and more folks that can do facilitation.
- setr 4y agoWhy do you think managers take over everything? Regardless of topic, any sufficiently large problem eventually becomes primarily a coordination problem
- epgui 4y agoPersonally, I’d much rather languages be designed from mathematical foundations and/or very careful theory.
- CoastalCoder 4y agoThere certainly are languages like that, but I think you'll find there are tradeoffs to consider. Especially in commercial software.
- epgui 4y agoThe ones I’m familiar with (clojure and haskell) are a joy to work with. These languages have conquered my heart!
- dragonwriter 4y agoPython decision-making is rather conservative, preferring deep exploration of implications, because its a big, established language with a lot of existing use to support, and because of the 2->3 experience. I don’t think lack of facilitation skill is an issue; its a deliberate policy choice.
- estebank 4y ago
- wpietri 4y agoI really appreciate both the good writeup here and the fact that so many people are thinking through changes so carefully.
- nijave 4y agoLazy imports don't really seem that useful. The only time I've found them useful (in a Ruby project) was for unit tests/local development where only a small subset of the application is loaded at a time. Anything long running you generally want the predictability of loading everything up front. For command line utilities, it seems like you're going to need to load the module at some point or another regardless (if you're actually using it) so I'm not sure how you'd see a gain unless there's some async/multi thread hacks.
- klyrs 4y agoIn really large projects (e.g. SciPy as mentioned in the article), lazy imports make sense. Especially with the popularity of decorators, importing a file without any apparent module-level code will actually need to run a nontrivial amount of code. Multiply that by a few thousand files in a library with a tree of "import * from ..." and you're looking at perhaps seconds of startup time. Lazy importing can short-circuit that, but still make symbols available for ease of use.
- kelsolaar 4y agoNumpy and Matplotlib were quite slow to import also, I haven’t timed them recently though.
- dalke 4y agoSome package developers don't want their users to have the two step process of "import" than "use." NumPy imports 137 modules with "import numpy", of which 94 are specifically in the NumPy hierarchy: >>> import sys >>> len(sys.modules) 83 >>> import numpy as np >>> len(sys.modules) 220 >>> sum(1 for k in sys.modules if "numpy" in k) 94 so people can write a one-liner like: >>> np.polynomial.chebyshev.Chebyshev([0,1,3])(np.linspace(-1.0, 1.0, 5)) array([ 2., -2., -3., -1., 4.]) without having to import np.polynomial.chebyshev.Chebyshev first. This API design requires importing most of NumPy at startup, which has a cost they didn't consider so important because their users are primarily doing long-term computing and notebook-style development, where startup cost is relatively small. I've complained about this because I live in the short-lived program world, where it's annoying to have a 0.1 second import overhead if I only need one function from NumPy: py310% time python -c 'pass' 0.025u 0.006s 0:00.03 66.6% 0+0k 0+0io 0pf+0w py310% time python -c 'import numpy' 0.142u 0.292s 0:00.14 307.1% 0+0k 0+0io 0pf+0w As I understand it, SciPy wants a similar API design goal, but has a lot more packages. They've developed lazy imports to try to have the best of both worlds. > For command line utilities, it seems like you're going to need to load the module at some point or another regardless (if you're actually using it) Thing is, you might not actually use it. If the command-line tool uses subcommands, each different subcommand might need only a subset of the full set of packages. Perhaps only one of the subcommands uses NumPy, while for 95% of uses, NuPy isn't used at all. As the discussion for this feature points out, this can be addressed by only importing when needed. (One of the reasons I've started using click over argparse is click does more of this separation for me.) However, it's somewhat fragile, in that it's easy to add an rarely-needed expensive import at top-level without noticing it, and requires some non-standard tooling to detect issues, like the non-predictability you mentioned. I personally want something like the lazy-/auto- importer in my package, so I can reduce the two step process. My last package released used module-level getattr functions, which gets me mostly there, except for notebook auto-completion of the lazy wrappers. (It works in the command-line shell though.) I can't import everything on startup because parts of my package depend on third-party packages, which might not be installed. I instead want to raise an ImportError when those lazy objects are accessed. Plus, one of the third-party packages is through a Python/Java bridge, which has its own startup costs that I want to avoid.
- isitmadeofglass 4y agoMight not grasp the full context here, but it’s trivial to lazily import modules in your own code. I know every beginners guide will advice you not to do that, but that’s just because it’s an easy footgun for new programmers. If you have some cli tool that only needs scipy for certain sub commands you can just move it to those subcommand calls so it’s loaded when needed instead of up front.
- coredog64 4y agoI usually only import argparse inside my ‘if __name__ == “__main__”’ stanza.
- dalke 4y agoThat's not really the issue with argparse and subcommands. argparse with subcommands generally requires specifying all of the options for all of the subcommands, even if you only want one subcommand. These in turn may require importing subcommand-specific modules, to handle things like the right 'type' handler in an an add_argument() parameter. This callback function might, depending on the input value, select one from a dozen different additional packages. It's possible to avoid this, by deferring argument->type processing until later, and having a single large module containing all of the help strings and epilogs, though this will separate your argparse code from your subcommand code, and in general make things more complicated. I did this for a while. Alternatively, you can create your own subcommand dispatch system using an nargs="?" to get the subcommand and an nargs=argparse.REMAINDER to capture the rest of the flags, to pass to a new ArgumentParser, and develop a top-level --help replacements. I tried this too. I've since decided to use click, which does a better job at compartmentalizing at least this level of subcommand imports.
- formerly_proven 4y agoI've written several large CLI tools with Python and always ended up doing something in this vein. --help - and invocation errors - just take too long otherwise.
- 4y ago
- overgard 4y agoPersonally, unless it has explicit "lazy" syntax I kind of hate the idea. One thing I always liked about python was how predictable and simple module imports are.
- slaymaker1907 4y agoI think such a mechanism already exists. You can just use the functional import syntax inside of a block. However, I think lazy imports could be ok so long as the language can show that a particular module is side effect free (i.e. no globals aside from something like constexpr).
- masklinn 4y ago> You can just use the functional import syntax inside of a block. You don't even need functional import syntax, but as TFA notes this comes at a cost as it has to invoke the entire import machinery, which it can only skip to an extent (once a module is loaded and cached) as import hooks can have odd behaviours.
- andreareina 4y agoI wouldn’t call different behaviour whether there’s __init__.py predictable nor simple. Also relative imports not working in an interactive session.
- bfung 4y agoThe only argument PEP 690 has is A) [performance on startup] or B) [when import functions is not used] in main body of the code. For B), easy enough to run one of many linters to detect this case and have people write less bad code. A) is way more subjective and can be fixed in many ways. With the many more Python coders these days with less coding experience, personal feeling is please stop throwing these production issue causing features in that I have to fix. Glad the PEP is rejected. Old programmer wisdom is to load all your configs and assumptions as early as possible to eliminate a whole space of problems with your code, making faster and easier to read/reason about later.
- rrdharan 4y agoI’m curious about the other ways to fix startup performance. I’ve seen a moderately sized (~300k LoC) Python CLI project that had a horrendous, anger-inducing startup time until they switched to the lazy import approach basically described/standardized by PEP 690 and the improvement was massive.
- lmeyerov 4y agoB sounds great but too much of the pydata ecosystem breaks here, like `class MyNeuralNet(nn.Module)`
- bfung 4y agoFor sure, pydata ecosystem is one of the worst offenders. Yeah, the libraries make it easy to do that 1 off model -- shove it in production and watch the next 3-6 months be a pagerduty fest for both the model writer and ops/backend people. It doesn't have to be like that though -- look at geohotz's tinygrad library for exampe: well tested, well written, and can do most of the things the bigger ecosystem libraries do.
- lmeyerov 4y agoThe goal of lang features like these is to automate the optimization especially when it is hard to do at scale without it. Scenarios that start with "If only everyone..." are great candidates. Conversely, agreed, it's not THAT hard to convert an individual project, just being unable to handle third-party, so import-shaming some top projects can likely go far. Features that enable framework providers to empower others are great... And none is needed here. It's messy to push through type checking, but doable. Feel like we are reliving when js got a more static module system. A lot of kicking and screaming, and still issues, but a lot of good came out of that.
- m000 4y agoNot a word though about the elephant in the room: circular imports. It is absurdly easy in Python to end up with a circular import situtation, where no real circular dependency exists. E.g. you can't have A.a1 -> B.b1 and B.b2 -> A.a2. So, you are forced to layout your code in some quite awkward ways.
- capableweb 4y agoIsn't having circular dependencies more awkward? Conceptually, it makes things more intertwined when instead you can build a better and more separated architecture.
- marius_k 4y agoIs there an elegant way to import type hints without circular imports?
- lopz 4y agoSomething like this does the trick: if TYPE_CHECKING: import WhateverClass https://docs.python.org/3/library/typing.html#typing.TYPE_CHECKING https://docs.python.org/3/library/typing.html#typing.TYPE_CH...
- nomel 4y agoI’m a fan of having a single state, used for everything. Splitting the code up into two states, one for the linter and one for the execution, seems like a recipe for incorrectness and confusion. I would hate to refactor something like that.
- closed 4y agoThe issue is that sometimes a function can take a type that is an optional dependency, so you don't want to import it unless you are type checking. (And some types are defined in the typeshed so only exist to be imported during type checking; eg the type checker lib itself is a dependency in this case)
- pard68 4y agoI didn't understand this when it first came up and I still don't. If you want to defer your imports than wait until it's needed. It might be useful if you need to load a behemoth of a module for some rarely used part of a CLI tool. Otherwise, an X ms load at startup is hardly any different than the same X ms load in the middle of the execution. And on a server it's actually worse.
- mort96 4y agoEvery single CLI tool gains from this. Most Python CLI tools have terrible UX since stuff that's as simple as running it with `--help` or `--version` requires the VM to go crawling around everywhere on your disk and executing bytecode.
- pard68 4y agoI get that, but that's poor design on the tool's part, not Python's problem. Hide your imports behind functions, don't import at the global level, unless it's needed globally.
- jonnycomputer 4y agoThat's a design problem, imo, not a Python problem.
- deleted 4y ago[deleted]
- kortex 4y agoIt's hugely different. The issue occurs when you have something which does not need to execute in every context. Let's day you have a CLI which does some machine learning with several models. You don't want to execute loading every model every time you start the app. The most egregious is when you just want to display the help/usage text and not actually execute any code at all. Instead, you have to either manually lazy import (what I usually do now), or eat huge startup costs each time you screw up the command syntax.
- kortex 4y agoI really hope a solution can be found and python can get true lazy imports (without doing the manual thing). I work in the ML/AI space and manual lazy imports are basically mandatory unless you want several seconds to see usage text from `footool --help`. Why "pay" for executing code you don't use? Manual lazy import meaning: def uses_foo(x): import foo return foo.bar(x) Manual lazy import sucks because: - it's just ugly, I like imports all at the top so I can see all the deps - bad for static analysis - performance hit every time the function is called Eschewing lazy imports has several problems: - you always pay the execution cost, even if you don't use it - also bad for static analysis and testing, since you have to eat the import time even if the code block you want to test doesn't execute the expensive path - sometimes you need lazy import to avoid circular import errors It's too bad that the main impediment to this is existing code which relies on side effects. Import-time side effects are an absolute pain in the ass. Avoid it at all costs.
- ashvardanian 4y agoGiven that modern ML libs can bring hundreds of megabytes of binaries with them, this is a great point. As for static analysis in Python, I am not sure it can get much worse :) The language is essentially built around the idea of making everything absolutely dynamic
- emptysea 4y agoTyping in python is pretty thorough nowadays, you’ve got overloads, literal types, strict null checking, unions It’s a more advanced type system than Go and Java
- dagmx 4y agoTyping is very surface level though and doesn’t negate what the person you’re replying to is saying: Python is über dynamic. It’s very trivial to have something pass the type hinting checks, but be completely different at time of use. Even if you hold on to a single instance of an object, it’s easy for any thing to monkey patch it.
- 4y ago
- im3w1l 4y agoIt may not feel very clean but one solution for slow imports is to write your own module that imports fast. To give an example: I had a some script whose startup was slowed down by importing numpy for one small (but important) thing. Replaced it with a 20 line c extension that imported basically instantly.
- deleted 4y ago[deleted]
- scaredginger 4y agoLibrary authors can make their modules initialize lazily. It's additional code complexity, but it's probably worth it in certain cases. At one point, I was on a team developing a complex extension module that would take several seconds to import since it produced a gigantic shared lib. Sure, that probably could've been made more modular, but lazy initialization was easier. https://docs.python.org/3/library/importlib.html#implementing-lazy-imports https://docs.python.org/3/library/importlib.html#implementin...
- lillecarl 4y agoThe people behind the xonsh shell has already implemented lazy loading of things:https://github.com/xonsh/lazyasd https://github.com/xonsh/lazyasd Xonsh shell is amazing.
- wadd1e 4y agoI wonder if it's worth adding a new `import lazy` statement for the next major release? I was about to suggest `import` being lazy by default and introducing an `import strict`, but that would fuel another 2->3 style nightmare.