10 ms·

PEP 810 – Explicit lazy imports

• sedatk 1y ago
  I wonder how much things would break if all imports were lazy by default.
  • simlevesque 1y ago
    Maybe nothing would break ? edit: ok well "xxx in sys.modules" would indeed be a problem
    • IshKebab 1y ago
      Yeah unfortunately in real world Python code people put side effects in their modules all the time.
      • kstrauser 1y ago
        In Python? I almost never see that. And I’ve certainly never signed off on a PR that did that.
        • oofbey 1y ago
          Maybe your code is awesome, but you're almost certainly using a library which does it.
          • lyu07282 1y ago
            This. The most egregious example I've ever seen is the inflect library, takes almost 2 seconds(!) to import: https://github.com/jaraco/inflect/issues/212 https://github.com/jaraco/inflect/issues/212
        • zahlman 1y ago
          Recursively importing other modules is itself a side effect. In fact, all the code you see in the module is "side effects", in a sense. A `class` body, for example, has to actually run at import time, creating the class object and attaching it as an attribute of the module object. Similarly for functions. Even a simple assignment of a constant actually has to run at module import. And all of these things add up. Further, if there isn't already cached bytecode available for the module, by default it will be written to disk as part of the import process. That's inarguably a side effect.
          • flykespice 1y ago
            Side-effect means you're changing state outside the scope the code is running. Sure thing you can declare globals variable and run anything on a module file global scope (outside funcs and class body), but even that 'global' scope is just an illusion, and everything declared there, as yourself said, is scoped to the module's namespace (and you can't leak the 'globals' when importing the module unless you explicity do so 'from foo import *'. Think of python's import as eval but safer because it doesn't leaks the results from the module execution) So for a module to have side-effect (for me) it would either: - Change/Create attributes from other modules - Call some other function that does side-effect (reflection builtins? IO stuff)
            • IshKebab 1y ago
              Often it's things like registering functions in a global registry, or patching things about how Python works (e.g. adding stuff to the module path).
            • kstrauser 1y ago
              You’re right. “Side effect” here means that your program’s behavior changes simply by importing something, eg by monkey patching something else. If’s not: * Importing other modules. * Taking a long time to import. * Writing .pyc files. If any program can “import foo” and still execute exactly the same bytecode afterward as before, you can say that foo doesn’t have side effects.
  • eesmith 1y ago
    All my code which uses import probing would fail, such as fallbacks: try: import module except ImportError: import slow_module as module Conditional support testing would also break, like having tests which only run if module2 is available: try: import module2 except ImportError: def if_has_module2(f): return unittest.skip("module2 not available")(f) else: def if_has_module2(f): return f @if_has_module2 class TestModule2Bindings(.... The proto-PEP also gives an example of using with suppress_warnings(): import module3 where some global configuration changes only during import. In general, "import this" and "import antigravity" - anything with import side-effects - would stop working. Oh, and as the proto-PEP points out, changes to sys.path and others can cause problems because of the delay between time of lazy import and time of resolution. Then there's code where you do a long computation then make use of a package which might not be present. import database # remember to install!! import qcd_simulation universe = qcd_simulation.run(seconds = 30*24*60*60) database.save(universe) All of these would be replaced with "import module; module.__name__" or something to force the import, or by an explicit use to __import__.
    • npage97 1y ago
      You have to opt in with the lazy import keyword no matter what. This pep also prevents lazy import in try catch. I think your concern matters if ‘module’ itself has a lazy import that you want to check exists. With this you now need to be more rigorous in check those sub dependencies. This can already happen with non top level imports so it is not a necessarily a new issue, but could become more prevalent if there is an overall uptake in this feature for optional dependencies.
      • eesmith 1y ago
        My post was a response to the sedatk conjectural "I wonder how much things would break if all imports were lazy by default." I have zero concerns about this PEP and look forward to its implementation.
    • zahlman 1y ago
      With the LazyLoader technique I described at https://news.ycombinator.com/item?id=45467489 https://news.ycombinator.com/item?id=45467489 , there is no problem: >>> import nonexistent_module Traceback (most recent call last): File "<python-input-2>", line 1, in <module> import nonexistent_module File "<frozen importlib._bootstrap>", line 1360, in _find_and_load File "<frozen importlib._bootstrap>", line 1322, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 1262, in _find_spec File "<python-input-0>", line 8, in find_spec base.loader = LazyLoader(base.loader) ^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'loader' The implementation should probably convert that exception back to ImportError for you, but the point is that the absence of an implementation can still be detected eagerly while the actual loading occurs lazily.
  • ashu1461 1y ago
    There will be chaos, all code bases usually have tons of side effects.
• TheMrZZ 1y ago
  Feels like a good feature, with a simple explanation, real world use cases, and a scoped solution (global only, pretty simple keyword). I like it!
  • 12_throw_away 1y ago
    Yeah, I think this is one of the cleanest PEPs to come around in quite a while, at least from the userspace perspective. Interested to see what happens after the traditional syntax bikeshedding ritual has been completed.
  • oofbey 1y ago
    Hopefully they learned lessons from why PEP-690 was rejected. I've spent quite a while trying to build this stuff for our codebase and it's never worked well enough to use.
    • dahart 1y ago
      690 is mentioned a few times, including a FAQ note on the differences https://pep-previews--4622.org.readthedocs.build/pep-0810/#faq https://pep-previews--4622.org.readthedocs.build/pep-0810/#f... Q: How does this differ from the rejected PEP 690? A: PEP 810 takes an explicit, opt-in approach instead of PEP 690’s implicit global approach. The key differences are: Explicit syntax: lazy import foo clearly marks which imports are lazy. Local scope: Laziness only affects the specific import statement, not cascading to dependencies. Simpler implementation: Uses proxy objects instead of modifying core dictionary behavior.
  • BiteCode_dev 1y ago
    Agree, they really did their homework, listed edge cases, made practical compromises, chose not to overdo it, reworked it again and again quite a bit and compared it to real life experience. It's really beautiful work, especially since touching the back bone (the import system) of a language as popular as Python with such a diverse community is super dangerous surgery. I'm impressed.
• Galanwe 1y ago
  That seems really great, .moving imports in-line for CLI tools startup and test discovery has always been a pain.
• hoten 1y ago
  For posterity, as the submitted link seems temporary: https://github.com/python/peps/pull/4622 https://github.com/python/peps/pull/4622
  • zahlman 1y ago
    The proper permanent link is https://peps.python.org/pep-0810/ https://peps.python.org/pep-0810/ .
• IshKebab 1y ago
  Wake me up when we can import a module by relative file path.
  • Spivak 1y ago
    You… can? I mean in the strictest sense you're technically not importing by file path but if you make your folder a module by slapping an __init__.py in there then your relative imports will follow the directory tree. I think as of Python 3.3 the init file is optional so it will do it by default but I can't remember if there are still some cases where it's required. The only thing you can't do is go "up" to a higher directory than the root module. Also if that doesn't strike your fancy all of the importlib machinery is at your disposal and it's really not very much work to write an import_path() function. It's one of the patterns plug-in systems use and so is stable and expected to be used by end users. No arcane magic required.
    • zahlman 1y ago
      > if you make your folder a module by slapping an __init__.py in there then your relative imports will follow the directory tree. `__init__.py` has nothing to do with making this work. It is neither necessary (as of 3.3, yes, you got it right: see https://peps.python.org/pep-0420/ https://peps.python.org/pep-0420/) nor sufficient (careless use of sys.path and absolute imports could make it so that the current folder hasn't been imported yet, so you can't even go "up into" it). The folder will already be represented by a module object. What `__init__.py` does is: 1. Prevents relative imports from also potentially checking in other paths. 2. Provides a space for code that runs before sub-modules, for example to set useful package attributes such as `__all__` (which controls star-imports).
    • IshKebab 1y ago
      > if you make your folder a module by slapping an __init__.py in there then your relative imports will follow the directory tree haha no > all of the importlib machinery is at your disposal This breaks all tooling. Awful option.
  • zahlman 1y ago
• wrmsr 1y ago
  If anyone's interested I've implemented a fairly user friendly lazy import mechanism in the form of context managers (auto_proxy_import/init) at https://pypi.org/project/lazyimp/ https://pypi.org/project/lazyimp/ that I use fairly heavily. Syntactically it's just wrapping otherwise unmodified import statements in a with block, so tools 'just work' and it can be easily disabled or told to import eagerly for debugging. It's powered primarily by swapping out the frame's f_builtins in a cext (as it needs more power than importlib hooks provide), but has a lame attempt at a threadsafe pure python version, and a super dumb global hook version. I was skeptical and cautious with it at first but I've since moved large chunks of my codebase to it - it's caused surprisingly few problems (honestly none besides forgetting to handle some import-time registration in some modules) and the speed boost is addictive.
  • NSPG911 1y ago
    looks very interesting! i might use this for some of my projects as well
• film42 1y ago
  I'm a fan because it's something you can explicitly turn on and off. For my Docker based app, I really want to verify the completeness of imports. Preferably, at build and test time. In fact, most of the time I will likely disable lazy loading outright. But, I would really appreciate a faster loading CLI tool. However, there is a pattern in python to raise an error if, say, pandas doesn't have an excel library installed, which is fine. In the future, will maintainers opt to include a bunch of unused libraries since they won't negatively impact startup time? (Think pandas including 3-4 excel parsers by default, since it will only be loaded when called). It's a much better UX, but, now if you opt out of lazy loading, your code will take longer to load than without it.
• the__alchemist 1y ago
  Does this fix the circular imports problem that comes up if you don't structural your programs in a hierarchical way?
  • colpabar 1y ago
    This is what I thought of too. I really only know python, do other languages not have that issue? In python it does not seem like a "problem" to me - whenever I have seen circular import issues it is because the code is organized poorly. I worry that this feature will lead to devs "fixing" circular import issues by using lazy imports.
    • matsemann 1y ago
      Sometimes it's hard to avoid cyclic imports, without blaming the design. Like if a Parent has a Child, and the Child needs to know of the parent. Only way to solve that in python is to put everything in the same file, which also feels like bad deisgn.
      • colpabar 1y ago
        I would say in that case, the Parent and Child shouldn't need to know about each other - some kind of handler in a different file should. Although I guess that doesn't work in all cases, like defining foreign key relationships when using an orm (like sqlalchemy) for example. But in the orm case, the way to get around that is... lazy resolution :^)
    • the__alchemist 1y ago
      Interesting. I find that I always have this problem in any non-trivial python project, and don't find this to be due to poorly organized code. I have only seen this requirement in Python. IME circular import errors aren't due to poor organization; they're due to an arbitrary restriction Python has.
  • Spivak 1y ago
    It should, when you do lazy imports on your own it fixes the problem.
• keeganpoppen 1y ago
  YES!!!
• behnamoh 1y ago
  Make Python more like Haskell, LFG!
• charliermarsh 1y ago
  Lazy imports have been proposed before, and were rejected most recently back in 2022: https://discuss.python.org/t/pep-690-lazy-imports-again/19661/26 https://discuss.python.org/t/pep-690-lazy-imports-again/1966.... If I recall correctly, lazy imports are a feature supported in Cinder, Meta's version of CPython, and the PEP was driven by folks that worked on Cinder. Last time, a lot of the discussion centered around questions like: Should this be opt-in or opt-out? At what level? Should it be a build-flag for CPython itself? Etc. The linked post suggests that the Steering Council ultimately rejected it because of the complexity it would introduce to have two divergent "modes" of importing. I hope this proposal succeeds. I would love to use this feature.
  • dheera 1y ago
    Oof. I wish they could support version imports import torch==2.6.0+cu124 import numpy>=1.2.6 and support having multiple simultaneous versions of any Python library installed. End this conda/virtualenv/docker/bazel/[pick your poison] mess
    • lblume 1y ago
      uv is good.
    • fouronnes3 1y ago
      The mess has ended thanks to uv.
    • maxloh 1y ago
      You could do that with uv. # /// script # dependencies = [ # "requests<3", # "rich", # ] # /// import requests from rich.pretty import pprint resp = requests.get("https://peps.python.org/api/peps.json") data = resp.json() pprint([(k, v["title"]) for k, v in data.items()][:10])
    • zahlman 1y ago
      It's been explained many times before why this is not possible: the library doesn't actually have a version number. The distribution of source code on PyPI has a version number, but the name of this is not connected to the name of any module or package you import in the source code. The distribution can validly define zero or more modules (packages are a subset of modules, represented using the same type in the Python type system). You got three other responses before me all pointing at uv. They are all wrong, because uv did not introduce this functionality to the Python ecosystem. It is a standard defined by https://peps.python.org/pep-0723/ https://peps.python.org/pep-0723/, implemented by multiple other tools, notably pipx.
• forrestthewoods 1y ago
  I don’t want lazy imports. That’s just makes performance shitty later and harder to debug. It’s a hacky workaround. What I want is for imports to not suck and be slow. I’ve had projects where it was faster to compile and run C++ than launch and start a Python CLI. It’s so bad.
  • dataangel 1y ago
    imports run arbitrary Python code, making them faster is the same as making every other language feature faster
    • forrestthewoods 1y ago
      Can you show me a profile that demonstrate the slowness is because imports are running large amounts of arbitrary Python code? Maybe they shouldn't do that. Maybe libraries should be pushed to not run expensive arbitrary code execution at import time.
• Alir3z4 1y ago
  I wish all imports were lazy by default. I know/heard there are "some" (which I haven't seen by the way) libraries that depend on import side effects, but the advantage is much bigger. First of all, the circular import problem will go away, especially on type hints. Although there was a PEP or recent addition to make the annotation not not cause such issue. Second and most important of all, is the launch time of Python applications. A CLI that uses many different parts of the app has to wait for all the imports to be done. The second point becomes a lot painful when you have a large application, like a Django project where the auto reload becomes several seconds. Not only auto reload crawls, the testing cycle is slow as well. Every time you want to run test command, it has to wait several seconds. Painful. So far the solution has been to do the lazy import by importing inside the methods where it's required. That is something, I never got to like to be honest. Maybe it will be fixed in Python 4, where the JIT uses the type hints as well /s
  • f311a 1y ago
    Lazy imports mean late errors. Fail fast is a good design principle.
    • zahlman 1y ago
      Top-level code should not be able to fail except in incredibly deterministic ways that are tested during development. Failing fast is not as good as not failing at all. Lazy imports mean the power to avoid importing things that don't need to be imported at all on this run. Good design also cares about performance to some extent. On my machine, asking pip to do literally nothing takes several times as long as creating a new virtual environment --without-pip .
    • josefx 1y ago
      Python really seems like a bad fit for that. So your imports succeed, what now? Do they have all the functions or fields your program needs? Those are still resolved at the last possible moment. If you want to be sure your program actually runs you will have to write and run tests with and without lazy imports.
• f311a 1y ago
  I don't like the idea of introducing a new keyword. We need a backward compatible solution. I feel like Python needs some kind of universal annotation syntax such as in go (comments) or in Rust (macros). New keyword means all parsers, lsps, editors should be updated. I’m pretty sure there will be new keywords in Python in the future that only solve one thing.
  • Alir3z4 1y ago
    Me neither. Introducing new keyword has become a recent thing in Python. Seems Python has a deep scare since Python2 to Python3 time and is scared to do anything that causes such drama again. For me, the worst of all is "async". If 2to3 didn't cause much division, the async definitely divided Python libraries in 2. Sync and Async. Maybe if they want backward compatible solution, this can be done by some compile or runtime flag like they did with free threading no-gil.
    • sigmoid10 1y ago
      Async or not in modules is a huge pain in the ass for web-dev, but thankfully a lot of Python still isn't web dev. As a Data Scientist you can live your life peacefully without ever worrying about this. But I can see why web devs like it, even though personally I really don't want to see anymore Javascript sneaking inty my Python. Especially when there is already support for IO bound concurrency elsewhere in the language. If I want to do JS syntax, I'll fucking use JS. And I really don't want to see Python go the C++ route where it just wants to be everything and do everything so that you end up with so many possible approaches to the same problem that two devs can't read each others code anymore.
      • vvpan 1y ago
        I'm relatively new to Python - how does one do concurrent IO without async/await? My main complaint, though, about Python async is - because it is opt-in I never know if I forgot a sync IO call somewhere that will block my worker. JS makes everything async by default and there is effectively no chance of blocking.
• jeremyscanvic 1y ago
  Really excited about this - we've recently been struggling with making imports lazy without completely messing up the code in DeepInverse https://deepinv.github.io/deepinv/ https://deepinv.github.io/deepinv/
• Spivak 1y ago
  I love the feature but I really dislike using the word lazy as a new language keyword. It just feels off somehow. I think maybe defer might be a better word. It is at least keeps the grammar right because it would be lazily. lazily import package.foo vs defer import package.foo Also the grammar is super weird for from imports. lazy from package import foo vs. from package defer import foo.
  • est 1y ago
    second this. Just might as well add `defer` keyword like Golang.
• the_mitsuhiko 1y ago
  I like the approach of ES6 where you pull in bindings that are generally lazily resolved. That is IMO the approach that should be the general strategy for Python.
  • 9dev 1y ago
    The import/require schism was necessary precisely of all the issues with module side effects the python community is rediscovering now. I suppose they won’t get around a similar solution eventually; but judging from years of discussions in the JS world, this is going to be dragging on for a while.
• simonw 1y ago
  Love this. My https://llm.datasette.io/ https://llm.datasette.io/ CLI tool supports plugins, and people were complaining about really slow start times even for commands like "llm --help" - it turned out there were popular plugins that did things like import pytorch at the base level, so the entire startup was blocked on heavy imports. I ended up adding a note to the plugin author docs suggesting lazy loading inside of functions - https://llm.datasette.io/en/stable/plugins/advanced-model-plugins.html#tip-lazily-load-expensive-dependencies https://llm.datasette.io/en/stable/plugins/advanced-model-pl... - but having a core Python language feature for this would be really nice.
  • zahlman 1y ago
    You can implement this from your tool today: https://news.ycombinator.com/item?id=45467489 https://news.ycombinator.com/item?id=45467489 Note that this is global to the entire process, so for example if you make an import of Numpy lazy this way, then so are the imports of all the sub-modules. Meaning that large parts of Numpy might not be imported at all if they aren't needed, but pauses for importing individual modules might be distributed unpredictably across the runtime. Edit: from further experimentation, it appears that if the source does something like `import foo.bar.baz` then `foo` and `foo.bar` will still be eagerly loaded, and only `foo.bar.baz` itself is deferred. This might be part of what the PEP meant by "mostly". But it might also be possible to improve my implementation to fix that.
    • dahart 1y ago
      Note the PEP does have a FAQ entry that mentions reasons they believe this proposed solution might be preferable to LazyLoader https://pep-previews--4622.org.readthedocs.build/pep-0810/#faq https://pep-previews--4622.org.readthedocs.build/pep-0810/#f... Q: Why not use importlib.util.LazyLoader instead? A: LazyLoader has significant limitations: Requires verbose setup code for each lazy import. Has ongoing performance overhead on every attribute access. Doesn’t work well with from ... import statements. Less clear and standard than dedicated syntax.
      • zahlman 1y ago
• aftbit 1y ago
  We tend to prefer explicit top-level imports specifically because they reveal dependency problems as soon as the program starts, rather than potentially hours or days later when a specific code path is executed.
  • maest 1y ago
    Who is "we"?
    • aftbit 1y ago
      My current team at my current company (see bio if you're really interested), though I should say I'm not authorized to speak on behalf of my employer, so I should really say something more like "I".
  • bityard 1y ago
    I haven't fully digested the PEP but perhaps there would be a command-line flag or external tool for dependency validation, a bit like how there are external tools for type annotations?
  • ashu1461 1y ago
    Right, Not sure why but a lot of code which claude generates also comes with local dependencies vs globally declared import statements. Don't promote that pattern because - It reduces visibility into a module’s dependencies. - It increases the risk of introducing circular dependencies later on.
  • zahlman 1y ago
    As a counterpoint, having all the imports automatically deferred would instantly dramatically speed up pip for short tasks. $ time pip install --disable-pip-version-check ERROR: You must give at least one requirement to install (see "pip help install") real 0m0.399s user 0m0.360s sys 0m0.041s Almost all of this time is spent importing (and later unloading) ultimately useless vendored code. From my testing (hacking the wrapper script to output some diagnostics), literally about 500 modules get imported in total (on top of the baseline for a default Python process), including almost a hundred modules related to Requests and its dependencies, even though no web request was necessary for this command.
• rplnt 1y ago
  Remember mercurial? Me neither. But what I remeber is this article I've read about all the hacks they had to do to achieve reasonable startup time for CLI in python. And the no #1 cause was loading the whole world you don't ever need. As I recall they somehow monkeypatched the interpreter to ignore imports and just remember their existence until they were actually needed, at which point the import happened. So all the dead paths were just skipped.
  • yegle 1y ago
    I recall chg was a must.
• jmward01 1y ago
  This is needed, but I don't like new keywords. What I would love, for many reasons, is if we could decorate statements. Then things like: import expensive_module could be: @lazy import expensive_module or you could do: @retry(3) x = failure_prone_call(y) lazy is needed, but maybe there is a more basic change that could give more power with more organic syntax, and not create a new keyword that is special purpose (and extending an already special purpose keyword)
  • jmward01 1y ago
    As a side note, It would be great to have a 'preview' for HN comments. I updated the above because I forgot to add line breaks. Sigh. I bet 'preview' would stop a lot of not well thought out comments too.
    • dijksterhuis 1y ago
      you can set “delay” in your profile to delay the live posting of a comment after you click the button. gives me ten minutes to edit (rewrite if i’m honest) before other people see it.
  • progval 1y ago
    For what it's worth, you can already do: x = retry(3)(failure_prone_call)(y)
    • zahlman 1y ago
      And, for that matter, expensive_module = lazy(importlib.import_module)('expensive_module') .
  • lyu07282 1y ago
    Would this statement decorator then manipulate the AST of the following statement or how would that work?
  • _ZeD_ 1y ago
    good luck with that syntax - it would be as possible as passing an inline-defined def as parameter instead of a lambda
• Boxxed 1y ago
  Ugh...I like the idea, but I wish lazy imports were the default. Python allows side effects in the top level though so that would be a breaking change. Soooo instead now we're going to be in a situation where you're going to be writing "lazy import ..." 99% of the time: unless you're a barbarian, you basically never have side effects at the top level.
• waynesonfire 1y ago
  why does this have to be a syntax feature and not a lazy code loader at the intepreter level?
• zahlman 1y ago
  > The standard library provides the LazyLoader class to solve some of these inefficiency problems. It permits imports at the module level to work mostly like inline imports do. The use of these sorts of Python import internals is highly non-obvious. The Stack Overflow Q&A I found about it (https://stackoverflow.com/questions/42703908/ https://stackoverflow.com/questions/42703908/) doesn't result in an especially nice-looking UX. So here's a proof of concept in existing Python for getting all imports to be lazy automatically, with no special syntax for the caller: import sys import threading # needed for python 3.13, at least at the REPL, because reasons from importlib.util import LazyLoader # this has to be eagerly imported! class LazyPathFinder(sys.meta_path[-1]): # <class '_frozen_importlib_external.PathFinder'> @classmethod def find_spec(cls, fullname, path=None, target=None): base = super().find_spec(fullname, path, target) base.loader = LazyLoader(base.loader) return base sys.meta_path[-1] = LazyPathFinder We've replaced the "meta path finder" (which implements the logic "when the module isn't in sys.modules, look on sys.path for source code and/or bytecode, including bytecode in __pycache__ subfolders, and create a 'spec' for it") with our own wrapper. The "loader" attached to the resulting spec is replaced with an importlib.util.LazyLoader instance, which wraps the base PathFinder's provided loader. When an import statement actually imports the module, the name will actually get bound to a <class 'importlib.util._LazyModule'> instance, rather than an ordinary module. Attempting to access any attribute of this instance will trigger the normal module loading procedure — which even replaces the global name. Now we can do: import this # nothing shows up print(type(this)) # <class 'importlib.util._LazyModule'> rot13 = this.s # the module is loaded, printing the Zen print(type(this)) # <class 'module'> That said, I don't know what the PEP means by "mostly" here.
  • redleader55 1y ago
    Can you explain why does "threading" needs to be loaded? The rest seems decently straightforward, but what initialization from threading is required and why only in 3.13+?
    • zahlman 1y ago
      I didn't investigate deeply (I just happened to stumble on the issue while figuring out the implementation), but it might be needed in older versions too. The loading process instantiates a `threading.RLock`, but from a source code file that doesn't import that (because it's in a file that has to bootstrap the import process). I'm not entirely sure how `threading` is supposed to get imported normally. The other complicating factor is that I was testing this at the REPL, and 3.13 has a new REPL implementation which might change how those initial libraries are brought in.
• comex 1y ago
  I don't hate it but I don't love it. It sounds like everyone will start writing `lazy` before essentially every single import, with rare exceptions where eager importing is actually needed. That makes Python code visually noisier. And with no plan to ever change the default, the noise will stay forever. I would have preferred a system where modules opt in to being lazy-loaded, with no extra syntax on the import side. That would simplify things since only large libraries would have to care about laziness. To be fair, in such a design, the interpreter would have to eagerly look up imports on the filesystem to decide whether they should be lazy-loaded. And there are probably other downsides I'm not thinking of.
  • BiteCode_dev 1y ago
    We heard that about types, the walrus, asyncio, dataclasses and so much more. But it didn't happen, if people don't need something (and many don't know it exists or what it does), it's unlikely they use it. In fact, half of the community basically uses only a modernized set of python 2.4 features and that's one of the beauties of the language. You don't need a lot to be productive, and if you want more, you can optionally reach for it. It has worked very well for the last 2 decades and it will likely work again.
    • hnlmorg 1y ago
      People said the same about Perl and its “there’s more than one way to do things” ethos, which gained much criticism. Same is true for C++. In this specific case, I think a lazy load directive isn’t a bad addition. But one does need to be careful about adding new language features just because you have an active community.
    • comex 1y ago
      Perhaps people won't use it. But I for one want it to be used, and I will certainly be using it in my own code if the PEP is accepted. Startup time is very important to me, more important than the cost of making the code noisier. I just wish I didn't have to make that tradeoff in the first place.
  • Spivak 1y ago
• enriquto 1y ago
  what is the point of this? you can just import inside function definitions: def antislash(A, b): from numpy.linalg import solve return solve(A, b) thus numpy.linalg is only imported the first time you call the antislash function. Much cleaner than a global import. Ignore wrong traditions. Put all imports in the innermost scopes of your code!
  • famouswaffles 1y ago
    That's a hack that forces you to duplicate and hide imports. The tradition is to specify imports at the top because it's that much better readability wise.
• gorgoiler 1y ago
  I don’t really agree with the premise: > The dominant convention in Python code is to place all imports at the beginning of the file. This avoids repetition, makes import dependencies clear and minimizes runtime overhead. > A somewhat common way to delay imports is to move the imports into functions, but this practice requires more work [and] obfuscates the full set of dependencies for a module. The first part is just saying the traditions exist because the traditions have always existed. Traditions are allowed to change! The second part is basically saying if you do your own logic-based lazy imports (inline imports in functions) then you’re going against the traditions. Again, traditions are allowed to change! The point about the import graph being obfuscated would ring more true if Python didn’t already provide lightning fast static analysis tools like ast. If you care about import graphs at the module level then you’re probably already automating everything with ast anyway, at which point you just walk the whole tree looking for imports rather than the top level. So, really, the whole argument for a new lazy keyword (instead of inlining giant imports where they are needed) is because people like to see import pytorch at the top of the file, and baulk at seeing it — and will refuse to even look for it! — anywhere else? Hmmm. What does seem like a pain in the ass is having to do this kind of repetitive crap (which they mention in their intro): def f1(): import pytorch … def f2(): import pytorch … def f3(): import pytorch … But perhaps the solution is a pattern where you put all your stuff like that in your own module and it’s that module which is lazy loaded instead?
  • famouswaffles 1y ago
    >The second part is basically saying if you do your own logic-based lazy imports (inline imports in functions) then you’re going against the traditions. Again, traditions are allowed to change! No, they are saying the tradition is there for a reason. Imports at the beginning of the file makes reasoning about the dependencies of a module much easier and faster. I've had to deal with both and I sure as hell know which I'd prefer. Lazy imports by functions is a sometimes necessary evil and it would be very nice if it became unnecessary.
  • fastasucan 1y ago
    >The second part is basically saying if you do your own logic-based lazy imports (inline imports in functions) then you’re going against the traditions. Think you need to read again.
• nodesocket 1y ago
  Won’t this have a very noticeable performance hit on the first request? Thinking web frameworks like Flask and Django.
• thayne 1y ago
  One thing the PEP doesn't really talk about, and that I find very annoying is that many python linters will complain if you don't put all of your imports at the top of the file, so you get lint warnings if you do the most obvious way to implement lazy imports. And that is actually a problem for more than just performance. In some cases, importing at the top might actually just fail. For example if you need a platform specific library, but only if it is running on that platform.
  • bcoates 1y ago
    I don't think there is any solution for that but "fix your broken linter".
    • thayne 1y ago
      It isn't just one though. Every linter I've used has warned about that. Probably because PEP 8 says > Imports are always put at the top of the file, just after any module comments and docstrings, and before module globals and constants
      • BHSPitMonkey 1y ago
        Ruff doesn't do this, and in fact even lets you specify modules that _must_ not be imported at the top level (banned-module-level-imports = [...]) I banished the worst/heaviest libraries to this list at my workplace and it's been really helpful at keeping startup times from regressing.
  • phainopepla2 1y ago
    It is annoying, but most linters will accept a `#noqa E402` comment to ignore it
    • zahlman 1y ago
      >most linters will accept So, they agreed on a common system of linting error codes? Is that documented somewhere?
• thijsvandien 1y ago
  Very unfortunate how from json lazy import dumps would result in backward compatibility issues. It reads much better and makes it easier to search for lazy imports, especially if in the future something else becomes optionally lazy as well.
• mekoka 1y ago
  Crossing my fingers that it goes through this time. Been in the top 3 of my Python wishlist for nearly a decade.
• nilslindemann 1y ago
  This is the wrong syntax, comparable to how "u" strings were the wrong syntax and "b" strings are the right syntax. They make that, what should be the default, a special case. Soon, every new code will use "lazy". The long term effect of such changes is a verbose language syntax. They should have had a period where one, if they want lazy imports, has to do "from __future__ import lazy_import". After that period, lazy imports become the default. For old-style immediate imports, introduce a syntax: "import now foo" and "from foo import now bar". All which authors of old code would have to do is run a provided fix script in the root directory of their code.
  • nilslindemann 1y ago
    In short, grandiose change, wrong syntax.
  • zahlman 1y ago
    I think your assessment of what's "the right/wrong" syntax is fair. But the transition you describe takes a long time, even now that the community has figured out a "deprecation cycle" process that seems satisfactory (i.e. won't lead to another Python 3.0 situation). > All which authors of old code would have to do is run a provided fix script in the root directory of their code. As I recall, `lib2to3` didn't do a lot to ease tensions. And `six` is still absurdly popular, mainly thanks to `python-dateutil` still attempting to support 2.7.
    • nilslindemann 1y ago
      Honestly, I don't care for 2 to 3 or six (which, btw, has just 1k stars on GitHub, compare to e.g. 90k stars for FastAPI). Someone who bases their code on 2.7 in 2025 must expect to run into trouble. If people do not run such upgrade scripts, it must be documented better. Python is free. In order to also stay elegant, they should say to their users: "We expect from you to run an upgrade script on your code once per Python upgrade"
  • nilslindemann 1y ago
    I have posted this Idea on Python Discussions, and Thomnas Wouters replied to it and to others: https://discuss.python.org/t/pep-810-explicit-lazy-imports/104131/80 https://discuss.python.org/t/pep-810-explicit-lazy-imports/1... The specific part of the PEP: https://github.com/python/peps/pull/4628/files#diff-ca0112672b06aae6c30c7d4213a58c42f2ad76dd1ec2a17bdc77e988fe75a6cbR1328-R1335 https://github.com/python/peps/pull/4628/files#diff-ca011267...
• dataangel 1y ago
  This would be a huge deal for Python startup time *if* it was applied to all the standard library packages recursively. Right now importing asyncio brings in half the standard library through transitive imports.
  • zahlman 1y ago
    It's bad, but it's not that bad. $ time python -c '' # baseline real 0m0.020s user 0m0.015s sys 0m0.005s $ time python -c 'import sys; old = len(sys.modules); import asyncio; print(len(sys.modules) - old)' 104 real 0m0.076s user 0m0.067s sys 0m0.009s For comparison, with the (seemingly optimized) Numpy included with my system: $ time python -c 'import sys; old = len(sys.modules); import numpy; print(len(sys.modules) - old)' 185 real 0m0.124s user 0m0.098s sys 0m0.026s
• romanows 1y ago
  Kinda related, I wish there was an easy way to exclude dependencies at pip-install time and mock them at runtime so an import doesn't cause an exception. Basically a way for me to approximate "extras" when the author isn't motivated to do it for me, even though it'd be super brittle.
  • zahlman 1y ago
    This sounds doable, actually. You'd want to pre-install (say, from a local wheel) a matching dummy dependency where the metadata claims that it's the right version of whatever package (so the installer will just see that the dependency is "already satisfied" and skip it), but the actual implementation code just exposes a hook to your mocking system. Doesn't work if version resolution decides to upgrade or downgrade your installed package, so you need to make sure the declared version is satisfactory, too.
• throwaway81523 1y ago
  We need some kind of unexec revival in Linux so we don't have to resort to crap like this. Maybe CRIU based. Worst case, some Python specific hack. But Python's import system is wacky enough without lazy loading. This sounds like combining the worst of multiple worlds.
• jacquesm 1y ago
  Lazy imports are a great way to create runtime errors far into the operation of a long lived service. Yes, it gives the superficial benefit of 'fast startup', but that upside is negated by the downside of not being sure that once something runs it will run to completion due to a failed import much further down the line. It also allows for some interesting edge cases with the items that are going to be imported no longer being what is on the tin at the time the program is started.
  • famouswaffles 1y ago
    That's fine, because this is still a genuine problem in need of a solution. It's not just about startup time for the sake of it (not that this is even a superficial concern - python startup time with large dependencies quickly gets awful). Large projects can have hefty dependencies that not every user will use. And bundling it all for everyone can sometimes be intractable. The work arounds people use already have the sort of problems you're talking about, on top of being diabolical and hacky. Not having to duplicate and hide imports in functions alone would be a big improvement. It's not like it isn't being proposed as an optional language feature.
  • BHSPitMonkey 1y ago
    An automated test mitigates the risk you describe, and is well worth the tradeoff for fast startup. I don't consider startup time "superficial" at all; I work in a Django monolith where this problem resulted in each and every management command, test invokation, and container reload incurring a 10-15sec penalty because of just a handful of heavy-to-import libraries used by certain tasks/views. Deferring these made a massive difference.
• d_burfoot 1y ago
  Looks good to me. I use a tab-completion trick where the tab-completer tool calls the script I'm about to invoke with special arguments, and the script reflects on itself and responds with possible completions. But because of slow imports, it often takes a while for the completion to respond. I could, and sometimes do, go through all the imports to figure out which ones are taking a long time to load, but it's a chore.
• bcoates 1y ago
  I think they're understating the thread safety risks here. The import is going to wind up happening at a random nondeterministic time, in who knows what thread holding who knows what locks (aside from the importer lock). Previously, if you had some thread hazardous code at module import time, it was highly likely to only run during the single threaded process startup phase, so it was likely harmless. Lazy loading is going to unearth these errors in the most inconvenient way (as Heisenbugs) (Function level import can trigger this as well, but the top of a function is at least a slightly more deterministic place for imports to happen, and an explicit line of syntax triggering the import/bug)
• Too 1y ago
  This is a great compromise given how much would break if this was the default. Making this the default would be better in the long-run, require taming the abuse of side-effects in imports too, win-win. If one could dream, modules should have to explicitly declare whether they have side-effects or not, with a marker at the top of the module. Not declaring this and trying anything except declaring a function or class should lead to type-checker errors. Modules declaring this pure-marker then automatically become lazy. Others should require explicit "import_with_side_effects" keyword. __pure__ = True import logging import threading app = Flask() # << ImpureError sys.path.append() # << ImpureError with open(.. ) # << ImpureError logging.basicSetup() # << ImpureError if foo: # << ImpureError (arguable) @app.route("/foo") # Questionable def serve(): # OK ... serve() # << ImpureError t = threading.Thread(target=serve) # << ImpureError All of this would be impossible today, given how much the Python relies on metaprogramming. Even standard library exposes functions to create classes on the fly like Enum and dataclasses, that are difficult to assert as either pure or impure. With more and more of the ecosystem embracing typed Python, this metaprogramming is reducing into a less dynamic subset. Type checkers and LSPs must have at least some awareness of these modules, without executing them as plain python-code.
• ajb 1y ago
  Given all the problems people are mentioning, it seems like this proposal is on the wrong side. There should be an easy way for a module to declare itself to be lazy loaded. The module author, not the user, is the one who knows whether lazy loading will break stuff.
  • Grikbdl 1y ago
    It's also simpler. It could even be a package level thing similar to typed.py marker. I don't want to pepper literally all my modules with loads of explicit lazy keywords.
  • zahlman 1y ago
    > There should be an easy way for a module to declare itself to be lazy loaded. It can just implement lazy loading itself today, by using module-level __getattr__ (https://docs.python.org/3/reference/datamodel.html#customizing-module-attribute-access https://docs.python.org/3/reference/datamodel.html#customizi...) to overwrite itself with a private implementation module at the appropriate time. Something like: # foo.py def __getattr__(name): # clean up the lazy loader before loading # this way it's cleaned up if the implementation doesn't replace it, # and not scrubbed if it does global __getattr__ del __getattr__ import sys self = sys.modules[__name__] from . import _foo # "star-import" by adding names that show up in __dir__ self.__dict__.update(**{k: getattr(_foo, k) for k in _foo.__dir__()}) # On future attribute lookups, everything will be in place. # But this time, we need to delegate to the normal lookup explicitly return getattr(self, name) Genericizing this is left as an exercise.
• pjjpo 1y ago
  I wonder if this proposal suffers an because of Python's extremely generous support period and perhaps the ship has sailed. - lazy imports are a hugely impactful feature - lazy imports are already possible without syntax This means any libraries that get large benefit from lazy imports already use import statements within functions. They can't really use the new feature since 3.14 EoL is _2030_, forever from now. The __lazy_modules__ syntax preserves compatibility only but being eager, not performance - libraries that need lazy imports can't use it until 2030. This means that the primary target for a long time is CLI authors, which can have a more strict python target and is mentioned many times in the PEP, and libraries that don't have broad Python version support (meaning not just works but works well), which indicates they are probably not major libraries. Unless the feature gets backported to 2+ versions, it feels not so compelling. But given how it modifies the interpreter to a reasonable degree, I wonder if even any backport is on the table.
  • Grikbdl 1y ago
    In at least the scientific python environment, there's "SPEC0" in which a lot of the de facto core libraries have basically agreed to support the last three versions of Python, no more. For other libraries they can of course choose as they want, but generally I don't think it's so common for libraries to be as generous with the support length as cpython.
  • deleted 1y ago
    [deleted]
• shiandow 1y ago
  I don't quite understand why they forbid this from being used in functions. I mean I get why that makes the most sense in most scenarios, but it is not as if the problem of having to choose between declaring dependencies up front or deferring expensive imports until needed does not happen in functions. Take for instance a function that quickly fails because the arguments are incorrect, it might do a whole bunch of imports that only make sense for that function but which are made immediately obsolete. It feels like it is forbidden just because someone thought it wasn't a good coding style but to me there is no obvious reason it couldn't work.
• Hendrikto 1y ago
  Python imports are already way too complicated. I don’t think even more syntax should be added.
• hhthrowaway1230 1y ago
  doesn't this make the language a little unpredictable in terms of loading times? requiring to touch all parts to fully load the app?
  • amdsn 1y ago
    I think a feature like this sees best use in short lived programs (where startup time is a disproportionate percentage of total run time) and programs where really fast startup is essential. There are plenty of places where I could imagine taking advantage of this in my code at work immediately, but I share your concern about unpredictability when libraries we use are also making use of it. It wouldn't be fun to have to dive into dependencies to see what needs to be touched to trigger lazy imports at the most convenient time. Unless I am misunderstanding and a normal import of a module means that all of its lazy imports also become non-lazy?
• f33d5173 1y ago
  It would be interesting if instead you added a syntax whereby a module could declare that it supported lazy importing. Maybe even after running some code with side effects that couldn't be done lazily. For one thing, this would have a much broader performance impact, since it would benefit all users of the library, not just those who explicitly tagged their imports as lazy. For another, it would minimize breakage, since a module author knows best whether, and which parts of, their module can be lazily loaded. On the other hand, it would create confusion for users of a library when the performance hit of importing a library was delayed to the site of usage. They might not expect, for example, a lag to occur there. I don't think it would cause outright breakage, but people might not like the way it behaved.
• deleted 1y ago
  [deleted]
• croemer 1y ago
  The link target should be https://peps.python.org/pep-0810/ https://peps.python.org/pep-0810/ not a preview build for a PR. @dang
• wmichelin 1y ago
  This seems unnecessary if we can run imports from functions, right? This feels like a layer of indirection and a potential source for confusion. Rather than implicitly importing a library when the variable is first used, why don't you just explicitly do it? Edit > A somewhat common way to delay imports is to move the imports into functions (inline imports), but this practice requires more work to implement and maintain, and can be subverted by a single inadvertent top-level import. Additionally, it obfuscates the full set of dependencies for a module. Analysis of the Python standard library shows that approximately 17% of all imports outside tests (nearly 3500 total imports across 730 files) are already placed inside functions or methods specifically to defer their execution. This demonstrates that developers are already manually implementing lazy imports in performance-sensitive code, but doing so requires scattering imports throughout the codebase and makes the full dependency graph harder to understand at a glance. I think this is a really weak foundation for this language feature. We don't need it.
  • tremon 1y ago
    On the contrary, I think "everyone is already doing this in their own way, so we will offer it as a language feature, which brings these extra benefits" is one of the stronger arguments for introducing a language feature.
  • wsve 1y ago
    > Rather than implicitly importing a library when the variable is first used, why don't you just explicitly do it? I think it's on you to explain why that's a better approach for everyone's use cases instead of this language feature
• huflungdung 1y ago
  [dead]
• thatxliner 1y ago
  I was looking at PEP 690 and I saw > A world in which Python only supported imports behaving in a lazy manner would likely be great...we do not envision the Python langauge transitioning to a world where lazy imports are the default...this concept would add complexity to our ecosystem. Why can't lazy be the default and instead propose an `eager` syntax? The only argument I can imagine is that there's some API that runs a side effect based on importing, but perhaps making it eager for modules with side effects would be a sufficient temporary fix?