7 ms·

Python Steering Council unanimously accepts "PEP 810, Explicit lazy imports"

170 points by Redoubts 11mo ago

Redoubts 11mo ago
I think HN is translating the link somehow? It should be directing to this post: https://discuss.python.org/t/pep-810-explicit-lazy-imports/104131/466 https://discuss.python.org/t/pep-810-explicit-lazy-imports/1... """ Dear PEP 810 authors. The Steering Council is happy to unanimously [4 votes, as Pablo cannot vote] accept “PEP 810, Explicit lazy imports”. Congratulations! We appreciate the way you were able to build on and improve the previously discussed (and rejected) attempt at lazy imports as proposed in PEP 690. We have recommendations about some of the PEP’s details, a few suggestions for filling a couple of small gaps, and we have made decisions on the alternatives that you’ve left to the SC, all of which I’ll outline below. If you have any questions, please do reach out to the SC for clarification, either here, on the SC tracker, or in office hours. Use lazy as the keyword. We debated many of the given alternatives (and some we came up with ourselves), and ultimately agreed with the PEP’s choice of the lazy keyword. The closest challenger was defer, but once we tried to use that in all the places where the term is visible, we ultimately didn’t think it was as good an overall fit. The same was true with all the other alternative keywords we could come up with, so… lazy it is! What about from foo lazy import bar? Nope! We like that in both module imports and from-imports that the lazy keyword is the first thing on the line. It helps to visually recognize lazy imports of both varieties. Leveraging a subclass of dict. We don’t see a need for this complicated alternative; please add this to the rejected ideas. Allowing ’*’ in __lazy_modules__. We agree with the rationale for rejecting this idea; it can always be added later if needed. One thing that the PEP does not mention is .pth files, which the site.py module processes, and which has some special handling for lines that begin with the string 'import' followed by a space or tab. It doesn’t make much sense for .pth files to support lazy imports, so we suggest that the PEP explicitly says that this special handling in .pth files will not be adapted to handle lazy imports. There currently is no way to get the active filter mode, so please add a sys.get_lazy_imports() function. Also, do you think appending _mode to their names makes the purpose of these functions clearer? We leave that up to the PEP authors. The PEP should be explicit about the precedence order between the different ways to set the mode, i.e. $PYTHON_LAZY_IMPORTS=<mode>, -X lazy_imports=<mode>, and sys.set_lazy_imports(). In all expectation, it will follow the same precedence order as other similar settings, but the PEP should be explicit. We agree that the PEP should take no position on any style recommendations for sorting lazy imports. While we generally like the idea of grouping lazy imports together, let’s leave that up to the linters and auto-formatters to decide the details. That should just about cover it. Again, thank you for your work on this, as it’s been a feature so many in the Python community have wanted for so long. Given the earlier attempts and existing workarounds, we think this strikes exactly the right balance. -Barry, on behalf of the Python Steering Council """
- lolpython 11mo ago
  HN is using the canonical URL for the page. See https://developer.mozilla.org/en-US/docs/Web/HTML/Attributes/rel#canonical https://developer.mozilla.org/en-US/docs/Web/HTML/Attributes...
  - NooneAtAll3 11mo ago
    so it's a misconfig on python.org side?
    - lolpython 11mo ago
      I think so? It caused the same issue when I cross posted to Lobste.rs
- rhat1512 11mo ago
  [flagged]
andrewmcwatters 11mo ago
I've noticed that instead of defining requires at the top of Lua files, if you can know your own program well enough, defining them right before the dependency is actually used makes large Lua programs much, much quicker. Generally speaking, startup times can be a sped up by a meaningful factor. I suspect this change in Python will dramatically improve the performance of such large programs as well.
- embedding-shape 11mo ago
  > I suspect this change in Python will dramatically improve the performance of such large programs as well. Makes packaging super fun too, where you need to hit every possible path so you don't miss anything imported in 1% of the execution paths :)
  - snovv_crash 11mo ago
    Can't you do some kind of static analysis instead?
    - fithisux 11mo ago
      Yes. It is much more clear to be explicit though.
      - jacquesm 11mo ago
        Safer too.
    - Figs 11mo ago
      Depends if your code has horrors like this lurking in it: m = importlib.import_module(requests.get("http://localhost:8000/package_name").content.strip().decode("ASCII"))
      - falcor84 11mo ago
        If you want even better nightmares, you can make localhost:8000 forward to a container running claude code with --dangerously-skip-permissions which uses an unkindness of mcp servers to control that endpoint on the fly based (amongst other sources) on 4chan's /b/.
ayhanfuat 11mo ago
That was fast. It was sent to SC 20 days ago. (Not complaining. I am happy with the outcome).
- zahlman 11mo ago
  Quite a few PEPs are accepted this quickly, actually. But some others take literally years.
xenator 11mo ago
Does it conform Occam's razor rule to have something that can be easily done very similar way without changing language?
- boothby 11mo ago
  Having some limited experience with lazy imports, yes, but this eliminates a lot of gross boilerplate. It also has the effect of "blessing" the practice of lazy imports which can have a cultural impact; it also prevents a situation wherein multiple subtly incompatible approaches to lazy imports become individually popular.
- contravariant 11mo ago
  Not sure that's Occam's razor any more. Regardless lazy loading needs widespread use to be most effective so having a unified syntax and no extra dependencies makes a lot of sense.
jauntywundrkind 11mo ago
Source phase imports are stage 3 (recommended for implementation, no major updates expected) in JS. "Import source" tells the runtime to go get the code, but not yet run it. Similar ideas seemingly to what's going on here in python! https://github.com/tc39/proposal-source-phase-imports https://github.com/tc39/proposal-source-phase-imports
- zahlman 11mo ago
  This proposal defers even looking for the code. The LazyLoader already in the standard library would eagerly look for code, but just record a file path and not actually store any bytecode data, never mind deserializing or running it. The rationale described in the PEP is that some systems try to `import`, for example, across a network share, so even searching the filesystem is slow and there is a desire to defer that (and avoid it on runs where the corresponding code isn't executed).
johnfn 11mo ago
This will be huge at the place I work! I’m unfamiliar with the PEP process. How long until this makes it into a Python version?
- joerick 11mo ago
  It should land in 3.15, so October next year. https://peps.python.org/pep-0790/ https://peps.python.org/pep-0790/
- zahlman 11mo ago
  This one is scheduled to land in the next "minor" version, 3.15. Python has an annual release cadence; 3.14 came out recently and 3.15 is due next October. In general, most PEPs are authored targeting the "next minor version" at the time of proposal; but they may be intentionally deferred at the start, and sometimes the process can take multiple years anyway. There are also PEPs that don't involve any change to the Python language, standard library or interpreter. In particular, there are PEPs that exist simply to document existing practice (or changes thereto), PEPs that concern governance (the Python Software Foundation, the Steering Council etc.), and PEPs that cover related special interests, such as packaging standards (which in turn can range from technical details about how metadata is formatted, to changes in PyPI's API). https://peps.python.org/pep-0000/ https://peps.python.org/pep-0000/
curiousgal 11mo ago
Some folks at HRT[0] will probably be unhappy about that lol 0.https://www.hudsonrivertrading.com/hrtbeat/inside-hrts-python-fork/ https://www.hudsonrivertrading.com/hrtbeat/inside-hrts-pytho...
- kccqzy 11mo ago
  Why would they? The accepted PEP just has extra keywords to mark imports explicit. That's a source level find-and-replace. They already did most of the real work in their codebase, such as finding where the code depended on the side effect of importing a module.
- seemaze 11mo ago
  I'm confused, wouldn't HRT be happy about this? The article you linked specifically states "..we hope to propose a revised lazy imports PEP that introduces an explicit lazy keyword.." Is that not exactly what PEP 810 proposes?
- CorrectHorseBat 11mo ago
  Look at the names in the PEP [1], this PEP is written by them [1] https://peps.python.org/pep-0810/ https://peps.python.org/pep-0810/
manbitesdog 11mo ago
This is great for building modules. One can now lazy import all interesting names on __init__.py, so that instead of having to remember `from module.some_submodule_that_you_need_to_remember import method` you can just do `from module import name`.
- maxbond 11mo ago
  https://peps.python.org/pep-0810/#what-about-star-imports-from-module-import https://peps.python.org/pep-0810/#what-about-star-imports-fr... > What about star imports (`from module import *`)? > Wild card (star) imports cannot be lazy - they remain eager. This is because the set of names being imported cannot be determined without loading the module. Using the lazy keyword with star imports will be a syntax error. If lazy imports are globally enabled, star imports will still be eager. Additionally, star imports can interfere with type checkers and IDEs and shadowing caused by star imports is a frequent and difficult to diagnose source of bugs (you analyze the function you think you're calling and find no issues, but you're actually calling a different function because the star import occurs after your explicit import). You might be able to workaround this limitation by doing a lazy import into an intermediate module (a prelude) on a name by name basis and then star import that intermediate module. But personally I solve this problem using IDE features. https://github.com/python-lsp/python-lsp-server/blob/develop/docs/autoimport.md https://github.com/python-lsp/python-lsp-server/blob/develop...
  - manbitesdog 11mo ago
    ? This has nothing to do with star imports
    - maxbond 11mo ago
      My bad for misreading you
alberth 11mo ago
Source link indicating PEP 810 was accepted: https://discuss.python.org/t/pep-810-explicit-lazy-imports/104131/466 https://discuss.python.org/t/pep-810-explicit-lazy-imports/1...
ant6n 11mo ago
Next we need lazy import *
- swiftcoder 11mo ago
  Is that not the purpose of the global switch outlined in the PEP?
  - cgriswald 11mo ago
    No. From the PEP: Where <mode> can be: "normal" (or unset): Only explicitly marked lazy imports are lazy "all": All module-level imports (except in try blocks and import *) become potentially lazy "none": No imports are lazy, even those explicitly marked with lazy keyword When the global flag is set to "all", all imports at the global level of all modules are potentially lazy except for those inside a try block or any wild card (from ... import *) import.
nothrowaways 11mo ago
Python is quickly turning into a crowded keyword junkyard
- riedel 11mo ago
  It is a 'soft keyword' as the PEP explains. I would not think that this has any major impact on anyone who just chooses to ignore this feature. Assuming that you want this behavior, I wonder how this could have been done in a better fashion without now having 'lazy' in the specific context of an import statement.
  - rrauenza 11mo ago
    soft keyword for anyone not familiar like I was ... "A new soft keyword lazy is added. A soft keyword is a context-sensitive keyword that only has special meaning in specific grammatical contexts; elsewhere it can be used as a regular identifier (e.g., as a variable name). The lazy keyword only has special meaning when it appears before import statements..."
- notatallshaw 11mo ago
  Python has about 40 keywords, I say I would regularly use about 30, and irregularly use about another 5. Hardly seems like a "junkyard". Further, this lack of first class support for lazy importing has spawned multiple CPython forks that implement their own lazy importing or a modified version of the prior rejected PEP 690. Reducing the real world need for forks seems worth the price of one keyword.
  - lairv 11mo ago
    For those curious here are the actual keywords (from https://docs.python.org/3/reference/lexical_analysis.html?utm_source=chatgpt.com#keywords https://docs.python.org/3/reference/lexical_analysis.html?ut... ) Hard Keywords: False await else import pass None break except in raise True class finally is return and continue for lambda try as def from nonlocal while assert del global not with async elif if or yield Soft Keywords: match case _ type I think nonlocal/global are the only hard keywords I now barely use, for the soft ones I rarely use pattern matching, so 5 seems like a good estimate
divbzero 11mo ago
How will lazy imports interact with PEP 8 which recommends grouping imports in order: 1. Standard library imports. 2. Related third party imports. 3. Local application/library specific imports. https://peps.python.org/pep-0008/#imports https://peps.python.org/pep-0008/#imports Based on the examples in PEP 810, I suppose each group of regular imports can be followed by a group of lazy imports? import os import sys lazy import json import fastapi lazy import numpy import myapi import mymodels lazy import myutils ...
ta371jashgG 11mo ago
This has been rejected several times before. Now that Google and Microsoft fired core developers, Python Inc. is owned by Meta. Meta supports lazy imports in Cinder and needs it for its atrocious and bloated scientific ecosystem so that the bloat loads in less than 5 min. If Meta also starts firing Python developers, development might return to normal.
mmis1000 11mo ago
Given you can already trigger import in a local scope, it's merely a syntax sugar though.

170 points by Redoubts 11mo ago

I think HN is translating the link somehow? It should be directing to this post: https://discuss.python.org/t/pep-810-explicit-lazy-imports/104131/466 https://discuss.python.org/t/pep-810-explicit-lazy-imports/1... """ Dear PEP 810 authors. The Steering Council is happy to unanimously [4 votes, as Pablo cannot vote] accept “PEP 810, Explicit lazy imports”. Congratulations! We appreciate the way you were able to build on and improve the previously discussed (and rejected) attempt at lazy imports as proposed in PEP 690. We have recommendations about some of the PEP’s details, a few suggestions for filling a couple of small gaps, and we have made decisions on the alternatives that you’ve left to the SC, all of which I’ll outline below. If you have any questions, please do reach out to the SC for clarification, either here, on the SC tracker, or in office hours. Use lazy as the keyword. We debated many of the given alternatives (and some we came up with ourselves), and ultimately agreed with the PEP’s choice of the lazy keyword. The closest challenger was defer, but once we tried to use that in all the places where the term is visible, we ultimately didn’t think it was as good an overall fit. The same was true with all the other alternative keywords we could come up with, so… lazy it is! What about from foo lazy import bar? Nope! We like that in both module imports and from-imports that the lazy keyword is the first thing on the line. It helps to visually recognize lazy imports of both varieties. Leveraging a subclass of dict. We don’t see a need for this complicated alternative; please add this to the rejected ideas. Allowing ’*’ in __lazy_modules__. We agree with the rationale for rejecting this idea; it can always be added later if needed. One thing that the PEP does not mention is .pth files, which the site.py module processes, and which has some special handling for lines that begin with the string 'import' followed by a space or tab. It doesn’t make much sense for .pth files to support lazy imports, so we suggest that the PEP explicitly says that this special handling in .pth files will not be adapted to handle lazy imports. There currently is no way to get the active filter mode, so please add a sys.get_lazy_imports() function. Also, do you think appending _mode to their names makes the purpose of these functions clearer? We leave that up to the PEP authors. The PEP should be explicit about the precedence order between the different ways to set the mode, i.e. $PYTHON_LAZY_IMPORTS=<mode>, -X lazy_imports=<mode>, and sys.set_lazy_imports(). In all expectation, it will follow the same precedence order as other similar settings, but the PEP should be explicit. We agree that the PEP should take no position on any style recommendations for sorting lazy imports. While we generally like the idea of grouping lazy imports together, let’s leave that up to the linters and auto-formatters to decide the details. That should just about cover it. Again, thank you for your work on this, as it’s been a feature so many in the Python community have wanted for so long. Given the earlier attempts and existing workarounds, we think this strikes exactly the right balance. -Barry, on behalf of the Python Steering Council """

lolpython 11mo ago
HN is using the canonical URL for the page. See https://developer.mozilla.org/en-US/docs/Web/HTML/Attributes/rel#canonical https://developer.mozilla.org/en-US/docs/Web/HTML/Attributes...
- NooneAtAll3 11mo ago
  so it's a misconfig on python.org side?
  - lolpython 11mo ago
    I think so? It caused the same issue when I cross posted to Lobste.rs
rhat1512 11mo ago
[flagged]

andrewmcwatters 11mo ago

I've noticed that instead of defining requires at the top of Lua files, if you can know your own program well enough, defining them right before the dependency is actually used makes large Lua programs much, much quicker. Generally speaking, startup times can be a sped up by a meaningful factor. I suspect this change in Python will dramatically improve the performance of such large programs as well.

embedding-shape 11mo ago
> I suspect this change in Python will dramatically improve the performance of such large programs as well. Makes packaging super fun too, where you need to hit every possible path so you don't miss anything imported in 1% of the execution paths :)
- snovv_crash 11mo ago
  Can't you do some kind of static analysis instead?
  - fithisux 11mo ago
    Yes. It is much more clear to be explicit though.
    - jacquesm 11mo ago
      Safer too.
  - Figs 11mo ago
    Depends if your code has horrors like this lurking in it: m = importlib.import_module(requests.get("http://localhost:8000/package_name").content.strip().decode("ASCII"))
    - falcor84 11mo ago
      If you want even better nightmares, you can make localhost:8000 forward to a container running claude code with --dangerously-skip-permissions which uses an unkindness of mcp servers to control that endpoint on the fly based (amongst other sources) on 4chan's /b/.

ayhanfuat 11mo ago

That was fast. It was sent to SC 20 days ago. (Not complaining. I am happy with the outcome).

zahlman 11mo ago
Quite a few PEPs are accepted this quickly, actually. But some others take literally years.

xenator 11mo ago

Does it conform Occam's razor rule to have something that can be easily done very similar way without changing language?

boothby 11mo ago
Having some limited experience with lazy imports, yes, but this eliminates a lot of gross boilerplate. It also has the effect of "blessing" the practice of lazy imports which can have a cultural impact; it also prevents a situation wherein multiple subtly incompatible approaches to lazy imports become individually popular.
contravariant 11mo ago
Not sure that's Occam's razor any more. Regardless lazy loading needs widespread use to be most effective so having a unified syntax and no extra dependencies makes a lot of sense.

jauntywundrkind 11mo ago

Source phase imports are stage 3 (recommended for implementation, no major updates expected) in JS. "Import source" tells the runtime to go get the code, but not yet run it. Similar ideas seemingly to what's going on here in python! https://github.com/tc39/proposal-source-phase-imports https://github.com/tc39/proposal-source-phase-imports

zahlman 11mo ago
This proposal defers even looking for the code. The LazyLoader already in the standard library would eagerly look for code, but just record a file path and not actually store any bytecode data, never mind deserializing or running it. The rationale described in the PEP is that some systems try to `import`, for example, across a network share, so even searching the filesystem is slow and there is a desire to defer that (and avoid it on runs where the corresponding code isn't executed).

johnfn 11mo ago

This will be huge at the place I work! I’m unfamiliar with the PEP process. How long until this makes it into a Python version?

joerick 11mo ago
It should land in 3.15, so October next year. https://peps.python.org/pep-0790/ https://peps.python.org/pep-0790/
zahlman 11mo ago
This one is scheduled to land in the next "minor" version, 3.15. Python has an annual release cadence; 3.14 came out recently and 3.15 is due next October. In general, most PEPs are authored targeting the "next minor version" at the time of proposal; but they may be intentionally deferred at the start, and sometimes the process can take multiple years anyway. There are also PEPs that don't involve any change to the Python language, standard library or interpreter. In particular, there are PEPs that exist simply to document existing practice (or changes thereto), PEPs that concern governance (the Python Software Foundation, the Steering Council etc.), and PEPs that cover related special interests, such as packaging standards (which in turn can range from technical details about how metadata is formatted, to changes in PyPI's API). https://peps.python.org/pep-0000/ https://peps.python.org/pep-0000/

curiousgal 11mo ago

Some folks at HRT[0] will probably be unhappy about that lol 0.https://www.hudsonrivertrading.com/hrtbeat/inside-hrts-python-fork/ https://www.hudsonrivertrading.com/hrtbeat/inside-hrts-pytho...

kccqzy 11mo ago
Why would they? The accepted PEP just has extra keywords to mark imports explicit. That's a source level find-and-replace. They already did most of the real work in their codebase, such as finding where the code depended on the side effect of importing a module.
seemaze 11mo ago
I'm confused, wouldn't HRT be happy about this? The article you linked specifically states "..we hope to propose a revised lazy imports PEP that introduces an explicit lazy keyword.." Is that not exactly what PEP 810 proposes?
CorrectHorseBat 11mo ago
Look at the names in the PEP [1], this PEP is written by them [1] https://peps.python.org/pep-0810/ https://peps.python.org/pep-0810/

manbitesdog 11mo ago

This is great for building modules. One can now lazy import all interesting names on __init__.py, so that instead of having to remember `from module.some_submodule_that_you_need_to_remember import method` you can just do `from module import name`.

maxbond 11mo ago
https://peps.python.org/pep-0810/#what-about-star-imports-from-module-import https://peps.python.org/pep-0810/#what-about-star-imports-fr... > What about star imports (`from module import *`)? > Wild card (star) imports cannot be lazy - they remain eager. This is because the set of names being imported cannot be determined without loading the module. Using the lazy keyword with star imports will be a syntax error. If lazy imports are globally enabled, star imports will still be eager. Additionally, star imports can interfere with type checkers and IDEs and shadowing caused by star imports is a frequent and difficult to diagnose source of bugs (you analyze the function you think you're calling and find no issues, but you're actually calling a different function because the star import occurs after your explicit import). You might be able to workaround this limitation by doing a lazy import into an intermediate module (a prelude) on a name by name basis and then star import that intermediate module. But personally I solve this problem using IDE features. https://github.com/python-lsp/python-lsp-server/blob/develop/docs/autoimport.md https://github.com/python-lsp/python-lsp-server/blob/develop...
- manbitesdog 11mo ago
  ? This has nothing to do with star imports
  - maxbond 11mo ago
    My bad for misreading you

alberth 11mo ago

Source link indicating PEP 810 was accepted: https://discuss.python.org/t/pep-810-explicit-lazy-imports/104131/466 https://discuss.python.org/t/pep-810-explicit-lazy-imports/1...

ant6n 11mo ago

Next we need lazy import *

swiftcoder 11mo ago
Is that not the purpose of the global switch outlined in the PEP?
- cgriswald 11mo ago
  No. From the PEP: Where <mode> can be: "normal" (or unset): Only explicitly marked lazy imports are lazy "all": All module-level imports (except in try blocks and import *) become potentially lazy "none": No imports are lazy, even those explicitly marked with lazy keyword When the global flag is set to "all", all imports at the global level of all modules are potentially lazy except for those inside a try block or any wild card (from ... import *) import.

nothrowaways 11mo ago

Python is quickly turning into a crowded keyword junkyard

riedel 11mo ago
It is a 'soft keyword' as the PEP explains. I would not think that this has any major impact on anyone who just chooses to ignore this feature. Assuming that you want this behavior, I wonder how this could have been done in a better fashion without now having 'lazy' in the specific context of an import statement.
- rrauenza 11mo ago
  soft keyword for anyone not familiar like I was ... "A new soft keyword lazy is added. A soft keyword is a context-sensitive keyword that only has special meaning in specific grammatical contexts; elsewhere it can be used as a regular identifier (e.g., as a variable name). The lazy keyword only has special meaning when it appears before import statements..."
notatallshaw 11mo ago
Python has about 40 keywords, I say I would regularly use about 30, and irregularly use about another 5. Hardly seems like a "junkyard". Further, this lack of first class support for lazy importing has spawned multiple CPython forks that implement their own lazy importing or a modified version of the prior rejected PEP 690. Reducing the real world need for forks seems worth the price of one keyword.
- lairv 11mo ago
  For those curious here are the actual keywords (from https://docs.python.org/3/reference/lexical_analysis.html?utm_source=chatgpt.com#keywords https://docs.python.org/3/reference/lexical_analysis.html?ut... ) Hard Keywords: False await else import pass None break except in raise True class finally is return and continue for lambda try as def from nonlocal while assert del global not with async elif if or yield Soft Keywords: match case _ type I think nonlocal/global are the only hard keywords I now barely use, for the soft ones I rarely use pattern matching, so 5 seems like a good estimate

divbzero 11mo ago

How will lazy imports interact with PEP 8 which recommends grouping imports in order: 1. Standard library imports. 2. Related third party imports. 3. Local application/library specific imports. https://peps.python.org/pep-0008/#imports https://peps.python.org/pep-0008/#imports Based on the examples in PEP 810, I suppose each group of regular imports can be followed by a group of lazy imports? import os import sys lazy import json import fastapi lazy import numpy import myapi import mymodels lazy import myutils ...

ta371jashgG 11mo ago

This has been rejected several times before. Now that Google and Microsoft fired core developers, Python Inc. is owned by Meta. Meta supports lazy imports in Cinder and needs it for its atrocious and bloated scientific ecosystem so that the bloat loads in less than 5 min. If Meta also starts firing Python developers, development might return to normal.

mmis1000 11mo ago

Given you can already trigger import in a local scope, it's merely a syntax sugar though.