5 ms·

Love this. My https://llm.datasette.io/ https://llm.datasette.io/ CLI tool supports plugins, and people were complaining about really slow start times even for

by simonw 1y ago

Love this. My https://llm.datasette.io/ https://llm.datasette.io/ CLI tool supports plugins, and people were complaining about really slow start times even for commands like "llm --help" - it turned out there were popular plugins that did things like import pytorch at the base level, so the entire startup was blocked on heavy imports. I ended up adding a note to the plugin author docs suggesting lazy loading inside of functions - https://llm.datasette.io/en/stable/plugins/advanced-model-plugins.html#tip-lazily-load-expensive-dependencies https://llm.datasette.io/en/stable/plugins/advanced-model-pl... - but having a core Python language feature for this would be really nice.

zahlman 1y ago
You can implement this from your tool today: https://news.ycombinator.com/item?id=45467489 https://news.ycombinator.com/item?id=45467489 Note that this is global to the entire process, so for example if you make an import of Numpy lazy this way, then so are the imports of all the sub-modules. Meaning that large parts of Numpy might not be imported at all if they aren't needed, but pauses for importing individual modules might be distributed unpredictably across the runtime. Edit: from further experimentation, it appears that if the source does something like `import foo.bar.baz` then `foo` and `foo.bar` will still be eagerly loaded, and only `foo.bar.baz` itself is deferred. This might be part of what the PEP meant by "mostly". But it might also be possible to improve my implementation to fix that.
- dahart 1y ago
  Note the PEP does have a FAQ entry that mentions reasons they believe this proposed solution might be preferable to LazyLoader https://pep-previews--4622.org.readthedocs.build/pep-0810/#faq https://pep-previews--4622.org.readthedocs.build/pep-0810/#f... Q: Why not use importlib.util.LazyLoader instead? A: LazyLoader has significant limitations: Requires verbose setup code for each lazy import. Has ongoing performance overhead on every attribute access. Doesn’t work well with from ... import statements. Less clear and standard than dedicated syntax.
  - zahlman 1y ago
    Thanks for pointing it out! > Has ongoing performance overhead on every attribute access. I would have expected so, but in my testing it seems like the lazy load does some kind of magic to replace the proxy with the real thing. I haven't properly dug into it, though. It appears this point is removed in the live version (https://peps.python.org/pep-0810 https://peps.python.org/pep-0810). > Doesn’t work well with from ... import statements. Hmm. The PEP doesn't seem to explain how reification works in this case. Per the above it's a solved problem for modules; I guess for the from-imports it could be made to work essentially the same way. Presumably this involves the proxy holding a reference to the namespace where the import occurred. That probably has a lot to do with restricting the syntax to top level. (Which is the opposite of how we've seen soft keywords used before!) > Requires verbose setup code for each lazy import. > Less clear and standard than dedicated syntax. If you want to use it in a fine-grained way, then sure.
- wslh 1y ago
  Is it another potential solution (until PEP 810 is accepted) to override the NameError exception, decide if it was triggered by an unloaded package from a list, and then running again that line of code? I understand the inefficiency of this solution (e.g. the same line could trigger NameError several times and you need to run it again until all modules are loaded) but this is a good brainstorming thread.
  - zahlman 1y ago
    That sounds very unpleasant. However nicely you wrapped it up, you'd still be referring to the code for that process everywhere that the NameError could occur.
peterfirefly 1y ago
Parse the command line and do things like "--help" without doing the imports. Only do imports when you know you need them -- or as an easy approximation, only if the easy command line options have been handled and there's still something to do.
- mr_mitm 1y ago
  In the llm project, plugins can modify the command line arguments, so it's not that simple.
  - simonw 1y ago
    Yea, that's the core problem here: plugins can add new CLI subcommands, which means they all need to be loaded on startup. https://llm.datasette.io/en/stable/plugins/plugin-hooks.html#register-commands-cli https://llm.datasette.io/en/stable/plugins/plugin-hooks.html...
    - deleted 1y ago
      [deleted]
    - pests 1y ago
      Could you cache the help doc after first full loaded run and only regenerate when new plugins are added / updated?
      - simonw 1y ago
        It's not just help - the plugins need to be imported so the root level CLI tool knows what to do if you type "llm subcommand ..." where that subcommand is defined by a plugin.
        Inufu 1y ago
        In that case the CLI only needs to import the plugin that defined that sub command, not all plugins?
what 1y ago
[flagged]
Neywiny 1y ago
I think really the problem is that packages like pytorch take so long to import. In my work I've tried a few packages (not AI stuff) that do a lot of work on import. It's actually quite detrimental because I have to setup environment variables to pass things that should be arguments of a setup function in. All things considered a python module shouldn't take any noticeable time to import
zvr 1y ago
It's not only in the case of plugins. If a tool has different capabilities that use different imports, why load all of them if only a subset is required? As a simple example, a tool that can generate output in various formats (e.g., json, csv, xml, ...) should only import the appropriate modules to handle the output format after having determined which ones will be used in this invocations.

by simonw 1y ago

zahlman 1y ago

You can implement this from your tool today: https://news.ycombinator.com/item?id=45467489 https://news.ycombinator.com/item?id=45467489 Note that this is global to the entire process, so for example if you make an import of Numpy lazy this way, then so are the imports of all the sub-modules. Meaning that large parts of Numpy might not be imported at all if they aren't needed, but pauses for importing individual modules might be distributed unpredictably across the runtime. Edit: from further experimentation, it appears that if the source does something like `import foo.bar.baz` then `foo` and `foo.bar` will still be eagerly loaded, and only `foo.bar.baz` itself is deferred. This might be part of what the PEP meant by "mostly". But it might also be possible to improve my implementation to fix that.

dahart 1y ago
Note the PEP does have a FAQ entry that mentions reasons they believe this proposed solution might be preferable to LazyLoader https://pep-previews--4622.org.readthedocs.build/pep-0810/#faq https://pep-previews--4622.org.readthedocs.build/pep-0810/#f... Q: Why not use importlib.util.LazyLoader instead? A: LazyLoader has significant limitations: Requires verbose setup code for each lazy import. Has ongoing performance overhead on every attribute access. Doesn’t work well with from ... import statements. Less clear and standard than dedicated syntax.
- zahlman 1y ago
  Thanks for pointing it out! > Has ongoing performance overhead on every attribute access. I would have expected so, but in my testing it seems like the lazy load does some kind of magic to replace the proxy with the real thing. I haven't properly dug into it, though. It appears this point is removed in the live version (https://peps.python.org/pep-0810 https://peps.python.org/pep-0810). > Doesn’t work well with from ... import statements. Hmm. The PEP doesn't seem to explain how reification works in this case. Per the above it's a solved problem for modules; I guess for the from-imports it could be made to work essentially the same way. Presumably this involves the proxy holding a reference to the namespace where the import occurred. That probably has a lot to do with restricting the syntax to top level. (Which is the opposite of how we've seen soft keywords used before!) > Requires verbose setup code for each lazy import. > Less clear and standard than dedicated syntax. If you want to use it in a fine-grained way, then sure.
wslh 1y ago
Is it another potential solution (until PEP 810 is accepted) to override the NameError exception, decide if it was triggered by an unloaded package from a list, and then running again that line of code? I understand the inefficiency of this solution (e.g. the same line could trigger NameError several times and you need to run it again until all modules are loaded) but this is a good brainstorming thread.
- zahlman 1y ago
  That sounds very unpleasant. However nicely you wrapped it up, you'd still be referring to the code for that process everywhere that the NameError could occur.

peterfirefly 1y ago

Parse the command line and do things like "--help" without doing the imports. Only do imports when you know you need them -- or as an easy approximation, only if the easy command line options have been handled and there's still something to do.

mr_mitm 1y ago
In the llm project, plugins can modify the command line arguments, so it's not that simple.
- simonw 1y ago
  Yea, that's the core problem here: plugins can add new CLI subcommands, which means they all need to be loaded on startup. https://llm.datasette.io/en/stable/plugins/plugin-hooks.html#register-commands-cli https://llm.datasette.io/en/stable/plugins/plugin-hooks.html...
  - deleted 1y ago
    [deleted]
  - pests 1y ago
    Could you cache the help doc after first full loaded run and only regenerate when new plugins are added / updated?
    - simonw 1y ago
      It's not just help - the plugins need to be imported so the root level CLI tool knows what to do if you type "llm subcommand ..." where that subcommand is defined by a plugin.
      - Inufu 1y ago
        In that case the CLI only needs to import the plugin that defined that sub command, not all plugins?

what 1y ago

[flagged]

Neywiny 1y ago

I think really the problem is that packages like pytorch take so long to import. In my work I've tried a few packages (not AI stuff) that do a lot of work on import. It's actually quite detrimental because I have to setup environment variables to pass things that should be arguments of a setup function in. All things considered a python module shouldn't take any noticeable time to import

zvr 1y ago

It's not only in the case of plugins. If a tool has different capabilities that use different imports, why load all of them if only a subset is required? As a simple example, a tool that can generate output in various formats (e.g., json, csv, xml, ...) should only import the appropriate modules to handle the output format after having determined which ones will be used in this invocations.