9 ms·
HRT's Python fork: Leveraging PEP 690 for faster imports
- davidteather 1y agoThe author interviewed me and talked about this project, so it was cool seeing a blog post posted about it
- rirze 1y agoOh, it's a trading firm. That's why they can fund an internal fork of Python... That sounds nice...
- rasjani 1y agoI know few modules that can take seconds to import but would have been nice to hear how much they actually gained? Also maybe, if this approach could yield stats on if some import was needed or not ?
- fabioz 1y agoIt'd have been really nice to have that PEP in as it'd have helped me not have to write local imports everywhere. As it is, top-level imports IMHO are only meant to be used for modules required to be used in the startup, everything else should be a local import -- getting everyone convinced of that is the main issue though as it really goes against the regular coding of most Python modules (but the time saved to start up apps I work on does definitely make it worth it).
- theLiminator 1y agoYeah, imo that's the way that python should've worked in the first place. Import-time side effects are definitely nasty though and I wonder what the implications on all downstream code would be. Perhaps a lazy import keyword is a better way forward.
- zzzeek 1y agoGonna call this an antipattern. Do you need all those modules imported in every script ? Well then you save nothing on loadup time, the time will be spent regardless. Does every script not need those imports ? Well they shouldn't be importing those things and this small set of top level imports should be curated into a better, more fine grained list (and if you want to write tools, you can certainly identify these patterns using tooling similar to that which you wrote for LazyImports).
- sunshowers 1y agoThere are often large programs where not every invocation imports every module. The lazy import approach was pioneered in Mercurial I believe, where it cut down startup times by 3x.
- mdaniel 1y agoOr, here's an idea: don't write a CLI on the hot path of a developer's flow in a scripting language. No wonder it lost out
- sunshowers 1y agoI don't disagree, but I mean it was either that or C + shell back in the early 2000s, and C + shell is notorious for its non-portability across Unix and Windows—partly why Git on Windows requires an entire MSYS installation. Today, it would be a mistake to use anything other than Rust (hence Jujutsu carrying the flame forward).
- spicybright 1y agoFor personal one file utility scripts, I'll sometimes only import a module on a code path that needs it. And make it global if the scope gets in the way. It's dirty, but speeds things up vs putting all imports at the top.
- its-summertime 1y agoimport argparse parser = argparse.ArgumentParser() parser.parse_args() import requests Is an annoying bodge that a programmer should not have to think about, as a random example
- ecshafer 1y agoI thought HRT was a Cpp shop? Is Python used in their main business applications, or more for quants / data scientists?
- almostgotcaught 1y agoevery quant shop has QR and QT people that can barely write passable python let alone cpp - then the QD people have to integrate that stuff with prod cpp pipelines.
- mhh__ 1y agoalpha in being good at both (if nothing else you can keep more of the desks pnl...)
- arn3n 1y agoIn my experience it tends to be the opposite — I am not a quant (QD) but having worked with a few teams there’s a negative selection for technical expertise. QRs who are good at programming are usually pushed into maintaining infrastructure, datasets, or just tooling for less technical members of their team, who then get to use those tools to further their own alpha generation. Orgs incentivize the final step in making alpha — spend too much time helping others or building reusable research, and your coworkers steal the thunder. That, or stop helping your coworkers/accommodating them… risky, as a career move. Only seen that work once.
- roadside_picnic 1y agoInterviewed with HRT awhile back. While I didn't get past the final round, their Python internals interview (which I did pass) was an absolute blast to prepare for, and required a really deep dive into implementation specific details of CPython around things like exactly how collisions are handled in dict, details about memory management, etc. Pretty much had to spend a few weeks in the CPython source to prep, and was, for me, worth the interview just to really learn what's going on. For most teams I would be pretty skeptical of a internal Python fork, but the Python devs at HRT really know their stuff.
- htrp 1y agowhen milliseconds mean millions
- PufPufPuf 1y agoIf that was the case, why use Python in the first place?
- dmoy 1y agoWell there's gonna be people writing code who can't do it in say a high performance C/C++ setup. Not professional programmers, but professional <some finance discipline>. Sometimes it will be worth the tradeoff to put that person and a programmer together to code up a solution in another language. Sometimes it will be worth it to have the non-programmer write it in Python and then do Herculean things in the background to make it fast enough.
- Spivak 1y ago> This process gets dramatically slower for … modules on distributed file systems, modules with slow side-effects Oh no. Look I'm not saying you're holding it wrong, it's perfectly valid to host your modules on what is presumably NFS as well as having modules with side effects but what if you didn't. I've been down this road with NFS (and SMB if it matters) and pain is the only thing that awaits you. It seems like they're feeling it. Storing what is spiritually executable code on shared storage was a never ending source of bugs and mysterious performance issues.
- globular-toast 1y agoImagine if these guys put their intelligence towards improving the world.
- bitfilped 1y agoNot sure why you're throwing shade on people making a living. Go lobby to your representative if you think the financial market should be changed instead of belittling folks doing a job.
- patrick91 1y agoI really really want lazy imports in Python, it's would be a godsend for CLIs
- nomel 1y agoLibraries for this have always existed, triggering import on first access. The problem was, they would break linters. But that's not an issue anymore with typing.TYPE_CHECKING. A PEP is very much welcome, but using lazy import libraries is a fairly common, very old, method of speeding things up. My pre PEP 690 code looks like this: import typing from lazy import LazyImport member = LazyImport('my_module.subpackage', 'member') member1, member2, = LazyImport('my_module', 'member1', 'member2') if typing.TYPE_CHECKING: # normal import, for linter/IDE/navigation. from my_module.subpackage import member from my_module import member1, member2
- formerly_proven 1y agoWell if you use argparse or one of the many argparse wrappers for a moderately complex CLI you end up lazyfing the CLI parser itself because just fully populating the argparse data structures can easily take half a second or more, so with other startup costs you easily end up with "program --help" taking >1s and any CLI parsing error also taking >1s.
- nasretdinov 1y agoI wonder how much can be saved by using a local file system for imports though. In my testing just a mere presense of a home directory on NFS already dramatically slows down imports (by ~10x) due to Python searching for modules in home directory too by default.
- gjvc 1y agoto prevent this, set PYTHONNOUSERSITE=1 will prevent searching for modules in ~/.local/ (for convenience, try calling python through a wrapper in your project, say bin/run-python, and there you can set all the python-specific environment variables you need, set at the time of execution and not have to worry about setting them in the user's shell etc)
- nasretdinov 1y agoThanks, yeah I know that it works, what I meant is that it may be quite easy to compare the module import times with and without that env variable to see how much impact an NFS home directory has (and it's a lot), and possibly draw similar conclusions about the distributed file system behaviour in general too
- its-summertime 1y ago> There’s also no way to make imports of the form from module import * lazy I'd say if you see from typing import Final [...] __all__: Final = ("a", "b", "c") Its probably 99% safe to pull that from a quick run over of the AST (and caching that for the later import if you want to be fancy) Of course, should one be doing a star import in a proper codebase?
- instig007 1y agoIf only compiled languages with dead code elimination existed...
- tracnar 1y agoWhile I see the usefulness of lazy imports, it always seemed a bit backward to me for the importer to ask for lazy import, especially if you make it an import keyword rather than a Python flag. Instead I'd expect the modules to declare (and maybe enforce) that they don't have side effects, that way you know they can be lazily imported, and it opens the door for more optimizations, like declaring the module immutable. That links to the performance barrier of Python due to its dynamic nature as discussed in https://news.ycombinator.com/item?id=44809387 https://news.ycombinator.com/item?id=44809387 Of course that doesn't solve the overhead of finding the modules, but that could be optimized without lazy import, for example by having a way to pre-compute the module locations at install time.
- instig007 1y ago> it always seemed a bit backward to me for the importer to ask for lazy import, especially if you make it an import keyword rather than a Python flag Exactly this. There must be zero side effects at module import time, not just for load times, but because the order of such effects is 1) undefined, 2) heavily dependent on a import protocol implementation, and 3) poses safety and security nightmares that Python devs don't seem to care much about until bad things happen at the most inconvenient time possible. > Of course that doesn't solve the overhead of finding the modules, but that could be optimized without lazy import, for example by having a way to pre-compute the module locations at install time. 1) opt for https://docs.python.org/3/reference/import.html#replacing-the-standard-import-system https://docs.python.org/3/reference/import.html#replacing-th... 2) pre-compute everything in CI by using a solution from (1) and doing universal toplevel import of the entire Python monorepo (safe, given no side effects). 3) This step can be used to scan all toplevel definitions too, to gather extra code meta useful for various dynamic dispatch at runtime without complex lookups. See for example: https://docs.pylonsproject.org/projects/venusian/en/latest/index.html https://docs.pylonsproject.org/projects/venusian/en/latest/i... 3) put the result of (2) and (3) as a machine-readable dump, read by (1) as the alternative optimised loading branch. 4) deploy (3) together with your program.
- tracnar 1y agoFor optimizing the module finding, using a custom import hook was indeed what I had in mind!
- procaryote 1y ago> python > monorepo > vast proliferation of imports > large modules > distributed file system > side-effects > many transitive imports This sounds like a very optional problem to have.
- mgaunard 1y agoTrading companies are really disciplined about their tech stack.
- skeledrew 1y agoHmm it strikes me that is they really wanted to go this lazy route, they could've implemented an import hook, instead of creating and maintaining an entire fork.
- deleted 1y ago[deleted]
- gkze 1y agoYeah I wonder what led them down the drastic fork trajectory rather than considering this approach… kind of interesting this wasn’t even acknowledged in the article
- rsyring 1y ago> we support the Steering Council in their rejection of PEP 690—the implicit lazy imports are not a good fit for upstream due to the same, subtle bugs we encountered during our migration. However, as time permits, we hope to propose a revised lazy imports PEP that introduces an explicit lazy keyword, e.g. lazy import foo or lazy from foo import bar. This approach will satisfy migration and compatibility concerns, allow users to opt-in gradually, and enable all Python users to reap the speed benefits of lazy imports in a safe way.
- throwaway81523 1y agoWe need something like the ancient unexec from Emacs to dump out Python images. More generally, we need something like that for generic checkpointing, maybe based on CRIU.