7 ms·

The mechanism is called "import hooks": https://docs.python.org/3/reference/import.html#import-hooks https://docs.python.org/3/reference/import.html#import-hoo

The mechanism is called "import hooks": https://docs.python.org/3/reference/import.html#import-hooks https://docs.python.org/3/reference/import.html#import-hooks https://www.python.org/dev/peps/pep-0302/ https://www.python.org/dev/peps/pep-0302/ You can see the import hook added here: https://github.com/kragniz/json-sempai/blob/master/jsonsempai/magic.py#L5 https://github.com/kragniz/json-sempai/blob/master/jsonsempa... Import hooks are a great feature in Python. Take the `import` mechanism in the language and reduce it to its barest theoretical formulation - what does it really do? (Name binding.) Think of all the other features in your application that can be reduced to this. (e.g., configuration management) Use the constraints of the import mechanism to guide the design of this feature, and use import hooks to implement it. You'll end up with an implementation that is very "close" to the core language. I would assert that this closeness strongly suggests correctness and composability. Less philosophically, think of the Python virtual machine as a system with lots of "safety valves" and "escape hatches." Import hooks are one such safety valve. Think about all the things you could do easily by hooking into module importing? (Real use-cases: loading code from bitemporal object databases, deprecation of modules, relocation of modules, configuration management, &c.) Of course, there are languages that are much more flexible than Python in this respect. Python aims for a practical flexibility, and I find that, in practice, Python strikes a nice balance.

Grue3 11y ago
This seems like a huge case of "implicit over explicit". Ideally if you import package foo, it shouldn't affect anything but foo.bar
- icebraining 11y ago
  Python itself never enforced that, though. It's the libraries' responsibility to have a "register_import_hook()" function or similar instead of just doing so unannounced.
jamesdutc 11y ago
A simple example of how import hooks use for module relocation/deprecation: https://gist.github.com/dutc/0f7498451d98e3114268 https://gist.github.com/dutc/0f7498451d98e3114268
orf 11y ago
Import hooks are awesome, one of the cool things you can do is re-write the code as you load it. I made a toy loader that automagically inlines (some) Python function calls[1]. There are also cool projects like MacroPy[2] that do much more extensive things. 1. https://tomforb.es/automatically-inline-python-function-calls https://tomforb.es/automatically-inline-python-function-call... 2. https://github.com/lihaoyi/macropy https://github.com/lihaoyi/macropy
- finyeates 11y ago
  Awesome, I saw your post a while back and it triggered me to write a full inliner for python, also added some IFDEF macros using context managers e.g. with IFDEF("DEBUG"): I'll have to dig around and find my code, your inline project was a great help in understanding the import hooks and walking the ast.
- dr_zoidberg 11y ago
  I read your experiment on inlining and it looks quite interesting. Did you do a benchmark on how it affects (improves?) performance? In my line of work I've found ocasionally places where inlining would've helped (functions that do masking with 64 bit masks, for example, are a mess to "inline by hand" and kill a lot of readability and clarity). Even with the current limitations of your "toy" implementation, it seems like it would help to avoid the costly function calls.
- jimbokun 11y ago
  Sounds a bit like the Ruby-fication of Python.
inglor 11y ago
For what it's worth - modern JavaScript does it too, it separates the syntax (import/export) and semantics of binding the reference from the actual loading mechanism. Not only can you "hook" into imports, you can replace them altogether if you'd like.
bdarnell 11y ago
The "loader protocol" (PEP 302) is a related trick that can be used locally, so it only affects imports from packages that opt in. I use this in https://github.com/bdarnell/codegenloader https://github.com/bdarnell/codegenloader to automagically process protobuf/thrift files and import the generated code.

Grue3 11y ago

This seems like a huge case of "implicit over explicit". Ideally if you import package foo, it shouldn't affect anything but foo.bar

icebraining 11y ago
Python itself never enforced that, though. It's the libraries' responsibility to have a "register_import_hook()" function or similar instead of just doing so unannounced.

jamesdutc 11y ago

A simple example of how import hooks use for module relocation/deprecation: https://gist.github.com/dutc/0f7498451d98e3114268 https://gist.github.com/dutc/0f7498451d98e3114268

orf 11y ago

Import hooks are awesome, one of the cool things you can do is re-write the code as you load it. I made a toy loader that automagically inlines (some) Python function calls[1]. There are also cool projects like MacroPy[2] that do much more extensive things. 1. https://tomforb.es/automatically-inline-python-function-calls https://tomforb.es/automatically-inline-python-function-call... 2. https://github.com/lihaoyi/macropy https://github.com/lihaoyi/macropy

finyeates 11y ago
Awesome, I saw your post a while back and it triggered me to write a full inliner for python, also added some IFDEF macros using context managers e.g. with IFDEF("DEBUG"): I'll have to dig around and find my code, your inline project was a great help in understanding the import hooks and walking the ast.
dr_zoidberg 11y ago
I read your experiment on inlining and it looks quite interesting. Did you do a benchmark on how it affects (improves?) performance? In my line of work I've found ocasionally places where inlining would've helped (functions that do masking with 64 bit masks, for example, are a mess to "inline by hand" and kill a lot of readability and clarity). Even with the current limitations of your "toy" implementation, it seems like it would help to avoid the costly function calls.
jimbokun 11y ago
Sounds a bit like the Ruby-fication of Python.

inglor 11y ago

For what it's worth - modern JavaScript does it too, it separates the syntax (import/export) and semantics of binding the reference from the actual loading mechanism. Not only can you "hook" into imports, you can replace them altogether if you'd like.

bdarnell 11y ago

The "loader protocol" (PEP 302) is a related trick that can be used locally, so it only affects imports from packages that opt in. I use this in https://github.com/bdarnell/codegenloader https://github.com/bdarnell/codegenloader to automagically process protobuf/thrift files and import the generated code.