4 ms·

I've been programming in Python professionally for more than five years. I consider myself quite a good programmer. I still don't have good grasp on Python's im

I've been programming in Python professionally for more than five years. I consider myself quite a good programmer. I still don't have good grasp on Python's import system. Does anyone else have similar experience?

benrbray 5y ago
When in doubt, put some more dots in front of your import statements. Or remove them? Maybe I need an extra __init__.py somewhere? Oh I'm importing from a subfolder, what do I need to do to get that work again? I can't remember.
- captaincaveman 5y ago
  This
abledon 5y ago
same! I've always been shielded from it with Django's conventions. (the ecosystem I mainly work in). I used a lot of '.' and '..' imports but I think something changed in python3 that made that strategy a lot less forgiving... now I _really_ should read the entirety of this article!
- BozeWolf 5y ago
  Always use absolute imports, do not use relative imports. Solves most problems. Also is recommended by pep8. Skip the relative imports in the article. No need to read it entirely anymore ;-)
jlund-molfese 5y ago
I’ve only been writing Python for about a year, but I’ve found it much harder to grasp how dependency resolution and imports work than other languages I’ve picked up (JVM, Node, Go, C).
agumonkey 5y ago
I thought not knowing that made me a noob.
isatty 5y ago
The average python programmer does not really need to deal with pythons import system that much (just be aware of how it does its module loading and that you can conditionally do stuff sometimes with __import__ etc). As someone who has messed around with it a lot (dynamically loading/unloading modules, modifying on the fly etc) I would NOT recommend doing that stuff for anything in production.
fullstop 5y ago
I sometimes fall into the trap of using pip to install dependencies and then things break after an os update. That is, my python version has changed from 3.8 to 3.9 and my dependencies are sitting in the wrong directory. I never know if I should use pip and requirements.txt or rely on Ubuntu's packaged versions.
- geofft 5y ago
  I used to have a rule of, never use pip and only use Ubuntu's/Debian's packaged versions. That works pretty well if you're happy with the packaged versions and you don't need unpackaged libraries. I now have the rule of, only ever use pip inside a venv. If your venv is more than a little bit complex, write a requirements.txt file so you can generate it. So it's something like $ cat > requirements.txt << EOF tensorflow Django==3.2.5 cryptography EOF $ echo venv > .gitignore $ python3 -m venv venv $ venv/bin/pip install -r requirements.txt $ venv/bin/python or, if you prefer, $ . venv/bin/activate (venv)$ pip install -r requirements.txt (venv)$ python Then when your Python version changes, or you get confused about what's installed, or whatever, you can just blow away the entire venv and recreate it: $ rm -r venv $ python3 -m venv venv $ venv/bin/pip install -r requirements.txt and you're in a known-good place. Either of these rules works fine. The thing that works poorly is using pip install outside of a venv (with or without root).
  - trulyme 5y ago
    For me the rule is to always use pipenv locally and pip + requirements.txt (generated by pipenv) for production (in docker container usually). No complaints.
- mixmastamyk 5y ago
  Just pip install again under new interpreter. Use distro versions if a script will primarily run there, and ok if an older version.
skindoe 5y ago
If you dig into importlib and try to write extensions for it then the underlying concepts of modules and packages will make a lot more sense. For a project I was working on where you could dynamically call distributed tasks (we we're using ecs) I added a subclass of module and package that dynamically created package structures and modules from a database call. So the new modules and packages would load from the database instead of from a python file and python the class would be dynamically generated to be something like albiet more advanced. Class MyDistributedTask: Def run(self, input, execution_engine=ecs): Run task So users of our package could import directly import their package_name.TaskName And run it directly as long as they imported our package which contained a custom module loader to our db to discover. Learned a ton about importing.
brilee 5y ago
I'm a python readability approver at Google and I don't understand how the import system works
- trulyme 5y ago
  Interesting... If someone wants to know more about what a readability approver does: https://www.pullrequest.com/blog/google-code-review-readability-certification/ https://www.pullrequest.com/blog/google-code-review-readabil...
- joshuamorton 5y ago
  To be fair, Google's python avoids ~99% of the complexity of Python's import system by making all imports absolute and doing most things through blaze/bazel.
wpietri 5y ago
Same. Occasionally I will get into some sort of mess, learn how it works under the hood enough to get myself out, and the promptly forget everything. And I think that's for the best. I'd much rather have a happy path that I stay on than use some sort of dark magic that nobody who comes after me will understand.
jmholla 5y ago
I've actually found the Python import documentation [0] to be really approachable. I'd give it a look if this article didn't clear things up. [0]: https://docs.python.org/3/reference/import.html https://docs.python.org/3/reference/import.html
kortex 5y ago
I have >10y python exp. Vanilla import system? I grasp it quite well. - Entrypoints? Getting there. - Namespace packages? Ehh. Murky - site-packages/mypackage.pth - I get it but I don't know why sometimes it appears and other times not - c extensions? Ehhh. - .so loading? Kinda magic. - the confluence of editable installs, namespace packages, foo.pth, PYTHONPATH, sys.path, relative imports, entry points, virtualenvs, LD_LIBRARY_PATH, PATH, shell initialization, setup.py vs pyproject.toml? Um yeah that's some heavy wizardry. Tbf, you don't need the vast majority of that to be effective in python.

benrbray 5y ago

When in doubt, put some more dots in front of your import statements. Or remove them? Maybe I need an extra __init__.py somewhere? Oh I'm importing from a subfolder, what do I need to do to get that work again? I can't remember.

captaincaveman 5y ago
This

abledon 5y ago

same! I've always been shielded from it with Django's conventions. (the ecosystem I mainly work in). I used a lot of '.' and '..' imports but I think something changed in python3 that made that strategy a lot less forgiving... now I _really_ should read the entirety of this article!

BozeWolf 5y ago
Always use absolute imports, do not use relative imports. Solves most problems. Also is recommended by pep8. Skip the relative imports in the article. No need to read it entirely anymore ;-)

jlund-molfese 5y ago

I’ve only been writing Python for about a year, but I’ve found it much harder to grasp how dependency resolution and imports work than other languages I’ve picked up (JVM, Node, Go, C).

agumonkey 5y ago

I thought not knowing that made me a noob.

isatty 5y ago

The average python programmer does not really need to deal with pythons import system that much (just be aware of how it does its module loading and that you can conditionally do stuff sometimes with __import__ etc). As someone who has messed around with it a lot (dynamically loading/unloading modules, modifying on the fly etc) I would NOT recommend doing that stuff for anything in production.

fullstop 5y ago

I sometimes fall into the trap of using pip to install dependencies and then things break after an os update. That is, my python version has changed from 3.8 to 3.9 and my dependencies are sitting in the wrong directory. I never know if I should use pip and requirements.txt or rely on Ubuntu's packaged versions.

geofft 5y ago
I used to have a rule of, never use pip and only use Ubuntu's/Debian's packaged versions. That works pretty well if you're happy with the packaged versions and you don't need unpackaged libraries. I now have the rule of, only ever use pip inside a venv. If your venv is more than a little bit complex, write a requirements.txt file so you can generate it. So it's something like $ cat > requirements.txt << EOF tensorflow Django==3.2.5 cryptography EOF $ echo venv > .gitignore $ python3 -m venv venv $ venv/bin/pip install -r requirements.txt $ venv/bin/python or, if you prefer, $ . venv/bin/activate (venv)$ pip install -r requirements.txt (venv)$ python Then when your Python version changes, or you get confused about what's installed, or whatever, you can just blow away the entire venv and recreate it: $ rm -r venv $ python3 -m venv venv $ venv/bin/pip install -r requirements.txt and you're in a known-good place. Either of these rules works fine. The thing that works poorly is using pip install outside of a venv (with or without root).
- trulyme 5y ago
  For me the rule is to always use pipenv locally and pip + requirements.txt (generated by pipenv) for production (in docker container usually). No complaints.
mixmastamyk 5y ago
Just pip install again under new interpreter. Use distro versions if a script will primarily run there, and ok if an older version.

skindoe 5y ago

If you dig into importlib and try to write extensions for it then the underlying concepts of modules and packages will make a lot more sense. For a project I was working on where you could dynamically call distributed tasks (we we're using ecs) I added a subclass of module and package that dynamically created package structures and modules from a database call. So the new modules and packages would load from the database instead of from a python file and python the class would be dynamically generated to be something like albiet more advanced. Class MyDistributedTask: Def run(self, input, execution_engine=ecs): Run task So users of our package could import directly import their package_name.TaskName And run it directly as long as they imported our package which contained a custom module loader to our db to discover. Learned a ton about importing.

brilee 5y ago

I'm a python readability approver at Google and I don't understand how the import system works

trulyme 5y ago
Interesting... If someone wants to know more about what a readability approver does: https://www.pullrequest.com/blog/google-code-review-readability-certification/ https://www.pullrequest.com/blog/google-code-review-readabil...
joshuamorton 5y ago
To be fair, Google's python avoids ~99% of the complexity of Python's import system by making all imports absolute and doing most things through blaze/bazel.

wpietri 5y ago

Same. Occasionally I will get into some sort of mess, learn how it works under the hood enough to get myself out, and the promptly forget everything. And I think that's for the best. I'd much rather have a happy path that I stay on than use some sort of dark magic that nobody who comes after me will understand.

jmholla 5y ago

I've actually found the Python import documentation [0] to be really approachable. I'd give it a look if this article didn't clear things up. [0]: https://docs.python.org/3/reference/import.html https://docs.python.org/3/reference/import.html

kortex 5y ago

I have >10y python exp. Vanilla import system? I grasp it quite well. - Entrypoints? Getting there. - Namespace packages? Ehh. Murky - site-packages/mypackage.pth - I get it but I don't know why sometimes it appears and other times not - c extensions? Ehhh. - .so loading? Kinda magic. - the confluence of editable installs, namespace packages, foo.pth, PYTHONPATH, sys.path, relative imports, entry points, virtualenvs, LD_LIBRARY_PATH, PATH, shell initialization, setup.py vs pyproject.toml? Um yeah that's some heavy wizardry. Tbf, you don't need the vast majority of that to be effective in python.