18 ms·

How the Python import system works

497 points by zikohh 5y ago

deleted 5y ago
[deleted]
johncena33 5y ago
I've been programming in Python professionally for more than five years. I consider myself quite a good programmer. I still don't have good grasp on Python's import system. Does anyone else have similar experience?
- benrbray 5y ago
  When in doubt, put some more dots in front of your import statements. Or remove them? Maybe I need an extra __init__.py somewhere? Oh I'm importing from a subfolder, what do I need to do to get that work again? I can't remember.
  - captaincaveman 5y ago
    This
- abledon 5y ago
  same! I've always been shielded from it with Django's conventions. (the ecosystem I mainly work in). I used a lot of '.' and '..' imports but I think something changed in python3 that made that strategy a lot less forgiving... now I _really_ should read the entirety of this article!
  - BozeWolf 5y ago
    Always use absolute imports, do not use relative imports. Solves most problems. Also is recommended by pep8. Skip the relative imports in the article. No need to read it entirely anymore ;-)
- jlund-molfese 5y ago
  I’ve only been writing Python for about a year, but I’ve found it much harder to grasp how dependency resolution and imports work than other languages I’ve picked up (JVM, Node, Go, C).
- agumonkey 5y ago
  I thought not knowing that made me a noob.
- isatty 5y ago
  The average python programmer does not really need to deal with pythons import system that much (just be aware of how it does its module loading and that you can conditionally do stuff sometimes with __import__ etc). As someone who has messed around with it a lot (dynamically loading/unloading modules, modifying on the fly etc) I would NOT recommend doing that stuff for anything in production.
- fullstop 5y ago
  I sometimes fall into the trap of using pip to install dependencies and then things break after an os update. That is, my python version has changed from 3.8 to 3.9 and my dependencies are sitting in the wrong directory. I never know if I should use pip and requirements.txt or rely on Ubuntu's packaged versions.
  - geofft 5y ago
    I used to have a rule of, never use pip and only use Ubuntu's/Debian's packaged versions. That works pretty well if you're happy with the packaged versions and you don't need unpackaged libraries. I now have the rule of, only ever use pip inside a venv. If your venv is more than a little bit complex, write a requirements.txt file so you can generate it. So it's something like $ cat > requirements.txt << EOF tensorflow Django==3.2.5 cryptography EOF $ echo venv > .gitignore $ python3 -m venv venv $ venv/bin/pip install -r requirements.txt $ venv/bin/python or, if you prefer, $ . venv/bin/activate (venv)$ pip install -r requirements.txt (venv)$ python Then when your Python version changes, or you get confused about what's installed, or whatever, you can just blow away the entire venv and recreate it: $ rm -r venv $ python3 -m venv venv $ venv/bin/pip install -r requirements.txt and you're in a known-good place. Either of these rules works fine. The thing that works poorly is using pip install outside of a venv (with or without root).
    - trulyme 5y ago
      For me the rule is to always use pipenv locally and pip + requirements.txt (generated by pipenv) for production (in docker container usually). No complaints.
  - mixmastamyk 5y ago
    Just pip install again under new interpreter. Use distro versions if a script will primarily run there, and ok if an older version.
- skindoe 5y ago
  If you dig into importlib and try to write extensions for it then the underlying concepts of modules and packages will make a lot more sense. For a project I was working on where you could dynamically call distributed tasks (we we're using ecs) I added a subclass of module and package that dynamically created package structures and modules from a database call. So the new modules and packages would load from the database instead of from a python file and python the class would be dynamically generated to be something like albiet more advanced. Class MyDistributedTask: Def run(self, input, execution_engine=ecs): Run task So users of our package could import directly import their package_name.TaskName And run it directly as long as they imported our package which contained a custom module loader to our db to discover. Learned a ton about importing.
- brilee 5y ago
  I'm a python readability approver at Google and I don't understand how the import system works
  - trulyme 5y ago
    Interesting... If someone wants to know more about what a readability approver does: https://www.pullrequest.com/blog/google-code-review-readability-certification/ https://www.pullrequest.com/blog/google-code-review-readabil...
  - joshuamorton 5y ago
    To be fair, Google's python avoids ~99% of the complexity of Python's import system by making all imports absolute and doing most things through blaze/bazel.
- wpietri 5y ago
  Same. Occasionally I will get into some sort of mess, learn how it works under the hood enough to get myself out, and the promptly forget everything. And I think that's for the best. I'd much rather have a happy path that I stay on than use some sort of dark magic that nobody who comes after me will understand.
- jmholla 5y ago
  I've actually found the Python import documentation [0] to be really approachable. I'd give it a look if this article didn't clear things up. [0]: https://docs.python.org/3/reference/import.html https://docs.python.org/3/reference/import.html
- kortex 5y ago
  I have >10y python exp. Vanilla import system? I grasp it quite well. - Entrypoints? Getting there. - Namespace packages? Ehh. Murky - site-packages/mypackage.pth - I get it but I don't know why sometimes it appears and other times not - c extensions? Ehhh. - .so loading? Kinda magic. - the confluence of editable installs, namespace packages, foo.pth, PYTHONPATH, sys.path, relative imports, entry points, virtualenvs, LD_LIBRARY_PATH, PATH, shell initialization, setup.py vs pyproject.toml? Um yeah that's some heavy wizardry. Tbf, you don't need the vast majority of that to be effective in python.
MeteorMarc 5y ago
Python importing quirks can be time consuming. Things beginners will encounter: - three modules cannot depend on each other in a circular way - relative imports are fragile ("module not found") - the __all__ definitions in the __init__ file make modules available under different full names - how to reload a module in a jupyter notebook if edited and so on.
- chrisseaton 5y ago
  > three modules cannot depend on each other in a circular way What would the purpose of circular modules like this be? You may as well collapse into a single module and the situation would not be any different would it?
  - gchamonlive 5y ago
    I don't think purposeful circular dependency, but you can end up with circular imports after refactoring for instance. The common approach to solving this is pulling everything that is used by all the modules into leaf libraries, effectively creating a directed acyclic graph, but this is not obvious nor easy to do the first time.
  - jvolkman 5y ago
    In my experience they creep in over time as the system grows. Coupling between parts of the system that was previously unnecessary is added, and the cycles form.
  - BurningFrog 5y ago
    The purpose is you want to use code in other modules. If you keep doing that for a while, a circular dependency will happen. This is the dumbest thing thing in Python. All other languages I know have solved it.
    - chrisseaton 5y ago
      > The purpose is you want to use code in other modules. So put them in the same module? Circular modules don’t give you the benefits of modules, do they? Not an expert in modularity.
jgwil2 5y ago
Related: https://instagram-engineering.com/python-at-scale-strict-modules-c0bb9245c834 https://instagram-engineering.com/python-at-scale-strict-mod...
shockfist 5y ago
Badly
maest 5y ago
Slightly off-topic: One of the simplest import systems I've seen were in q/kdb (like with most things in that language, everything is as simple as possible) Imports work by simply calling `\l my_script.q` which is similar to simply taking the file `my_script.q` and running it line by line (iirc, it does so eagerly, so it reruns the entire file whenever you do `\l my_script.q`, even if the file has been loaded before, which may affect you state. By contrast, Python `import` statements are no-ops if the module has already been imported). The main disadvantage is that you risk your imported script overwriting your global variables. This is solved by following the strong (unenforced) conversion that scripts should only affect their own namespaces (which works by having the script write declare \d .my_namespace at the top of the script) I never found this system limiting and always appreciated its simplicity - whenever things go wrong debugging is fairly easy. What does Python gain by having a more sophisticated import system?
- wizzwizz4 5y ago
  Everything is an object, and it “just works” for most purposes. (Not nearly as many as I would like, but still; it has backwards-compatibility to consider, so I'll cut it some slack.) If you need to start installing your own user packages, you need `pip` and then `venv` and then things get ugly, but for the usual case where the sysadmin deals with all that (or you're on Windows), it works quite well.
- __float 5y ago
  Python imports aren't necessarily always importing Python source code -- they can be pyc (bytecode) files, or C API extensions, etc. These are slightly more complicated than "load the script at this path". There's probably a more detailed answer, in that historically, decisions were made that we're now stuck with. Python packages can and sometimes intentionally have import-time side effects, for example. They must be only run once, without relying on convention, or we break existing code.
- btilly 5y ago
  Python avoids having to rerun the import over and over again. Suppose that you are importing a larger project. Where your one import (say of your standard database interface) pulls in a group of modules (say one for each table in your database), all of which import a couple of base libraries (to make your database objects provide a common interface) that themselves pull in common Python modules (like psychopg2) which themselves do further imports of standard Python modules. The combinatorial explosion of ways to get to those standard Python modules would make the redundant work of parsing them into a significant time commitment without a caching scheme. From the point of view of a small script, this kind of design may seem like overkill. But in a large project, it is both reasonable and common.
roenxi 5y ago
But despite being terrible the Python import system is remarkably easy to get started in. And generally easy for beginners to work with (put all the code in the same folder or "pip install"). There are some lessons here that other languages would do well to learn. Trouble importing 3rd party libraries must be a kiss of death for beginner engagement.
- awestroke 5y ago
  I disagree. I find it hard to get started in python. There are loads of package managers, so I don't know which one to pick. There are multiple different rules for for how imports work with local files. The standard library is full of of functions that one should not use any more, and you need to know which is which. Defining a main entrypoint consists of checking a magic __NAME__ constant. You have internalised all these quirks, and know how to work with/around them. Beginners haven't.
  - pnt12 5y ago
    Counterpoints: Start with pip, the official package manager. You don't have to start with virtual envs: I didn't. I haven't ran into any function which I "should not use". You don't need an entry point, you can just write code in a file. Besides that, I don't see how much it differs from other languages with implicit entry points, where you need to match a certain name in your function. Imports: yes, those are annoying and confusing. I still struggle with those. Aditionally: package managers and virtual envs are a pain in the ass. Every year we're getting a new one which is supposed to solve the problems from the previous one, but doesn't, and the cycle goes on. The language should really solve this at the core instead of requiring community fixing, as it is a core part of any serious development.
    - kortex 5y ago
      Honestly I have had very few issues with poetry, and the ones I've had, it's because I'm trying to use plugins, which are alpha right now. Dependency resolution just works. Editable installs just work. Building just works. Before that, I only had problems with virtual envs once, and it was due to bad hygiene with system python libs and deps. Moral of the story: don't. Unless it's necessary to bootstrap virtualenv or compiled libs, don't system install python. Good ol' get-pip.py and virtualenv.
jvolkman 5y ago
My least favorite is that import ordering matters in some situations. Like if I just run "organize imports", all of a sudden dependency cycles pop up and everything is broken. Certainly a sign of things being misimported/misorganized, but stuff happens has systems grow fast. And solving these issues is always incredibly time consuming. Unless anyone knows of magical tools to help solve import issues?
- matheusmoreira 5y ago
  > Unless anyone knows of magical tools to help solve import issues? Topological sorting? Always wondered why programming languages can't do what package managers do.
  - sltkr 5y ago
    The breakage described can only occur if there are dependency cycles, so topological sorting can't fix it. If there are no cycles, then the order of imports doesn't matter. edit: Actually I'm not even sure what kind of error we're talking about here. If two modules import each other and they both need access to the other's contents upon initialization, there is no ordering that will work. And if at most one needs access to the other, it will always work, no matter in which order they are imported. So I don't really know what the OP was talking about.
- colonwqbang 5y ago
  Yes, it is unfortunate. Loading modules can have side effects as the loaded module is allowed to execute arbitrary code at load time. This is also a source of ordering issues. Maybe some think this is only a theoretical problem and doesn't happen with "well-written" libraries. Well, here is one example which bit me in the past: https://stackoverflow.com/a/4706614/767442 https://stackoverflow.com/a/4706614/767442
matheusmoreira 5y ago
It's certainly a lot better than Ruby's require which just executes code and alters global virtual machine state. Not too different from C's #include. My favorite is Javascript's. Modules are objects containing functions and data, require returns such objects. Simple, elegant. Completely reifies the behind-the-scenes complexity of Python's import, making it easily understandable.
- brundolf 5y ago
  Though that's not how JS's new module system works, which I would say is also elegant in its own ways, but juggling two different systems is less so
  - matheusmoreira 5y ago
    I have no idea why they added a module system to the Javascript language itself. The old one was so awesome. Not sure what advantages the new system offers.
- osmarks 5y ago
  Lua does the same thing as JS. It's basically "dofile" with some indirection and extra path resolution logic.
sillysaurusx 5y ago
Fun fact: you can overload the Python import system to work with other languages that you create. I use this for my Python-based Lisp: https://github.com/shawwn/pymen/blob/ml/importer.py https://github.com/shawwn/pymen/blob/ml/importer.py import foo checks for "foo.l", and if found, compiles it to foo.py on the fly and imports that instead. It's so cursed. I love it.
- toomuchtodo 5y ago
  A master of the dark arts I presume.
- th0ma5 5y ago
  JPype does this for importing Java classes.
- Fordec 5y ago
  Wait, what? > that you create As in existing language supplied eg. Perl, Java, etc., or literally anything? Like bootstrapping your own home made language from scratch?
  - sillysaurusx 5y ago
    Literally anything. 'Tis a homemade homegrown lisp, grown by Scott Bell for several years till I took it all for myself. Nom nom. It starts with reader.l: https://github.com/shawwn/pymen/blob/ml/reader.l https://github.com/shawwn/pymen/blob/ml/reader.l where the raw character stream is turned into a bunch of nested arrays. E.g. (+ 1 2) becomes ["+", 1, 2] Then it's punted over to compiler.l https://github.com/shawwn/pymen/blob/ml/compiler.l https://github.com/shawwn/pymen/blob/ml/compiler.l where it's passed through `expand`, which does a `macroexpand` followed by a `lower`. E.g. (do (do (do (print 'hi)))) becomes ["print", "\"hi\""] Then the final thing is thrown to the compile function, which spits out print("hi") -- the final valid Python code that gets passed into the standard python `exec` function. Works with all the standard python things, like async functions and `with` contexts. Been screwing around with it for a few years.
    - 5y ago
abhisuri97 5y ago
Despite using python for the past 4 years, it still takes me several tries to set up packages and imports correctly when I make them myself. Honestly, I wish that python had an import system similar to JS (where you can just say “I want this file” <insert path to file> and specify the exports yourself). For me, it just feels more intuitive and less “magic”-like when dealing with custom scripts you want to import.
- josephorjoe 5y ago
  I have seen way too many `ModuleNotFoundError`s. It is moderately infuriating when the two files are in the same directory and python can't find the module. Honestly that error is misnamed. It should be `ModuleImportRefusedError`. And the frustration caused by getting PyTest to work in a project is likely responsible for a large percentage of the untested python projects in the world...
  - mixmastamyk 5y ago
    I’ve not had either of these issues. Maybe a detail left out?
    - josephorjoe 5y ago
      the first occurs if you haven't created __init__.py. the second occurs if you want your test files outside of the package directory (e.g., project/src/foo_app/foo.py, project/test/test_foo.py)
GregJJ 5y ago
Python import system is by far the worst one I dealt with. Using Setup.py and regular or namespace packages, relative import, having complex sub packages and cross importing, running a script from somewhere inside one of your sub packages, and many more craps like these. Import system must be intuitive and easy to use!
- CapmCrackaWaka 5y ago
  Yeah it really tripped me up as a beginner. I think the hardest part to get used to was that the import syntax actually changes based on how, and from where, you are running your code. So depending on how you call your code, your imports might or might not work. This is ESPECIALLY painful when you are building a distribution. There is no syntax that works for all situations, which seems like it would be pretty important for an import system. I had to bookmark this tab, and still refer to it often. https://stackoverflow.com/questions/14132789/relative-imports-for-the-billionth-time https://stackoverflow.com/questions/14132789/relative-import...
- t-writescode 5y ago
  … have you tried Rails’ autoloading? Especially when running rake tasks?
- paulddraper 5y ago
  What import system does not have these problems?
simonw 5y ago
This is great: I'm learning so much reading this. It lead me to read the source of the Python "types" standard library module, which really does just create a bunch of different Python objects and then use type() to extract their types: https://github.com/python/cpython/blob/v3.9.6/Lib/types.py https://github.com/python/cpython/blob/v3.9.6/Lib/types.py Some examples from that file: async def _ag(): yield _ag = _ag() AsyncGeneratorType = type(_ag) class _C: def _m(self): pass MethodType = type(_C()._m) BuiltinFunctionType = type(len) BuiltinMethodType = type([].append) # Same as BuiltinFunctionType
captainmuon 5y ago
There are two things that are annoying about Python's import system. Number one is that relative imports are weird. My intuition about imports is good enough that I never bothered to learn all the rules explicitly, but sometimes something simple is just not possible and it bites me. I think the case is importing files relative to a script (and not running with python -m ...). Number two is, in order to do package management, you have to create a fake python installation and bend PYTHONPATH. Virtualenvs are the canonical way to do it, but to me it feels like a hack - the core language seems to wants all packages installed in /usr. So now I have all these virtualenvs lying around and they are detached from the scripts. Why couldn't the import system resolve versions, too? You could say `import foo >= 1.0` and it would download it to some global cache, and the import the correct versions from the cache.
- Denvercoder9 5y ago
  > the core language seems to wants all packages installed in /usr. There's also ~/.local/lib/python3/site-packages (or whatever your distribution made of that). Virtualenvs are only necessary if you want to isolate dependencies between environments. That's useful if you have projects with conflicting dependencies, because Python doesn't allow you to install multiple versions of the same package, for better or worse. However, if you've written some simple scripts that don't care much about the exact version of their dependencies, it's perfectly fine to install their dependencies glboally.
  - chriswarbo 5y ago
    Quick shout-out to nix-shell shebangs, which allow a script to specify exact versions of all dependencies, Python or otherwise, which will be cached into a temporary sandbox. https://nixos.org/manual/nix/stable/#use-as-a-interpreter https://nixos.org/manual/nix/stable/#use-as-a-interpreter I wrote a Python script yesterday which calls out to a couple of external commands (`mlr` and `xq`), with a shebang like this: #!/usr/bin/env nix-shell #!nix-shell -i python3 -p python3 -p miller -p yq
- kristaps 5y ago
  Yeah, saner relative imports might be one of the few things that I envy the JS ecosystem for.
tyingq 5y ago
There's a very cool (and succinct) blog post[1] showing how to abuse this in an interesting way where you can put the code of a module into a string and load it that way. Not something I'd use in production, but it's a very clear way to see how both "finding a module" and "loading a module" works under the covers. [1] https://cprohm.de/blog/python-packages-in-a-single-file/ https://cprohm.de/blog/python-packages-in-a-single-file/ Edit: As an aside, I much prefer the way Perl does things in this space. It's much easier to define multiple packages either within a single file or across different files, and much clearer about what's happening underneath.
hirowan 5y ago
I'm just grateful I'm not a core Python dev after reading this thread. I've never seen so much negativity concentrated in one place in quite some time, for a feature of a programming language which is fairly innocuous.
- deleted 5y ago
  [deleted]
- ASalazarMX 5y ago
  I love Python, but to be fair, the dependency/import system has not aged well, and the various projects trying to fix it are proof of that.
  - mixmastamyk 5y ago
    They're not trying to fix the import system.
- ledauphin 5y ago
  Python is the language everyone at HN loves to hate. One presumes it has something to do with the fact that it's Y Combinator's recommendation for most startups.
Bukhmanizer 5y ago
This is so needed. I feel like a lot of people know enough about python imports, but everyone has pains with it.
grawprog 5y ago
Years ago, I thought I'd try learning Python, I'd heard it was supposed to be easy, good for beginners and everything. I read one of those beginner Python type books and followed along with a roguelike tutorial. Everything was going pretty alright, until I started trying to split everything into different files and use imports. I ended up just giving up. I read programming in lua, and rewrote my entire project in lua and actually finished it. Some day I'd like to go back and maybe learn Python but I really didn't enjoy my experience with it. I even found C headers easier to figure out than Python imports.
- toxik 5y ago
  Strange, Python’s import is not very difficult to use.
  - grawprog 5y ago
    It was probably something I did. The original tutorial I followed had everything in one file and didn't get into anything about imports. I started splitting everything up arbitrarily and started tossing imports into the files that complained about missing dependencies and ended up getting overwhelmed because nothing worked. I'm sure if I'd taken the time to try and fix it I eventually could have and at this point i've had more experience with a bunch of different languages, so I'm sure it's not as bad as I remember. I imagine it's one of those cases where if i were to go back and laugh about how stupid I was, but ya know, those first impressions.
    - trulyme 5y ago
      No, your impression was right. Reading this blog post made me realize how little I know about the python import system (and I use python daily), and at the same time how little I want to learn it. It is completely unintuitive and probably one of the worst aspects of otherwise beautiful and useful language. Fortunately, sys.path hack works reliably - one can just add that one line and imports work as expected.
      - 5y ago
dheera 5y ago
I created this fun hack that taught me a LOT about the import system. Basically it allows you to import anything you want, even specify a version, and it will fetch it from PyPI live. Might be interesting to flesh this out in a way that's deployable. Basically instead of import tornado and hoping and praying that the user has read you README and pip installed the right version of tornado, you can do from magicimport import magicimport tornado = magicimport("tornado", version = "4.5") and you have exactly what you need, as long as there is an internet connection. https://github.com/dheera/magicimport.py https://github.com/dheera/magicimport.py
- kristjansson 5y ago
  Cool! But why not just throw the code a package with a version dependency on tornado?
  - dheera 5y ago
    Because then you'd still have to install the package. It's nice to have something that "just works" without having to install it. I like to call it Level 5 autonomous software -- software that can figure out how to run itself with zero complaints. I actually use this for a lot of personal non-production scripts, I can just clone the script on any system and just run it, it will figure itself out. Also, packages with version dependencies fuck up other packages with other version dependencies, unless you set up virtualenvs or dockers or condas for them, and those take additional steps. magicimport.py uses virtualenv behind the scenes, so when it imports tornado 4.5 because some script wants 4.5, it won't mess up the 6.0 that's already on your system. In some sense it just automates the virtualenv and pip install process on-the-fly, to give the effect that the script "just works".
random_upvoter 5y ago
One of the reasons I gradually fell out of love with Python. To get Python right you need to remember more protocol than Queen Victoria's master of tea. And it is truly protocol in the sense that there is always this arbitrariness hanging around it.
- danuker 5y ago
  What do you use now?
  - random_upvoter 5y ago
    Python, hehe. But I just don't like it that much anymore.
yongjik 5y ago
Probably half of the commenters here know this, but since we're here, this is my go-to boilerplate for starting a python script. (Probably won't work on Windows.) #!/bin/sh # Run the interpreter with -u so that stdout isn't buffered. "exec" "python3" "-u" "$0" "$@" import os import sys curdir = os.path.dirname(os.path.realpath(sys.argv[0])) # Add enough .. to point to the top-level project directory. sys.path.insert(0, '%s/../..' % curdir) Your main program starts here ...
- mdaniel 5y ago
  That's crafty; it reminds me of the suggested similar tactic with tclsh since tcl honors backslashes in comment strings(!): https://wiki.tcl-lang.org/page/exec+magic https://wiki.tcl-lang.org/page/exec+magic
- kbenson 5y ago
  There's no well known module to do most this for you? In perl, the recent canonical way is to use the FindBin module to find the current binary's running dir, and the the local::lib module to set the import path (or just use lib for older style library dirs). That always seemed cumbersome to me at 2-3 lines that weren't very clean looking. Also, say what you will about Perl and esoteric global variables, but it's kinda nice to be able to toggle buffered output on and off on the fly. Is there really no way to do this in python without re-executing the script like that?
  - icegreentea2 5y ago
    Ya... if you're trying to get the path of the script, you can use `__file__` special variable (instead of loading it from bash $0 and grabbing sys.argv[0]). For adding current directory to front of path, the sys.path.insert() call is a pretty sound way of doing it.
    - yongjik 5y ago
      Yeah, I think you're right - didn't know about __file__. (To clarify, using "$0" with bash is just standard method to invoke the same script with an interpreter - sys.argv[0] will work with or without bash exec part.)
moonbug 5y ago
"badly"
AdrianoKF 5y ago
One interesting use case of overwriting `builtins.__import__` I've encountered was the automatic hooking by ClearML [0] (experiment tracking, ...) into all sorts of common libraries like Matplotlib, Tensorflow, Pytorch, and friends. The implementation is surprisingly straightforward, once you've come to terms with the basic idea, see [1] and the rest of the `clearml.binding` package. [0]: https://clear.ml https://clear.ml [1]: https://github.com/allegroai/clearml/blob/master/clearml/binding/import_bind.py https://github.com/allegroai/clearml/blob/master/clearml/bin...
bredren 5y ago
Carl Meyer, a Python developer at Instagram, has ~recently discussed open source contributions to Python he has worked on. He specifically describes “strict modules” and efforts to improve the efficiency of Python imports at 16:30 of this episode of Django Chat: https://django-chat.simplecast.com/episodes/django-instagram-carl-meyer-lvSQm3aL https://django-chat.simplecast.com/episodes/django-instagram...
xiaodai 5y ago
I am so glad this post exists
r4victor 5y ago
Hi! I'm the author of this article. Thanks for posting it. I've been programming in Python for quite a while but didn't really understand how the import system works: what modules and packages are exactly; what relative imports are relative to; what's in sys.path and so on. My goal with this post was to answer all these questions. I welcome your feedback and questions. Thanks!

497 points by zikohh 5y ago

deleted 5y ago

[deleted]

I've been programming in Python professionally for more than five years. I consider myself quite a good programmer. I still don't have good grasp on Python's import system. Does anyone else have similar experience?

benrbray 5y ago
When in doubt, put some more dots in front of your import statements. Or remove them? Maybe I need an extra __init__.py somewhere? Oh I'm importing from a subfolder, what do I need to do to get that work again? I can't remember.
- captaincaveman 5y ago
  This
abledon 5y ago
same! I've always been shielded from it with Django's conventions. (the ecosystem I mainly work in). I used a lot of '.' and '..' imports but I think something changed in python3 that made that strategy a lot less forgiving... now I _really_ should read the entirety of this article!
- BozeWolf 5y ago
  Always use absolute imports, do not use relative imports. Solves most problems. Also is recommended by pep8. Skip the relative imports in the article. No need to read it entirely anymore ;-)
jlund-molfese 5y ago
I’ve only been writing Python for about a year, but I’ve found it much harder to grasp how dependency resolution and imports work than other languages I’ve picked up (JVM, Node, Go, C).
agumonkey 5y ago
I thought not knowing that made me a noob.
isatty 5y ago
The average python programmer does not really need to deal with pythons import system that much (just be aware of how it does its module loading and that you can conditionally do stuff sometimes with __import__ etc). As someone who has messed around with it a lot (dynamically loading/unloading modules, modifying on the fly etc) I would NOT recommend doing that stuff for anything in production.
fullstop 5y ago
I sometimes fall into the trap of using pip to install dependencies and then things break after an os update. That is, my python version has changed from 3.8 to 3.9 and my dependencies are sitting in the wrong directory. I never know if I should use pip and requirements.txt or rely on Ubuntu's packaged versions.
- geofft 5y ago
  I used to have a rule of, never use pip and only use Ubuntu's/Debian's packaged versions. That works pretty well if you're happy with the packaged versions and you don't need unpackaged libraries. I now have the rule of, only ever use pip inside a venv. If your venv is more than a little bit complex, write a requirements.txt file so you can generate it. So it's something like $ cat > requirements.txt << EOF tensorflow Django==3.2.5 cryptography EOF $ echo venv > .gitignore $ python3 -m venv venv $ venv/bin/pip install -r requirements.txt $ venv/bin/python or, if you prefer, $ . venv/bin/activate (venv)$ pip install -r requirements.txt (venv)$ python Then when your Python version changes, or you get confused about what's installed, or whatever, you can just blow away the entire venv and recreate it: $ rm -r venv $ python3 -m venv venv $ venv/bin/pip install -r requirements.txt and you're in a known-good place. Either of these rules works fine. The thing that works poorly is using pip install outside of a venv (with or without root).
  - trulyme 5y ago
    For me the rule is to always use pipenv locally and pip + requirements.txt (generated by pipenv) for production (in docker container usually). No complaints.
- mixmastamyk 5y ago
  Just pip install again under new interpreter. Use distro versions if a script will primarily run there, and ok if an older version.
skindoe 5y ago
If you dig into importlib and try to write extensions for it then the underlying concepts of modules and packages will make a lot more sense. For a project I was working on where you could dynamically call distributed tasks (we we're using ecs) I added a subclass of module and package that dynamically created package structures and modules from a database call. So the new modules and packages would load from the database instead of from a python file and python the class would be dynamically generated to be something like albiet more advanced. Class MyDistributedTask: Def run(self, input, execution_engine=ecs): Run task So users of our package could import directly import their package_name.TaskName And run it directly as long as they imported our package which contained a custom module loader to our db to discover. Learned a ton about importing.
brilee 5y ago
I'm a python readability approver at Google and I don't understand how the import system works
- trulyme 5y ago
  Interesting... If someone wants to know more about what a readability approver does: https://www.pullrequest.com/blog/google-code-review-readability-certification/ https://www.pullrequest.com/blog/google-code-review-readabil...
- joshuamorton 5y ago
  To be fair, Google's python avoids ~99% of the complexity of Python's import system by making all imports absolute and doing most things through blaze/bazel.
wpietri 5y ago
Same. Occasionally I will get into some sort of mess, learn how it works under the hood enough to get myself out, and the promptly forget everything. And I think that's for the best. I'd much rather have a happy path that I stay on than use some sort of dark magic that nobody who comes after me will understand.
jmholla 5y ago
I've actually found the Python import documentation [0] to be really approachable. I'd give it a look if this article didn't clear things up. [0]: https://docs.python.org/3/reference/import.html https://docs.python.org/3/reference/import.html
kortex 5y ago
I have >10y python exp. Vanilla import system? I grasp it quite well. - Entrypoints? Getting there. - Namespace packages? Ehh. Murky - site-packages/mypackage.pth - I get it but I don't know why sometimes it appears and other times not - c extensions? Ehhh. - .so loading? Kinda magic. - the confluence of editable installs, namespace packages, foo.pth, PYTHONPATH, sys.path, relative imports, entry points, virtualenvs, LD_LIBRARY_PATH, PATH, shell initialization, setup.py vs pyproject.toml? Um yeah that's some heavy wizardry. Tbf, you don't need the vast majority of that to be effective in python.

MeteorMarc 5y ago

Python importing quirks can be time consuming. Things beginners will encounter: - three modules cannot depend on each other in a circular way - relative imports are fragile ("module not found") - the __all__ definitions in the __init__ file make modules available under different full names - how to reload a module in a jupyter notebook if edited and so on.

chrisseaton 5y ago
> three modules cannot depend on each other in a circular way What would the purpose of circular modules like this be? You may as well collapse into a single module and the situation would not be any different would it?
- gchamonlive 5y ago
  I don't think purposeful circular dependency, but you can end up with circular imports after refactoring for instance. The common approach to solving this is pulling everything that is used by all the modules into leaf libraries, effectively creating a directed acyclic graph, but this is not obvious nor easy to do the first time.
- jvolkman 5y ago
  In my experience they creep in over time as the system grows. Coupling between parts of the system that was previously unnecessary is added, and the cycles form.
- BurningFrog 5y ago
  The purpose is you want to use code in other modules. If you keep doing that for a while, a circular dependency will happen. This is the dumbest thing thing in Python. All other languages I know have solved it.
  - chrisseaton 5y ago
    > The purpose is you want to use code in other modules. So put them in the same module? Circular modules don’t give you the benefits of modules, do they? Not an expert in modularity.

jgwil2 5y ago

Related: https://instagram-engineering.com/python-at-scale-strict-modules-c0bb9245c834 https://instagram-engineering.com/python-at-scale-strict-mod...

shockfist 5y ago

Badly

maest 5y ago

Slightly off-topic: One of the simplest import systems I've seen were in q/kdb (like with most things in that language, everything is as simple as possible) Imports work by simply calling `\l my_script.q` which is similar to simply taking the file `my_script.q` and running it line by line (iirc, it does so eagerly, so it reruns the entire file whenever you do `\l my_script.q`, even if the file has been loaded before, which may affect you state. By contrast, Python `import` statements are no-ops if the module has already been imported). The main disadvantage is that you risk your imported script overwriting your global variables. This is solved by following the strong (unenforced) conversion that scripts should only affect their own namespaces (which works by having the script write declare \d .my_namespace at the top of the script) I never found this system limiting and always appreciated its simplicity - whenever things go wrong debugging is fairly easy. What does Python gain by having a more sophisticated import system?

wizzwizz4 5y ago
Everything is an object, and it “just works” for most purposes. (Not nearly as many as I would like, but still; it has backwards-compatibility to consider, so I'll cut it some slack.) If you need to start installing your own user packages, you need `pip` and then `venv` and then things get ugly, but for the usual case where the sysadmin deals with all that (or you're on Windows), it works quite well.
__float 5y ago
Python imports aren't necessarily always importing Python source code -- they can be pyc (bytecode) files, or C API extensions, etc. These are slightly more complicated than "load the script at this path". There's probably a more detailed answer, in that historically, decisions were made that we're now stuck with. Python packages can and sometimes intentionally have import-time side effects, for example. They must be only run once, without relying on convention, or we break existing code.
btilly 5y ago
Python avoids having to rerun the import over and over again. Suppose that you are importing a larger project. Where your one import (say of your standard database interface) pulls in a group of modules (say one for each table in your database), all of which import a couple of base libraries (to make your database objects provide a common interface) that themselves pull in common Python modules (like psychopg2) which themselves do further imports of standard Python modules. The combinatorial explosion of ways to get to those standard Python modules would make the redundant work of parsing them into a significant time commitment without a caching scheme. From the point of view of a small script, this kind of design may seem like overkill. But in a large project, it is both reasonable and common.

roenxi 5y ago

But despite being terrible the Python import system is remarkably easy to get started in. And generally easy for beginners to work with (put all the code in the same folder or "pip install"). There are some lessons here that other languages would do well to learn. Trouble importing 3rd party libraries must be a kiss of death for beginner engagement.

awestroke 5y ago
I disagree. I find it hard to get started in python. There are loads of package managers, so I don't know which one to pick. There are multiple different rules for for how imports work with local files. The standard library is full of of functions that one should not use any more, and you need to know which is which. Defining a main entrypoint consists of checking a magic __NAME__ constant. You have internalised all these quirks, and know how to work with/around them. Beginners haven't.
- pnt12 5y ago
  Counterpoints: Start with pip, the official package manager. You don't have to start with virtual envs: I didn't. I haven't ran into any function which I "should not use". You don't need an entry point, you can just write code in a file. Besides that, I don't see how much it differs from other languages with implicit entry points, where you need to match a certain name in your function. Imports: yes, those are annoying and confusing. I still struggle with those. Aditionally: package managers and virtual envs are a pain in the ass. Every year we're getting a new one which is supposed to solve the problems from the previous one, but doesn't, and the cycle goes on. The language should really solve this at the core instead of requiring community fixing, as it is a core part of any serious development.
  - kortex 5y ago
    Honestly I have had very few issues with poetry, and the ones I've had, it's because I'm trying to use plugins, which are alpha right now. Dependency resolution just works. Editable installs just work. Building just works. Before that, I only had problems with virtual envs once, and it was due to bad hygiene with system python libs and deps. Moral of the story: don't. Unless it's necessary to bootstrap virtualenv or compiled libs, don't system install python. Good ol' get-pip.py and virtualenv.

jvolkman 5y ago

My least favorite is that import ordering matters in some situations. Like if I just run "organize imports", all of a sudden dependency cycles pop up and everything is broken. Certainly a sign of things being misimported/misorganized, but stuff happens has systems grow fast. And solving these issues is always incredibly time consuming. Unless anyone knows of magical tools to help solve import issues?

matheusmoreira 5y ago
> Unless anyone knows of magical tools to help solve import issues? Topological sorting? Always wondered why programming languages can't do what package managers do.
- sltkr 5y ago
  The breakage described can only occur if there are dependency cycles, so topological sorting can't fix it. If there are no cycles, then the order of imports doesn't matter. edit: Actually I'm not even sure what kind of error we're talking about here. If two modules import each other and they both need access to the other's contents upon initialization, there is no ordering that will work. And if at most one needs access to the other, it will always work, no matter in which order they are imported. So I don't really know what the OP was talking about.
colonwqbang 5y ago
Yes, it is unfortunate. Loading modules can have side effects as the loaded module is allowed to execute arbitrary code at load time. This is also a source of ordering issues. Maybe some think this is only a theoretical problem and doesn't happen with "well-written" libraries. Well, here is one example which bit me in the past: https://stackoverflow.com/a/4706614/767442 https://stackoverflow.com/a/4706614/767442

matheusmoreira 5y ago

It's certainly a lot better than Ruby's require which just executes code and alters global virtual machine state. Not too different from C's #include. My favorite is Javascript's. Modules are objects containing functions and data, require returns such objects. Simple, elegant. Completely reifies the behind-the-scenes complexity of Python's import, making it easily understandable.

brundolf 5y ago
Though that's not how JS's new module system works, which I would say is also elegant in its own ways, but juggling two different systems is less so
- matheusmoreira 5y ago
  I have no idea why they added a module system to the Javascript language itself. The old one was so awesome. Not sure what advantages the new system offers.
osmarks 5y ago
Lua does the same thing as JS. It's basically "dofile" with some indirection and extra path resolution logic.

sillysaurusx 5y ago

Fun fact: you can overload the Python import system to work with other languages that you create. I use this for my Python-based Lisp: https://github.com/shawwn/pymen/blob/ml/importer.py https://github.com/shawwn/pymen/blob/ml/importer.py import foo checks for "foo.l", and if found, compiles it to foo.py on the fly and imports that instead. It's so cursed. I love it.

toomuchtodo 5y ago
A master of the dark arts I presume.
th0ma5 5y ago
JPype does this for importing Java classes.
Fordec 5y ago
Wait, what? > that you create As in existing language supplied eg. Perl, Java, etc., or literally anything? Like bootstrapping your own home made language from scratch?
- sillysaurusx 5y ago
  Literally anything. 'Tis a homemade homegrown lisp, grown by Scott Bell for several years till I took it all for myself. Nom nom. It starts with reader.l: https://github.com/shawwn/pymen/blob/ml/reader.l https://github.com/shawwn/pymen/blob/ml/reader.l where the raw character stream is turned into a bunch of nested arrays. E.g. (+ 1 2) becomes ["+", 1, 2] Then it's punted over to compiler.l https://github.com/shawwn/pymen/blob/ml/compiler.l https://github.com/shawwn/pymen/blob/ml/compiler.l where it's passed through `expand`, which does a `macroexpand` followed by a `lower`. E.g. (do (do (do (print 'hi)))) becomes ["print", "\"hi\""] Then the final thing is thrown to the compile function, which spits out print("hi") -- the final valid Python code that gets passed into the standard python `exec` function. Works with all the standard python things, like async functions and `with` contexts. Been screwing around with it for a few years.
  - 5y ago

abhisuri97 5y ago

Despite using python for the past 4 years, it still takes me several tries to set up packages and imports correctly when I make them myself. Honestly, I wish that python had an import system similar to JS (where you can just say “I want this file” <insert path to file> and specify the exports yourself). For me, it just feels more intuitive and less “magic”-like when dealing with custom scripts you want to import.

josephorjoe 5y ago
I have seen way too many `ModuleNotFoundError`s. It is moderately infuriating when the two files are in the same directory and python can't find the module. Honestly that error is misnamed. It should be `ModuleImportRefusedError`. And the frustration caused by getting PyTest to work in a project is likely responsible for a large percentage of the untested python projects in the world...
- mixmastamyk 5y ago
  I’ve not had either of these issues. Maybe a detail left out?
  - josephorjoe 5y ago
    the first occurs if you haven't created __init__.py. the second occurs if you want your test files outside of the package directory (e.g., project/src/foo_app/foo.py, project/test/test_foo.py)

GregJJ 5y ago

Python import system is by far the worst one I dealt with. Using Setup.py and regular or namespace packages, relative import, having complex sub packages and cross importing, running a script from somewhere inside one of your sub packages, and many more craps like these. Import system must be intuitive and easy to use!

CapmCrackaWaka 5y ago
Yeah it really tripped me up as a beginner. I think the hardest part to get used to was that the import syntax actually changes based on how, and from where, you are running your code. So depending on how you call your code, your imports might or might not work. This is ESPECIALLY painful when you are building a distribution. There is no syntax that works for all situations, which seems like it would be pretty important for an import system. I had to bookmark this tab, and still refer to it often. https://stackoverflow.com/questions/14132789/relative-imports-for-the-billionth-time https://stackoverflow.com/questions/14132789/relative-import...
t-writescode 5y ago
… have you tried Rails’ autoloading? Especially when running rake tasks?
paulddraper 5y ago
What import system does not have these problems?

simonw 5y ago

This is great: I'm learning so much reading this. It lead me to read the source of the Python "types" standard library module, which really does just create a bunch of different Python objects and then use type() to extract their types: https://github.com/python/cpython/blob/v3.9.6/Lib/types.py https://github.com/python/cpython/blob/v3.9.6/Lib/types.py Some examples from that file: async def _ag(): yield _ag = _ag() AsyncGeneratorType = type(_ag) class _C: def _m(self): pass MethodType = type(_C()._m) BuiltinFunctionType = type(len) BuiltinMethodType = type([].append) # Same as BuiltinFunctionType

captainmuon 5y ago

There are two things that are annoying about Python's import system. Number one is that relative imports are weird. My intuition about imports is good enough that I never bothered to learn all the rules explicitly, but sometimes something simple is just not possible and it bites me. I think the case is importing files relative to a script (and not running with python -m ...). Number two is, in order to do package management, you have to create a fake python installation and bend PYTHONPATH. Virtualenvs are the canonical way to do it, but to me it feels like a hack - the core language seems to wants all packages installed in /usr. So now I have all these virtualenvs lying around and they are detached from the scripts. Why couldn't the import system resolve versions, too? You could say `import foo >= 1.0` and it would download it to some global cache, and the import the correct versions from the cache.

Denvercoder9 5y ago
> the core language seems to wants all packages installed in /usr. There's also ~/.local/lib/python3/site-packages (or whatever your distribution made of that). Virtualenvs are only necessary if you want to isolate dependencies between environments. That's useful if you have projects with conflicting dependencies, because Python doesn't allow you to install multiple versions of the same package, for better or worse. However, if you've written some simple scripts that don't care much about the exact version of their dependencies, it's perfectly fine to install their dependencies glboally.
- chriswarbo 5y ago
  Quick shout-out to nix-shell shebangs, which allow a script to specify exact versions of all dependencies, Python or otherwise, which will be cached into a temporary sandbox. https://nixos.org/manual/nix/stable/#use-as-a-interpreter https://nixos.org/manual/nix/stable/#use-as-a-interpreter I wrote a Python script yesterday which calls out to a couple of external commands (`mlr` and `xq`), with a shebang like this: #!/usr/bin/env nix-shell #!nix-shell -i python3 -p python3 -p miller -p yq
kristaps 5y ago
Yeah, saner relative imports might be one of the few things that I envy the JS ecosystem for.

tyingq 5y ago

There's a very cool (and succinct) blog post[1] showing how to abuse this in an interesting way where you can put the code of a module into a string and load it that way. Not something I'd use in production, but it's a very clear way to see how both "finding a module" and "loading a module" works under the covers. [1] https://cprohm.de/blog/python-packages-in-a-single-file/ https://cprohm.de/blog/python-packages-in-a-single-file/ Edit: As an aside, I much prefer the way Perl does things in this space. It's much easier to define multiple packages either within a single file or across different files, and much clearer about what's happening underneath.

hirowan 5y ago

I'm just grateful I'm not a core Python dev after reading this thread. I've never seen so much negativity concentrated in one place in quite some time, for a feature of a programming language which is fairly innocuous.

deleted 5y ago
[deleted]
ASalazarMX 5y ago
I love Python, but to be fair, the dependency/import system has not aged well, and the various projects trying to fix it are proof of that.
- mixmastamyk 5y ago
  They're not trying to fix the import system.
ledauphin 5y ago
Python is the language everyone at HN loves to hate. One presumes it has something to do with the fact that it's Y Combinator's recommendation for most startups.

Bukhmanizer 5y ago

This is so needed. I feel like a lot of people know enough about python imports, but everyone has pains with it.

grawprog 5y ago

Years ago, I thought I'd try learning Python, I'd heard it was supposed to be easy, good for beginners and everything. I read one of those beginner Python type books and followed along with a roguelike tutorial. Everything was going pretty alright, until I started trying to split everything into different files and use imports. I ended up just giving up. I read programming in lua, and rewrote my entire project in lua and actually finished it. Some day I'd like to go back and maybe learn Python but I really didn't enjoy my experience with it. I even found C headers easier to figure out than Python imports.

toxik 5y ago
Strange, Python’s import is not very difficult to use.
- grawprog 5y ago
  It was probably something I did. The original tutorial I followed had everything in one file and didn't get into anything about imports. I started splitting everything up arbitrarily and started tossing imports into the files that complained about missing dependencies and ended up getting overwhelmed because nothing worked. I'm sure if I'd taken the time to try and fix it I eventually could have and at this point i've had more experience with a bunch of different languages, so I'm sure it's not as bad as I remember. I imagine it's one of those cases where if i were to go back and laugh about how stupid I was, but ya know, those first impressions.
  - trulyme 5y ago
    No, your impression was right. Reading this blog post made me realize how little I know about the python import system (and I use python daily), and at the same time how little I want to learn it. It is completely unintuitive and probably one of the worst aspects of otherwise beautiful and useful language. Fortunately, sys.path hack works reliably - one can just add that one line and imports work as expected.
    - 5y ago

dheera 5y ago

I created this fun hack that taught me a LOT about the import system. Basically it allows you to import anything you want, even specify a version, and it will fetch it from PyPI live. Might be interesting to flesh this out in a way that's deployable. Basically instead of import tornado and hoping and praying that the user has read you README and pip installed the right version of tornado, you can do from magicimport import magicimport tornado = magicimport("tornado", version = "4.5") and you have exactly what you need, as long as there is an internet connection. https://github.com/dheera/magicimport.py https://github.com/dheera/magicimport.py

kristjansson 5y ago
Cool! But why not just throw the code a package with a version dependency on tornado?
- dheera 5y ago
  Because then you'd still have to install the package. It's nice to have something that "just works" without having to install it. I like to call it Level 5 autonomous software -- software that can figure out how to run itself with zero complaints. I actually use this for a lot of personal non-production scripts, I can just clone the script on any system and just run it, it will figure itself out. Also, packages with version dependencies fuck up other packages with other version dependencies, unless you set up virtualenvs or dockers or condas for them, and those take additional steps. magicimport.py uses virtualenv behind the scenes, so when it imports tornado 4.5 because some script wants 4.5, it won't mess up the 6.0 that's already on your system. In some sense it just automates the virtualenv and pip install process on-the-fly, to give the effect that the script "just works".

random_upvoter 5y ago

One of the reasons I gradually fell out of love with Python. To get Python right you need to remember more protocol than Queen Victoria's master of tea. And it is truly protocol in the sense that there is always this arbitrariness hanging around it.

danuker 5y ago
What do you use now?
- random_upvoter 5y ago
  Python, hehe. But I just don't like it that much anymore.

yongjik 5y ago

Probably half of the commenters here know this, but since we're here, this is my go-to boilerplate for starting a python script. (Probably won't work on Windows.) #!/bin/sh # Run the interpreter with -u so that stdout isn't buffered. "exec" "python3" "-u" "$0" "$@" import os import sys curdir = os.path.dirname(os.path.realpath(sys.argv[0])) # Add enough .. to point to the top-level project directory. sys.path.insert(0, '%s/../..' % curdir) Your main program starts here ...

mdaniel 5y ago
That's crafty; it reminds me of the suggested similar tactic with tclsh since tcl honors backslashes in comment strings(!): https://wiki.tcl-lang.org/page/exec+magic https://wiki.tcl-lang.org/page/exec+magic
kbenson 5y ago
There's no well known module to do most this for you? In perl, the recent canonical way is to use the FindBin module to find the current binary's running dir, and the the local::lib module to set the import path (or just use lib for older style library dirs). That always seemed cumbersome to me at 2-3 lines that weren't very clean looking. Also, say what you will about Perl and esoteric global variables, but it's kinda nice to be able to toggle buffered output on and off on the fly. Is there really no way to do this in python without re-executing the script like that?
- icegreentea2 5y ago
  Ya... if you're trying to get the path of the script, you can use `__file__` special variable (instead of loading it from bash $0 and grabbing sys.argv[0]). For adding current directory to front of path, the sys.path.insert() call is a pretty sound way of doing it.
  - yongjik 5y ago
    Yeah, I think you're right - didn't know about __file__. (To clarify, using "$0" with bash is just standard method to invoke the same script with an interpreter - sys.argv[0] will work with or without bash exec part.)

moonbug 5y ago

"badly"

AdrianoKF 5y ago

One interesting use case of overwriting `builtins.__import__` I've encountered was the automatic hooking by ClearML [0] (experiment tracking, ...) into all sorts of common libraries like Matplotlib, Tensorflow, Pytorch, and friends. The implementation is surprisingly straightforward, once you've come to terms with the basic idea, see [1] and the rest of the `clearml.binding` package. [0]: https://clear.ml https://clear.ml [1]: https://github.com/allegroai/clearml/blob/master/clearml/binding/import_bind.py https://github.com/allegroai/clearml/blob/master/clearml/bin...

bredren 5y ago

Carl Meyer, a Python developer at Instagram, has ~recently discussed open source contributions to Python he has worked on. He specifically describes “strict modules” and efforts to improve the efficiency of Python imports at 16:30 of this episode of Django Chat: https://django-chat.simplecast.com/episodes/django-instagram-carl-meyer-lvSQm3aL https://django-chat.simplecast.com/episodes/django-instagram...

xiaodai 5y ago

I am so glad this post exists

r4victor 5y ago

Hi! I'm the author of this article. Thanks for posting it. I've been programming in Python for quite a while but didn't really understand how the import system works: what modules and packages are exactly; what relative imports are relative to; what's in sys.path and so on. My goal with this post was to answer all these questions. I welcome your feedback and questions. Thanks!