7 ms·

There are two things that are annoying about Python's import system. Number one is that relative imports are weird. My intuition about imports is good enough t

There are two things that are annoying about Python's import system. Number one is that relative imports are weird. My intuition about imports is good enough that I never bothered to learn all the rules explicitly, but sometimes something simple is just not possible and it bites me. I think the case is importing files relative to a script (and not running with python -m ...). Number two is, in order to do package management, you have to create a fake python installation and bend PYTHONPATH. Virtualenvs are the canonical way to do it, but to me it feels like a hack - the core language seems to wants all packages installed in /usr. So now I have all these virtualenvs lying around and they are detached from the scripts. Why couldn't the import system resolve versions, too? You could say `import foo >= 1.0` and it would download it to some global cache, and the import the correct versions from the cache.

Denvercoder9 5y ago
> the core language seems to wants all packages installed in /usr. There's also ~/.local/lib/python3/site-packages (or whatever your distribution made of that). Virtualenvs are only necessary if you want to isolate dependencies between environments. That's useful if you have projects with conflicting dependencies, because Python doesn't allow you to install multiple versions of the same package, for better or worse. However, if you've written some simple scripts that don't care much about the exact version of their dependencies, it's perfectly fine to install their dependencies glboally.
- chriswarbo 5y ago
  Quick shout-out to nix-shell shebangs, which allow a script to specify exact versions of all dependencies, Python or otherwise, which will be cached into a temporary sandbox. https://nixos.org/manual/nix/stable/#use-as-a-interpreter https://nixos.org/manual/nix/stable/#use-as-a-interpreter I wrote a Python script yesterday which calls out to a couple of external commands (`mlr` and `xq`), with a shebang like this: #!/usr/bin/env nix-shell #!nix-shell -i python3 -p python3 -p miller -p yq
kristaps 5y ago
Yeah, saner relative imports might be one of the few things that I envy the JS ecosystem for.
foresto 5y ago
> Number two is, in order to do package management, you have to create a fake python installation and bend PYTHONPATH. If I understand what you mean by package management in this context, I wonder if editable installs will help you. https://www.python.org/dev/peps/pep-0660/#abstract https://www.python.org/dev/peps/pep-0660/#abstract https://discuss.python.org/t/pronouncement-on-peps-660-and-662-editable-installs/9450 https://discuss.python.org/t/pronouncement-on-peps-660-and-6...
Xavdidtheshadow 5y ago
Once I wrapped my head around when you can and can't use relative imports, I've been pretty ok with them. The think that irks me is whether they work changes based on where you've invoked Python from. `./bin/my_script.py` behaves differently from `./my_script.py`. Coming from JS, that was a pretty frustrating realization.
- codethief 5y ago
  > The think that irks me is whether they work changes based on where you've invoked Python from. `./bin/my_script.py` behaves differently from `./my_script.py`. This. IMO that's also one of the main issues of Bash – you can't modularize a script unless you make sure your working directory is the directory containing the script. (And good luck with finding out the latter! [0]) [0]: https://stackoverflow.com/questions/4774054/reliable-way-for-a-bash-script-to-get-the-full-path-to-itself https://stackoverflow.com/questions/4774054/reliable-way-for...
  - deleted 5y ago
    [deleted]
- not_skynet 5y ago
  I actually spent time wrestling with exactly this earlier today, and I think I finally have a 2 line solution for it: https://pastebin.com/5WCXb2Gg https://pastebin.com/5WCXb2Gg Assuming `foo.bar` is a script inside the package `foo` that you want to both import and run directly (without `python -m ...`), this lets you do so without too much hassle.
- masklinn 5y ago
  > The think that irks me is whether they work changes based on where you've invoked Python from. `./bin/my_script.py` behaves differently from `./my_script.py`. It does not though, unless you have altered the default sys.path to always contains `.`. When running a file / script, Python will add that file's directory as the first lookup path, so these two invocations should have the exact same sys.path, and thus the same module resolution throughout[0]. If you have added `.` to sys.path (which is not the default), then the first invocation will also have the "grand-parent" folder on the sys.path, while the second won't. This also doesn't seem to have anything to do with relative imports, you need to already be inside a package for relative imports to do anything. [0] that is not the case of `-c` and `-m`, both add CWD to the path, and they differ in what they do: `-c` adds `''` to sys.path, which is the in-process CWD. `-m` stores the actual value of CWD at invocation, so changes to CWD while the program runs don't influence module resolution
robochat 5y ago
Relative imports used to work much more naturally IMHO in python2 but then they broke it in python3 because Guido wanted scripts and modules to always be separate codebases. So, whereas it used to be easy to have a module that could also be run as a script inside a package, this is now very difficult to implement. To the extent that any python2 code that does this, should probably be refactored when being ported to python3.
- rocqua 5y ago
  This decision has bit me in the ass so often. I want to organize my code logically in directories. As a script grows, I want the ability to spin out parts of that file to separate files. In order to do that in python, I need separate directories between the script and the spun-out functionality. This ends with a script that says "do function from module" and all code being in the module. Having code in different directories for no reason except "the import system" sucks. How is this supposed to go?
  - eesmith 5y ago
    FWIW, my command-line tools and any support code are all in a subdirectory under my package directory. I used to include a simple stub Python program (~10 lines) as a "script" in my setup.py that would import the right code and call it. Then I learned that "entry_points" implemented most of the dispatch behavior I wanted. I no longer have those scripts, just an entry that basically says "from abc.cli.prog1 import main; main()". The prog1.py, prog2.py, etc. look like normal command-line scripts, assuming the usual: if __name__ == "__main__": main() The major downside is I can't suppress SIGPIPE and SIGINT until main() is called, leaving a wider window where something like ^C gives a unwanted Python stack trace.
  - Doxin 5y ago
    So as your script grows, don't you eventually rename it to whatever/__main__.py and run it with -m? seems like a fairly trivial transform that then allows you to spin out as many modules as you need.
- masklinn 5y ago
blakesley 5y ago
Earnest question: why are you all trying to use relative imports? What problem is that solving for you? I've never even bothered to try it out because it seems potentially problematic in the way all relative references can be, e.g., relative file paths.
- scaryclam 5y ago
  So much this. There's no good reason to use relative imports. They're less readable, more dangerous and don't solve a single problem.
  - zarzavat 5y ago
    In JS land everything is a relative import. It's nice because you can move a whole directory of code from one project to another (or around in the same project) and it still works because all the imports pointing to other files inside that directory were relative, you only have to fix imports that go outside the directory. Also the way that JS imports are just relative paths is very nice because it means that the imports are statically determinable, your editor can understand them and fix them automatically and you can trust that refactoring. Python has turing complete imports because there's so much dynamic messing about with sys.path that goes on in Python due to inadequacies of the import system.
    - noptd 5y ago
      Given most editors handle absolute imports as good as if not better than relative imports, I don't see any real benefit to using the latter.
      - true_religion 5y ago
        It stops some circular imports if you use relative imports.
        np_tedious 5y ago
whoknowswhat11 5y ago
I never followed the changes around imports in 3 … but def still run into issues especially trying to move code from dev to deploy
bobbylarrybobby 5y ago
Always an informative (and fun) read: https://stackoverflow.com/questions/14132789/relative-imports-for-the-billionth-time https://stackoverflow.com/questions/14132789/relative-import...
- kdmccormick 5y ago
  Nice, do you know of anything similarly-comprehensive that is updated for Python 3? IIRC, Python 3 simplified things a good deal by removing "implicit" relative imports, but I'm a little foggy on exactly what that means.
  - Izkata 5y ago
    Explicit relative imports use a "." to indicate the file/package is from the current directory and not found elsewhere in sys.path: from . import foo from .foo import bar Implicit relative imports don't have such an indicator: from foo import bar In py2, that second one could be a relative import or from anywhere in sys.path, while in py3 those implicit relative imports were removed so that means it'll only only look in sys.path and not the local directory. https://stackoverflow.com/questions/48716943/what-is-python-implicit-relative-import https://stackoverflow.com/questions/48716943/what-is-python-...
  - masklinn 5y ago
    > IIRC, Python 3 simplified things a good deal by removing "implicit" relative imports, but I'm a little foggy on exactly what that means. Exactly that: in Python 3, if an import doesn’t start with a . it will always be absolute aka looked up from sys.path. In python 2, it would try performing a relative import (so importing a sibling module) first.
rcfox 5y ago
> Why couldn't the import system resolve versions, too? You could say `import foo >= 1.0` and it would download it to some global cache, and the import the correct versions from the cache. What do you do about conflicts? Or say you have `import foo >= 1.0` in a file, and `import foo == 2.4`, but the latest version is 2.5, so the first import grabbed the latest version, and you later realize you need 2.4? Imagine running a report generator for 5 hours, only to have the formatting module require a conflicting version of something and erroring out at run time...
heavyset_go 5y ago
> Number one is that relative imports are weird. My intuition about imports is good enough that I never bothered to learn all the rules explicitly, but sometimes something simple is just not possible and it bites me. I think the case is importing files relative to a script (and not running with python -m ...). I think the thing to remember about using relative imports is that it requires modules. Using relative imports to import something into a script will fail because the scripts don't belong to modules. > Number two is, in order to do package management, you have to create a fake python installation and bend PYTHONPATH. Virtualenvs are the canonical way to do it, but to me it feels like a hack - the core language seems to wants all packages installed in /usr. So now I have all these virtualenvs lying around and they are detached from the scripts. This is a pain. You can abstract this away with with pyenv or virtualenvwrapper, though.
Galanwe 5y ago
> Number two is, in order to do package management, you have to create a fake python installation and bend PYTHONPATH. Well whatever the language or runtime, you need to tell it how to find its dependencies. Be it with a configuration file, environment variables or parameters. Your statement is not exactly correct either. You don't need a fake installation _and_ bend PYTHONPATH. Virtualenv leverages the fact that "../lib/python<version>/site-packages/" relative to the interpreter is a default import path (default value of PYTHONHOME). It doesn't use PYTHONPATH. > Why couldn't the import system resolve versions, too? Not sure that would be really great. I prefer to have my dependencies all grouped together in a setup.py rather than scattered in various files at import time.

Denvercoder9 5y ago

> the core language seems to wants all packages installed in /usr. There's also ~/.local/lib/python3/site-packages (or whatever your distribution made of that). Virtualenvs are only necessary if you want to isolate dependencies between environments. That's useful if you have projects with conflicting dependencies, because Python doesn't allow you to install multiple versions of the same package, for better or worse. However, if you've written some simple scripts that don't care much about the exact version of their dependencies, it's perfectly fine to install their dependencies glboally.

chriswarbo 5y ago
Quick shout-out to nix-shell shebangs, which allow a script to specify exact versions of all dependencies, Python or otherwise, which will be cached into a temporary sandbox. https://nixos.org/manual/nix/stable/#use-as-a-interpreter https://nixos.org/manual/nix/stable/#use-as-a-interpreter I wrote a Python script yesterday which calls out to a couple of external commands (`mlr` and `xq`), with a shebang like this: #!/usr/bin/env nix-shell #!nix-shell -i python3 -p python3 -p miller -p yq

kristaps 5y ago

Yeah, saner relative imports might be one of the few things that I envy the JS ecosystem for.

foresto 5y ago

> Number two is, in order to do package management, you have to create a fake python installation and bend PYTHONPATH. If I understand what you mean by package management in this context, I wonder if editable installs will help you. https://www.python.org/dev/peps/pep-0660/#abstract https://www.python.org/dev/peps/pep-0660/#abstract https://discuss.python.org/t/pronouncement-on-peps-660-and-662-editable-installs/9450 https://discuss.python.org/t/pronouncement-on-peps-660-and-6...

Xavdidtheshadow 5y ago

Once I wrapped my head around when you can and can't use relative imports, I've been pretty ok with them. The think that irks me is whether they work changes based on where you've invoked Python from. `./bin/my_script.py` behaves differently from `./my_script.py`. Coming from JS, that was a pretty frustrating realization.

codethief 5y ago
> The think that irks me is whether they work changes based on where you've invoked Python from. `./bin/my_script.py` behaves differently from `./my_script.py`. This. IMO that's also one of the main issues of Bash – you can't modularize a script unless you make sure your working directory is the directory containing the script. (And good luck with finding out the latter! [0]) [0]: https://stackoverflow.com/questions/4774054/reliable-way-for-a-bash-script-to-get-the-full-path-to-itself https://stackoverflow.com/questions/4774054/reliable-way-for...
- deleted 5y ago
  [deleted]
not_skynet 5y ago
I actually spent time wrestling with exactly this earlier today, and I think I finally have a 2 line solution for it: https://pastebin.com/5WCXb2Gg https://pastebin.com/5WCXb2Gg Assuming `foo.bar` is a script inside the package `foo` that you want to both import and run directly (without `python -m ...`), this lets you do so without too much hassle.
masklinn 5y ago
> The think that irks me is whether they work changes based on where you've invoked Python from. `./bin/my_script.py` behaves differently from `./my_script.py`. It does not though, unless you have altered the default sys.path to always contains `.`. When running a file / script, Python will add that file's directory as the first lookup path, so these two invocations should have the exact same sys.path, and thus the same module resolution throughout[0]. If you have added `.` to sys.path (which is not the default), then the first invocation will also have the "grand-parent" folder on the sys.path, while the second won't. This also doesn't seem to have anything to do with relative imports, you need to already be inside a package for relative imports to do anything. [0] that is not the case of `-c` and `-m`, both add CWD to the path, and they differ in what they do: `-c` adds `''` to sys.path, which is the in-process CWD. `-m` stores the actual value of CWD at invocation, so changes to CWD while the program runs don't influence module resolution

robochat 5y ago

Relative imports used to work much more naturally IMHO in python2 but then they broke it in python3 because Guido wanted scripts and modules to always be separate codebases. So, whereas it used to be easy to have a module that could also be run as a script inside a package, this is now very difficult to implement. To the extent that any python2 code that does this, should probably be refactored when being ported to python3.

rocqua 5y ago
This decision has bit me in the ass so often. I want to organize my code logically in directories. As a script grows, I want the ability to spin out parts of that file to separate files. In order to do that in python, I need separate directories between the script and the spun-out functionality. This ends with a script that says "do function from module" and all code being in the module. Having code in different directories for no reason except "the import system" sucks. How is this supposed to go?
- eesmith 5y ago
  FWIW, my command-line tools and any support code are all in a subdirectory under my package directory. I used to include a simple stub Python program (~10 lines) as a "script" in my setup.py that would import the right code and call it. Then I learned that "entry_points" implemented most of the dispatch behavior I wanted. I no longer have those scripts, just an entry that basically says "from abc.cli.prog1 import main; main()". The prog1.py, prog2.py, etc. look like normal command-line scripts, assuming the usual: if __name__ == "__main__": main() The major downside is I can't suppress SIGPIPE and SIGINT until main() is called, leaving a wider window where something like ^C gives a unwanted Python stack trace.
- Doxin 5y ago
  So as your script grows, don't you eventually rename it to whatever/__main__.py and run it with -m? seems like a fairly trivial transform that then allows you to spin out as many modules as you need.
masklinn 5y ago

blakesley 5y ago

Earnest question: why are you all trying to use relative imports? What problem is that solving for you? I've never even bothered to try it out because it seems potentially problematic in the way all relative references can be, e.g., relative file paths.

scaryclam 5y ago
So much this. There's no good reason to use relative imports. They're less readable, more dangerous and don't solve a single problem.
- zarzavat 5y ago
  In JS land everything is a relative import. It's nice because you can move a whole directory of code from one project to another (or around in the same project) and it still works because all the imports pointing to other files inside that directory were relative, you only have to fix imports that go outside the directory. Also the way that JS imports are just relative paths is very nice because it means that the imports are statically determinable, your editor can understand them and fix them automatically and you can trust that refactoring. Python has turing complete imports because there's so much dynamic messing about with sys.path that goes on in Python due to inadequacies of the import system.
  - noptd 5y ago
    Given most editors handle absolute imports as good as if not better than relative imports, I don't see any real benefit to using the latter.
    - true_religion 5y ago
      It stops some circular imports if you use relative imports.
      - np_tedious 5y ago

whoknowswhat11 5y ago

I never followed the changes around imports in 3 … but def still run into issues especially trying to move code from dev to deploy

bobbylarrybobby 5y ago

Always an informative (and fun) read: https://stackoverflow.com/questions/14132789/relative-imports-for-the-billionth-time https://stackoverflow.com/questions/14132789/relative-import...

kdmccormick 5y ago
Nice, do you know of anything similarly-comprehensive that is updated for Python 3? IIRC, Python 3 simplified things a good deal by removing "implicit" relative imports, but I'm a little foggy on exactly what that means.
- Izkata 5y ago
  Explicit relative imports use a "." to indicate the file/package is from the current directory and not found elsewhere in sys.path: from . import foo from .foo import bar Implicit relative imports don't have such an indicator: from foo import bar In py2, that second one could be a relative import or from anywhere in sys.path, while in py3 those implicit relative imports were removed so that means it'll only only look in sys.path and not the local directory. https://stackoverflow.com/questions/48716943/what-is-python-implicit-relative-import https://stackoverflow.com/questions/48716943/what-is-python-...
- masklinn 5y ago
  > IIRC, Python 3 simplified things a good deal by removing "implicit" relative imports, but I'm a little foggy on exactly what that means. Exactly that: in Python 3, if an import doesn’t start with a . it will always be absolute aka looked up from sys.path. In python 2, it would try performing a relative import (so importing a sibling module) first.

rcfox 5y ago

> Why couldn't the import system resolve versions, too? You could say `import foo >= 1.0` and it would download it to some global cache, and the import the correct versions from the cache. What do you do about conflicts? Or say you have `import foo >= 1.0` in a file, and `import foo == 2.4`, but the latest version is 2.5, so the first import grabbed the latest version, and you later realize you need 2.4? Imagine running a report generator for 5 hours, only to have the formatting module require a conflicting version of something and erroring out at run time...

heavyset_go 5y ago

> Number one is that relative imports are weird. My intuition about imports is good enough that I never bothered to learn all the rules explicitly, but sometimes something simple is just not possible and it bites me. I think the case is importing files relative to a script (and not running with python -m ...). I think the thing to remember about using relative imports is that it requires modules. Using relative imports to import something into a script will fail because the scripts don't belong to modules. > Number two is, in order to do package management, you have to create a fake python installation and bend PYTHONPATH. Virtualenvs are the canonical way to do it, but to me it feels like a hack - the core language seems to wants all packages installed in /usr. So now I have all these virtualenvs lying around and they are detached from the scripts. This is a pain. You can abstract this away with with pyenv or virtualenvwrapper, though.

Galanwe 5y ago

> Number two is, in order to do package management, you have to create a fake python installation and bend PYTHONPATH. Well whatever the language or runtime, you need to tell it how to find its dependencies. Be it with a configuration file, environment variables or parameters. Your statement is not exactly correct either. You don't need a fake installation _and_ bend PYTHONPATH. Virtualenv leverages the fact that "../lib/python<version>/site-packages/" relative to the interpreter is a default import path (default value of PYTHONHOME). It doesn't use PYTHONPATH. > Why couldn't the import system resolve versions, too? Not sure that would be really great. I prefer to have my dependencies all grouped together in a setup.py rather than scattered in various files at import time.