7 ms·

Python's import system would be my biggest complaint about the language. I've used Python for more than ten years, but there are so many pitfalls with regards t

Python's import system would be my biggest complaint about the language. I've used Python for more than ten years, but there are so many pitfalls with regards to imports that I still have to look things up on a regular basis. What makes things even worse is that packages sometimes even meddle with how imports work. (See e.g. Tensorflow – I recently had to debug the import magic they're doing. It was a nightmare.) Don't get me wrong, I LOVE Python! It's my go-to language for both small and big tasks. But sometimes I do wish Python had a function like PHP's require()… (Obviously for package-local imports, not for importing other packages.)

kokooo 6y ago
I've seen this complaint for almost every language, and the few languages i don't see this complaint for, people end up complaining about something else. It's too late to change how imports work and it's not especially complicated so no use complaining.
- Trasmatta 6y ago
  I pretty much never have problems with imports in Ruby. It's not like it's an unsolvable problem. Nothing would ever get better if we just took the "no use complaining" approach.
  - BiteCode_dev 6y ago
    What's the differences between python and ruby import systems ?
    - masklinn 6y ago
      That they never had a problem with one. I’ve never had a problem with python’s import system either.
- pabs3 6y ago
  It is never too late to change a language's default import paths. Perl removed . from @INC a couple of years ago.
staticautomatic 6y ago
I recently spent an entire day trying to import AllenNLP and its dependencies. I gave up after trying multiple interpreters, virtual environments, and operating systems.
- chishaku 6y ago
  Before Docker, I remember yak shaving like this with lxml, numpy, scipy and other libraries. Now that I do everything in a container, life is much improved. Try Docker as documented in the AllenNLP README: $ mkdir -p $HOME/.allennlp/ $ docker run --rm -v $HOME/.allennlp:/root/.allennlp allennlp/allennlp:latest
  - hinkley 6y ago
    That was probably the first things that attracted me to Docker. If I started a new job or new project, it was the same story every time. “Do these five things and then you should be able to run the code” after half a day of back and forth I’d have a list of ten things, and the next new person would discover it is actually 12. Docker doesn’t just change your experience, it forces the author to actually capture most of their prerequisites. To the point where even if you don’t use docker, you benefit from a project having at least tried to use it.
- jhayward 6y ago
  Interesting, I just went to the repo's README and followed the directions for conda and had it installed and running in 3 commands.
  - staticautomatic 6y ago
    Frankly I was just trying to get a pre-trained ELMO model deployed via Cortex. As best I could tell the problem boiled down to a failure related to installing tensorflow with gpu disabled. I am a NN noob but still, it just shouldn’t be this hard.
    - FridgeSeal 6y ago
nonbirithm 6y ago
Lua uses a "require()" function instead of an import statement for loading modules. I think that this is a crucial difference from a lot of programming languages that is vastly understated. What this means is that you can completely replace "require()" with your own function that has complete control over how the module importing system works. Because of this it is possible to build your own runtime code hotloader entirely in Lua without needing to modify the source of the compiler, and thus Lua is the only "mainstream" programming language listed in the list of notable live coding environments on Wikipedia[0]. (I'm surprised that Clojure isn't listed there, though.) The only way this can be accomplished is by designing the import system from the start to use a function instead of a statement. It's too late for Python to redo things and change its import system, unfortunately. It really is the one language feature you have to get right the first time or you'll miss out on those kinds of enhancements forever. [0] https://en.m.wikipedia.org/wiki/Live_coding#Notable_live_coding_environments https://en.m.wikipedia.org/wiki/Live_coding#Notable_live_cod...
- sakex 6y ago
  Nodejs has require too
- yzmtf2008 6y ago
  Untrue, you could literally replace `__import__` in python and achieve the same thing: https://docs.python.org/3.8/library/functions.html#__import__ https://docs.python.org/3.8/library/functions.html#__import_...
  - Izkata 6y ago
    Unless something has changed or there's a detail involving builtins I'm not aware of, I don't think this would work, for the same reason as the most common pitfall when mocking functions in python: You have to replace it where it's used, not where it's defined. To do that on a module, you have to first import that module - and then all the imports at the top of it would've already run.
    - yzmtf2008 6y ago
yongjik 6y ago
Well, to be fair, Tensorflow is a Google library - pretty much any Google-made library will have five levels of indirection before it does anything, regardless of language. It's in their DNA.
- hinkley 6y ago
  Maybe I should convince a couple coworkers to apply to Google. They’d fit right in.
  - Lammy 6y ago
    The different styles of programmers need each other! Checks and balances :)
    - kristopolous 6y ago
      Once a company gets sufficiently large it's indistinguishable from a cult.
- kinkrtyavimoodh 6y ago
  Is there actually any veracity to this claim or is it general anti-google shit-flinging? I ask because from what I understand from Google's monorepo, it should be easy for engineers to take direct dependencies on whatever they want. Of course TF is an exported out library so things might be different, but surely it won't then be a good example of Google's DNA?
  - galkk 6y ago
    > I ask because from what I understand from Google's monorepo, it should be easy for engineers to take direct dependencies on whatever they want. Not correct. The dependency system supports visibility restrictions - you can take direct dependencies to anything only if you're in your own experimental tree. Otherwise package authors can control access. You can see that even in the Bazel: https://docs.bazel.build/versions/master/visibility.html https://docs.bazel.build/versions/master/visibility.html (google employee)
bobbyi_settv 6y ago
Does anyone know of a write-up that describes Tensorflow's import magic? I'd be curious to read about what they're doing and why.
- codethief 6y ago
  I wish I had a write-up but here's a starting point at least: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/api_template.__init__.py https://github.com/tensorflow/tensorflow/blob/master/tensorf... Inside the source code files, they then use a decorator (@tf_export) to define where in the package/module hierarchy a function/class should get placed, see e.g. https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ops/summary_ops_v2.py https://github.com/tensorflow/tensorflow/blob/master/tensorf... For instance, the write() function in that module actually gets exposed as `tf.summary.write()` even though it's contained in the file tensorflow/python/ops/summary_ops_v2.py. Needless to say, this even brings auto-completion of an IDE like IntelliJ to its knees at times. To make things even worse, things like Tensorboard also make use of global state everywhere, see e.g. the function _should_record_summaries_internal() in the above file. But now the question is: Which module's global state does the function actually access if it gets "moved" to a different module by TF? At the end of the day, I gave up hunting down that bug (_should_record_summaries_internal() always returned False, no matter what) and resorted to mocking the function out whenever I used Tensorboard. To do that (i.e. to find its parent module in the first place), I in turn used the `inspect` package. This is what I meant by "import magic". :)
  - shp0ngle 6y ago
    this hurts.
lmuench 6y ago
Python’s import system is weird but it does make sense when you use absolute imports: project ├── main.py ├── package1 │ ├── module1.py │ └── module2.py └── package2 ├── __init__.py ├── module3.py ├── module4.py └── subpackage1 └── module5.py As long as the file containing the main() function resides in the top-level folder you can use absolute imports in every file, e.g. from package1 import module1 from package1.module2 import function1 from package2.subpackage1.module5 import function2
Galanwe 6y ago
Agree, python import system is annoyingly configurable, and way too much libraries use these features to do magic. I regularly have to spend some time on pandas source code just to untangle the mess of import aliases magic they do. But the standard library is no exception, importing os will dynamically import os.path.
x86_64Ubuntu 6y ago
I ran into that issue a few weeks ago. I had a small project that was supposed to migrate rows from a MySQL server into a SQL Server instance. For some reason that I was too exhausted to figure out, the program would only work if I imported the mysql connector before the pydobc stuff. That was very frustrating to say the least.

kokooo 6y ago

I've seen this complaint for almost every language, and the few languages i don't see this complaint for, people end up complaining about something else. It's too late to change how imports work and it's not especially complicated so no use complaining.

Trasmatta 6y ago
I pretty much never have problems with imports in Ruby. It's not like it's an unsolvable problem. Nothing would ever get better if we just took the "no use complaining" approach.
- BiteCode_dev 6y ago
  What's the differences between python and ruby import systems ?
  - masklinn 6y ago
    That they never had a problem with one. I’ve never had a problem with python’s import system either.
pabs3 6y ago
It is never too late to change a language's default import paths. Perl removed . from @INC a couple of years ago.

staticautomatic 6y ago

I recently spent an entire day trying to import AllenNLP and its dependencies. I gave up after trying multiple interpreters, virtual environments, and operating systems.

chishaku 6y ago
Before Docker, I remember yak shaving like this with lxml, numpy, scipy and other libraries. Now that I do everything in a container, life is much improved. Try Docker as documented in the AllenNLP README: $ mkdir -p $HOME/.allennlp/ $ docker run --rm -v $HOME/.allennlp:/root/.allennlp allennlp/allennlp:latest
- hinkley 6y ago
  That was probably the first things that attracted me to Docker. If I started a new job or new project, it was the same story every time. “Do these five things and then you should be able to run the code” after half a day of back and forth I’d have a list of ten things, and the next new person would discover it is actually 12. Docker doesn’t just change your experience, it forces the author to actually capture most of their prerequisites. To the point where even if you don’t use docker, you benefit from a project having at least tried to use it.
jhayward 6y ago
Interesting, I just went to the repo's README and followed the directions for conda and had it installed and running in 3 commands.
- staticautomatic 6y ago
  Frankly I was just trying to get a pre-trained ELMO model deployed via Cortex. As best I could tell the problem boiled down to a failure related to installing tensorflow with gpu disabled. I am a NN noob but still, it just shouldn’t be this hard.
  - FridgeSeal 6y ago

nonbirithm 6y ago

Lua uses a "require()" function instead of an import statement for loading modules. I think that this is a crucial difference from a lot of programming languages that is vastly understated. What this means is that you can completely replace "require()" with your own function that has complete control over how the module importing system works. Because of this it is possible to build your own runtime code hotloader entirely in Lua without needing to modify the source of the compiler, and thus Lua is the only "mainstream" programming language listed in the list of notable live coding environments on Wikipedia[0]. (I'm surprised that Clojure isn't listed there, though.) The only way this can be accomplished is by designing the import system from the start to use a function instead of a statement. It's too late for Python to redo things and change its import system, unfortunately. It really is the one language feature you have to get right the first time or you'll miss out on those kinds of enhancements forever. [0] https://en.m.wikipedia.org/wiki/Live_coding#Notable_live_coding_environments https://en.m.wikipedia.org/wiki/Live_coding#Notable_live_cod...

sakex 6y ago
Nodejs has require too
yzmtf2008 6y ago
Untrue, you could literally replace `__import__` in python and achieve the same thing: https://docs.python.org/3.8/library/functions.html#__import__ https://docs.python.org/3.8/library/functions.html#__import_...
- Izkata 6y ago
  Unless something has changed or there's a detail involving builtins I'm not aware of, I don't think this would work, for the same reason as the most common pitfall when mocking functions in python: You have to replace it where it's used, not where it's defined. To do that on a module, you have to first import that module - and then all the imports at the top of it would've already run.
  - yzmtf2008 6y ago

yongjik 6y ago

Well, to be fair, Tensorflow is a Google library - pretty much any Google-made library will have five levels of indirection before it does anything, regardless of language. It's in their DNA.

hinkley 6y ago
Maybe I should convince a couple coworkers to apply to Google. They’d fit right in.
- Lammy 6y ago
  The different styles of programmers need each other! Checks and balances :)
  - kristopolous 6y ago
    Once a company gets sufficiently large it's indistinguishable from a cult.
kinkrtyavimoodh 6y ago
Is there actually any veracity to this claim or is it general anti-google shit-flinging? I ask because from what I understand from Google's monorepo, it should be easy for engineers to take direct dependencies on whatever they want. Of course TF is an exported out library so things might be different, but surely it won't then be a good example of Google's DNA?
- galkk 6y ago
  > I ask because from what I understand from Google's monorepo, it should be easy for engineers to take direct dependencies on whatever they want. Not correct. The dependency system supports visibility restrictions - you can take direct dependencies to anything only if you're in your own experimental tree. Otherwise package authors can control access. You can see that even in the Bazel: https://docs.bazel.build/versions/master/visibility.html https://docs.bazel.build/versions/master/visibility.html (google employee)

bobbyi_settv 6y ago

Does anyone know of a write-up that describes Tensorflow's import magic? I'd be curious to read about what they're doing and why.

codethief 6y ago
I wish I had a write-up but here's a starting point at least: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/api_template.__init__.py https://github.com/tensorflow/tensorflow/blob/master/tensorf... Inside the source code files, they then use a decorator (@tf_export) to define where in the package/module hierarchy a function/class should get placed, see e.g. https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ops/summary_ops_v2.py https://github.com/tensorflow/tensorflow/blob/master/tensorf... For instance, the write() function in that module actually gets exposed as `tf.summary.write()` even though it's contained in the file tensorflow/python/ops/summary_ops_v2.py. Needless to say, this even brings auto-completion of an IDE like IntelliJ to its knees at times. To make things even worse, things like Tensorboard also make use of global state everywhere, see e.g. the function _should_record_summaries_internal() in the above file. But now the question is: Which module's global state does the function actually access if it gets "moved" to a different module by TF? At the end of the day, I gave up hunting down that bug (_should_record_summaries_internal() always returned False, no matter what) and resorted to mocking the function out whenever I used Tensorboard. To do that (i.e. to find its parent module in the first place), I in turn used the `inspect` package. This is what I meant by "import magic". :)
- shp0ngle 6y ago
  this hurts.

lmuench 6y ago

Python’s import system is weird but it does make sense when you use absolute imports: project ├── main.py ├── package1 │ ├── module1.py │ └── module2.py └── package2 ├── __init__.py ├── module3.py ├── module4.py └── subpackage1 └── module5.py As long as the file containing the main() function resides in the top-level folder you can use absolute imports in every file, e.g. from package1 import module1 from package1.module2 import function1 from package2.subpackage1.module5 import function2

Galanwe 6y ago

Agree, python import system is annoyingly configurable, and way too much libraries use these features to do magic. I regularly have to spend some time on pandas source code just to untangle the mess of import aliases magic they do. But the standard library is no exception, importing os will dynamically import os.path.

x86_64Ubuntu 6y ago

I ran into that issue a few weeks ago. I had a small project that was supposed to migrate rows from a MySQL server into a SQL Server instance. For some reason that I was too exhausted to figure out, the program would only work if I imported the mysql connector before the pydobc stuff. That was very frustrating to say the least.