7 ms·
Python importing quirks can be time consuming. Things beginners will encounter: - three modules cannot depend on each other in a circular way - relative impor
by MeteorMarc 5y ago
Python importing quirks can be time consuming. Things beginners will encounter:
- three modules cannot depend on each other in a circular way
- relative imports are fragile ("module not found")
- the __all__ definitions in the __init__ file make modules available under different full names
- how to reload a module in a jupyter notebook if edited
and so on.
- chrisseaton 5y ago> three modules cannot depend on each other in a circular way What would the purpose of circular modules like this be? You may as well collapse into a single module and the situation would not be any different would it?
- gchamonlive 5y agoI don't think purposeful circular dependency, but you can end up with circular imports after refactoring for instance. The common approach to solving this is pulling everything that is used by all the modules into leaf libraries, effectively creating a directed acyclic graph, but this is not obvious nor easy to do the first time.
- jvolkman 5y agoIn my experience they creep in over time as the system grows. Coupling between parts of the system that was previously unnecessary is added, and the cycles form.
- BurningFrog 5y agoThe purpose is you want to use code in other modules. If you keep doing that for a while, a circular dependency will happen. This is the dumbest thing thing in Python. All other languages I know have solved it.
- chrisseaton 5y ago> The purpose is you want to use code in other modules. So put them in the same module? Circular modules don’t give you the benefits of modules, do they? Not an expert in modularity.
- BurningFrog 5y agoPutting all your code in one file does indeed solve all import problems, but creates far bigger ones. In case you didn't know, each source code file is a module in Python.
- maleldil 5y agoIf you have A -> B -> C -> A, then you need all three of A, B and C to be defined if you're going to use any one of those modules. When that's the case, the only thing you're gaining by using modules is the organisation of the file.
- BurningFrog 5y agoSure, but organizing code in files is one of the most important parts of programming!
- kortex 5y agoI've never had a circular dependency I couldn't resolve. Just organize your modules into a DAG. I have a pretty standard flow: - protocols and abcs. Depend only on builtins, this sits at the top of the hierarchy and can be used anywhere - top-level types/models, data structures. Only depends on external libraries - config and logging. Depends on pydantic, which handles all init config validation, sets up log level. Many things import this - specific class implementations, pure functions, transforms. imports from all the above - dependency-injected stuff - application logic - routers and endpoints I've had some close calls since I type check heavily, but protocols (since 3...6? 3.7 for sure) are a fantastic tool for both structuring and appeasing mypy.
- btilly 5y agoWhat would the purpose of circular modules like this be? You may as well collapse into a single module and the situation would not be any different would it? What is the purpose of modules? You may as well collapse into a single script and the situation would not be any different, would it? I'm not being facetious here. The answer to the second is the answer to the first. A common example might go like this. You have a module for each kind of thing you have in the database. But now if someone loads a Company, they need to get to Employees. And if someone loads Employee they need to get to Accounting for the salary, payments, etc. And Accounting needs to be able to get to Company. Those are all large and complicated enough that it makes sense to make them modules. But you just created a circular dependency! The standard solution is to load a base library that loads all kinds of objects so they all can assume that all the others already exist and don't need a circular dependency. But of course someone won't like wasting all that memory for things you don't need and ...
- chrisseaton 5y ago> What is the purpose of modules? So parts of the system can be managed independently. > The answer to the second is the answer to the first. Clearly not - since circular modules cannot be managed independently!
- wpietri 5y agoTo me the purpose of models is to help humans manage code. Our brains don't hold much at once, so the more we can forget about in a given circumstance, the easier it is. So I think btilly is correct: the reason I want modules, ignoring things I don't care about, is the same reason I want them to deal reasonably with circular references.
- btilly 5y agoExactly. In the example that I gave, the design described will handle complex edge cases such as a part time employee working for multiple companies. And will do so without programmers having to think through the whole system at all points. Independence of modules has no importance in a codebase that ships as a whole. But modularity does.
- nemetroid 5y agoIn my experience, typically a codebase that has grown organically in a different direction than the original design, and where the cost of refactoring is not deemed worth it.
- lp251 5y agoSuppose you have two classes, A and B. They are sufficiently complex to merit their own modules. Suppose you have some method of A which does something special if it gets an instance of B, and vice versa. Now you have a circular import problem; glhf
- chrisseaton 5y ago> Now you have a circular import problem; glhf But why not collapse into a single module at this point if you can’t avoid dependency? What are the separate modules adding at this stage forward?
- lp251 5y agoYou can, but sometimes it’s not ideal. I ran into this when A and B had many derived classes. I wanted to put A and it’s derived classes in one module, and B and it’s derived classes in another. It was messy. I wound up putting A and B in a single module and having a separate one for the derived classes. Not ideal.
- nemetroid 5y agoIt does sound ideal to me, or at least better than the initial proposal. A and B both need to know about the other's base definition, neither cares about the details about the other's derived classes. Splitting it into three modules shares as little surface area as possible.
- 0xC0ncord 5y agoI generally solve this problem by having a module specifically containing the abstract base classes of each of the classes I will be working with that implements either no or bare minimum functionality for these objects. That way, any other module can import this module and have visibility of every other class I will be working with.
- scbrg 5y agoWon't this work just fine if you instead of writing: from a import A write import a and in the code (which is presumably not at module level) check against isinstance(obj, a.A) ?
- perlgeek 5y agoA Customer has a BillingContact, which references a Person, which has primary Customer. Boom, circular dependency. Happens in basically all corporate code bases that grow over the years, with varying path lengths. Throwing all potentially circular types into one big module isn't a great solution. (In practice, we tend to rely on run-time imports to make it work. Not really great, but better than throwing several 10k or 100k lines of code into a single module).
- deleted 5y ago[deleted]
- chrisseaton 5y agoI guess I don’t get why so you want to use separate modules if you aren’t getting the benefits of modularisation?
- perlgeek 5y agoI still get some benefits of modularisation, even if there are some cross-dependencies.
- Waterluvian 5y agoI’m always amazed just how tolerant javascript’s import system is when I have circular imports. I guess maybe because it doesn’t care about modules and just cares about specific elements that are being imported/exported. When I do have a nasty circular dependency Webpack usually does a bad job telling me what’s wrong. Though I should still treat circular imports as, at the very least, an organization code smell.
- brundolf 5y agoCircular imports in JS matter when the imported code is being called immediately at import time. If a file defines functions that later call functions from another file (and vice-versa), but those symbols will be populated before the function actually gets called, there's no problem
- Waterluvian 5y agoYeah. It’s so flexible. I get frustrated at Python (Django) serializers that legitimately need to depend on each other. And the answer on the forums is to create a near duplicate class.
- nyrikki 5y agoJavaScript does not provide namespaces by default which allows this. Python is built upon namespaces and cycles in dependencies, either viewed as graph theory or kSAT introduce that fun NP-complete problem of version hell. Using `need` in Javascript maintains the directed acyclic graph structure, but if you get fancy you will run into the same problems with circular depends in Python. Karp's 21 and/or SAT will catch up with you at some point if you don't respect the constraints that make the problem tractable. Note I am not saying I prefer or like pythons choices...but that they had to make one.
- Waterluvian 5y agoSorry what’s ‘need’ ? I can’t seem to find it as a keyword.
- maxnoe 5y ago> __all__ No it doesn't. __all__ just defines which objects are imported when doing a star import.
- nooorofe 5y agoprobably, you may add to the list - understand difference between Python in IDE and Python in shell So many times people do `pip install <>` and still not able to use in IDE
- blooalien 5y agoI usually deal with this by defining a cell with the actual contents of the class/module code so that I can just re-execute that cell any time I make changes to it. Then I simply copy/paste all the code back into the module.py file once I'm done tweaking it and playing with it. Thus for me Jupyter sort of operates as almost an IDE of sorts.