7 ms·

> Good dependency management means ... > it is possible to safely evolve and update your environment when new versions of dependencies become available. The po

• zahlman 2y ago
  >I'm blocked from upgrading A with the normal tooling, unless I either ditch or modify B.... I think this is also most of the time in principle possible in python, but the ecosystem doesn't really support or encourage it. No, the main problems with this are at the language level. Languages like Java can do it because the import is resolved statically. Python's `import` statement is incredibly dynamic, offering a variety of rarely-used hooks (start at https://docs.python.org/3/library/sys.html#sys.meta_path https://docs.python.org/3/library/sys.html#sys.meta_path and follow the links, if you dare) that ultimately (by default) put imported modules into a single global dictionary, keyed by name. If you want to work around that, you can, but you'll definitely have to roll up your sleeves a bit. Even if you just want to choose between two different versions at runtime, the base `import` syntax doesn't have a way to specify a version. If the package version isn't explicitly namespaced (which causes more pain for everyone who doesn't care about the exact version) then you need to make special arrangements so that the one you want will be found first (normally, by manipulating `sys.path`). But if you want to use both during the same run, you'll encounter far more problems. The global dict strategy is very deliberate. Of course it offers a performance benefit (caching) but it's also required for correctness in many normal cases. It makes sure that different parts of the code importing the same module see the same module object, and therefore the same module-level globals. That is: in the Python world, it's common that the design of C involves mutating its global state. If you "let B use its older version" then it would have to be an entirely separate object with its own state, and mutating that would result in changes not seen by A. Whether or not that's desirable would depend on your individual use case. A lot of the time, you would want the change to be shared. But then, even if you could propagate the changes automatically, you'd have to deal with the fact that the two versions of C don't necessarily represent their state identically. > by letting B use its older version but re-package everything under a non-colliding name. Or perhaps I depend on library B and B uses C but I don't use the parts of B that interact with C. I could build a fat-jar for my project, and effectively drop the parts of B that I don't need. In principle, the B author can make the dependency on C optional ("extra"), for those users who do need that part of B. Or even for B to optionally depend on B-prime, which encapsulates the C-using parts. This is actually not very difficult (and the above-described dynamic nature of Python is very helpful here), but in practice it isn't done very much (perhaps Python developers are wary of creating another left-pad situation). But I do wish e.g. Numpy were more like that. But the "fat-jar" analog is also possible in Python, as long as it isn't a problem that you aren't sharing state between C-old and C-new. It's called "vendoring"; Pip does a lot of it; and the tooling that Pip uses to enable it is also made available (https://pypi.org/project/vendoring/ https://pypi.org/project/vendoring/) by one of the main Pip devs (Pradyun Gedam).
  • abeppu 2y ago
    > No, the main problems with this are at the language level. Languages like Java can do it because the import is resolved statically. > put imported modules into a single global dictionary, keyed by name. > If the package version isn't explicitly namespaced (which causes more pain for everyone who doesn't care about the exact version) then you need to make special arrangements so that the one you want will be found first (normally, by manipulating `sys.path`). > That is: in the Python world, it's common that the design of C involves mutating its global state. If you "let B use its older version" then it would have to be an entirely separate object with its own state, and mutating that would result in changes not seen by A. Whether or not that's desirable would depend on your individual use case I haven't worked on a java project that needed this in some years so I may be out of date, but all the same things you describe are also (mostly) true of and relevant to the java ecosystem (or at least were), and these are the same considerations that come to the choices around shading in java land. - the class name, method names etc are statically known at compile time, but version is not indicated by import or use. The compiler finds (or doesn't) the requisite definitions from name, based on what's available on the classpath at compile time. At runtime, we load whichever version of that class is on the classpath. If between compilation and running, some part of your build process or environment changed to make a different version of the class available, and names/signatures align, nothing at runtime knows of the change. Order still matters, and just as with python, java (outside of custom classloader shenanigans) also works hard to only load each class once, associated to a qualified name. - shading works around these constraints by renaming. E.g. you might have two SDKs which wrap API calls to different vendors, which depend on different major versions of a serialization library. No functionality depends on passing any of internal classes of the serialization library between calls to these two vendors, so you're safe to shade one SDK's use of that library (package) to a new, unambiguous name. The key point is both the conflicted library C and the SDK that uses it B get rewritten. Note, this would break at runtime if either library had code that e.g. constructed a string which was then used as a classname, but this would already be pretty abusive. - similarly in python, if in your project you use two libraries which each separately use e.g. pydantic at 1.x and 2.x, and your own code isolates these from each other (i.e. classes from B don't get passed through methods in A, etc), then you could pretty safely rename one version (e.g. `pydantic` 1.x in library gets renamed as `pydanticlegacy`) -- but the common tooling doesn't generally support this. Just as in the python case, if the library code does something weird (e.g. `eval`ing a string that uses the original name), stuff will break at runtime, very analogously to the java situation. In both cases, the language on its own doesn't support a concept of import of a specified version, and at runtime each unique name is associated with code which is resolved by a search through an ordered collection of paths, and the first version found wins. What differs is the level of tooling support for consistent package/module renaming. If anything, I think the actual requirements here on the python side should be lower; because java shading must work on class files (so you don't have to recompile upstream stuff), it needs to use ASM.