3 ms·

I don’t pretend to know the answer here, but python is unavoidable in my work and the packaging is a constant source of irritation for me. Disclaimer: I am not

I don’t pretend to know the answer here, but python is unavoidable in my work and the packaging is a constant source of irritation for me. Disclaimer: I am not pretending to be a python expert here but coming from a C background this anecdote is baffling to me. I am writing some lambda OAuth glue logic using python because python is the best choice for this particular implementation. I need to package “jwt” library which worked absolutely fine for a while. Then I upgrade python versions and the particular pre-built container I was using to run pip and zip the packages up ended up with totally borked versions of jwt which seemed to not have functions I was previously using fine. Dig in, finally figure out that my older version was actually importing PyJWT even though I had specified “jwt.” The new container was breaking because it was actually installing a different “jwt” library. So, the solution was to specifically specify PyJWT and which version I wanted in my pip install. Great! That’s how I think it should work and I was a little baffled that pip had made that decision for me previously. Anyway, it now has my missing functions but is still crashing in my blue deployments. Wtf? Oh, this PyJWT import is missing an algorithm. To fix that, I also need to pip install “cryptography” (making sure to get the compatible version matrix here, I at this point had stopped trusting pip). So maybe this is all very obvious and “duh” to veterans out there but this was impossibly silly and wasted a stupid amount of time for something that should be dead simple. yea, wrangling makefiles and fussing with linkers can be annoying but I’d take that any day over this bs.

zahlman 2y ago
> Dig in, finally figure out that my older version was actually importing PyJWT even though I had specified “jwt.” The new container was breaking because it was actually installing a different “jwt” library. Names on PyPI have no necessary correlation with names used to `import` modules in the code. This is known and by design and has always been so; and it is thoroughly documented (e.g. https://packaging.python.org/en/latest/discussions/distribution-package-vs-import-package/ https://packaging.python.org/en/latest/discussions/distribut...). There are at important architectural reasons for it: * A package installed from PyPI can legally provide any number of importable packages and/or modules - including zero. * Making it work the way you imagine would require additional verification work, with no clear idea of who's responsible for that work nor what to do when the assumption is violated. * Sometimes projects have valid reasons to rename what you `import` at top level. It could be to avoid common conflicts; it could be because of internal open-source politics (see: PIL/Pillow); it could be because a legacy name was needed for some disambiguation purpose with Conda; for whatever rare reasons it might just make more logical sense. In the extreme, some name could become a keyword in a later Python version. (There's an actual `async` package on PyPI, although it was last updated 10 years ago.) In all of these cases, there's value in keeping the name used on PyPI the same. * There are valid reasons why people might want to swap implementations of the same library. The PyPI system allows you to just change the name provided to `pip install` without modifying the code. You actually ran into that last point: both `jwt` and `PyJWT` on PyPI refer to implementations of JSON Web Tokens that expect you to `import jwt` in the code. They may or may not have different interfaces and supply different functionality, and are maintained by different people. You can't seriously expect to forbid everyone else from publishing such an implementation just because one already exists. > This PyJWT import is missing an algorithm. To fix that, I also need to pip install “cryptography” There are a few possibilities: * The maintainer messed up and forgot to list it as a dependency in the manifest. These things happen. Other people's projects are allowed to have their own dependencies. * The maintainer listed `cryptography` as an optional dependency or "extra" - so that you don't have to grab it if you aren't using that algorithm. You're expected to read the documentation to understand what your project will need. * You really were intended to use the `jwt` package, but a new version of that project removed functionality you were depending on. Other people's projects are allowed to change their interfaces and go through deprecation cycles. In this case, you would presumably find that your code also needed to be rewritten to invoke the `cryptography` functionality explicitly/directly. >making sure to get the compatible version Pip solves dependencies for you. If you're running into version incompatibility issues, generally it's either the fault of the package maintainers, or the environment you want to create is fundamentally not possible due to conflicting version requirements of your transitive dependencies. (Pip is designed to only install one version of a given package in a given environment, because Python is designed to only `import` a given module from one place and then reuse the cached module object.) >So maybe this is all very obvious and “duh” to veterans out there but this was impossibly silly and wasted a stupid amount of time for something that should be dead simple Yes, it's obvious. Yes, normally it is dead simple. Veterans commonly run into hairy problems regardless. There are plenty of faults in the system - this just isn't generally considered one of them. The scope of the problem solved (or at least undertaken) by the Pip/PyPI system is much larger than you seem to imagine. In particular, it often has to do that same wrangling with linkers and Makefiles that you describe, and in an automated fashion at that. (Python code commonly has to interoperate with extensions written in C - and FORTRAN, and other languages - which is provided as part of the same package.)
- JohnMakin 2y ago
  Thank you for the detailed explanation - this makes a lot of sense.
  - stevenhuang 2y ago
    This is also documented by the library https://pyjwt.readthedocs.io/en/stable/installation.html https://pyjwt.readthedocs.io/en/stable/installation.html > If you are planning on encoding or decoding tokens using certain digital signature algorithms (like RSA or ECDSA), you will need to install the cryptography library. This can be installed explicitly, or as a required extra in the pyjwt requirement: > pip install pyjwt[crypto]

zahlman 2y ago

> Dig in, finally figure out that my older version was actually importing PyJWT even though I had specified “jwt.” The new container was breaking because it was actually installing a different “jwt” library. Names on PyPI have no necessary correlation with names used to `import` modules in the code. This is known and by design and has always been so; and it is thoroughly documented (e.g. https://packaging.python.org/en/latest/discussions/distribution-package-vs-import-package/ https://packaging.python.org/en/latest/discussions/distribut...). There are at important architectural reasons for it: * A package installed from PyPI can legally provide any number of importable packages and/or modules - including zero. * Making it work the way you imagine would require additional verification work, with no clear idea of who's responsible for that work nor what to do when the assumption is violated. * Sometimes projects have valid reasons to rename what you `import` at top level. It could be to avoid common conflicts; it could be because of internal open-source politics (see: PIL/Pillow); it could be because a legacy name was needed for some disambiguation purpose with Conda; for whatever rare reasons it might just make more logical sense. In the extreme, some name could become a keyword in a later Python version. (There's an actual `async` package on PyPI, although it was last updated 10 years ago.) In all of these cases, there's value in keeping the name used on PyPI the same. * There are valid reasons why people might want to swap implementations of the same library. The PyPI system allows you to just change the name provided to `pip install` without modifying the code. You actually ran into that last point: both `jwt` and `PyJWT` on PyPI refer to implementations of JSON Web Tokens that expect you to `import jwt` in the code. They may or may not have different interfaces and supply different functionality, and are maintained by different people. You can't seriously expect to forbid everyone else from publishing such an implementation just because one already exists. > This PyJWT import is missing an algorithm. To fix that, I also need to pip install “cryptography” There are a few possibilities: * The maintainer messed up and forgot to list it as a dependency in the manifest. These things happen. Other people's projects are allowed to have their own dependencies. * The maintainer listed `cryptography` as an optional dependency or "extra" - so that you don't have to grab it if you aren't using that algorithm. You're expected to read the documentation to understand what your project will need. * You really were intended to use the `jwt` package, but a new version of that project removed functionality you were depending on. Other people's projects are allowed to change their interfaces and go through deprecation cycles. In this case, you would presumably find that your code also needed to be rewritten to invoke the `cryptography` functionality explicitly/directly. >making sure to get the compatible version Pip solves dependencies for you. If you're running into version incompatibility issues, generally it's either the fault of the package maintainers, or the environment you want to create is fundamentally not possible due to conflicting version requirements of your transitive dependencies. (Pip is designed to only install one version of a given package in a given environment, because Python is designed to only `import` a given module from one place and then reuse the cached module object.) >So maybe this is all very obvious and “duh” to veterans out there but this was impossibly silly and wasted a stupid amount of time for something that should be dead simple Yes, it's obvious. Yes, normally it is dead simple. Veterans commonly run into hairy problems regardless. There are plenty of faults in the system - this just isn't generally considered one of them. The scope of the problem solved (or at least undertaken) by the Pip/PyPI system is much larger than you seem to imagine. In particular, it often has to do that same wrangling with linkers and Makefiles that you describe, and in an automated fashion at that. (Python code commonly has to interoperate with extensions written in C - and FORTRAN, and other languages - which is provided as part of the same package.)

JohnMakin 2y ago
Thank you for the detailed explanation - this makes a lot of sense.
- stevenhuang 2y ago
  This is also documented by the library https://pyjwt.readthedocs.io/en/stable/installation.html https://pyjwt.readthedocs.io/en/stable/installation.html > If you are planning on encoding or decoding tokens using certain digital signature algorithms (like RSA or ECDSA), you will need to install the cryptography library. This can be installed explicitly, or as a required extra in the pyjwt requirement: > pip install pyjwt[crypto]