5 ms·

> And yet, 90% of my interactions with Python via the CLI involve it crashing because the version that the current thing I'm trying to compile or execute wants

by _bxg1 6y ago

> And yet, 90% of my interactions with Python via the CLI involve it crashing because the version that the current thing I'm trying to compile or execute wants isn't the one the OS happens to be vending for it. Do python releases introduce backwards-incompatible changes (ignoring the 2.x/3.x divide)? Would it not be as simple has making sure you have the latest runtime?

somehnguy 6y ago
Clearly you have never found a random python script on the internet that does exactly what you need and tried to run it. It's a nightmare every single time.
- lstamour 6y ago
  Could be worse, try to run a C++ project next time! But yeah, the same applies to Ruby, Node, even C# (msbuild or CSX) and PowerShell modules. You have to know enough to know enough to get started. About the only thing that reliably works besides "install Docker" is getting a bash shell going and pasting in a curl link, assuming your operating system is supported by whatever it is. That said, a close runner-up, that doesn't require bash and curl to be installed, would be Node's npx. Just run it, and "it works" even if whatever it is isn't installed. (Unless whatever it is doesn't have a precompiled binary for your system published and you then need to install CMake and compile tool chains like Xcode or Tools for VS)
jcrawfordor 6y ago
This issue is largely OS X related rather than Python-related per se. OS X ships with vendored copies of many common Python libraries which you cannot update, and so if you need to use newer libraries you have to install them in some other location and fiddle with the path to make sure your versions are loaded before the OS X versions. Because the versions included in OS X (including the version of the Python runtime) are pretty old almost everyone ends up doing this. In theory homebrew takes care of the whole thing for you but, in practice... as an only part-time OS X user I have run into a lot of problems with the path getting set in various ways that end up in various copies of the Python interpreter or library paths getting loaded, resulting in runtime errors due to library versions being inconsistent with the interpreter version or expected libraries not being loaded... A further complication is that, the specific way that Apple vendored a set of python packages, it is possible but not particularly easy to add new packages to the OS-provided Python runtime, so most users, if they want some package that Apple didn't include, seem to end up installing a whole new environment just to get that package, even if the OS-provided environment were otherwise acceptable to them. It's a very real problem and very frustrating, but I'm not quite willing to blame it on the Python ecosystem because it seems to be pretty much a result of Apple's decision to ship an interpreter and set of libraries that most users will end up being unhappy with. Anecdotally at least I run into these issues a lot less on Linux because even with fairly stable distributions (e.g. RHEL7.5/8) the included interpreter and library are more recent and the package repositories include most of the Python packages you would want, maintained to match the OS interpreter version. So while some people do end up using virtual environments it's less necessary than it is on OS X, where it frankly seems near impossible to do anything without installing a whole second Python environment and munging paths. An extra complication is that, as a casual OS X user, it seems to me like OS X terminal emulators and tools are more likely to munge with the path than Linux terminal emulators and tools, so even if you get everything set up "just right" you end up having it not work properly in certain contexts like plugins because something has modified the path for some reason. Edited to expand that, I guess what I mean to say but failed to articulate, is that the fundamental problem here (in my opinion) is that OS X ships with a Python distribution but not with a package manager. That's the fundamental difference from Linux where these problems are much less common. Third-party package managers like Homebrew and Macports are used, and in order to have full control of the Python environment they have to install a whole second copy because they are not allowed to tamper with the OS X Python environment. This pretty inevitably leads to frustration. The best solution would seem to be for Apple to provide some kind of "first-party-ish" package management functionality, perhaps something like Chocalatey which is not a Microsoft product but is built on Windows-provided primitives, so that the package managers that users are actually using can play nicely with the OS X Python environment.
- pvg 6y ago
  An extra complication is that, as a casual OS X user, it seems to me like OS X terminal emulators and tools are more likely to munge with the path This might be a function of the casualness rather than the tools. Here's a thing to check out on OS X: man path_helper I didn't know about this for ages and it makes your path life much saner - at least you'll have some idea where seemingly magical paths come from. Personally, I use it for everything and don't let other profile scripts eff with the global path at all.
  - tptacek 6y ago
    You set up your personal shell paths by adding files to /etc/paths.d/?
    - pvg 6y ago
      With neither shame nor ruth. Is there some non-aesthetic reason not to?
      - tptacek 6y ago
        No! Well other than not having your whole shell environment in your dotfiles. It just wouldn’t have occurred to me.
- _bxg1 6y ago
  So it sounds like more of a packages thing than an interpreter thing; basically "DLL hell" all over again. Setting aside all other axes of comparison, it sounds like Node really learned from Python with regards to this problem. Install Node itself, do an NPM install, and you're golden.
  - desert_boi 6y ago
    Until your global installs get stale/conflict with new global installs. npx is pretty handy for this problem though.
maple3142 6y ago
Just a example for a breaking change between Python 3.7 and before: https://stackoverflow.com/q/51196568/6885801 https://stackoverflow.com/q/51196568/6885801 It is because Python 3.7 makes "async" a reserved word, which was a valid identifier name before.
- oefrha 6y ago
  py34 asyncio was provisional (read: experimental, API in flux). It was very clearly labeled as such in both py34 and py35.[1][2] In py36 it was no longer provisional, but async/await syntax was established in py35 and asyncio.async was already deprecated back then. Using an example of a clearly experimental and evolving feature as breaking backwards-compatibility is disingenuous. Now, there are actual backwards-incompatible changes that aren't made after at least two versions of deprecation, but those tend to be low level and subtle. [1] https://docs.python.org/3.4/library/asyncio.html https://docs.python.org/3.4/library/asyncio.html [2] https://docs.python.org/3.5/library/asyncio.html https://docs.python.org/3.5/library/asyncio.html More on provisional API: https://docs.python.org/3.8/glossary.html#term-provisional-api https://docs.python.org/3.8/glossary.html#term-provisional-a... > A provisional API is one which has been deliberately excluded from the standard library’s backwards compatibility guarantees. While major changes to such interfaces are not expected, as long as they are marked provisional, backwards incompatible changes (up to and including removal of the interface) may occur if deemed necessary by core developers. Such changes will not be made gratuitously – they will occur only if serious fundamental flaws are uncovered that were missed prior to the inclusion of the API.
  - Xixi 6y ago
    I think you misunderstood maple3142 comment: he is only saying that from Python 3.7 onward you could not use "async" as an identifier in your code, as it became a reserved word. So maybe you were using "async" as a variable name, or as a function name in your code, and suddenly that wouldn't work anymore. It definitely is a backwards-incompatible change, and that has nothing to do with provisional APIs. Since I have never use "async" as an identifier in my own code, I have no idea if warnings were issued, though probably?
stevehb 6y ago
Even after the 2.x/3.x debacle, it's not as simple as keeping everything current. For example, until very recently, Tensorflow would work on Python 3.7 but not on 3.8. TF is just one well-known example, but version incompatibilities are common enough that many Python devs keep multiple virtual envs.

by _bxg1 6y ago

somehnguy 6y ago

Clearly you have never found a random python script on the internet that does exactly what you need and tried to run it. It's a nightmare every single time.

lstamour 6y ago
Could be worse, try to run a C++ project next time! But yeah, the same applies to Ruby, Node, even C# (msbuild or CSX) and PowerShell modules. You have to know enough to know enough to get started. About the only thing that reliably works besides "install Docker" is getting a bash shell going and pasting in a curl link, assuming your operating system is supported by whatever it is. That said, a close runner-up, that doesn't require bash and curl to be installed, would be Node's npx. Just run it, and "it works" even if whatever it is isn't installed. (Unless whatever it is doesn't have a precompiled binary for your system published and you then need to install CMake and compile tool chains like Xcode or Tools for VS)

jcrawfordor 6y ago

This issue is largely OS X related rather than Python-related per se. OS X ships with vendored copies of many common Python libraries which you cannot update, and so if you need to use newer libraries you have to install them in some other location and fiddle with the path to make sure your versions are loaded before the OS X versions. Because the versions included in OS X (including the version of the Python runtime) are pretty old almost everyone ends up doing this. In theory homebrew takes care of the whole thing for you but, in practice... as an only part-time OS X user I have run into a lot of problems with the path getting set in various ways that end up in various copies of the Python interpreter or library paths getting loaded, resulting in runtime errors due to library versions being inconsistent with the interpreter version or expected libraries not being loaded... A further complication is that, the specific way that Apple vendored a set of python packages, it is possible but not particularly easy to add new packages to the OS-provided Python runtime, so most users, if they want some package that Apple didn't include, seem to end up installing a whole new environment just to get that package, even if the OS-provided environment were otherwise acceptable to them. It's a very real problem and very frustrating, but I'm not quite willing to blame it on the Python ecosystem because it seems to be pretty much a result of Apple's decision to ship an interpreter and set of libraries that most users will end up being unhappy with. Anecdotally at least I run into these issues a lot less on Linux because even with fairly stable distributions (e.g. RHEL7.5/8) the included interpreter and library are more recent and the package repositories include most of the Python packages you would want, maintained to match the OS interpreter version. So while some people do end up using virtual environments it's less necessary than it is on OS X, where it frankly seems near impossible to do anything without installing a whole second Python environment and munging paths. An extra complication is that, as a casual OS X user, it seems to me like OS X terminal emulators and tools are more likely to munge with the path than Linux terminal emulators and tools, so even if you get everything set up "just right" you end up having it not work properly in certain contexts like plugins because something has modified the path for some reason. Edited to expand that, I guess what I mean to say but failed to articulate, is that the fundamental problem here (in my opinion) is that OS X ships with a Python distribution but not with a package manager. That's the fundamental difference from Linux where these problems are much less common. Third-party package managers like Homebrew and Macports are used, and in order to have full control of the Python environment they have to install a whole second copy because they are not allowed to tamper with the OS X Python environment. This pretty inevitably leads to frustration. The best solution would seem to be for Apple to provide some kind of "first-party-ish" package management functionality, perhaps something like Chocalatey which is not a Microsoft product but is built on Windows-provided primitives, so that the package managers that users are actually using can play nicely with the OS X Python environment.

pvg 6y ago
An extra complication is that, as a casual OS X user, it seems to me like OS X terminal emulators and tools are more likely to munge with the path This might be a function of the casualness rather than the tools. Here's a thing to check out on OS X: man path_helper I didn't know about this for ages and it makes your path life much saner - at least you'll have some idea where seemingly magical paths come from. Personally, I use it for everything and don't let other profile scripts eff with the global path at all.
- tptacek 6y ago
  You set up your personal shell paths by adding files to /etc/paths.d/?
  - pvg 6y ago
    With neither shame nor ruth. Is there some non-aesthetic reason not to?
    - tptacek 6y ago
      No! Well other than not having your whole shell environment in your dotfiles. It just wouldn’t have occurred to me.
_bxg1 6y ago
So it sounds like more of a packages thing than an interpreter thing; basically "DLL hell" all over again. Setting aside all other axes of comparison, it sounds like Node really learned from Python with regards to this problem. Install Node itself, do an NPM install, and you're golden.
- desert_boi 6y ago
  Until your global installs get stale/conflict with new global installs. npx is pretty handy for this problem though.

maple3142 6y ago

Just a example for a breaking change between Python 3.7 and before: https://stackoverflow.com/q/51196568/6885801 https://stackoverflow.com/q/51196568/6885801 It is because Python 3.7 makes "async" a reserved word, which was a valid identifier name before.

oefrha 6y ago
py34 asyncio was provisional (read: experimental, API in flux). It was very clearly labeled as such in both py34 and py35.[1][2] In py36 it was no longer provisional, but async/await syntax was established in py35 and asyncio.async was already deprecated back then. Using an example of a clearly experimental and evolving feature as breaking backwards-compatibility is disingenuous. Now, there are actual backwards-incompatible changes that aren't made after at least two versions of deprecation, but those tend to be low level and subtle. [1] https://docs.python.org/3.4/library/asyncio.html https://docs.python.org/3.4/library/asyncio.html [2] https://docs.python.org/3.5/library/asyncio.html https://docs.python.org/3.5/library/asyncio.html More on provisional API: https://docs.python.org/3.8/glossary.html#term-provisional-api https://docs.python.org/3.8/glossary.html#term-provisional-a... > A provisional API is one which has been deliberately excluded from the standard library’s backwards compatibility guarantees. While major changes to such interfaces are not expected, as long as they are marked provisional, backwards incompatible changes (up to and including removal of the interface) may occur if deemed necessary by core developers. Such changes will not be made gratuitously – they will occur only if serious fundamental flaws are uncovered that were missed prior to the inclusion of the API.
- Xixi 6y ago
  I think you misunderstood maple3142 comment: he is only saying that from Python 3.7 onward you could not use "async" as an identifier in your code, as it became a reserved word. So maybe you were using "async" as a variable name, or as a function name in your code, and suddenly that wouldn't work anymore. It definitely is a backwards-incompatible change, and that has nothing to do with provisional APIs. Since I have never use "async" as an identifier in my own code, I have no idea if warnings were issued, though probably?

stevehb 6y ago

Even after the 2.x/3.x debacle, it's not as simple as keeping everything current. For example, until very recently, Tensorflow would work on Python 3.7 but not on 3.8. TF is just one well-known example, but version incompatibilities are common enough that many Python devs keep multiple virtual envs.