7 ms·

I'm a developer who's never had to ship serious production systems with Python, but I have had to modify the occasional library or open source package. i.e. I k

I'm a developer who's never had to ship serious production systems with Python, but I have had to modify the occasional library or open source package. i.e. I know just enough to get myself into trouble... I develop on a Mac where in theory everything should be easier: * Comes with Python out of the box * Platform has ubiquitous and wonderful package manager (Homewbrew) And yet, 90% of my interactions with Python via the CLI involve it crashing because the version that the current thing I'm trying to compile or execute wants isn't the one the OS happens to be vending for it. Some OSS projects suggest using things like pyenv to help. I don't know, it just seems to make things even worse for me because now even 'which python' doesn't tell me anything because it directs to some shell script shims that I can't parse or understand. Python the language is wonderful, as are the libraries that exist for it. Python the runtime/execution environment is actively hostile to users, and it's honestly keeping me from actually building anything real with it from the ground up. I'm not surprised that Microsoft went with this approach to TRY to help it's users. I'm also not surprised that Python devs are pissed because I've also been in those shoes where FINALLY everything is working, and you think you have that one version of python 2 and one version of python 3 both running and coexisting and everything you need to run runs, and then something small changes and WHAM the whole house of cards collapses. But I wouldn't be angry at Microsoft, I'd be angry at the Python community for letting things get so bad.

downerending 6y ago
> I'm not surprised that Microsoft went with this approach to TRY to help it's users. This approach doesn't help--it makes things worse. I see this occasionally on Linux as well. A novice programmer will think: My program depends on programs X, Y, and Z, and if they're not there, I'll download them from the Internet to make things easier for my user. Aside from being a major security hole, this just makes a mess. The user has no idea of what's going on, and it's 10x harder to debug when something goes wrong. I do relate to your pain. For starters, I'd recommend not using a Mac for Python development (or really any development). If you must, something like a VirtualBox machine running Linux would be good. But probably better to just make a Linux instance in the cloud.
_bxg1 6y ago
> And yet, 90% of my interactions with Python via the CLI involve it crashing because the version that the current thing I'm trying to compile or execute wants isn't the one the OS happens to be vending for it. Do python releases introduce backwards-incompatible changes (ignoring the 2.x/3.x divide)? Would it not be as simple has making sure you have the latest runtime?
- somehnguy 6y ago
  Clearly you have never found a random python script on the internet that does exactly what you need and tried to run it. It's a nightmare every single time.
  - lstamour 6y ago
    Could be worse, try to run a C++ project next time! But yeah, the same applies to Ruby, Node, even C# (msbuild or CSX) and PowerShell modules. You have to know enough to know enough to get started. About the only thing that reliably works besides "install Docker" is getting a bash shell going and pasting in a curl link, assuming your operating system is supported by whatever it is. That said, a close runner-up, that doesn't require bash and curl to be installed, would be Node's npx. Just run it, and "it works" even if whatever it is isn't installed. (Unless whatever it is doesn't have a precompiled binary for your system published and you then need to install CMake and compile tool chains like Xcode or Tools for VS)
- jcrawfordor 6y ago
  This issue is largely OS X related rather than Python-related per se. OS X ships with vendored copies of many common Python libraries which you cannot update, and so if you need to use newer libraries you have to install them in some other location and fiddle with the path to make sure your versions are loaded before the OS X versions. Because the versions included in OS X (including the version of the Python runtime) are pretty old almost everyone ends up doing this. In theory homebrew takes care of the whole thing for you but, in practice... as an only part-time OS X user I have run into a lot of problems with the path getting set in various ways that end up in various copies of the Python interpreter or library paths getting loaded, resulting in runtime errors due to library versions being inconsistent with the interpreter version or expected libraries not being loaded... A further complication is that, the specific way that Apple vendored a set of python packages, it is possible but not particularly easy to add new packages to the OS-provided Python runtime, so most users, if they want some package that Apple didn't include, seem to end up installing a whole new environment just to get that package, even if the OS-provided environment were otherwise acceptable to them. It's a very real problem and very frustrating, but I'm not quite willing to blame it on the Python ecosystem because it seems to be pretty much a result of Apple's decision to ship an interpreter and set of libraries that most users will end up being unhappy with. Anecdotally at least I run into these issues a lot less on Linux because even with fairly stable distributions (e.g. RHEL7.5/8) the included interpreter and library are more recent and the package repositories include most of the Python packages you would want, maintained to match the OS interpreter version. So while some people do end up using virtual environments it's less necessary than it is on OS X, where it frankly seems near impossible to do anything without installing a whole second Python environment and munging paths. An extra complication is that, as a casual OS X user, it seems to me like OS X terminal emulators and tools are more likely to munge with the path than Linux terminal emulators and tools, so even if you get everything set up "just right" you end up having it not work properly in certain contexts like plugins because something has modified the path for some reason. Edited to expand that, I guess what I mean to say but failed to articulate, is that the fundamental problem here (in my opinion) is that OS X ships with a Python distribution but not with a package manager. That's the fundamental difference from Linux where these problems are much less common. Third-party package managers like Homebrew and Macports are used, and in order to have full control of the Python environment they have to install a whole second copy because they are not allowed to tamper with the OS X Python environment. This pretty inevitably leads to frustration. The best solution would seem to be for Apple to provide some kind of "first-party-ish" package management functionality, perhaps something like Chocalatey which is not a Microsoft product but is built on Windows-provided primitives, so that the package managers that users are actually using can play nicely with the OS X Python environment.
wombatpm 6y ago
If you are distributing to someone who is not a developer Pyinstaller or py2exe are your friends on windows. Build your exe, and wrap it up with a MSI installer using wix.
- happytoexplain 6y ago
  As a developer, I would qualify this even further by specifying someone who is not a developer who works extensively with Python.
softwarejosh 6y ago
homebrew and mac application management is far from wonderful. you should just start using pipenv.
MagnumPIG 6y ago
I've had a very similar experience on Linux. Love to build projects with Python, hate to use other's Python apps (unless they're packaged, in which case I don't even have to know).
geerlingguy 6y ago
I don't call myself a Python dev, but work with it enough... and I am tripped up by environment issues on a weekly basis. (Not just on my main workstation, either.)
- teraku 6y ago
  I never have issues with my Python environments, and I use either requirements, venv or pipenv, depending on the project. And I use it on Windows and Linux, server and desktop. You might be interested in: https://snarky.ca/what-the-heck-is-pyproject-toml/ https://snarky.ca/what-the-heck-is-pyproject-toml/
distantaidenn 6y ago
I’ve had similar experiences with Ruby (and Rails). I loved the language and it paid the bills for a large part of my career. But the tooling and distribution are abysmal. I can count on one hand the number of times “bundle install” has actually worked on the first try. Getting an existing project working on a new machine (sans docker) is an exercise in frustration.
mroche 6y ago
> Some OSS projects suggest using things like pyenv to help. I don't know, it just seems to make things even worse for me because now even 'which python' doesn't tell me anything because it directs to some shell script shims that I can't parse or understand. Quick tip: pyenv which [python,pip,...] pyenv help The shimming of Pyenv can bite you sometimes, but for the most part it has the tools necessary built in to help you figure out your env.
pacala 6y ago
conda
rlayton2 6y ago
I get it, but if you are developing on python, you should consider the "OS python" to be a completely separate thing you don't touch. The reason is that programs in your OS are using that python, and if you muck it up, you risk damaging your OS. Updating a package could stop your computer being able to log in (although that risk is only slight). Developers should be using pyenv or similar (try Anaconda if you want "easy") and preferably one environment per project (so you can lock in library versions for the project). I'd personally prefer the "OS python" to be more hidden, but I don't know enough about linux/mac inner workings to know if thats even a thing (maybe a libpython instead?). But it is hard to understand if you aren't working with python all the time. I do basically just python, and still spent several years with outdated Ubuntu-shipped libraries before getting properly setup.
- pmontra 6y ago
  Agreed. The OS shipping version is not a thing in every single language I've been working with. It's either outdated or not the version my project needs, and what if I need many different versions for different projects? I always use a language manager for Elixir, Erlang, Node, PHP, Python and Ruby, either the language's usual one or asdf. I'm even using different versions of databases, usually with docker. Java is somewhat easier to deal with, unzip in a directory and set JAVA_HOME but I'm not working much with it so I could miss some nuances.
- greggman3 6y ago
  I would argue if not using the OS python is the correct way to do things then python by default should not install at an OS level. Installing in the OS should be left to the OS only and the default should be that python installs per project which special instructions for installing it as a default for a user and special special instructions for how to replace the OS level one. I really wish this was the default for all languages but the majority default to "install in your system" and then I run into the problems you and the OP mention. As it is, finding out/figuring out how to work around the default is always magic incantations and spells buried on page 947 of appendix G if you're lucky or else spewed around the internet in drips and drabs by unreliable and or outdated sources. As an example I wanted to contribute to some conformance tests for some browser API. The test suite used selenium which uses python all of which wanted global installs. I tried for a few hours to get it to run with local installs but I failed to find the correct secret ritual to perform and after a few hours decided I no longer felt like searching for the correct oracle to enlighten me. ps: For my particular needs a VM was not a solution
mixmastamyk 6y ago
pythonX.Y -m module
wutbrodo 6y ago
> * Platform has ubiquitous and wonderful package manager (Homewbrew) Is this the case? I thought Homebrew was considered fairly mediocre compared to more established package managers on (eg) Linux systems. But I've never been required to use a Mac, so my firsthand familiarity is limited.
- saagarjha 6y ago
  Even ignoring any potential issues with Homebrew, it has the inherent disadvantage that every macOS package manager has in that it does not ship with the system by default.
- sesuximo 6y ago
  It's extremely slow so even when it works I'm not satisfied
rovr138 6y ago
> it just seems to make things even worse for me because now even 'which python' doesn't tell me anything because it directs to some shell script shims that I can't parse or understand. I understand it’s another layer and for someone not working in it daily, it’s hard. Some info, the command which python Wouldn’t tell you much anyway. It would just return something like /usr/bin/python Reading a binary wouldn’t lead to much either. If you’re trying to get the version of python, this still works, python --version Pyenv also has this command, pyenv version That also works with pyenv virtualenv (if you’re using it) If you want all versions installed, pyenv versions
- jjgreen 6y ago
  Helpfully, Python 3.4 switched from writing --version output from stderr to stdout, so you need capture both :-|
kevin_thibedeau 6y ago
Debian fixes this issue by having dist-packages for OS vendored packages and leaving site-packages as a customizable sandbox for the user. This makes it easy to install things without stomping on OS packages.
genmon 6y ago
For non-system users, there's no good reason to use python outside a virtual environment now, and IMO the devs shouldn't encourage it. Perhaps python should issue a warning if used outside a virtual environment, unless that warning is deliberately suppressed by a flag? That would be worth a point release on all older versions.
- fest 6y ago
  How's the situation with python packages that have native code parts? Previously they had to be compiled during installation, which wasn't easy on Windows.
zakk 6y ago
> Platform has ubiquitous and wonderful package manager (Homewbrew) Probably "wonderful" is subjective (Homebrew to me is far from being a wonderful piece of software), but "ubiquitous" is factually wrong. MacPorts, even though less popular than Homebrew, has a large userbase.
whywhywhywhy 6y ago
> * Comes with Python out of the box Not for much longer. "Scripting language runtimes such as Python, Ruby, and Perl are included in macOS for compatibility with legacy software. Future versions of macOS won’t include scripting language runtimes by default, and might require you to install additional packages. If your software depends on scripting languages, it’s recommended that you bundle the runtime within the app." https://developer.apple.com/documentation/macos_release_notes/macos_catalina_10_15_release_notes https://developer.apple.com/documentation/macos_release_note...
- tonyedgecombe 6y ago
  I think this is a good thing. I've had regular problems with Ruby and Python versioning on macOS.
  - musicale 6y ago
    I really wish Python's developers weren't hellbent on breaking backward compatibility.
dr-detroit 6y ago
Im pretty sure installing homebrew is a violation of the terms of your various apple agreements. I dont think there are many python devs on Windows. Most of their stuff like Anaconda doesn't work it does really gnarly stuff with PATH that leads me to believe they just "port" it, slap a Windows badge on their website, and move on.

downerending 6y ago

> I'm not surprised that Microsoft went with this approach to TRY to help it's users. This approach doesn't help--it makes things worse. I see this occasionally on Linux as well. A novice programmer will think: My program depends on programs X, Y, and Z, and if they're not there, I'll download them from the Internet to make things easier for my user. Aside from being a major security hole, this just makes a mess. The user has no idea of what's going on, and it's 10x harder to debug when something goes wrong. I do relate to your pain. For starters, I'd recommend not using a Mac for Python development (or really any development). If you must, something like a VirtualBox machine running Linux would be good. But probably better to just make a Linux instance in the cloud.

_bxg1 6y ago

> And yet, 90% of my interactions with Python via the CLI involve it crashing because the version that the current thing I'm trying to compile or execute wants isn't the one the OS happens to be vending for it. Do python releases introduce backwards-incompatible changes (ignoring the 2.x/3.x divide)? Would it not be as simple has making sure you have the latest runtime?

somehnguy 6y ago
Clearly you have never found a random python script on the internet that does exactly what you need and tried to run it. It's a nightmare every single time.
- lstamour 6y ago
  Could be worse, try to run a C++ project next time! But yeah, the same applies to Ruby, Node, even C# (msbuild or CSX) and PowerShell modules. You have to know enough to know enough to get started. About the only thing that reliably works besides "install Docker" is getting a bash shell going and pasting in a curl link, assuming your operating system is supported by whatever it is. That said, a close runner-up, that doesn't require bash and curl to be installed, would be Node's npx. Just run it, and "it works" even if whatever it is isn't installed. (Unless whatever it is doesn't have a precompiled binary for your system published and you then need to install CMake and compile tool chains like Xcode or Tools for VS)
jcrawfordor 6y ago
This issue is largely OS X related rather than Python-related per se. OS X ships with vendored copies of many common Python libraries which you cannot update, and so if you need to use newer libraries you have to install them in some other location and fiddle with the path to make sure your versions are loaded before the OS X versions. Because the versions included in OS X (including the version of the Python runtime) are pretty old almost everyone ends up doing this. In theory homebrew takes care of the whole thing for you but, in practice... as an only part-time OS X user I have run into a lot of problems with the path getting set in various ways that end up in various copies of the Python interpreter or library paths getting loaded, resulting in runtime errors due to library versions being inconsistent with the interpreter version or expected libraries not being loaded... A further complication is that, the specific way that Apple vendored a set of python packages, it is possible but not particularly easy to add new packages to the OS-provided Python runtime, so most users, if they want some package that Apple didn't include, seem to end up installing a whole new environment just to get that package, even if the OS-provided environment were otherwise acceptable to them. It's a very real problem and very frustrating, but I'm not quite willing to blame it on the Python ecosystem because it seems to be pretty much a result of Apple's decision to ship an interpreter and set of libraries that most users will end up being unhappy with. Anecdotally at least I run into these issues a lot less on Linux because even with fairly stable distributions (e.g. RHEL7.5/8) the included interpreter and library are more recent and the package repositories include most of the Python packages you would want, maintained to match the OS interpreter version. So while some people do end up using virtual environments it's less necessary than it is on OS X, where it frankly seems near impossible to do anything without installing a whole second Python environment and munging paths. An extra complication is that, as a casual OS X user, it seems to me like OS X terminal emulators and tools are more likely to munge with the path than Linux terminal emulators and tools, so even if you get everything set up "just right" you end up having it not work properly in certain contexts like plugins because something has modified the path for some reason. Edited to expand that, I guess what I mean to say but failed to articulate, is that the fundamental problem here (in my opinion) is that OS X ships with a Python distribution but not with a package manager. That's the fundamental difference from Linux where these problems are much less common. Third-party package managers like Homebrew and Macports are used, and in order to have full control of the Python environment they have to install a whole second copy because they are not allowed to tamper with the OS X Python environment. This pretty inevitably leads to frustration. The best solution would seem to be for Apple to provide some kind of "first-party-ish" package management functionality, perhaps something like Chocalatey which is not a Microsoft product but is built on Windows-provided primitives, so that the package managers that users are actually using can play nicely with the OS X Python environment.

wombatpm 6y ago

If you are distributing to someone who is not a developer Pyinstaller or py2exe are your friends on windows. Build your exe, and wrap it up with a MSI installer using wix.

happytoexplain 6y ago
As a developer, I would qualify this even further by specifying someone who is not a developer who works extensively with Python.

softwarejosh 6y ago

homebrew and mac application management is far from wonderful. you should just start using pipenv.

MagnumPIG 6y ago

I've had a very similar experience on Linux. Love to build projects with Python, hate to use other's Python apps (unless they're packaged, in which case I don't even have to know).

geerlingguy 6y ago

I don't call myself a Python dev, but work with it enough... and I am tripped up by environment issues on a weekly basis. (Not just on my main workstation, either.)

teraku 6y ago
I never have issues with my Python environments, and I use either requirements, venv or pipenv, depending on the project. And I use it on Windows and Linux, server and desktop. You might be interested in: https://snarky.ca/what-the-heck-is-pyproject-toml/ https://snarky.ca/what-the-heck-is-pyproject-toml/

distantaidenn 6y ago

I’ve had similar experiences with Ruby (and Rails). I loved the language and it paid the bills for a large part of my career. But the tooling and distribution are abysmal. I can count on one hand the number of times “bundle install” has actually worked on the first try. Getting an existing project working on a new machine (sans docker) is an exercise in frustration.

mroche 6y ago

> Some OSS projects suggest using things like pyenv to help. I don't know, it just seems to make things even worse for me because now even 'which python' doesn't tell me anything because it directs to some shell script shims that I can't parse or understand. Quick tip: pyenv which [python,pip,...] pyenv help The shimming of Pyenv can bite you sometimes, but for the most part it has the tools necessary built in to help you figure out your env.

pacala 6y ago

conda

rlayton2 6y ago

I get it, but if you are developing on python, you should consider the "OS python" to be a completely separate thing you don't touch. The reason is that programs in your OS are using that python, and if you muck it up, you risk damaging your OS. Updating a package could stop your computer being able to log in (although that risk is only slight). Developers should be using pyenv or similar (try Anaconda if you want "easy") and preferably one environment per project (so you can lock in library versions for the project). I'd personally prefer the "OS python" to be more hidden, but I don't know enough about linux/mac inner workings to know if thats even a thing (maybe a libpython instead?). But it is hard to understand if you aren't working with python all the time. I do basically just python, and still spent several years with outdated Ubuntu-shipped libraries before getting properly setup.

pmontra 6y ago
Agreed. The OS shipping version is not a thing in every single language I've been working with. It's either outdated or not the version my project needs, and what if I need many different versions for different projects? I always use a language manager for Elixir, Erlang, Node, PHP, Python and Ruby, either the language's usual one or asdf. I'm even using different versions of databases, usually with docker. Java is somewhat easier to deal with, unzip in a directory and set JAVA_HOME but I'm not working much with it so I could miss some nuances.
greggman3 6y ago
I would argue if not using the OS python is the correct way to do things then python by default should not install at an OS level. Installing in the OS should be left to the OS only and the default should be that python installs per project which special instructions for installing it as a default for a user and special special instructions for how to replace the OS level one. I really wish this was the default for all languages but the majority default to "install in your system" and then I run into the problems you and the OP mention. As it is, finding out/figuring out how to work around the default is always magic incantations and spells buried on page 947 of appendix G if you're lucky or else spewed around the internet in drips and drabs by unreliable and or outdated sources. As an example I wanted to contribute to some conformance tests for some browser API. The test suite used selenium which uses python all of which wanted global installs. I tried for a few hours to get it to run with local installs but I failed to find the correct secret ritual to perform and after a few hours decided I no longer felt like searching for the correct oracle to enlighten me. ps: For my particular needs a VM was not a solution

mixmastamyk 6y ago

pythonX.Y -m module

wutbrodo 6y ago

> * Platform has ubiquitous and wonderful package manager (Homewbrew) Is this the case? I thought Homebrew was considered fairly mediocre compared to more established package managers on (eg) Linux systems. But I've never been required to use a Mac, so my firsthand familiarity is limited.

saagarjha 6y ago
Even ignoring any potential issues with Homebrew, it has the inherent disadvantage that every macOS package manager has in that it does not ship with the system by default.
sesuximo 6y ago
It's extremely slow so even when it works I'm not satisfied

rovr138 6y ago

> it just seems to make things even worse for me because now even 'which python' doesn't tell me anything because it directs to some shell script shims that I can't parse or understand. I understand it’s another layer and for someone not working in it daily, it’s hard. Some info, the command which python Wouldn’t tell you much anyway. It would just return something like /usr/bin/python Reading a binary wouldn’t lead to much either. If you’re trying to get the version of python, this still works, python --version Pyenv also has this command, pyenv version That also works with pyenv virtualenv (if you’re using it) If you want all versions installed, pyenv versions

jjgreen 6y ago
Helpfully, Python 3.4 switched from writing --version output from stderr to stdout, so you need capture both :-|

kevin_thibedeau 6y ago

Debian fixes this issue by having dist-packages for OS vendored packages and leaving site-packages as a customizable sandbox for the user. This makes it easy to install things without stomping on OS packages.

genmon 6y ago

For non-system users, there's no good reason to use python outside a virtual environment now, and IMO the devs shouldn't encourage it. Perhaps python should issue a warning if used outside a virtual environment, unless that warning is deliberately suppressed by a flag? That would be worth a point release on all older versions.

fest 6y ago
How's the situation with python packages that have native code parts? Previously they had to be compiled during installation, which wasn't easy on Windows.

zakk 6y ago

> Platform has ubiquitous and wonderful package manager (Homewbrew) Probably "wonderful" is subjective (Homebrew to me is far from being a wonderful piece of software), but "ubiquitous" is factually wrong. MacPorts, even though less popular than Homebrew, has a large userbase.

whywhywhywhy 6y ago

> * Comes with Python out of the box Not for much longer. "Scripting language runtimes such as Python, Ruby, and Perl are included in macOS for compatibility with legacy software. Future versions of macOS won’t include scripting language runtimes by default, and might require you to install additional packages. If your software depends on scripting languages, it’s recommended that you bundle the runtime within the app." https://developer.apple.com/documentation/macos_release_notes/macos_catalina_10_15_release_notes https://developer.apple.com/documentation/macos_release_note...

tonyedgecombe 6y ago
I think this is a good thing. I've had regular problems with Ruby and Python versioning on macOS.
- musicale 6y ago
  I really wish Python's developers weren't hellbent on breaking backward compatibility.

dr-detroit 6y ago

Im pretty sure installing homebrew is a violation of the terms of your various apple agreements. I dont think there are many python devs on Windows. Most of their stuff like Anaconda doesn't work it does really gnarly stuff with PATH that leads me to believe they just "port" it, slap a Windows badge on their website, and move on.