9 ms·
Changes to Homebrew adversely affect Python development
- draw_down 6y agoWhy does everything related to versioning and packaging have to be so difficult with Python? I never understood that.
- thenewwazoo 6y agoI had a whole comment written up saying basically this, but in less-kind terms. Python just seems so incredibly brittle.
- artful-hacker 6y agoI'd hazard a guess that it is because Python is so old. It was literally created in the 80s. Package management, explosion of dependencies, multiple implementations of the interpreter, being added to the base operating system, and all that fancy stuff came after. I remember doing versioning and packaging in C++ in early 2000s, and it being just as annoying but for different reasons. People are trying to fix it, and their are some great packages out there that make it easier. Pyenv, Poetry, this asdf tool the fine article mentions, all seem like good solutions.
- coldtea 6y ago>I'd hazard a guess that it is because Python is so old. It was literally created in the 80s. Package management, explosion of dependencies, multiple implementations of the interpreter, being added to the base operating system, and all that fancy stuff came after. So? You can still create nice tools for it, even decades after a language has been created...
- acdha 6y agoIt's not. People arbitrarily complicate their personal workflows, especially having years of habit predating the existence of modern tools, but this is a voluntary decision to make local development complicated and you can choose not to work that way. As an example, any time Homebrew ships a new update I run “pipx reinstall-all” and, for local projects, I let pipenv / poetry rebuild the virtualenv the next time I use it. This is automated and takes far less of my time than, say, managing dependency updates.
- ilikepi 6y agoI think part of the challenge (from my perspective, as someone who only rarely needs to mess with python) is not that the workflow you describe is particularly complicated in isolation, it's the shear number of different tools that ostensibly have the same goal: to manage python virtual environments and packages. Because there are so many different tools, there is a near-limitless supply of different articles and SO posts in which people describe the "best" way to accomplish the most basic setup and maintenance of a virtual environment. It's quite difficult for new and/or occasional users to choose a good way to get started. Even if there is an "official" way documented on python.org for each Python version, when you run into a problem with it, inevitably you're stuck filtering through lots of advice to "just use <third-party tool xyz> instead".
- draw_down 6y agoAh, my mistake! I had thought it was a huge problem and a constant pain in the ass, but it turns out that actually everyone is just stupid. Whew, what a relief.
- coldtea 6y ago>As an example, any time Homebrew ships a new update I run “pipx reinstall-all” and, for local projects, I let pipenv / poetry rebuild the virtualenv the next time I use it. This is automated and takes far less of my time than, say, managing dependency updates. So, giving up (and re-installing everything from scratch or rebuilding virtualenvs, using several third party tools (pipx on top of pip, pipenv and/or poetry in lieu/in place of virtualenv) is your argument to how "it's not complicated" and people merely "arbitrarily complicate their personal workflows"? Oh, the irony!
- acdha 6y agoThe key part which you missed is that everything is automated, always installed in an isolated virtual environment, the package are automatically tracked and updated, and the tools which make this happen are installed from Homebrew and update at the same time. As to why there are two tools, it's because they have separate goals: pipx provides a robust way to manage global installs of Python utilities in per-tool virtualenvs while tools like poetry or pipenv are aimed at projects you're developing and handle things like automating the process of installing updates & updating the corresponding metadata files so you can ship clean updates. All of them use virtualenvs and pip underneath — it's just a modern workflow. The key part is recognizing that the problem comes from trying to fight updates: software changes and accepting that means moving to a workflow which automates that rather than spending time trying to freeze everything in amber. Learning to use a tool every decade or so is a LOT less time consuming than doing this by hand. The average poster in this thread has already spent more time on this topic than my projects have needed for the mechanics of installing Python packages in any given year.
- qbasic_forever 6y agoPython was created in the 90s when most unix users were on huge minicomputer-like shared systems (mostly at universities). In those days you had entire IT staff dedicated to curating and maintaining all of the software on the system, making sure it ran well, keeping it updated, etc. Guido explicitly did not want to bog the language development down with time spent on building packaging, deployment, etc.--those were matters for the system admins. We've all kind of been dealing with the consequences of that decision for many years.
- bluedino 6y agoI loved Homebrew, but it really seems to have gone to shit the last couple years. Are they spreading themselves too thin?
- plorkyeran 6y agoThe general trend I've noticed over the years is that they've been reducing edge-cases at the cost of making normal behavior worse. The change noted in TFA is one of these: always upgrading a package's dependencies when upgrading a package greatly reduces the numbers of combinations of versions that users can possibly have and so probably reduces the number of bug reports that come down to local environment things, but it comes at the cost of effectively removing the ability to upgrade a specific tool without fucking with the rest of your system. Similarly I'm sure that making `brew install X` implicitly perform `brew upgrade` simplifies things a lot for them, but it's been a major source of hassle for me and it's making me consider switching away from homebrew.
- mdaniel 6y agoHave you explored the `$HOMEBREW_NO_AUTO_UPDATE=1` env var? https://github.com/Homebrew/brew/blob/2.7.7/docs/Manpage.md#environment https://github.com/Homebrew/brew/blob/2.7.7/docs/Manpage.md#... I can't tell if that would influence the behavior you're experiencing, but it sounds like it makes the brew command more deterministic
- plorkyeran 6y agoThat sounds unrelated. It disables automatically calling `brew update` (which I'm fine with), but not `brew upgrade`.
- heavyset_go 6y agoThis was one of the pain points that made me move back to Linux on my devices, and I'm glad I did it. Homebrew is infuriating when you're used to real package managers.
- blt 6y agoWhen you switch from Linux to Mac/Windows, you think "wow, this OS was actually made for GUIs. Everything just works. I don't have to restart ALSA every 3 days." When you switch from Mac/Windows to Linux, you think "wow, this OS was actually made for software development. Everything just works. I don't have to resolve a library version conflict every 3 days."
- happymellon 6y agoI have never needed to restart ALSA, but I agree that it can be a bit rougher around the edges. Bluetooth is flakey everywhere, but I do seem to encounter more "edge cases" in Linux than my Mac. If you can live with the rough edges, I find the general experience with applications or my printer suddenly being incompatible under MacOS for "reasons" is much less of an issue under Linux.
- blt 6y agoAgree that Mac OS backwards compatibility is atrocious! I think another big challenge for Linux is battery life and sleep/wake on laptops. Last time I considered replacing Mac OS with Ubuntu on my laptop, I got scared away by reports that battery life would decrease by half.
- happymellon 6y agoMy Lenovo laptop gets better battery life under Linux than it did under Windows, and I've generally found the battery life estimations (for my use cases anyway) to be complete bullshit even under MacOS. I'm excited to be able to upgrade to the M1 series Mac's as it sounds like I might actually get battery life, though I have the 2019 16" Pro and can't justify the upgrade just yet, even if that thing overheats with just a Slack video.
- surbas 6y agoRan into this myself. Starting using `pyenv` instead. Never looked back. Allows you to maintain multiple versions of python outside brew's control and the system's. https://github.com/pyenv/pyenv https://github.com/pyenv/pyenv What's nice is you can find analog programs for a lot of different languages and platforms including Node: https://github.com/ekalinin/nodeenv Ruby: https://github.com/rbenv/rbenv Terraform: https://github.com/tfutils/tfenv
- justinmayer 6y agoThe asdf Python plugin uses python-build, the same back-end that pyenv uses. So the tools are very similar. I prefer asdf to the collection of programs you mentioned because with asdf you get a unified CLI syntax for adding/updating plugins and installing/removing versions. I like being able to use the same toolchain and UI to manage different versions of Python, Node.js, Ruby, etc. Check out the list of asdf plugins [1] — it's quite impressive. [1] https://asdf-vm.com/#/plugins-all?id=plugin-list https://asdf-vm.com/#/plugins-all?id=plugin-list
- baryphonic 6y agoNot trying to spam, but have you looked at `anyenv`? It installs and updates all of these installers (and more), and keeps plugins up to date. I've been very happy with it.
- waffletower 6y agoOh noes, I will soon need to use tfenv too!?!
- lars-b2018 6y agoI believe I have noticed this as well with Ruby as well. All Gems work great, then, after a brew operation, I need to reinstall/rebuild against an updated Ruby. It's certainly frustrating and can bite you, especially since it is unexpected.
- binarymax 6y agoWhere my conda crew at?
- ksm1717 6y agoI was worried that even miniconda seemed kinda bloaty, but it was the only way I could consistently use numpy on apple silicon. Really good option for local dev environments
- qbasic_forever 6y agoIf you haven't yet, try out mamba. It's more or less a drop-in conda replacement but written in C++ using a super fast SAT solver. The speed difference is incredible--I rarely even have time to finish reading the output before it's done installing a conda environment.
- lolpython 6y agoThank you very much for this suggestion. My team's workflow is conda heavy and this will speed up my development by orders of magnitude.
- 1123581321 6y agoUse Homebrew to install pyenv and pyenv-virtualenv. Then use those to install pythons and collections of packages. You'll have stable development environments.
- systemvoltage 6y agoNot on M1 Apple Silicon. So many issues with pyenv on the M1. Brew python works fine natively.
- 1123581321 6y agoI have this setup running on M1 and Intel. I'm not saying you are having no issues, but I don't think that's the reason. If you're having trouble installing something with brew on M1, run arch x86_64 brew instead of brew to do a Rosetta installation.
- ArchOversight 6y agoMake sure that Xcode is installed, and available, then these instructions work a treat: https://github.com/pyenv/pyenv/issues/1643#issuecomment-729336054 https://github.com/pyenv/pyenv/issues/1643#issuecomment-7293... For installing Python on the M1 Apple Silicon. This will install x86_64 versions, but those will work just fine.
- easton 6y agoWhat? I did "pyenv install 3.9.1" on M1 and it worked absolutely fine, it built a arm64 build and everything. Maybe it's because I had Xcode installed?
- reaperhulk 6y agoMany of the problems people appear to be having with pyenv on M1 are related to it pulling include paths for homebrew that include homebrew libffi, but not passing the library path. For cases like cffi this results in header/library mismatch and you get runtime dynamic linker errors. Other replies here appear to assume basic compilation issues, but the issues I've seen on various GitHub projects are mostly related to these more insidious C toolchain challenges. They're not M1-specific, but they are manifesting on M1 machines.
- ogre_codes 6y agoI know this is frustrating, and I expect it affects more than just Python developers, but I think the author sums it up pretty well here: > Homebrew’s Python is not for you. It exists to serve Homebrew, or more accurately, Homebrew’s other formulae. The primary purpose of Homebrew’s Python formula is to enable other Python-dependent Homebrew packages to work. If you want to develop Python (likely Ruby/ PHP/ whatever), you are best keeping your dev environment separate from the interpreters installed for the purpose of running apps managed by Home-brew (or apt/ rpm/ etc).
- woodruffw 6y agoSpeaking as a Homebrew maintainer (and separately as someone who does both Ruby and Python development locally): this is the correct answer. Homebrew's guarantees are no different from those of apt or any other system package manager; the only difference is that we don't have a stable distribution version to lock our Python or Ruby versions against. Individual developers will always be best served to isolate their development environments from the overarching runtime environment that's been distributed to them.
- justinmayer 6y agoI think it would be wonderful if this last sentence could be expanded into its own section of the "Homebrew and Python" docs. [1] [1] https://docs.brew.sh/Homebrew-and-Python.html https://docs.brew.sh/Homebrew-and-Python.html
- woodruffw 6y agoThanks, I'll base the language in the PR off of it. If you give me your GitHub handle, I'll tag you when I create the PR.
- justinmayer 6y agoCertainly — that would be great. You can find me at: http://github.com/justinmayer http://github.com/justinmayer
- justinmayer 6y agoArticle author here. I think my biggest complaint is that there’s no disclaimer on the box [1]. Homebrew folks know about these problems, but they don’t warn users that their virtual environments will be obliterated. If Homebrew explained that these problems can happen, and that developers might want to use a different source for installing Python interpreters, my ire would be lessened considerably. [1] https://docs.brew.sh/Homebrew-and-Python.html https://docs.brew.sh/Homebrew-and-Python.html
- Hamuko 6y agoI don't think the Homebrew team cares much about UX. I still don't appreciate how pretty much all build options were deprecated.
- woodruffw 6y agoWe actually care a great deal about UX! The decision to remove build options was informed (in part) by user experience: for every one user who was using them correctly to configure a package slightly differently for their uses, there were fifteen users following outdated SO answers and reporting their own breakages to us. Like every other package manager that distributes binary builds, we made a decision to improve the 99% use case.
- woodruffw 6y agoHi there! I'm one of the Homebrew maintainers, albeit not a very active one for the past year or so. I agree that we could definitely improve our documentation on what people can and should expect from a Homebrewed version of Python; I'll look into opening a PR that improves these docs tonight.
- justinmayer 6y agoThat would be really helpful. In another thread you mentioned that "individual developers will always be best served to isolate their development environments from the overarching runtime environment that's been distributed to them". I agree, but it took me way longer to figure that out than I think was necessary, precisely because the "Homebrew and Python" docs don't mention that. I imagine a lot of other developers have been similarly affected, so many thanks in advance for looking into opening a pull request to enhance the documentation.
- jonbaer 6y agoI gave up and just started using the OSX system copy (along w/ Google Colab), the main reason being that nearly every time there was a system update something would break and I would have to pin/unpin python@3.x
- CamJN 6y agoThis all stems from the official policy of only caring about themselves and putting themselves above the users. https://docs.brew.sh/Maintainers-Avoiding-Burnout#3-prioritise-maintainers-over-users https://docs.brew.sh/Maintainers-Avoiding-Burnout#3-prioriti...
- Lammy 6y agoAs someone who has suffered burnout at various points I think that policy is a great idea.
- mynameisvlad 6y agoIs this somehow a bad thing? Your comment certainly seems to imply that it is. At the end of the day, they're people volunteering their time for little to no benefit. Of course they should put themselves above their users; their users aren't even paying them for said time.
- CamJN 6y agoIt certainly is a bad thing, it has led to them making a lot of user hostile changes that have made Homebrew a much worse project than it used to be. This article describes just one example of the kind of things that are considered ok by the Homebrew maintainers.
- mynameisvlad 6y agoOnce again, they're volunteers doing unpaid work. They're not beholden to anyone's interests except their own. If you don't like their decisions, you're free to criticize them, but to imply that (rightly) putting themselves above their users is a bad thing is an entirely different beast when they don't owe anything to their users. At the end of the day, if you don't like their decisions, you're free to work towards becoming a maintainer and make your voice heard, or fork it and do whatever you want and think is right. But nobody owes you or any other user anything unless you explicitly are paying them for the time and effort.
- jrjsmrtn 6y agoYou should give MacPorts a try... MacPorts allows you to install multiple Python versions at the same time and I personally have several venvs with Python 2.7, 3.6, 3.7, 3.8, 3.9... all running without a glitch.
- bede 6y agoHomebrew and its excellent community was once indispensible on the Mac, but various pain points including this flakiness made me switch to (mini)conda for nearly everything, despite initial apprehension. There are some wrinkles and the resolver is slow, but in my hands conda is just so much more reliable than Homebrew ever was.
- lordgilman 6y agoMacPorts is installing Python just fine here, fwiw. There's a package for each Python 3.x version so you can have whatever random versions you need installed. They can be used in virtualenvs easily and they don't break when you have to install an update for a 3.x.y release.
- sirn 6y agoNot only that, MacPorts also kept all versions installed even after upgrade unless it's explicitly uninstalled (port echo inactive / port uninstall inactive); suffix every binaries with versions by default (e.g. pip-3.8, ansible-3.8) and only linked them as unversioned when explicitly set (port select --set python3 python38) which should be more familiar to people coming from Linux/BSD world. I wish more people try out MacPorts.
- hultner 6y agoHow well does macports and home brew play together on the same system these days? I used to use ports back in the days and appreciated it coming from FreeBSD previously but at some point most things I wanted were simply only available in brew, at that time brew and ports didn’t play well together on the same system so sometime around SL/Lion I switched to brew completely.
- l2dy 6y agoDefault prefix of Homebrew is now /opt/homebrew on macOS on ARM, which should not conflict with MacPorts anymore. For macOS on Intel, MacPorts does not play well with Homebrew because it took /usr/local, so ports built from source might use Homebrew files and break.
- hultner 6y agoI see so as an Intel user I'd better stay away from mac ports if I want to use software only published on brew. I probably won't buy a new computer in the coming 5 years, so arm isn't relevant for me at this moment. But good to know for the future.
- tomc1985 6y agoA postgres update seriously fucked up my work Ruby install as well. We were using an old version and homebrew updated openssl on us (from 1.0.1 to 1.1 I think?) with no way of going back. (The Ruby install I was running was linked to 1.0.1, so it was not happy with having its library deleted.) I spent a month trying to figure out how to get that shit working again. I really like Homebrew but its dependency management is horrible. The moment I saw that dyld "image not found" error message I didn't even have to finish reading the article to know what happened.
- madrox 6y agoI totally get that package management is Hard. However, I feel like Homebrew should be able to track what's there because it's a dependency vs what's there because the user wants it to be and act accordingly. Granted, I've only lived in these tools just long enough to install what I need, so I'm sure I'm naively overlooking a lot of nuance.
- montebicyclelo 6y agoI have had good experiences with Conda [1], and generally use Miniconda. [1] https://docs.conda.io/en/latest/ https://docs.conda.io/en/latest/
- pansa2 6y agoSo many different ways to install Python - doesn’t anybody just use the official installer from python.org/downloads?
- qbasic_forever 6y agoPython's insistence that it be a system-level utility in one global location, with one global set of libraries, etc. is kind of why we're in this mess of many different ways to install and use python (and the myriad of workarounds like virtual envs, pyenv, pipx, etc.). The official CPython installer is fine for some simple scripting or learning the language, but as soon as you need to tightly control and define multiple different python installs, different versions of libraries, etc. to satisfy different constraints it becomes a huge headache.
- coldtea 6y agoMeanwhile, they could have fixed this shit with 3... had time until 3.4 or so (when few were using it anyway).
- coldtea 6y agoPeople doing first-timer Python classes in school.
- maliker 6y agoI'd be perfectly happy with homebrew if it didn't auto-update all packages when installing something new. E.g. I do `brew install ffmpeg`, and as part of that it upgrades python from 3.8 to 3.9 (???), breaking imports in all my scripts. I know how to work around this, but still, surprising choice for the default behavior.
- ArchOversight 6y agoThat's what the author is talking about. The Python that comes with HomeBrew is not for you, it's only for the dependencies in the HomeBrew world.
- maliker 6y agoEverything in homebrew auto-updates on new installs, not just python. Databases, CLI tools, etc.
- mdaniel 6y ago> I know how to work around this For others similarly curious, it appears setting [`$HOMEBREW_NO_AUTO_UPDATE=1`](https://github.com/Homebrew/brew/blob/2.7.7/docs/Manpage.md#environment https://github.com/Homebrew/brew/blob/2.7.7/docs/Manpage.md#...) is the magic sauce (or the perhaps more accurate link: https://github.com/Homebrew/brew/blob/2.7.7/Library/Homebrew/env_config.rb#L201 https://github.com/Homebrew/brew/blob/2.7.7/Library/Homebrew... )
- ignoranceprior 6y agoRelevant xkcd: https://xkcd.com/1987/ https://xkcd.com/1987/
- renewiltord 6y agoAh interesting. It appears that the reason for this difference is that Python on Homebrew is delivered as a single package for development and execution. The true intention should have been to have had Python-Homebrew to create the environment for Homebrew and Python-Dev which you can use for your development. Sort of analogous to the way the JVM is distributed usually as separate packages with the JRE and the JDK. I don't write very much prod-grade Python, so I just recreate the venv with my requirements.txt instead.
- baryphonic 6y agoI'm a little surprised more people aren't using `anyenv`. That lets you install & update pyenv, nodenv, rbenv, etc. (though some support for certain languages is spotty) and thus get everything built/installed the way you want it. Also, when using pyenv, I almost always set `env PYTHON_CONFIGURE_OPTS="--enable-framework" pyenv install -v ${pylatest}` (macOS) or `env PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install -v ${pylatest}` (Linux). (I also set some CFLAGS, but these are personal preference.) I've had plenty of situations where not having the shared/framework build has screwed something up (and in fact that was my original reason for switching from Homebrew or package manager-installed Python years ago), and I've never had issues where the shared build screwed something up.
- sirn 6y agoasdf[1] mentioned in the article is similar to anyenv but instead of being a wrapper around *env, it implements its own simple framework for creating a version manager (e.g. [2]). Usually asdf plugin uses the same underlying installer as its *env equivalent (e.g. asdf-python uses python-build, asdf-ruby uses ruby-build, asdf-erlang uses kerl). [1]: https://asdf-vm.com/ https://asdf-vm.com/ [1]: https://asdf-vm.com/#/plugins-all https://asdf-vm.com/#/plugins-all
- waffletower 6y agoI will almost resist the urge to truncate the title to "Python is Not for You" -- but I'd like to second the benefits of using a python version manager such as pyenv. Yes, all the versions waste an ungodly amount of space, but you probably can't/won't use Clojure instead I guess.
- yarcob 6y agoCan someone explain why nobody seems to use the official Python installers from python.org? In my experience they work great, you can install multiple versions, and even pip3 seems to just work with them. I'm not a heavy Python user, but ever since I started using the official installers it just works.
- justinmayer 6y agoArticle author here. I think folks like asdf and pyenv because of the ease with which new versions can be installed and removed. For example, I can run asdf install python 3.9.1 and have a new version installed in minutes, without leaving the terminal console. I don't think that's possible via the official python.org installers.
- happymellon 6y agoMy experience on MacOS was that it doesn't update and cant be uninstalled without using 3rd party scripts. I could be mistaken, but I didn't find any documentation on how to do it and the fact that other people had projects to try and do it mean that I wasnt alone on finding this a bit of a mess.
- satysin 6y agoAs someone who doesn't often use Python what is the current "best" solution for installing and managing your environment? Last time I used Python on macOS and Windows I manually installed the official version from python.org and used pip and virtualenv. It wasn't too bad but it seems things have changed in the last 6 years or so? Or is this still the most flexible way to go? I would be grateful for any guidance.
- qbasic_forever 6y agoFor totally greenfield python projects check out poetry, it is an opinionated python project workflow that's very close to the nodejs/npm ecosystem's package.json. It does a virtualenv behind the scenes, pins your python dependencies with a lock file, and even has niceties like dev scripts as first class citizens. Otherwise if you're trying to use some python project with particular dependencies pyenv is a good way to get a specific python version. But if the project has a lot of native dependencies (think scientific computing with lots of fortran, C++, etc. code to compile) then you're almost certainly better off with conda as a higher-level entire binary system manager.
- justinmayer 6y agoI use asdf and its Python plugin [1] to install Python interpreter versions. For managing virtual environments on my preferred shell (Fish), I use (and maintain) VirtualFish [2]. For managing project dependencies, I activate environments via VirtualFish and then use Poetry [3] to update the dependencies within the environments. [1]: https://github.com/danhper/asdf-python https://github.com/danhper/asdf-python [2]: http://github.com/justinmayer/virtualfish http://github.com/justinmayer/virtualfish [3]: http://python-poetry.org http://python-poetry.org
- tzs 6y agoHomebrew Python recently had a problem with pasting in the REPL, because it uses GNU Readline and Homebrew recently updated that from 8.0 to 8.1. GNU Readline changed the default for bracketed paste mode from off in 8.0 to on in 8.1. Python does not explicitly set this to what it wants (off), so if you didn't happen to have "set enable-bracketed-paste off" in your ~/.inputrc, pasting got messed up. This isn't just a Homebrew thing. It also hit Arch Linux, and probably others. As part of trying to figure out who to file a bug report with, I installed Apple's Python3 and I installed the Mac version of Python3 from python.org. It worked fine in the later two, so I filed the bug with Homebrew [1] (and they figured out that it is really a Python bug and they submitted a pull request upstream to have Python explicitly turn off bracketed paste). The python.org Python3 was easy to install, and was the same version the Homebrew installed. This raises a question. The article talks about alternate systems to manage Python installations instead of using Homebrew or using Apple's Python. But why do I want any of them? The python.org install seemed fine. Why not just use that and not have to deal with any third party Python installation managers? [1] https://github.com/Homebrew/homebrew-core/issues/68193 https://github.com/Homebrew/homebrew-core/issues/68193
- happymellon 6y agoBecause the Python.org installer does not have an uninstaller or an upgrader, so you end up with files dumped in various locations on your system.
- smitty1e 6y ago>https://asdf-vm.com/ https://asdf-vm.com/ I have successfully confused people who sought meaning in 'asdf' when the string is absolutely meaningless, actually being the left-hand home keys. 'asdf' could be interpreted as "totally blew off that 'naming' thing".
- teilo 6y agoI gave up on Homebrew for Python development and now use Conda. Never looked back.