8 ms·

Homebrew project leader here: I hope you're able to find a package manager that better fits your needs and I'm sorry that Homebrew is not currently doing so. -

Homebrew project leader here: I hope you're able to find a package manager that better fits your needs and I'm sorry that Homebrew is not currently doing so. --- Homebrew upgrades dependencies and dependents of those dependencies (which, admittedly, can feel like unrelated) on installation and upgrade. As mentioned in other comments, you can customise this behaviour with `HOMEBREW_NO_INSTALL_UPGRADE` or `HOMEBREW_NO_AUTO_UPDATE`. Homebrew does this because the alternative is sometimes breaking things. An example: - you want to install `virtualenv` that depends on `python@3.10` - the binary package for `virtualenv` you want requires the newest `python@3.10` - this upgrades `python@3.10` on installation Now, Homebrew has a choice. Either we upgrade _everything_ that depends on `python@3.10` that you have installed or we knowingly break some of the things you have installed that depend on `python@3.10`. We choose the safer option by default. The more time left between updating/upgrading, the more likely to have more dependencies updated which requires more dependents to be updated. --- Regardless, I appreciate this is a problem and we're still figuring out potential solutions for this problem. A reminder that we're a volunteer run project so it's not always as easy as we'd like it to be to get these changes out quickly.

taylodl 5y ago
For the record I love Homebrew. I even use Homebrew to install regular applications whenever possible because I love how easy it is to update them. I appreciate the complexities of dependency management and yes, it sucked a bit when I installed postgresql. That isn't a problem with Homebrew though, that's just the nature of the beast. Anyway, I just wanted to say you guys are doing a great job!
mikemcquaid 5y ago
I've opened https://github.com/Homebrew/brew/pull/12413 https://github.com/Homebrew/brew/pull/12413 to improve the discoverability and documentation for all this stuff.
dcchambers 5y ago
Mike - just wanted to say thanks for all you (and the other Homebrew volunteers) do! My mac dev experience is 100x better with Homebrew - it's literally the first thing I install when I get a new machine. Perhaps in situations like this Homebrew could alert the user "Hey, this package hasn't been updated in a while and 40 other packages depend on it so this is going to take a while. Press enter to confirm or cancel and use an alternative method (HOMEBREW_NO_INSTALL_UPGRADE) to just upgrade the one package. Caveat - this may break things."
- johnboiles 5y ago
  +1 I love homebrew. I just set up my new M1 Max machine and half of the setup work was just brew commands to install cli stuff and casks <3. It's such a joy to not need to go to a bazillion websites to download dmgs
  - xchaotic 5y ago
    what a flex on your new M1 MAX ;) But yeah, homebrew should ship as default with any OS X at this point...
  - fragmede 5y ago
    did brew leaves from your old machine not hey you a transferrable list of packages?
    - katbyte 5y ago
      Every time I setup OS X I like to go through my osxsetup.sh bit by bit to see if there’s anything I didn’t use, change or update. Takes a little longer but it’s nice to take that time every couple years I go through it
    - mikemcquaid 5y ago
      ‘brew bundle dump’ does an even nicer job of this. Check it out!
      - deleted 5y ago
        [deleted]
- bin_bash 5y ago
  It's not as simple as you think: consider headless setups.
  - FranksTV 5y ago
    Generally programs can figure out whether they should run interactively or not.
oldtablet 5y ago
Wondering if this would be solved by using brew pin? http://www.valentinourbano.com/brew-pin.html http://www.valentinourbano.com/brew-pin.html
beepbooptheory 5y ago
yeah really unfair to blame a package manager for being unexpected when the context is installing/updating a python package. Don't care what package manager you use, python is always going to be a mess
- ethbr0 5y ago
  > python is always going to be a mess The irony is that poster was trying to install a solution to this very problem (virtualenv). The broader problem is that, at this point, python is effectively its own OS. Coupled with macOS being schizophrenic between "We want to provide a posix environment for users" and "No user should ever look under the hood, so no need to tell them about all the duct taped bits." Which brew attempts to bridge. Admirably, but never perfectly. Obligatory: https://xkcd.com/1987/ https://xkcd.com/1987/
  - matthewmc3 5y ago
    The bigger irony is that Python has had its own venv built in since 3.3. I haven’t needed virtualenv for years.
  - pxc 5y ago
    No, the problem is that Python's packaging system (like Homebrew's!) was created without adequate reference to prior art, has undergone much more evolution than design, and has suffered in every attempt to improve it from the fact of its adoption by a large userbase which democratically manages it so that changes must be conservative. As a result, Python packaging has an utterly insane design that it can't evolve its way out of. Problems that have proven more or less intractable for resolution by evolutionary change: - total lack of static metadata— you must download a source package and attempt to build it in order to determine what its actual dependencies are - relatedly, package setup scripts are way too powerful and can do literally anything, which makes automating Python packaging a nightmare - any possible vendorization is deeply limited by the fact that Python processes can't include multiple versions of a single library - the behavior of all Python packaging tools relies on the implicit state of PyPI, forcing a high degree of non-determinism unless you go to great lengths to take snapshots of PyPI - native dependencies (i.e., dependencies on C libraries) are either just totally undeclared, or packaged abuse setup.py in order to manually compile them in an ad-hoc, unmanaged fashion There's no way out of this that doesn't involve breaking things for Python developers who are, whether out of shortsightedness or resigned pragmatism, relying on or working around the many, various bad practices that the Python packaging 'system' currently allows. This is why programmatically generating Python packages for general-purpose package managers like Homebrew ranges (depending on the resources made available by the package manager's design) ranges from impossible to stupidly complex and computationally expensive to kinda works but has no hope of reliability/correctness/completeness.
pchm 5y ago
OP here, thanks for clarifying this - I can imagine it's a super complex problem and I truly appreciate all the work that you (and other contributors) have put into this project. My only point (and a source of confusion) here is that I never anticipated the cascading effect that installing a single package can have on seemingly unrelated parts of the system. I've been using homebrew for many years and as far as I remember, the only time anything broke like this was when I explicitly ran update/upgrade. I'm going to try HOMEBREW_NO_INSTALL_UPGRADE/HOMEBREW_NO_AUTO_UPDATE — it does seem like a more intuitive default, but again, I don't know anything about how homebrew works under the hood. Thanks again!
- RNCTX 5y ago
  There's more than one way to skin a cat. Personally, I have moved to downloading large things (Python, Postgres by your example) as standalone apps, and only use Homebrew for small libraries and CLI tools. I think that makes more sense. You can download all of the Mac Python versions you want from python.org and you can get Postgres from postgresapp.com which provides a nice toolbar interface to start and shutdown servers of various versions.
beowulfey 5y ago
First, wanted to say thanks for heading a wonderful project. I personally love Homebrew! What’s the difference in what Homebrew is doing from what normally happens with something like Pacman or Apt? My impression is that “core” binaries, like Python might be, don’t get updated as often as they would on Homebrew, and perhaps those dependency chains are less frequent. Is it because (e.g.) python versions change rarely on linux package managers? Or something about how the binaries are compiled?
- pxc 5y ago
  `brew` lives across (at least) one fundamental divide from `apt`. Homebrew is a source-based package manager, where packages are built from a single shared source of build recipes, and distribution of software happens by distributing the whole collection of recipes. Support for a binary cache is added through the identification of build recipes with the hashes of archives of binaries. APT is what's called a binary package management system. While build recipes can be stored in a central repository, they don't have to be, as binary package management systems typically identify two kinds of packages: source packages and binary packages, which can be distributed separately. (Some repositories may not even include source packages.) Source packages contain metadata like build recipes, but things like archives of the upstream source code, patches, and sometimes helper scripts that might be usd in building. Binary packages include the actual built artifacts; they're installable packages. In the source-based package management world, if packages can be added to the main collection through external sources, this is typically done by pretending that those packages were embedded in (some particular version of) the main repository of build recipes. Collections of packages that are ‘injected’ into the main package collection like this are typically called ‘overlays’. In binary package management systems, an external package is not expected to have access to some copy of a complete collection of build recipes from some ‘master’ repository. Instead, source packages are distributed individually using the same mechanisms as binary packages. This means that the global namespace of terms which refer to packages can't be navigated implicitly at build time like on source-based systems, the identification of actual source code and build instructions with some package name happens both at build time and at install time. And in both cases, explicit metadata (package names, package versions, equivalency classes, lists of conflicting packages, etc.) is used to determine what versions of dependencies get pulled in. As a result, binary package management systems always include dependency solving at install time, and usually include some explicit version constraint for every single dependency declared in a package's build recipe. This means they have the resources to look at what's going to be installed, what's already installed, and say ‘oh, the new installee requires Python <= 3.10.4, and since your installed packages together only need Python >= 3.10.2 and you already have python-3.10.3 installed, we don't have to touch Python to install this new package unless you ask us to’. Pacman is somewhat atypical. The build system is essentially source-based and monolithic (there's no source package format) and the AUR is essentially an overlay, but the high-level CLI tool acts like a binary packager and uses the language of repositories to describe dealing with multiple package sources. Pacman's dependency resolution behavior is also unusually unreliable, and probably closer to what `brew` does than what `apt` does. Pacman has the resources to describe and test version constraints, both in its build system and in its CLI tools, but they're underutilized in a way that's more typical of source-based package management systems than binary ones. Homebrew is similar in that such constraints are radically underutilized (they're totally absent). The main difference between Homebrew's behavior and Pacman's in this instance is that by default, just `pacman -S` doesn't update pacman's collection of package definitions (pacman -Sy does) and Homebrew always tries to install from the latest build recipes. I imagine that Pacman's capacity for handling version constraints means that the story with `pacman` will usually be a bit better than with `brew`, but that otherwise `brew install` will be similar to `pacman -Sy` and `HOMEBREW_NO_INSTALL_UPGRADE=1 brew install` will be similar to `pacman -S`. (Pacman also sometimes makes the opposite tradeoff that Howell describes for Homebrew sometimes; it's not unheard of for installing new software or updating software using Pacman to break installed software.)
fredsted 5y ago
On Debian-based systems, I get a preview of what's going to be installed/updated when I do something with `apt`. With homebrew, I never know how many packages will be updated and how long it will take, so sometimes when I just need a small utility, suddenly it'll update a bunch of packages and it takes 45 minutes. It's a UI problem.
- kristjansson 5y ago
  An apt-style 'hey here's what you going to do, confirm/deny' would be _great_ in homebrew
  - mikemcquaid 5y ago
    Not sure how to add that prompt by default without breaking existing scripts. Making it non-default seems like it wouldn’t help (given comments here).
    - Fire-Dragon-DoL 5y ago
      The prompt should appear only in interactive mode, so it shouldn't break existing scripts
    - kristjansson 5y ago
      Fair enough. Like the sibling says, checking the moral equivalent of [ -t 0 ] probably distinguishes enough interactive vs non-interactive use to add an interactive-only default. I'd add that script-y usage of brew already can block on user input (e.g. if a cask asks for sudo), so adding a go/no-go check probably only breaks the most determined automators, while the benefits accrue to a wide swath of users (both new and fat-fingered). Regardless, thanks for all the work you (and the community!) do on Homebrew! I wouldn't like macOS half as much (or have bought half as many) without it.
      - mikemcquaid 5y ago
        Yeh, that’s the interactive check we’ve used in the past and unfortunately it still has given false positives and false negatives enough I’m hesitant to make a change this big due to it.
systemvoltage 5y ago
Been using homebrew for many years, thanks. Although, I think the permissions model is broken. I wish Homebrew supported multiple accounts. On my macbook m1, all accounts point to /opt/homebrew and it is a giant ball of mess. Modifying the prefix to ~/homebrew leads to other packages that hard code the path such as pyenv. I think we need a new package manager since its not the fault of homebrew that many packages have hardcoded stuff in there.
EVa5I7bHFq9mnYK 5y ago
Can't you keep both python 3.10 and python 3.09, for older apps that rely on it? Sure, not optimal, but hard disk space is cheap these days .. Ideally, each app would list hashes of packages it depends on, so that everyone is happy.
- MontyCarloHall 5y ago
  What about applications that just reference “python” (or any other dependency), without specifying a particular version? AFAIK homebrew cannot automatically modify $PATH before starting a program such that $PATH points to the correct dependencies for that program. If you have multiple versions of some dependency installed, it is up to you to run `brew switch` before running your program to manually set the correct versions for each dependency.
karmajunkie 5y ago
Just wanted to echo the appreciation for everything you and other volunteers do on this project. Using a Brewfile has completely changed my setup process for a new machine, taking me from weeks of tinkering and going to find some tool I'd forgotten I use once every month or two down to a couple hours of downloading packages automatically. There may be a few rough edges here and there, but homebrew is one of the few truly essential tools for development on a Mac today IMHO.
cpach 5y ago
FWIW: Very happy Homebrew camper here, thanks a lot for all effort you’re putting in. At least for my tastes, Homebrew is a very nice and convenient package manager.
willseth 5y ago
I appreciate all your hard work and I like Homebrew, but I think updating high-level packages when a descendent dependency is updated is the wrong behavior - at least in this example. If Postgres has an unversioned dependency to PackageX, the only time I might expect Postgres to be updated automatically is if I update PackageX across a major version boundary, risking breaking changes. Even then, I'm not sure I'd want that behavior by default - I'd at least like to be prompted - since often this still won't break anything, or it will likely cause less damage than updating every package with a dependency to PackageX. And if Postgres breaks after I update PackageX I can easily reason about what just happened. The alternative can be incredibly hard to troubleshoot because so many changes have occurred. Overall, Homebrew feels far too aggressive here, and I think the result, for people who have some understanding of Homebrew behavior, is that they're afraid to touch anything.
- pxc 5y ago
  > If Postgres has an unversioned dependency to PackageX, the only time I might expect Postgres to be updated automatically is if I update PackageX across a major version boundary, risking breaking changes. Homebrew doesn't meaningfully have versioned or unversioned dependencies, because it doesn't incorporate any notion of version into its dependency solver. When you declare a dependency on `python@3.9`, you're not telling `brew` that ‘I need a copy of the package `python` which satisfies the constraint “version is 3.9.x”.’ Homebrew just treats `python@3.9` as the name of a package which is a whole copy of python, and it can exist alongside other, similar packages like `python@3.8` and `python@3.10`. Homebrew only incorporates this kind of `@` syntax for select packages of which maintainers have elected to maintain multiple versions. By my count, there are 81 such packages in homebrew-core, and the following is a complete list of all of those which appear in a `depends_on` declaration: antlr atkmm autoconf cairomm db elasticsearch erlang gcc ghc glibmm go gradle guile hdf5 headers helm isl libfuse llvm lua mbedtls mysql node numpy openexr openjdk openssl pangomm php proj pyqt pyside python qt ruby swig synth tbb vtk wxwidgets So when you look at a Postgres formula in the Homebrew repos and see that it declares an ‘unversioned’ dependency on openldap, for example[1], that does not mean that the formula author has declared ‘any version is fine’, i.e., it does not mean that the dependency is unversioned. It just means that the dependency's version constraints are totally unspecified except implicitly as ‘the only version of that package which is simultaneously present in the glorious monorepo of build recipes’. This is a feature it shares with some source-based package management systems, like Nix and Guix. Those package managers solve this problem by basically vendorizing everything, so installing a new package never upgrades anything else you have installed. Other source-based package management systems, like Portage and Pkgsrc, have extended their model to include some handling of version constraints, but they also sometimes declare dependencies without specifying a version. That seems dubious to me, since in reality there pretty much always are version constraints that ought to be specified there, but maybe those two projects are mature enough that they've hammered out the problematic cases by now. — https://github.com/Homebrew/homebrew-core/blob/master/Formula/postgresql@13.rb https://github.com/Homebrew/homebrew-core/blob/master/Formul...
pxc 5y ago
I didn't know before this post that Homebrew's infamous ‘auto-update’ only automatically updated the dependency closure of whatever you were installing/upgrading, which sounds more or less correct to me. > Now, Homebrew has a choice. Either we upgrade _everything_ that depends on `python@3.10` that you have installed or we knowingly break some of the things you have installed that depend on `python@3.10`. We choose the safer option by default. A look at the formulae in homebrew-core shows that Homebrew basically doesn't consider versions in its dependency management at all, so formula are just using the ‘@x.y.z’ in place of all manner of version constraints. So you get wild stuff like this line here[1]: depends_on "curl" # curl >= 7.55 is required wat. Of course the recursive update policy is going to be weirdly painful for some users! Homebrew doesn't even attempt to encode some very basic aspects of dependency, so now (a) you can't know what you're going to break if you do upgrade a common dependency and (b) you can't know when you can afford not to upgrade a dependency shared by a package you're about to install and the already-installed system. The choice you outline above is one that is not faced by most package managers, because they don't make this mistake. The naive wheel reinvention with Homebrew is so eternally disappointing, and it explains a lot of the pain users experience with it. > Homebrew upgrades dependencies and dependents of those dependencies (which, admittedly, can feel like unrelated) One relatively non-disruptive thing you might be able to do to make this behavior less surprising to users is (offer a way to?) print the dependency tree for package installations/upgrades that pull in upgrades of their ‘siblings’. You'd probably want to just do it in a textual way, but this project seems to model the kind of logic you'd want for printing dependency trees with Homebrew as it exists.[2] — 1: https://github.com/Homebrew/homebrew-core/blob/master/Formula/saldl.rb#L21 https://github.com/Homebrew/homebrew-core/blob/master/Formul... 2: https://github.com/martido/homebrew-graph/blob/master/cmd/brew-graph.rb https://github.com/martido/homebrew-graph/blob/master/cmd/br...
- mikemcquaid 5y ago
  That “wat” is because we use the system ‘curl’ when we can here. The comment is stating why we can’t in this case. Not ideal, I’ll admit.
_benj 5y ago
this is why I love HN! Thanks for all the work you put in homebrew, I really appreciate it. Just upgraded to Monte Rey a few days ago and I was kind of fearful that some of the things that I needed where not going to work... nop, every singe package that I've needed so far was installed successfully. Thanks!
saghm 5y ago
I don't use homebrew too much (I'm usually on Linux, and when I am on macos, my development environment tends to be pretty lightweight), but based on my experience with Linux package managers, I wonder if this is mostly a pain point due to the performance issues brew sometimes has. On Linux, I tend check for upgrades fairly often (at least once a day, and always right before trying to install something), so I don't run into this issue much there. However, with my Linux package manager, checking for updates is always really fast (maybe a second or two at most unless I'm on very spotty wifi), whereas sometimes trying to do the same with brew takes significantly longer (and that's before even applying the updates). It seems plausible that users who are frustrated with slow updates start updating less often, which then causes updates to take even longer the next time they do, until eventually they have an experience like OP where they try to install one thing and it takes 45 minutes. I imagine there isn't an easy solution if this is in fact the core issue, but I think it's worth considering that the core issue isn't actually the update strategy you describe, as that's fairly similar to what I've seen with Linux package managers, and I can't remember ever seeing any complaints like OP's about any of them.
smsm42 5y ago
Thank you for all your work. Homebrew is one of the most useful tools in macos development ecosystem, at least for me. You're doing an awesome job. That said, I did notice recently the same effects as the OP. I.e. I try to install something small and it depends on something like python, which triggers "upgrade the whole universe" loop, which breaks a ton of things (e.g. python upgrade doesn't carry over installed modules, but does change current version - which breaks many local python scripts since their dependency modules are now in the old version). Auto-cleanup tends to break a lot of things too - especially bad for commonly-used things like openssl or ICU which commonly are linked as shared libs. I think there are settings for changing all of this, but you only find out once you've already been bit. Maybe it's worth considering making it somehow explicit - e.g. ask the user "do you really want to upgrade the whole universe" before going in? In many cases, I'm completely fine with using python 3.7 even if python 3.10 is already out. And most of the tools that depend on python would be fine with it too. It would be nicer if low-friction mode would be the default.
musicale 5y ago
> HOMEBREW_NO_AUTO_UPDATE My life became much better after setting this. I wouldn't be averse to a HOMEBREW_AUTO_UPDATE_INTERVAL=180d either, or HOMEBREW_AUTO_UPDATE_ASK=y .
chrisgoman 5y ago
homebrew is the BEST!

taylodl 5y ago

For the record I love Homebrew. I even use Homebrew to install regular applications whenever possible because I love how easy it is to update them. I appreciate the complexities of dependency management and yes, it sucked a bit when I installed postgresql. That isn't a problem with Homebrew though, that's just the nature of the beast. Anyway, I just wanted to say you guys are doing a great job!

mikemcquaid 5y ago

I've opened https://github.com/Homebrew/brew/pull/12413 https://github.com/Homebrew/brew/pull/12413 to improve the discoverability and documentation for all this stuff.

dcchambers 5y ago

Mike - just wanted to say thanks for all you (and the other Homebrew volunteers) do! My mac dev experience is 100x better with Homebrew - it's literally the first thing I install when I get a new machine. Perhaps in situations like this Homebrew could alert the user "Hey, this package hasn't been updated in a while and 40 other packages depend on it so this is going to take a while. Press enter to confirm or cancel and use an alternative method (HOMEBREW_NO_INSTALL_UPGRADE) to just upgrade the one package. Caveat - this may break things."

johnboiles 5y ago
+1 I love homebrew. I just set up my new M1 Max machine and half of the setup work was just brew commands to install cli stuff and casks <3. It's such a joy to not need to go to a bazillion websites to download dmgs
- xchaotic 5y ago
  what a flex on your new M1 MAX ;) But yeah, homebrew should ship as default with any OS X at this point...
- fragmede 5y ago
  did brew leaves from your old machine not hey you a transferrable list of packages?
  - katbyte 5y ago
    Every time I setup OS X I like to go through my osxsetup.sh bit by bit to see if there’s anything I didn’t use, change or update. Takes a little longer but it’s nice to take that time every couple years I go through it
  - mikemcquaid 5y ago
    ‘brew bundle dump’ does an even nicer job of this. Check it out!
    - deleted 5y ago
      [deleted]
bin_bash 5y ago
It's not as simple as you think: consider headless setups.
- FranksTV 5y ago
  Generally programs can figure out whether they should run interactively or not.

oldtablet 5y ago

Wondering if this would be solved by using brew pin? http://www.valentinourbano.com/brew-pin.html http://www.valentinourbano.com/brew-pin.html

beepbooptheory 5y ago

yeah really unfair to blame a package manager for being unexpected when the context is installing/updating a python package. Don't care what package manager you use, python is always going to be a mess

ethbr0 5y ago
> python is always going to be a mess The irony is that poster was trying to install a solution to this very problem (virtualenv). The broader problem is that, at this point, python is effectively its own OS. Coupled with macOS being schizophrenic between "We want to provide a posix environment for users" and "No user should ever look under the hood, so no need to tell them about all the duct taped bits." Which brew attempts to bridge. Admirably, but never perfectly. Obligatory: https://xkcd.com/1987/ https://xkcd.com/1987/
- matthewmc3 5y ago
  The bigger irony is that Python has had its own venv built in since 3.3. I haven’t needed virtualenv for years.
- pxc 5y ago
  No, the problem is that Python's packaging system (like Homebrew's!) was created without adequate reference to prior art, has undergone much more evolution than design, and has suffered in every attempt to improve it from the fact of its adoption by a large userbase which democratically manages it so that changes must be conservative. As a result, Python packaging has an utterly insane design that it can't evolve its way out of. Problems that have proven more or less intractable for resolution by evolutionary change: - total lack of static metadata— you must download a source package and attempt to build it in order to determine what its actual dependencies are - relatedly, package setup scripts are way too powerful and can do literally anything, which makes automating Python packaging a nightmare - any possible vendorization is deeply limited by the fact that Python processes can't include multiple versions of a single library - the behavior of all Python packaging tools relies on the implicit state of PyPI, forcing a high degree of non-determinism unless you go to great lengths to take snapshots of PyPI - native dependencies (i.e., dependencies on C libraries) are either just totally undeclared, or packaged abuse setup.py in order to manually compile them in an ad-hoc, unmanaged fashion There's no way out of this that doesn't involve breaking things for Python developers who are, whether out of shortsightedness or resigned pragmatism, relying on or working around the many, various bad practices that the Python packaging 'system' currently allows. This is why programmatically generating Python packages for general-purpose package managers like Homebrew ranges (depending on the resources made available by the package manager's design) ranges from impossible to stupidly complex and computationally expensive to kinda works but has no hope of reliability/correctness/completeness.

pchm 5y ago

OP here, thanks for clarifying this - I can imagine it's a super complex problem and I truly appreciate all the work that you (and other contributors) have put into this project. My only point (and a source of confusion) here is that I never anticipated the cascading effect that installing a single package can have on seemingly unrelated parts of the system. I've been using homebrew for many years and as far as I remember, the only time anything broke like this was when I explicitly ran update/upgrade. I'm going to try HOMEBREW_NO_INSTALL_UPGRADE/HOMEBREW_NO_AUTO_UPDATE — it does seem like a more intuitive default, but again, I don't know anything about how homebrew works under the hood. Thanks again!

RNCTX 5y ago
There's more than one way to skin a cat. Personally, I have moved to downloading large things (Python, Postgres by your example) as standalone apps, and only use Homebrew for small libraries and CLI tools. I think that makes more sense. You can download all of the Mac Python versions you want from python.org and you can get Postgres from postgresapp.com which provides a nice toolbar interface to start and shutdown servers of various versions.

beowulfey 5y ago

First, wanted to say thanks for heading a wonderful project. I personally love Homebrew! What’s the difference in what Homebrew is doing from what normally happens with something like Pacman or Apt? My impression is that “core” binaries, like Python might be, don’t get updated as often as they would on Homebrew, and perhaps those dependency chains are less frequent. Is it because (e.g.) python versions change rarely on linux package managers? Or something about how the binaries are compiled?

pxc 5y ago
`brew` lives across (at least) one fundamental divide from `apt`. Homebrew is a source-based package manager, where packages are built from a single shared source of build recipes, and distribution of software happens by distributing the whole collection of recipes. Support for a binary cache is added through the identification of build recipes with the hashes of archives of binaries. APT is what's called a binary package management system. While build recipes can be stored in a central repository, they don't have to be, as binary package management systems typically identify two kinds of packages: source packages and binary packages, which can be distributed separately. (Some repositories may not even include source packages.) Source packages contain metadata like build recipes, but things like archives of the upstream source code, patches, and sometimes helper scripts that might be usd in building. Binary packages include the actual built artifacts; they're installable packages. In the source-based package management world, if packages can be added to the main collection through external sources, this is typically done by pretending that those packages were embedded in (some particular version of) the main repository of build recipes. Collections of packages that are ‘injected’ into the main package collection like this are typically called ‘overlays’. In binary package management systems, an external package is not expected to have access to some copy of a complete collection of build recipes from some ‘master’ repository. Instead, source packages are distributed individually using the same mechanisms as binary packages. This means that the global namespace of terms which refer to packages can't be navigated implicitly at build time like on source-based systems, the identification of actual source code and build instructions with some package name happens both at build time and at install time. And in both cases, explicit metadata (package names, package versions, equivalency classes, lists of conflicting packages, etc.) is used to determine what versions of dependencies get pulled in. As a result, binary package management systems always include dependency solving at install time, and usually include some explicit version constraint for every single dependency declared in a package's build recipe. This means they have the resources to look at what's going to be installed, what's already installed, and say ‘oh, the new installee requires Python <= 3.10.4, and since your installed packages together only need Python >= 3.10.2 and you already have python-3.10.3 installed, we don't have to touch Python to install this new package unless you ask us to’. Pacman is somewhat atypical. The build system is essentially source-based and monolithic (there's no source package format) and the AUR is essentially an overlay, but the high-level CLI tool acts like a binary packager and uses the language of repositories to describe dealing with multiple package sources. Pacman's dependency resolution behavior is also unusually unreliable, and probably closer to what `brew` does than what `apt` does. Pacman has the resources to describe and test version constraints, both in its build system and in its CLI tools, but they're underutilized in a way that's more typical of source-based package management systems than binary ones. Homebrew is similar in that such constraints are radically underutilized (they're totally absent). The main difference between Homebrew's behavior and Pacman's in this instance is that by default, just `pacman -S` doesn't update pacman's collection of package definitions (pacman -Sy does) and Homebrew always tries to install from the latest build recipes. I imagine that Pacman's capacity for handling version constraints means that the story with `pacman` will usually be a bit better than with `brew`, but that otherwise `brew install` will be similar to `pacman -Sy` and `HOMEBREW_NO_INSTALL_UPGRADE=1 brew install` will be similar to `pacman -S`. (Pacman also sometimes makes the opposite tradeoff that Howell describes for Homebrew sometimes; it's not unheard of for installing new software or updating software using Pacman to break installed software.)

fredsted 5y ago

On Debian-based systems, I get a preview of what's going to be installed/updated when I do something with `apt`. With homebrew, I never know how many packages will be updated and how long it will take, so sometimes when I just need a small utility, suddenly it'll update a bunch of packages and it takes 45 minutes. It's a UI problem.

kristjansson 5y ago
An apt-style 'hey here's what you going to do, confirm/deny' would be _great_ in homebrew
- mikemcquaid 5y ago
  Not sure how to add that prompt by default without breaking existing scripts. Making it non-default seems like it wouldn’t help (given comments here).
  - Fire-Dragon-DoL 5y ago
    The prompt should appear only in interactive mode, so it shouldn't break existing scripts
  - kristjansson 5y ago
    Fair enough. Like the sibling says, checking the moral equivalent of [ -t 0 ] probably distinguishes enough interactive vs non-interactive use to add an interactive-only default. I'd add that script-y usage of brew already can block on user input (e.g. if a cask asks for sudo), so adding a go/no-go check probably only breaks the most determined automators, while the benefits accrue to a wide swath of users (both new and fat-fingered). Regardless, thanks for all the work you (and the community!) do on Homebrew! I wouldn't like macOS half as much (or have bought half as many) without it.
    - mikemcquaid 5y ago
      Yeh, that’s the interactive check we’ve used in the past and unfortunately it still has given false positives and false negatives enough I’m hesitant to make a change this big due to it.

systemvoltage 5y ago

Been using homebrew for many years, thanks. Although, I think the permissions model is broken. I wish Homebrew supported multiple accounts. On my macbook m1, all accounts point to /opt/homebrew and it is a giant ball of mess. Modifying the prefix to ~/homebrew leads to other packages that hard code the path such as pyenv. I think we need a new package manager since its not the fault of homebrew that many packages have hardcoded stuff in there.

EVa5I7bHFq9mnYK 5y ago

Can't you keep both python 3.10 and python 3.09, for older apps that rely on it? Sure, not optimal, but hard disk space is cheap these days .. Ideally, each app would list hashes of packages it depends on, so that everyone is happy.

MontyCarloHall 5y ago
What about applications that just reference “python” (or any other dependency), without specifying a particular version? AFAIK homebrew cannot automatically modify $PATH before starting a program such that $PATH points to the correct dependencies for that program. If you have multiple versions of some dependency installed, it is up to you to run `brew switch` before running your program to manually set the correct versions for each dependency.

karmajunkie 5y ago

Just wanted to echo the appreciation for everything you and other volunteers do on this project. Using a Brewfile has completely changed my setup process for a new machine, taking me from weeks of tinkering and going to find some tool I'd forgotten I use once every month or two down to a couple hours of downloading packages automatically. There may be a few rough edges here and there, but homebrew is one of the few truly essential tools for development on a Mac today IMHO.

cpach 5y ago

FWIW: Very happy Homebrew camper here, thanks a lot for all effort you’re putting in. At least for my tastes, Homebrew is a very nice and convenient package manager.

willseth 5y ago

I appreciate all your hard work and I like Homebrew, but I think updating high-level packages when a descendent dependency is updated is the wrong behavior - at least in this example. If Postgres has an unversioned dependency to PackageX, the only time I might expect Postgres to be updated automatically is if I update PackageX across a major version boundary, risking breaking changes. Even then, I'm not sure I'd want that behavior by default - I'd at least like to be prompted - since often this still won't break anything, or it will likely cause less damage than updating every package with a dependency to PackageX. And if Postgres breaks after I update PackageX I can easily reason about what just happened. The alternative can be incredibly hard to troubleshoot because so many changes have occurred. Overall, Homebrew feels far too aggressive here, and I think the result, for people who have some understanding of Homebrew behavior, is that they're afraid to touch anything.

pxc 5y ago
> If Postgres has an unversioned dependency to PackageX, the only time I might expect Postgres to be updated automatically is if I update PackageX across a major version boundary, risking breaking changes. Homebrew doesn't meaningfully have versioned or unversioned dependencies, because it doesn't incorporate any notion of version into its dependency solver. When you declare a dependency on `python@3.9`, you're not telling `brew` that ‘I need a copy of the package `python` which satisfies the constraint “version is 3.9.x”.’ Homebrew just treats `python@3.9` as the name of a package which is a whole copy of python, and it can exist alongside other, similar packages like `python@3.8` and `python@3.10`. Homebrew only incorporates this kind of `@` syntax for select packages of which maintainers have elected to maintain multiple versions. By my count, there are 81 such packages in homebrew-core, and the following is a complete list of all of those which appear in a `depends_on` declaration: antlr atkmm autoconf cairomm db elasticsearch erlang gcc ghc glibmm go gradle guile hdf5 headers helm isl libfuse llvm lua mbedtls mysql node numpy openexr openjdk openssl pangomm php proj pyqt pyside python qt ruby swig synth tbb vtk wxwidgets So when you look at a Postgres formula in the Homebrew repos and see that it declares an ‘unversioned’ dependency on openldap, for example[1], that does not mean that the formula author has declared ‘any version is fine’, i.e., it does not mean that the dependency is unversioned. It just means that the dependency's version constraints are totally unspecified except implicitly as ‘the only version of that package which is simultaneously present in the glorious monorepo of build recipes’. This is a feature it shares with some source-based package management systems, like Nix and Guix. Those package managers solve this problem by basically vendorizing everything, so installing a new package never upgrades anything else you have installed. Other source-based package management systems, like Portage and Pkgsrc, have extended their model to include some handling of version constraints, but they also sometimes declare dependencies without specifying a version. That seems dubious to me, since in reality there pretty much always are version constraints that ought to be specified there, but maybe those two projects are mature enough that they've hammered out the problematic cases by now. — https://github.com/Homebrew/homebrew-core/blob/master/Formula/postgresql@13.rb https://github.com/Homebrew/homebrew-core/blob/master/Formul...

pxc 5y ago

I didn't know before this post that Homebrew's infamous ‘auto-update’ only automatically updated the dependency closure of whatever you were installing/upgrading, which sounds more or less correct to me. > Now, Homebrew has a choice. Either we upgrade _everything_ that depends on `python@3.10` that you have installed or we knowingly break some of the things you have installed that depend on `python@3.10`. We choose the safer option by default. A look at the formulae in homebrew-core shows that Homebrew basically doesn't consider versions in its dependency management at all, so formula are just using the ‘@x.y.z’ in place of all manner of version constraints. So you get wild stuff like this line here[1]: depends_on "curl" # curl >= 7.55 is required wat. Of course the recursive update policy is going to be weirdly painful for some users! Homebrew doesn't even attempt to encode some very basic aspects of dependency, so now (a) you can't know what you're going to break if you do upgrade a common dependency and (b) you can't know when you can afford not to upgrade a dependency shared by a package you're about to install and the already-installed system. The choice you outline above is one that is not faced by most package managers, because they don't make this mistake. The naive wheel reinvention with Homebrew is so eternally disappointing, and it explains a lot of the pain users experience with it. > Homebrew upgrades dependencies and dependents of those dependencies (which, admittedly, can feel like unrelated) One relatively non-disruptive thing you might be able to do to make this behavior less surprising to users is (offer a way to?) print the dependency tree for package installations/upgrades that pull in upgrades of their ‘siblings’. You'd probably want to just do it in a textual way, but this project seems to model the kind of logic you'd want for printing dependency trees with Homebrew as it exists.[2] — 1: https://github.com/Homebrew/homebrew-core/blob/master/Formula/saldl.rb#L21 https://github.com/Homebrew/homebrew-core/blob/master/Formul... 2: https://github.com/martido/homebrew-graph/blob/master/cmd/brew-graph.rb https://github.com/martido/homebrew-graph/blob/master/cmd/br...

mikemcquaid 5y ago
That “wat” is because we use the system ‘curl’ when we can here. The comment is stating why we can’t in this case. Not ideal, I’ll admit.

_benj 5y ago

this is why I love HN! Thanks for all the work you put in homebrew, I really appreciate it. Just upgraded to Monte Rey a few days ago and I was kind of fearful that some of the things that I needed where not going to work... nop, every singe package that I've needed so far was installed successfully. Thanks!

saghm 5y ago

I don't use homebrew too much (I'm usually on Linux, and when I am on macos, my development environment tends to be pretty lightweight), but based on my experience with Linux package managers, I wonder if this is mostly a pain point due to the performance issues brew sometimes has. On Linux, I tend check for upgrades fairly often (at least once a day, and always right before trying to install something), so I don't run into this issue much there. However, with my Linux package manager, checking for updates is always really fast (maybe a second or two at most unless I'm on very spotty wifi), whereas sometimes trying to do the same with brew takes significantly longer (and that's before even applying the updates). It seems plausible that users who are frustrated with slow updates start updating less often, which then causes updates to take even longer the next time they do, until eventually they have an experience like OP where they try to install one thing and it takes 45 minutes. I imagine there isn't an easy solution if this is in fact the core issue, but I think it's worth considering that the core issue isn't actually the update strategy you describe, as that's fairly similar to what I've seen with Linux package managers, and I can't remember ever seeing any complaints like OP's about any of them.

smsm42 5y ago

Thank you for all your work. Homebrew is one of the most useful tools in macos development ecosystem, at least for me. You're doing an awesome job. That said, I did notice recently the same effects as the OP. I.e. I try to install something small and it depends on something like python, which triggers "upgrade the whole universe" loop, which breaks a ton of things (e.g. python upgrade doesn't carry over installed modules, but does change current version - which breaks many local python scripts since their dependency modules are now in the old version). Auto-cleanup tends to break a lot of things too - especially bad for commonly-used things like openssl or ICU which commonly are linked as shared libs. I think there are settings for changing all of this, but you only find out once you've already been bit. Maybe it's worth considering making it somehow explicit - e.g. ask the user "do you really want to upgrade the whole universe" before going in? In many cases, I'm completely fine with using python 3.7 even if python 3.10 is already out. And most of the tools that depend on python would be fine with it too. It would be nicer if low-friction mode would be the default.

musicale 5y ago

> HOMEBREW_NO_AUTO_UPDATE My life became much better after setting this. I wouldn't be averse to a HOMEBREW_AUTO_UPDATE_INTERVAL=180d either, or HOMEBREW_AUTO_UPDATE_ASK=y .

chrisgoman 5y ago

homebrew is the BEST!