4 ms·

With my Homebrew hat on, but not speaking for others: I think this is pretty cool, and demonstrates something that we've discussed indirectly for years. At its

With my Homebrew hat on, but not speaking for others: I think this is pretty cool, and demonstrates something that we've discussed indirectly for years. At its core, there are really two parts to Homebrew: 1. There's the client side, i.e. `brew`, which 99.9% of users stick to happy paths (bottle installs, supported platforms) within. These users could be supported with relative ease by a small native-code installer, since the majority of the work done by the installer in the happy path is fetching bottles, exploding them, and doing a bit of relocation. 2. There's literally everything else, i.e. all of the developer, repository, and CI/CD machinery that keeps homebrew-core humming. This is the largely invisible infrastructure that makes `brew install` work smoothly, and it's very hard to RIIR (in a large part because it's tied heavily to the formula DSL, which is arbitrary Ruby). (1) is a nice experimental space, because Homebrew does (IMO) a decent job of isolating the client-facing side from the complexity of (2). However, (2) is where the meat-and-potatoes of packaging happens, and where Homebrew's differentiators really lie (specifically, in how easy it is to contribute new packages and bump existing ones). Edit: Another noteworthy aspect here around performance: I mentioned this in another comment[1], but parallel downloads of things like bottles and DMGs is not an architectural limitation of Homebrew itself, but instead a conscious decision to trade some install speed for courtesy towards the services we fetch from (including GitHub itself). Smaller projects can sidestep this because they're not directing nearly the same degree of volume; I think this project will discover if/when its volumes grow that it will need to serialize downloads to avoid being throttled or outright limited. [1]: https://news.ycombinator.com/item?id=43765605 https://news.ycombinator.com/item?id=43765605

IshKebab 2y ago
> but parallel downloads of things like bottles and DMGs is not an architectural limitation of Homebrew itself, but instead a conscious decision to trade some install speed for courtesy towards the services we fetch from That doesn't make sense. As you say you're directing a huge volume of traffic so it makes no difference exactly when a user downloads a byte. It all gets smeared out. Only the total amount of data matters and that is unaffected by parallelism.
- woodruffw 2y ago
  > As you say you're directing a huge volume of traffic so it makes no difference exactly when a user downloads a byte. It all gets smeared out. Homebrew's traffic pattern is not a uniform distribution. Package updates go out, and users download those packages in structured ways: there are spikes for MDM-managed Homebrew installations, spikes for cronjobs and CI/CD systems, spikes at 9AM on different coasts when developers sign into their machines, etc. (How much this matters is also not uniform: it matters somewhat less for GitHub Packages - they should have a hefty CDN - and it matters somewhat more for Casks, which tend to be large DMGs hosted on individual servers.)
  - internetter 2y ago
    Could you parallelize intelligently based on these heuristics? Like, say, whitelist GitHub and S3 for parallelizing
    - SonOfLilit 2y ago
      When I asked a few years ago, the brew maintainer said "parallelizing at this scale would hurt github, source: I work for github"
      - IshKebab 2y ago
        Definitely sounds like one of those "I don't want to do it because it's work, but I don't want to admit that so here's a spurious technical reason" excuses to me. If spikes at midnight etc are an issue just automatically disabled parallel downloads around midnight. Or only use them when running in a terminal. I really doubt it is an issue for GitHub though.
        woodruffw 2y ago
        Can you think of any more charitable reason than this? I think it beggars belief that Homebrew isn’t interested in improving performance; it’s something we talk about constantly and is one of our largest and most consistent user asks. (It’s not just midnight, or terminals, as mentioned, not that either of these is really a “just”. And it’s not just GitHub, as mentioned.)
xiphias2 2y ago
It's not _just_ the infrastructure that is awesome for homebrew. The help I got from the team, answering in real-time when I didn't know how to get through the CI bugs is amazing. It's the boring maintainence work that makes it so special for me. I also feel that there could be a lot of automation in the backend part, catching bugs early (maybe even on local machine before CI run) for example.
- KolenCh 2y ago
  I think you mean “the hell I got from the team”. I wonder if anything changed substantially over the years to make you say that?
  - saagarjha 2y ago
    I feel like it's quite rude to deny to someone's lived experience and then ask what might have changed to make them feel that way.
    - deleted 2y ago
      [deleted]
  - KolenCh 2y ago
    To those who downvote, care to explain? Homebrew community is notorious in how they treat users on their GitHub Issues and I wonder why anyone is downvoting without showing how that situation have changed.
chrisweekly 2y ago
> "it's very hard to RIIR" RIIR - "Rewrite It In Rust" (maybe obvious in context? sharing in case not)
- montag 2y ago
  IWNO, thanks.
  - OJFord 2y ago
    I was not.. oware either, also thanks.
    - j16sdiz 2y ago
      it was not obvious, thanks
- deleted 2y ago
  [deleted]
fastball 2y ago
Yeah, to me the language (ruby) has nothing to do with the success of Homebrew, it is entirely about the DX around backend packaging. The main reason I switched from MacPorts to Homebrew back in the day is because when a new version of [insert software here] was released, I could expect the brew package to be updated within days. MacPorts always lagged significantly behind, and oftentimes didn't have certain packages at all.
- woodruffw 2y ago
  I completely agree. I think Ruby has actually been a pretty great boon of a language choice because of the formula DSL, but that goes back to the DX point and not something fundamental to the language.
  - mdaniel 2y ago
    Merely for your consideration, Ruby is great if you live in that codebase all the time, but Ruby -- in general, not just Homebrew -- has indescribably bad discoverability. I mean that both in terms of "what can I write where my cursor is?" and its friend "where the hell did this symbol under my cursor come from?" I'm acutely aware someone's going to say LSP something or Rubymine something else but as for drive-by contributions, ... anyway, like I said, just for your consideration
    - kusokurae 2y ago
      I've tried to love Ruby and use it daily for money, but by far the best justifications I have for the RoR dev experience are: it keeps me employed, and it's great if you're the one who wrote everything.
- mdaniel 2y ago
  As best I can tell, that's partially due to the size of the community, but also because Homebrew went out of their way to teach the system to fish via the 3(?) different kinds of "check for updates" built into the Formulae: - GitHub release sniffing https://github.com/Homebrew/homebrew-core/blob/b331b99b9f24f84d1de579558b5c6839361c6bf0/Formula/a/apko.rb#L12-L15 https://github.com/Homebrew/homebrew-core/blob/b331b99b9f24f... - page scraping https://github.com/Homebrew/homebrew-core/blob/b331b99b9f24f84d1de579558b5c6839361c6bf0/Formula/a/acme.rb#L8-L12 https://github.com/Homebrew/homebrew-core/blob/b331b99b9f24f... (and also per-content-type flavors json and presumably xml) - links to other formulae (IOW cascading updates): https://github.com/Homebrew/homebrew-core/blob/b331b99b9f24f84d1de579558b5c6839361c6bf0/Formula/a/aarch64-elf-binutils.rb#L10 https://github.com/Homebrew/homebrew-core/blob/b331b99b9f24f... and then the $(brew livecheck) invocation which will do a subset, a curated list, or hypothetically all of them I can't imagine why MacPorts or our new Sapphire friend couldn't adopt a similar strategy

IshKebab 2y ago

> but parallel downloads of things like bottles and DMGs is not an architectural limitation of Homebrew itself, but instead a conscious decision to trade some install speed for courtesy towards the services we fetch from That doesn't make sense. As you say you're directing a huge volume of traffic so it makes no difference exactly when a user downloads a byte. It all gets smeared out. Only the total amount of data matters and that is unaffected by parallelism.

woodruffw 2y ago
> As you say you're directing a huge volume of traffic so it makes no difference exactly when a user downloads a byte. It all gets smeared out. Homebrew's traffic pattern is not a uniform distribution. Package updates go out, and users download those packages in structured ways: there are spikes for MDM-managed Homebrew installations, spikes for cronjobs and CI/CD systems, spikes at 9AM on different coasts when developers sign into their machines, etc. (How much this matters is also not uniform: it matters somewhat less for GitHub Packages - they should have a hefty CDN - and it matters somewhat more for Casks, which tend to be large DMGs hosted on individual servers.)
- internetter 2y ago
  Could you parallelize intelligently based on these heuristics? Like, say, whitelist GitHub and S3 for parallelizing
  - SonOfLilit 2y ago
    When I asked a few years ago, the brew maintainer said "parallelizing at this scale would hurt github, source: I work for github"
    - IshKebab 2y ago
      Definitely sounds like one of those "I don't want to do it because it's work, but I don't want to admit that so here's a spurious technical reason" excuses to me. If spikes at midnight etc are an issue just automatically disabled parallel downloads around midnight. Or only use them when running in a terminal. I really doubt it is an issue for GitHub though.
      - woodruffw 2y ago
        Can you think of any more charitable reason than this? I think it beggars belief that Homebrew isn’t interested in improving performance; it’s something we talk about constantly and is one of our largest and most consistent user asks. (It’s not just midnight, or terminals, as mentioned, not that either of these is really a “just”. And it’s not just GitHub, as mentioned.)

xiphias2 2y ago

It's not _just_ the infrastructure that is awesome for homebrew. The help I got from the team, answering in real-time when I didn't know how to get through the CI bugs is amazing. It's the boring maintainence work that makes it so special for me. I also feel that there could be a lot of automation in the backend part, catching bugs early (maybe even on local machine before CI run) for example.

KolenCh 2y ago
I think you mean “the hell I got from the team”. I wonder if anything changed substantially over the years to make you say that?
- saagarjha 2y ago
  I feel like it's quite rude to deny to someone's lived experience and then ask what might have changed to make them feel that way.
  - deleted 2y ago
    [deleted]
- KolenCh 2y ago
  To those who downvote, care to explain? Homebrew community is notorious in how they treat users on their GitHub Issues and I wonder why anyone is downvoting without showing how that situation have changed.

chrisweekly 2y ago

> "it's very hard to RIIR" RIIR - "Rewrite It In Rust" (maybe obvious in context? sharing in case not)

montag 2y ago
IWNO, thanks.
- OJFord 2y ago
  I was not.. oware either, also thanks.
  - j16sdiz 2y ago
    it was not obvious, thanks
deleted 2y ago
[deleted]

fastball 2y ago

Yeah, to me the language (ruby) has nothing to do with the success of Homebrew, it is entirely about the DX around backend packaging. The main reason I switched from MacPorts to Homebrew back in the day is because when a new version of [insert software here] was released, I could expect the brew package to be updated within days. MacPorts always lagged significantly behind, and oftentimes didn't have certain packages at all.

woodruffw 2y ago
I completely agree. I think Ruby has actually been a pretty great boon of a language choice because of the formula DSL, but that goes back to the DX point and not something fundamental to the language.
- mdaniel 2y ago
  Merely for your consideration, Ruby is great if you live in that codebase all the time, but Ruby -- in general, not just Homebrew -- has indescribably bad discoverability. I mean that both in terms of "what can I write where my cursor is?" and its friend "where the hell did this symbol under my cursor come from?" I'm acutely aware someone's going to say LSP something or Rubymine something else but as for drive-by contributions, ... anyway, like I said, just for your consideration
  - kusokurae 2y ago
    I've tried to love Ruby and use it daily for money, but by far the best justifications I have for the RoR dev experience are: it keeps me employed, and it's great if you're the one who wrote everything.
mdaniel 2y ago
As best I can tell, that's partially due to the size of the community, but also because Homebrew went out of their way to teach the system to fish via the 3(?) different kinds of "check for updates" built into the Formulae: - GitHub release sniffing https://github.com/Homebrew/homebrew-core/blob/b331b99b9f24f84d1de579558b5c6839361c6bf0/Formula/a/apko.rb#L12-L15 https://github.com/Homebrew/homebrew-core/blob/b331b99b9f24f... - page scraping https://github.com/Homebrew/homebrew-core/blob/b331b99b9f24f84d1de579558b5c6839361c6bf0/Formula/a/acme.rb#L8-L12 https://github.com/Homebrew/homebrew-core/blob/b331b99b9f24f... (and also per-content-type flavors json and presumably xml) - links to other formulae (IOW cascading updates): https://github.com/Homebrew/homebrew-core/blob/b331b99b9f24f84d1de579558b5c6839361c6bf0/Formula/a/aarch64-elf-binutils.rb#L10 https://github.com/Homebrew/homebrew-core/blob/b331b99b9f24f... and then the $(brew livecheck) invocation which will do a subset, a curated list, or hypothetically all of them I can't imagine why MacPorts or our new Sapphire friend couldn't adopt a similar strategy