3 ms·

> As you say you're directing a huge volume of traffic so it makes no difference exactly when a user downloads a byte. It all gets smeared out. Homebrew's traf

> As you say you're directing a huge volume of traffic so it makes no difference exactly when a user downloads a byte. It all gets smeared out. Homebrew's traffic pattern is not a uniform distribution. Package updates go out, and users download those packages in structured ways: there are spikes for MDM-managed Homebrew installations, spikes for cronjobs and CI/CD systems, spikes at 9AM on different coasts when developers sign into their machines, etc. (How much this matters is also not uniform: it matters somewhat less for GitHub Packages - they should have a hefty CDN - and it matters somewhat more for Casks, which tend to be large DMGs hosted on individual servers.)

internetter 2y ago
Could you parallelize intelligently based on these heuristics? Like, say, whitelist GitHub and S3 for parallelizing
- SonOfLilit 2y ago
  When I asked a few years ago, the brew maintainer said "parallelizing at this scale would hurt github, source: I work for github"
  - IshKebab 2y ago
    Definitely sounds like one of those "I don't want to do it because it's work, but I don't want to admit that so here's a spurious technical reason" excuses to me. If spikes at midnight etc are an issue just automatically disabled parallel downloads around midnight. Or only use them when running in a terminal. I really doubt it is an issue for GitHub though.
    - woodruffw 2y ago
      Can you think of any more charitable reason than this? I think it beggars belief that Homebrew isn’t interested in improving performance; it’s something we talk about constantly and is one of our largest and most consistent user asks. (It’s not just midnight, or terminals, as mentioned, not that either of these is really a “just”. And it’s not just GitHub, as mentioned.)
      - SonOfLilit 2y ago
        (In this subthread it is just github because we were discussing enabling parallelism for github only. The terminal proposal meant "keep it single thread for CI")
        woodruffw 2y ago
        Okay. The problem is still present in the GH case, although to a lesser extent. These kinds of complications on a "same experience for everyone" flow are also nontrivial: single-threading on CI is hard to explain to users who see very fast local installs and slow CI installs - the latter costs them extra money! - and complicates the story with mirrors, download groups that include non-GH origins, etc. That's not to say it's impossible; as linked in another thread, it's something that's being actively worked on. But I think it should be understandable why "just parallelize it and you're wrong/stupid/lazy because you haven't already" is neither a contextualized or charitable response.
        SonOfLilit 2y ago
        I agree that the brew devs are not stupid or lazy (as can be seen by the great popularity of the tool they developed among the complaining demographic). I suspect they are wrong in this specific care based on the "every other package manager solved this" heuristic, but I have no doubt this is not a straightforward issue. Personally, I dropped the case after offering to send a PR and getting back a "we won't accept this, it's not because we didn't think of it, from where we're standing it's a bad idea". They may or may not be wrong, but X am definitely wrong here, knowing almost nothing about it.
      - IshKebab 2y ago
        To be clear I think "it's a lot of work and we don't want to deal with the complexity" is a totally valid reason. I have that reason for not doing a ton of things. I just don't think "it shouldn't be done" is really the true reason. It's very tempting to say that though because it sounds like a stronger defense than "I don't want to". I think some people that do it don't even realise.

internetter 2y ago

Could you parallelize intelligently based on these heuristics? Like, say, whitelist GitHub and S3 for parallelizing

SonOfLilit 2y ago
When I asked a few years ago, the brew maintainer said "parallelizing at this scale would hurt github, source: I work for github"
- IshKebab 2y ago
  Definitely sounds like one of those "I don't want to do it because it's work, but I don't want to admit that so here's a spurious technical reason" excuses to me. If spikes at midnight etc are an issue just automatically disabled parallel downloads around midnight. Or only use them when running in a terminal. I really doubt it is an issue for GitHub though.
  - woodruffw 2y ago
    Can you think of any more charitable reason than this? I think it beggars belief that Homebrew isn’t interested in improving performance; it’s something we talk about constantly and is one of our largest and most consistent user asks. (It’s not just midnight, or terminals, as mentioned, not that either of these is really a “just”. And it’s not just GitHub, as mentioned.)
    - SonOfLilit 2y ago
      (In this subthread it is just github because we were discussing enabling parallelism for github only. The terminal proposal meant "keep it single thread for CI")
      - woodruffw 2y ago
        Okay. The problem is still present in the GH case, although to a lesser extent. These kinds of complications on a "same experience for everyone" flow are also nontrivial: single-threading on CI is hard to explain to users who see very fast local installs and slow CI installs - the latter costs them extra money! - and complicates the story with mirrors, download groups that include non-GH origins, etc. That's not to say it's impossible; as linked in another thread, it's something that's being actively worked on. But I think it should be understandable why "just parallelize it and you're wrong/stupid/lazy because you haven't already" is neither a contextualized or charitable response.
        SonOfLilit 2y ago
        I agree that the brew devs are not stupid or lazy (as can be seen by the great popularity of the tool they developed among the complaining demographic). I suspect they are wrong in this specific care based on the "every other package manager solved this" heuristic, but I have no doubt this is not a straightforward issue. Personally, I dropped the case after offering to send a PR and getting back a "we won't accept this, it's not because we didn't think of it, from where we're standing it's a bad idea". They may or may not be wrong, but X am definitely wrong here, knowing almost nothing about it.
    - IshKebab 2y ago
      To be clear I think "it's a lot of work and we don't want to deal with the complexity" is a totally valid reason. I have that reason for not doing a ton of things. I just don't think "it shouldn't be done" is really the true reason. It's very tempting to say that though because it sounds like a stronger defense than "I don't want to". I think some people that do it don't even realise.