5 ms·

FWIW: a longstanding limitation for parallel downloads within Homebrew isn't architectural (it's not too hard to add!) but structural with respect to Homebrew's

FWIW: a longstanding limitation for parallel downloads within Homebrew isn't architectural (it's not too hard to add!) but structural with respect to Homebrew's download sources: GitHub and others are very gracious with the amount of traffic we send their way, and we don't want to overtax services that have other major consuming parties. (This is a perverse countereffect: small projects can make performance decisions that Homebrew and other larger projects can't make, because they don't have a large install base that reveals the limitations of those decisions.)

bhouston 2y ago
> FWIW: a longstanding limitation for parallel downloads within Homebrew isn't architectural (it's not too hard to add!) but structural with respect to Homebrew's download sources I have heard that before. Hmm.... I wonder if you can get away with not doing parallel downloads, but just keep the sequential downloads going in the background while it is installing a package? It is the pause in downloads during an install that I find is the main slow down in brew.
- woodruffw 2y ago
  > I wonder if you can get away with not doing parallel downloads, but just keep the sequential downloads going in the background while it is installing a package? I could be wrong, but I believe multiple people, including maintainers, have looked into exactly that :-) (I also need to correct myself: there is some work ongoing into concurrent downloads[1]. That work hasn't hit `brew install` yet, where I imagine the question of concurrent traffic volume will become more pressing.) [1]: https://github.com/Homebrew/brew/issues/18278 https://github.com/Homebrew/brew/issues/18278
hackingonempty 2y ago
P2P protocols like BitTorrent are made for this kind of problem.
- woodruffw 2y ago
  Sure. There's also a reason no major OSS packaging ecosystem uses these protocols: the only thing worse than a slow distribution scheme is an unreliable one. Combine that with the (reasonable) lack of a reward scheme for seeding an OSS packaging ecosystem, and you have a distribution mechanism that's significantly more brittle than the current "throw a lot of bandwidth at it" approach. (Among other technical challenges, like updating the P2P broadcast for each new bottle.)
  - hackingonempty 2y ago
    I think you are misinformed as BitTorrent, for instance, is much more reliable than https alone. The reward scheme is built in already: the client uploads while it's downloading and installing and prioritizes the clients it is downloading from. At worst, the reliability and performance are the same as the web seed. Generating additional metadata at bottle build time doesn't appear to be much of a technical challenge either.
    - woodruffw 2y ago
      > The reward scheme is built in already: the client uploads while it's downloading and installing and prioritizes the clients it is downloading from. These are asymmetric: brew runs at a point in time, and most people decidedly do not want brew running in the background or blocking while leechers are still being serviced. They want it to exit quickly once the task at hand is done. > Generating additional metadata at bottle build time doesn't appear to be much of a technical challenge either. That's not the challenge. The challenge is distributing those updates. My understanding is that there's no standard way to update a torrent file; you re-roll a new file with the changes. That means staggered delivery, which in turn means a long tail of clients that see different, incompatible views of the same majority-equal files.
      - bhouston 2y ago
        > My understanding is that there's no standard way to update a torrent file; you re-roll a new file with the changes. You should only re-distribute the original file that was downloaded and thus one can just advertise the original torrent that was downloaded. But as you said earlier, brew is a point in time command and this BitTorrent solution would only really work if brew switched to an always-on service. And I am not sure that many people want to do that, although I am sure some would.

bhouston 2y ago

> FWIW: a longstanding limitation for parallel downloads within Homebrew isn't architectural (it's not too hard to add!) but structural with respect to Homebrew's download sources I have heard that before. Hmm.... I wonder if you can get away with not doing parallel downloads, but just keep the sequential downloads going in the background while it is installing a package? It is the pause in downloads during an install that I find is the main slow down in brew.

woodruffw 2y ago
> I wonder if you can get away with not doing parallel downloads, but just keep the sequential downloads going in the background while it is installing a package? I could be wrong, but I believe multiple people, including maintainers, have looked into exactly that :-) (I also need to correct myself: there is some work ongoing into concurrent downloads[1]. That work hasn't hit `brew install` yet, where I imagine the question of concurrent traffic volume will become more pressing.) [1]: https://github.com/Homebrew/brew/issues/18278 https://github.com/Homebrew/brew/issues/18278

hackingonempty 2y ago

P2P protocols like BitTorrent are made for this kind of problem.

woodruffw 2y ago
Sure. There's also a reason no major OSS packaging ecosystem uses these protocols: the only thing worse than a slow distribution scheme is an unreliable one. Combine that with the (reasonable) lack of a reward scheme for seeding an OSS packaging ecosystem, and you have a distribution mechanism that's significantly more brittle than the current "throw a lot of bandwidth at it" approach. (Among other technical challenges, like updating the P2P broadcast for each new bottle.)
- hackingonempty 2y ago
  I think you are misinformed as BitTorrent, for instance, is much more reliable than https alone. The reward scheme is built in already: the client uploads while it's downloading and installing and prioritizes the clients it is downloading from. At worst, the reliability and performance are the same as the web seed. Generating additional metadata at bottle build time doesn't appear to be much of a technical challenge either.
  - woodruffw 2y ago
    > The reward scheme is built in already: the client uploads while it's downloading and installing and prioritizes the clients it is downloading from. These are asymmetric: brew runs at a point in time, and most people decidedly do not want brew running in the background or blocking while leechers are still being serviced. They want it to exit quickly once the task at hand is done. > Generating additional metadata at bottle build time doesn't appear to be much of a technical challenge either. That's not the challenge. The challenge is distributing those updates. My understanding is that there's no standard way to update a torrent file; you re-roll a new file with the changes. That means staggered delivery, which in turn means a long tail of clients that see different, incompatible views of the same majority-equal files.
    - bhouston 2y ago
      > My understanding is that there's no standard way to update a torrent file; you re-roll a new file with the changes. You should only re-distribute the original file that was downloaded and thus one can just advertise the original torrent that was downloaded. But as you said earlier, brew is a point in time command and this BitTorrent solution would only really work if brew switched to an always-on service. And I am not sure that many people want to do that, although I am sure some would.