4 ms·

My thoughts exactly. The main reason I find brew to be a bit slow is just a lack of parallelism with regards to downloads and installs. Rather brew does altern

My thoughts exactly. The main reason I find brew to be a bit slow is just a lack of parallelism with regards to downloads and installs. Rather brew does alternating and sequential D, I, D, I, D, I, when I wish it just kept downloading in the background when it is doing the installs. It would cut down the brew upgrade time by 30% or more at the cost of more disk space used during the process. But this one qualm I have isn't a result of its language implementation at all.

woodruffw 2y ago
FWIW: a longstanding limitation for parallel downloads within Homebrew isn't architectural (it's not too hard to add!) but structural with respect to Homebrew's download sources: GitHub and others are very gracious with the amount of traffic we send their way, and we don't want to overtax services that have other major consuming parties. (This is a perverse countereffect: small projects can make performance decisions that Homebrew and other larger projects can't make, because they don't have a large install base that reveals the limitations of those decisions.)
- bhouston 2y ago
  > FWIW: a longstanding limitation for parallel downloads within Homebrew isn't architectural (it's not too hard to add!) but structural with respect to Homebrew's download sources I have heard that before. Hmm.... I wonder if you can get away with not doing parallel downloads, but just keep the sequential downloads going in the background while it is installing a package? It is the pause in downloads during an install that I find is the main slow down in brew.
  - woodruffw 2y ago
    > I wonder if you can get away with not doing parallel downloads, but just keep the sequential downloads going in the background while it is installing a package? I could be wrong, but I believe multiple people, including maintainers, have looked into exactly that :-) (I also need to correct myself: there is some work ongoing into concurrent downloads[1]. That work hasn't hit `brew install` yet, where I imagine the question of concurrent traffic volume will become more pressing.) [1]: https://github.com/Homebrew/brew/issues/18278 https://github.com/Homebrew/brew/issues/18278
- hackingonempty 2y ago
  P2P protocols like BitTorrent are made for this kind of problem.
  - woodruffw 2y ago
    Sure. There's also a reason no major OSS packaging ecosystem uses these protocols: the only thing worse than a slow distribution scheme is an unreliable one. Combine that with the (reasonable) lack of a reward scheme for seeding an OSS packaging ecosystem, and you have a distribution mechanism that's significantly more brittle than the current "throw a lot of bandwidth at it" approach. (Among other technical challenges, like updating the P2P broadcast for each new bottle.)
    - hackingonempty 2y ago
      I think you are misinformed as BitTorrent, for instance, is much more reliable than https alone. The reward scheme is built in already: the client uploads while it's downloading and installing and prioritizes the clients it is downloading from. At worst, the reliability and performance are the same as the web seed. Generating additional metadata at bottle build time doesn't appear to be much of a technical challenge either.
      - woodruffw 2y ago
        > The reward scheme is built in already: the client uploads while it's downloading and installing and prioritizes the clients it is downloading from. These are asymmetric: brew runs at a point in time, and most people decidedly do not want brew running in the background or blocking while leechers are still being serviced. They want it to exit quickly once the task at hand is done. > Generating additional metadata at bottle build time doesn't appear to be much of a technical challenge either. That's not the challenge. The challenge is distributing those updates. My understanding is that there's no standard way to update a torrent file; you re-roll a new file with the changes. That means staggered delivery, which in turn means a long tail of clients that see different, incompatible views of the same majority-equal files.
elcritch 2y ago
Good points, background downloads would speed it up. And the language doesn’t have much to do with that. This project looks to be someone just toying around with Rust or their own PM. Props for that, but the headline has extra implications on HN. I recently rewrote a big portion of Atlas [1]. It’s a Nim based dependency manager that clones Nim packages to a `deps/` folder. Initially I was worried about using reference types, etc, for performance reasons. It’s a general habit of mine. Then I remembered that stuff would be negligible compared to the download times and git overhead. Well aside from the SAT solver. 1: https://github.com/nim-lang/atlas https://github.com/nim-lang/atlas

woodruffw 2y ago

FWIW: a longstanding limitation for parallel downloads within Homebrew isn't architectural (it's not too hard to add!) but structural with respect to Homebrew's download sources: GitHub and others are very gracious with the amount of traffic we send their way, and we don't want to overtax services that have other major consuming parties. (This is a perverse countereffect: small projects can make performance decisions that Homebrew and other larger projects can't make, because they don't have a large install base that reveals the limitations of those decisions.)

bhouston 2y ago
> FWIW: a longstanding limitation for parallel downloads within Homebrew isn't architectural (it's not too hard to add!) but structural with respect to Homebrew's download sources I have heard that before. Hmm.... I wonder if you can get away with not doing parallel downloads, but just keep the sequential downloads going in the background while it is installing a package? It is the pause in downloads during an install that I find is the main slow down in brew.
- woodruffw 2y ago
  > I wonder if you can get away with not doing parallel downloads, but just keep the sequential downloads going in the background while it is installing a package? I could be wrong, but I believe multiple people, including maintainers, have looked into exactly that :-) (I also need to correct myself: there is some work ongoing into concurrent downloads[1]. That work hasn't hit `brew install` yet, where I imagine the question of concurrent traffic volume will become more pressing.) [1]: https://github.com/Homebrew/brew/issues/18278 https://github.com/Homebrew/brew/issues/18278
hackingonempty 2y ago
P2P protocols like BitTorrent are made for this kind of problem.
- woodruffw 2y ago
  Sure. There's also a reason no major OSS packaging ecosystem uses these protocols: the only thing worse than a slow distribution scheme is an unreliable one. Combine that with the (reasonable) lack of a reward scheme for seeding an OSS packaging ecosystem, and you have a distribution mechanism that's significantly more brittle than the current "throw a lot of bandwidth at it" approach. (Among other technical challenges, like updating the P2P broadcast for each new bottle.)
  - hackingonempty 2y ago
    I think you are misinformed as BitTorrent, for instance, is much more reliable than https alone. The reward scheme is built in already: the client uploads while it's downloading and installing and prioritizes the clients it is downloading from. At worst, the reliability and performance are the same as the web seed. Generating additional metadata at bottle build time doesn't appear to be much of a technical challenge either.
    - woodruffw 2y ago
      > The reward scheme is built in already: the client uploads while it's downloading and installing and prioritizes the clients it is downloading from. These are asymmetric: brew runs at a point in time, and most people decidedly do not want brew running in the background or blocking while leechers are still being serviced. They want it to exit quickly once the task at hand is done. > Generating additional metadata at bottle build time doesn't appear to be much of a technical challenge either. That's not the challenge. The challenge is distributing those updates. My understanding is that there's no standard way to update a torrent file; you re-roll a new file with the changes. That means staggered delivery, which in turn means a long tail of clients that see different, incompatible views of the same majority-equal files.

elcritch 2y ago

Good points, background downloads would speed it up. And the language doesn’t have much to do with that. This project looks to be someone just toying around with Rust or their own PM. Props for that, but the headline has extra implications on HN. I recently rewrote a big portion of Atlas [1]. It’s a Nim based dependency manager that clones Nim packages to a `deps/` folder. Initially I was worried about using reference types, etc, for performance reasons. It’s a general habit of mine. Then I remembered that stuff would be negligible compared to the download times and git overhead. Well aside from the SAT solver. 1: https://github.com/nim-lang/atlas https://github.com/nim-lang/atlas