4 ms·

> WARNING: ALPHA SOFTWARE > Sapphire is experimental, under heavy development, and may be unstable. Use at your own risk! Ruby seems fine for brew. Does this d

> WARNING: ALPHA SOFTWARE > Sapphire is experimental, under heavy development, and may be unstable. Use at your own risk! Ruby seems fine for brew. Does this do anything else better? Ruby makes it easy to write recipes for it which is a huge boon for a package manager.

bhouston 2y ago
My thoughts exactly. The main reason I find brew to be a bit slow is just a lack of parallelism with regards to downloads and installs. Rather brew does alternating and sequential D, I, D, I, D, I, when I wish it just kept downloading in the background when it is doing the installs. It would cut down the brew upgrade time by 30% or more at the cost of more disk space used during the process. But this one qualm I have isn't a result of its language implementation at all.
- woodruffw 2y ago
  FWIW: a longstanding limitation for parallel downloads within Homebrew isn't architectural (it's not too hard to add!) but structural with respect to Homebrew's download sources: GitHub and others are very gracious with the amount of traffic we send their way, and we don't want to overtax services that have other major consuming parties. (This is a perverse countereffect: small projects can make performance decisions that Homebrew and other larger projects can't make, because they don't have a large install base that reveals the limitations of those decisions.)
  - bhouston 2y ago
    > FWIW: a longstanding limitation for parallel downloads within Homebrew isn't architectural (it's not too hard to add!) but structural with respect to Homebrew's download sources I have heard that before. Hmm.... I wonder if you can get away with not doing parallel downloads, but just keep the sequential downloads going in the background while it is installing a package? It is the pause in downloads during an install that I find is the main slow down in brew.
    - woodruffw 2y ago
      > I wonder if you can get away with not doing parallel downloads, but just keep the sequential downloads going in the background while it is installing a package? I could be wrong, but I believe multiple people, including maintainers, have looked into exactly that :-) (I also need to correct myself: there is some work ongoing into concurrent downloads[1]. That work hasn't hit `brew install` yet, where I imagine the question of concurrent traffic volume will become more pressing.) [1]: https://github.com/Homebrew/brew/issues/18278 https://github.com/Homebrew/brew/issues/18278
  - hackingonempty 2y ago
    P2P protocols like BitTorrent are made for this kind of problem.
    - woodruffw 2y ago
      Sure. There's also a reason no major OSS packaging ecosystem uses these protocols: the only thing worse than a slow distribution scheme is an unreliable one. Combine that with the (reasonable) lack of a reward scheme for seeding an OSS packaging ecosystem, and you have a distribution mechanism that's significantly more brittle than the current "throw a lot of bandwidth at it" approach. (Among other technical challenges, like updating the P2P broadcast for each new bottle.)
      - hackingonempty 2y ago
        I think you are misinformed as BitTorrent, for instance, is much more reliable than https alone. The reward scheme is built in already: the client uploads while it's downloading and installing and prioritizes the clients it is downloading from. At worst, the reliability and performance are the same as the web seed. Generating additional metadata at bottle build time doesn't appear to be much of a technical challenge either.
        woodruffw 2y ago
        > The reward scheme is built in already: the client uploads while it's downloading and installing and prioritizes the clients it is downloading from. These are asymmetric: brew runs at a point in time, and most people decidedly do not want brew running in the background or blocking while leechers are still being serviced. They want it to exit quickly once the task at hand is done. > Generating additional metadata at bottle build time doesn't appear to be much of a technical challenge either. That's not the challenge. The challenge is distributing those updates. My understanding is that there's no standard way to update a torrent file; you re-roll a new file with the changes. That means staggered delivery, which in turn means a long tail of clients that see different, incompatible views of the same majority-equal files.
- elcritch 2y ago
  Good points, background downloads would speed it up. And the language doesn’t have much to do with that. This project looks to be someone just toying around with Rust or their own PM. Props for that, but the headline has extra implications on HN. I recently rewrote a big portion of Atlas [1]. It’s a Nim based dependency manager that clones Nim packages to a `deps/` folder. Initially I was worried about using reference types, etc, for performance reasons. It’s a general habit of mine. Then I remembered that stuff would be negligible compared to the download times and git overhead. Well aside from the SAT solver. 1: https://github.com/nim-lang/atlas https://github.com/nim-lang/atlas
lopatin 2y ago
Ruby isn't web scale. Rust is web scale.
MrBuddyCasino 2y ago
Homebrew is dog-slow. If this becomes yet another Rust tool that is 10x faster than the one that it replaces: great.
- bhouston 2y ago
  I use homebrew constantly but the only part I find slow is the custom installations of the software and the lack of downloading while installing? Neither of these is related to the implementation language. What part of the home brew experience do you find slow? eg... % time brew upgrade brew upgrade 0.75s user 0.16s system 68% cpu 1.337 total % time brew list brew list 0.01s user 0.02s system 57% cpu 0.054 total
  - silverwind 2y ago
    The slowness comes from its serialized downloads and package installations. If you install a thing that has 10 dependencies, those 10 dependencies will serially install, likely taking many seconds.
- rafram 2y ago
  Homebrew is 99% IO-bound and this will be too. Installing (prebuilt) packages doesn't require much logic. If this tool supports parallel downloads, it will be 10x faster than Homebrew, but it won't have anything to do with the language. The issue is finding a hosting provider willing to be DDOS'd for the good of the open-source community.
  - MrBuddyCasino 2y ago
    > If this tool supports parallel downloads, it will be 10x faster than Homebrew, but it won't have anything to do with the language. If this is true, why are the Rust tools, on average, so much faster than the JS (or whatever) tools they replace? Hint: the answer hasn't just to do with the theoretical technical merits of the underlying platform, but also what type of developer they attract.
    - 2y ago
deleted 2y ago
[deleted]

bhouston 2y ago

My thoughts exactly. The main reason I find brew to be a bit slow is just a lack of parallelism with regards to downloads and installs. Rather brew does alternating and sequential D, I, D, I, D, I, when I wish it just kept downloading in the background when it is doing the installs. It would cut down the brew upgrade time by 30% or more at the cost of more disk space used during the process. But this one qualm I have isn't a result of its language implementation at all.

woodruffw 2y ago
FWIW: a longstanding limitation for parallel downloads within Homebrew isn't architectural (it's not too hard to add!) but structural with respect to Homebrew's download sources: GitHub and others are very gracious with the amount of traffic we send their way, and we don't want to overtax services that have other major consuming parties. (This is a perverse countereffect: small projects can make performance decisions that Homebrew and other larger projects can't make, because they don't have a large install base that reveals the limitations of those decisions.)
- bhouston 2y ago
  > FWIW: a longstanding limitation for parallel downloads within Homebrew isn't architectural (it's not too hard to add!) but structural with respect to Homebrew's download sources I have heard that before. Hmm.... I wonder if you can get away with not doing parallel downloads, but just keep the sequential downloads going in the background while it is installing a package? It is the pause in downloads during an install that I find is the main slow down in brew.
  - woodruffw 2y ago
    > I wonder if you can get away with not doing parallel downloads, but just keep the sequential downloads going in the background while it is installing a package? I could be wrong, but I believe multiple people, including maintainers, have looked into exactly that :-) (I also need to correct myself: there is some work ongoing into concurrent downloads[1]. That work hasn't hit `brew install` yet, where I imagine the question of concurrent traffic volume will become more pressing.) [1]: https://github.com/Homebrew/brew/issues/18278 https://github.com/Homebrew/brew/issues/18278
- hackingonempty 2y ago
  P2P protocols like BitTorrent are made for this kind of problem.
  - woodruffw 2y ago
    Sure. There's also a reason no major OSS packaging ecosystem uses these protocols: the only thing worse than a slow distribution scheme is an unreliable one. Combine that with the (reasonable) lack of a reward scheme for seeding an OSS packaging ecosystem, and you have a distribution mechanism that's significantly more brittle than the current "throw a lot of bandwidth at it" approach. (Among other technical challenges, like updating the P2P broadcast for each new bottle.)
    - hackingonempty 2y ago
      I think you are misinformed as BitTorrent, for instance, is much more reliable than https alone. The reward scheme is built in already: the client uploads while it's downloading and installing and prioritizes the clients it is downloading from. At worst, the reliability and performance are the same as the web seed. Generating additional metadata at bottle build time doesn't appear to be much of a technical challenge either.
      - woodruffw 2y ago
        > The reward scheme is built in already: the client uploads while it's downloading and installing and prioritizes the clients it is downloading from. These are asymmetric: brew runs at a point in time, and most people decidedly do not want brew running in the background or blocking while leechers are still being serviced. They want it to exit quickly once the task at hand is done. > Generating additional metadata at bottle build time doesn't appear to be much of a technical challenge either. That's not the challenge. The challenge is distributing those updates. My understanding is that there's no standard way to update a torrent file; you re-roll a new file with the changes. That means staggered delivery, which in turn means a long tail of clients that see different, incompatible views of the same majority-equal files.
elcritch 2y ago
Good points, background downloads would speed it up. And the language doesn’t have much to do with that. This project looks to be someone just toying around with Rust or their own PM. Props for that, but the headline has extra implications on HN. I recently rewrote a big portion of Atlas [1]. It’s a Nim based dependency manager that clones Nim packages to a `deps/` folder. Initially I was worried about using reference types, etc, for performance reasons. It’s a general habit of mine. Then I remembered that stuff would be negligible compared to the download times and git overhead. Well aside from the SAT solver. 1: https://github.com/nim-lang/atlas https://github.com/nim-lang/atlas

lopatin 2y ago

Ruby isn't web scale. Rust is web scale.

MrBuddyCasino 2y ago

Homebrew is dog-slow. If this becomes yet another Rust tool that is 10x faster than the one that it replaces: great.

bhouston 2y ago
I use homebrew constantly but the only part I find slow is the custom installations of the software and the lack of downloading while installing? Neither of these is related to the implementation language. What part of the home brew experience do you find slow? eg... % time brew upgrade brew upgrade 0.75s user 0.16s system 68% cpu 1.337 total % time brew list brew list 0.01s user 0.02s system 57% cpu 0.054 total
- silverwind 2y ago
  The slowness comes from its serialized downloads and package installations. If you install a thing that has 10 dependencies, those 10 dependencies will serially install, likely taking many seconds.
rafram 2y ago
Homebrew is 99% IO-bound and this will be too. Installing (prebuilt) packages doesn't require much logic. If this tool supports parallel downloads, it will be 10x faster than Homebrew, but it won't have anything to do with the language. The issue is finding a hosting provider willing to be DDOS'd for the good of the open-source community.
- MrBuddyCasino 2y ago
  > If this tool supports parallel downloads, it will be 10x faster than Homebrew, but it won't have anything to do with the language. If this is true, why are the Rust tools, on average, so much faster than the JS (or whatever) tools they replace? Hint: the answer hasn't just to do with the theoretical technical merits of the underlying platform, but also what type of developer they attract.
  - 2y ago

deleted 2y ago

[deleted]