5 ms·

I think you are misinformed as BitTorrent, for instance, is much more reliable than https alone. The reward scheme is built in already: the client uploads whil

I think you are misinformed as BitTorrent, for instance, is much more reliable than https alone. The reward scheme is built in already: the client uploads while it's downloading and installing and prioritizes the clients it is downloading from. At worst, the reliability and performance are the same as the web seed. Generating additional metadata at bottle build time doesn't appear to be much of a technical challenge either.

woodruffw 1y ago
> The reward scheme is built in already: the client uploads while it's downloading and installing and prioritizes the clients it is downloading from. These are asymmetric: brew runs at a point in time, and most people decidedly do not want brew running in the background or blocking while leechers are still being serviced. They want it to exit quickly once the task at hand is done. > Generating additional metadata at bottle build time doesn't appear to be much of a technical challenge either. That's not the challenge. The challenge is distributing those updates. My understanding is that there's no standard way to update a torrent file; you re-roll a new file with the changes. That means staggered delivery, which in turn means a long tail of clients that see different, incompatible views of the same majority-equal files.
- bhouston 1y ago
  > My understanding is that there's no standard way to update a torrent file; you re-roll a new file with the changes. You should only re-distribute the original file that was downloaded and thus one can just advertise the original torrent that was downloaded. But as you said earlier, brew is a point in time command and this BitTorrent solution would only really work if brew switched to an always-on service. And I am not sure that many people want to do that, although I am sure some would.
- hackingonempty 1y ago
  > These are asymmetric: brew runs at a point in time, and most people decidedly do not want brew running in the background or blocking while leechers are still being serviced. They want it to exit quickly once the task at hand is done. Yes, brew exits when it is done installing, nothing would need to change about that if you used BT protocol to speed up downloads. I'm sure you do have some helpful users who would volunteer to seed their cache though, which would become feasible. > That's not the challenge. The challenge is distributing those updates. The metadata goes in the formula alongside the current metadata (URLs and hashes.)
- Lammy 1y ago
  > My understanding is that there's no standard way to update a torrent file; you re-roll a new file with the changes. Kinda. You do create a new torrent, but you distribute it in a way that to a swarm member is functionally equivalent to updating an old one. Check out BEP-0039 and BEP-0046 which respectively cover the HTTP and DHT mechanisms for updating torrents: https://www.bittorrent.org/beps/bep_0039.html https://www.bittorrent.org/beps/bep_0039.html https://www.bittorrent.org/beps/bep_0046.html https://www.bittorrent.org/beps/bep_0046.html If that updated torrent is a BEP-0052 (v2) torrent it will hash per-file, and so the updated v2 torrent will have identical hashes for files which aren't changed: https://www.bittorrent.org/beps/bep_0052.html https://www.bittorrent.org/beps/bep_0052.html This combines with BEP-0038 so the updated torrent can refer to the infohash of the older torrent with which it shares files, so if you already have the old one you only have to download files that have changed: https://www.bittorrent.org/beps/bep_0038.html https://www.bittorrent.org/beps/bep_0038.html
  - woodruffw 1y ago
    That’s very cool! That addresses the basic update issue, although I would be surprised if there was a production-ready Ruby library for torrents that included these. The state of HTTP(S) in Ruby is sad enough :-) (There’s also still the state/seeding problem and its collision with user expectations around brew getting faster, or at least not any slower.)
    - Lammy 1y ago
      I agree with you about package manager usage patterns being a poor fit for seeding by end users. I definitely wouldn't want my computer to participate. I could see institutional seeders doing it as a way to donate bandwidth though, like a CDN that's built into the distribution protocol instead of getting load-balanced to Microsoft's nearest PoP when hitting a GitHub `ghcr.io` URI like Homebrew does today. Or even better, use that as an HTTP Seed (BEP-0019) to combine benefits of both :) https://www.bittorrent.org/beps/bep_0019.html https://www.bittorrent.org/beps/bep_0019.html
      - woodruffw 1y ago

woodruffw 1y ago

> The reward scheme is built in already: the client uploads while it's downloading and installing and prioritizes the clients it is downloading from. These are asymmetric: brew runs at a point in time, and most people decidedly do not want brew running in the background or blocking while leechers are still being serviced. They want it to exit quickly once the task at hand is done. > Generating additional metadata at bottle build time doesn't appear to be much of a technical challenge either. That's not the challenge. The challenge is distributing those updates. My understanding is that there's no standard way to update a torrent file; you re-roll a new file with the changes. That means staggered delivery, which in turn means a long tail of clients that see different, incompatible views of the same majority-equal files.

bhouston 1y ago
> My understanding is that there's no standard way to update a torrent file; you re-roll a new file with the changes. You should only re-distribute the original file that was downloaded and thus one can just advertise the original torrent that was downloaded. But as you said earlier, brew is a point in time command and this BitTorrent solution would only really work if brew switched to an always-on service. And I am not sure that many people want to do that, although I am sure some would.
hackingonempty 1y ago
> These are asymmetric: brew runs at a point in time, and most people decidedly do not want brew running in the background or blocking while leechers are still being serviced. They want it to exit quickly once the task at hand is done. Yes, brew exits when it is done installing, nothing would need to change about that if you used BT protocol to speed up downloads. I'm sure you do have some helpful users who would volunteer to seed their cache though, which would become feasible. > That's not the challenge. The challenge is distributing those updates. The metadata goes in the formula alongside the current metadata (URLs and hashes.)
Lammy 1y ago
> My understanding is that there's no standard way to update a torrent file; you re-roll a new file with the changes. Kinda. You do create a new torrent, but you distribute it in a way that to a swarm member is functionally equivalent to updating an old one. Check out BEP-0039 and BEP-0046 which respectively cover the HTTP and DHT mechanisms for updating torrents: https://www.bittorrent.org/beps/bep_0039.html https://www.bittorrent.org/beps/bep_0039.html https://www.bittorrent.org/beps/bep_0046.html https://www.bittorrent.org/beps/bep_0046.html If that updated torrent is a BEP-0052 (v2) torrent it will hash per-file, and so the updated v2 torrent will have identical hashes for files which aren't changed: https://www.bittorrent.org/beps/bep_0052.html https://www.bittorrent.org/beps/bep_0052.html This combines with BEP-0038 so the updated torrent can refer to the infohash of the older torrent with which it shares files, so if you already have the old one you only have to download files that have changed: https://www.bittorrent.org/beps/bep_0038.html https://www.bittorrent.org/beps/bep_0038.html
- woodruffw 1y ago
  That’s very cool! That addresses the basic update issue, although I would be surprised if there was a production-ready Ruby library for torrents that included these. The state of HTTP(S) in Ruby is sad enough :-) (There’s also still the state/seeding problem and its collision with user expectations around brew getting faster, or at least not any slower.)
  - Lammy 1y ago
    I agree with you about package manager usage patterns being a poor fit for seeding by end users. I definitely wouldn't want my computer to participate. I could see institutional seeders doing it as a way to donate bandwidth though, like a CDN that's built into the distribution protocol instead of getting load-balanced to Microsoft's nearest PoP when hitting a GitHub `ghcr.io` URI like Homebrew does today. Or even better, use that as an HTTP Seed (BEP-0019) to combine benefits of both :) https://www.bittorrent.org/beps/bep_0019.html https://www.bittorrent.org/beps/bep_0019.html
    - woodruffw 1y ago