4 ms·

I'm disappointed that this is an issue for some package management systems. 20 years ago I helped run a mirroring service, it's still running today. Distributio

I'm disappointed that this is an issue for some package management systems. 20 years ago I helped run a mirroring service, it's still running today. Distributions such as Debian have hundreds of mirrors. This is a solved problems, but we just decided to but everything in the hands of one for-profit company.

BonoboIO 4y ago
Move fast and break things ... /s
naikrovek 4y ago
just yesterday I was stuck for an hour because a debian package mirror went down. took a long time to talk the user through changing their sources.list so that another mirror was chosen, and the mirror chosen out of that pool was down also. finally I had to manually check for a good mirror and give them the URLs. the user's take was "why don't they use GitHub packages?" "still running today" doesn't mean 100.0% uptime.
- ddtaylor 4y ago
  This is not how this works anymore. The system that is behaving this way must be relatively old at this point since almost all modern Debian based distros use the "mirror://" URI syntax now that automatically falls back to another mirror if one fails.
  - pxc 4y ago
    I don't think a clean Debian stable install uses that today. But even so, at least the mirrorlist.txt file that appears in the mirror:// URI must be available for it to work, right?
    - ddtaylor 4y ago
      You are correct. While it's supported and part of the APT version in Debian, they don't make much use of it themselves, whereas most downstream distros are making use of it (eg. Ubuntu) https://manpages.debian.org/bullseye/apt/apt-transport-mirror.1.en.html https://manpages.debian.org/bullseye/apt/apt-transport-mirro... You can still use it in vanilla Debian, but they don't make their mirror list available easily in the correct format, so you would have to basically curl + awk the URLs into a text file and use that. My guess is that Debian itself probably sees less than 1% of the traffic on their mirrors compared to Ubuntu and they haven't been as motivated to make this change.
Karellen 4y ago
If only there were some way to make git distributed! /s
- crest 4y ago
  You mean something like a git annex enabled branch tracking mirror locations of each release artifact like HTTP URLs, (webseeded) torrents, maybe even something content addressed like IPFS? sigh
  - booi 4y ago
    uhhh wait can you explain what you mean?
Xylakant 4y ago
I can’t speak for homebrew, but cargo/crate.io is fundamentally different from debian. The rate of churn is significantly higher, packages are published constantly and people expect them to be available. You can’t really do that with a system of mirrors like debian does. You can do edge caching and crates does that, but you want some central authority of which package versions are available. And every cargo run queries that index. It’s acceptable for debian mirrors lag a few minutes or hours behind. The same thing is much harder to accept when the rate of change is much higher. Different requirements, different tradeoffs.
- mrweasel 4y ago
  I guess I don't really understand the need for something like Cargo to be up to date to the seconds, or even minutes. My assumption is that you build your code with certain package versions in mind and release that to testing. Unless it's a security update, if won't matter if your 24 hours behind. Say that version 1.2.1 of a library is release right as you do your build, that won't go into production within that 24 hour window anyway. If it is a security fix, then, like Debian, you pull that from another repository, which is under tighter control.
  - kibwen 4y ago
    This thread is glossing over some important details. A package repo has two distinct storage concerns: the index (a list of which versions of packages have ever been released), and the actual packages themselves. It's convenient to have the index centralized, for maximum consistency. But the packages themselves can be stored however you want, and if you try to access a stale mirror without the most recent version of a package, then the client should have the option of using a different mirror or else accepting the old version. For crates.io specifically, the packages are stored in S3, whereas the index is currently stored as a bog-standard Github repo (not as a Github Package), and in the near future the crates.io index will also move to crates.io itself (https://blog.rust-lang.org/inside-rust/2023/01/30/cargo-sparse-protocol.html https://blog.rust-lang.org/inside-rust/2023/01/30/cargo-spar...).
    - 4y ago

BonoboIO 4y ago

Move fast and break things ... /s

naikrovek 4y ago

just yesterday I was stuck for an hour because a debian package mirror went down. took a long time to talk the user through changing their sources.list so that another mirror was chosen, and the mirror chosen out of that pool was down also. finally I had to manually check for a good mirror and give them the URLs. the user's take was "why don't they use GitHub packages?" "still running today" doesn't mean 100.0% uptime.

ddtaylor 4y ago
This is not how this works anymore. The system that is behaving this way must be relatively old at this point since almost all modern Debian based distros use the "mirror://" URI syntax now that automatically falls back to another mirror if one fails.
- pxc 4y ago
  I don't think a clean Debian stable install uses that today. But even so, at least the mirrorlist.txt file that appears in the mirror:// URI must be available for it to work, right?
  - ddtaylor 4y ago
    You are correct. While it's supported and part of the APT version in Debian, they don't make much use of it themselves, whereas most downstream distros are making use of it (eg. Ubuntu) https://manpages.debian.org/bullseye/apt/apt-transport-mirror.1.en.html https://manpages.debian.org/bullseye/apt/apt-transport-mirro... You can still use it in vanilla Debian, but they don't make their mirror list available easily in the correct format, so you would have to basically curl + awk the URLs into a text file and use that. My guess is that Debian itself probably sees less than 1% of the traffic on their mirrors compared to Ubuntu and they haven't been as motivated to make this change.

Karellen 4y ago

If only there were some way to make git distributed! /s

crest 4y ago
You mean something like a git annex enabled branch tracking mirror locations of each release artifact like HTTP URLs, (webseeded) torrents, maybe even something content addressed like IPFS? sigh
- booi 4y ago
  uhhh wait can you explain what you mean?

Xylakant 4y ago

I can’t speak for homebrew, but cargo/crate.io is fundamentally different from debian. The rate of churn is significantly higher, packages are published constantly and people expect them to be available. You can’t really do that with a system of mirrors like debian does. You can do edge caching and crates does that, but you want some central authority of which package versions are available. And every cargo run queries that index. It’s acceptable for debian mirrors lag a few minutes or hours behind. The same thing is much harder to accept when the rate of change is much higher. Different requirements, different tradeoffs.

mrweasel 4y ago
I guess I don't really understand the need for something like Cargo to be up to date to the seconds, or even minutes. My assumption is that you build your code with certain package versions in mind and release that to testing. Unless it's a security update, if won't matter if your 24 hours behind. Say that version 1.2.1 of a library is release right as you do your build, that won't go into production within that 24 hour window anyway. If it is a security fix, then, like Debian, you pull that from another repository, which is under tighter control.
- kibwen 4y ago
  This thread is glossing over some important details. A package repo has two distinct storage concerns: the index (a list of which versions of packages have ever been released), and the actual packages themselves. It's convenient to have the index centralized, for maximum consistency. But the packages themselves can be stored however you want, and if you try to access a stale mirror without the most recent version of a package, then the client should have the option of using a different mirror or else accepting the old version. For crates.io specifically, the packages are stored in S3, whereas the index is currently stored as a bog-standard Github repo (not as a Github Package), and in the near future the crates.io index will also move to crates.io itself (https://blog.rust-lang.org/inside-rust/2023/01/30/cargo-sparse-protocol.html https://blog.rust-lang.org/inside-rust/2023/01/30/cargo-spar...).
  - 4y ago