8 ms·

Cloudflare Package Repository Mirrors

88 points by Foxboron 6y ago

tiernano 6y ago
Been looking for something like this for a while. We see if more repos get added soon (Ubuntu and fedora would be nice!)
dennajort 6y ago
Is it me, or it downloads so fast that the progress output doesn't even show in the terminal ? On Archlinux.
- deleted 6y ago
  [deleted]
- husam212 6y ago
  I think the header Content-Length is missing.
  - mercora 6y ago
    range requests to resume downloads do not work either... but when encountered i can see the progress of downloading that package with the failed resume attempt... quite weird behavior
- rwha 6y ago
  I saw the same. My best initial guess is that this is somehow caused by cloudflare either setting the content-type header to text/plain or not setting the accept-ranges header. Or maybe both?
  - p00f 6y ago
    > not setting the accept-ranges header Mostly that, because it doesn't support resuming partial downloads either (it says http server does not support byte ranges)
  - dennajort 6y ago
    Oh ! totally makes sense, nice guess. and very interesting
SXX 6y ago
Is anyone from CloudFlare or Canonical reading? Please parter to make mirrors for Launchpad PPAs. They're so painfully slow around the globe and there is tons of open source projects that depend on PPAs...
- adonese 6y ago
  Second this. Cloudflare package repo gets me so excited about it and i wish they extend to ubuntu too.
- Ayesh 6y ago
  I recently moved to Vietnam, and I sometimes get awfully slow connection speeds to the mirrors nearby my city, to a point that switching to US mirrors is faster. (ftp.[kr|sg].debian.org)
  - SXX 6y ago
    I also live in Vietnam for 3-6 months every year so it's part of the reason I posted it. PPA is real pain since as far as I understand they're not mirrored at all or at least they haven't been before. In my experience mirrors in Japan work best in Vietnam for whatever reason.
  - stingraycharles 6y ago
    How is Docker hub from Vietnam nowadays? I lived in Cambodia from 2014-2016, and it was nearly impossible to pull Docker images which have large single layers. Would be great to have a Cloudflare solution for this.
    - momothereal 6y ago
      Off-topic. I visited Cambodia for 3 weeks in 2017. It's a beautiful country and I enjoyed every minute. My mind has pondered over living there long-term -- it seems so affordable. How was your experience living there for a while, especially as a developer?
      - 6y ago
james412 6y ago
As much as I'd love to use CloudFlare's uniformly fast network for package update, I'd never use this simply because I already depend on them for too much stuff. It's not like letting them gather logs on my computers updating would be terrible somehow, but if it's so easy to avoid donating more logs and more dependence on them, why not do it for the sake of preserving diversity? Growing companies often exhibit a long term pattern that usually seems innocent at first manifestation: - Offer something for free - Become hugely popular at offering that free thing - Alternatives slowly dwindle in popularity and/or begin to die off - User 'choice' increasingly becomes concentrated in that single option - Company introduces surprising policy change, but users can switch to an alternative if they don't like it - Company introduces differentiating functionality setting its option apart from the alternatives. - Users grow additional dependence on the new functionality, alternatives look increasingly obsolete - Company becomes increasingly aggressive at introducing changes, as they now own the primary alternative It's hard to imagine CloudFlare are somehow going to monopolize package management :) But nonetheless, free services like this generally signal the beginning of this pattern At some point I'm sure e.g. DejaNews seemed like an innocent dependency, and that attempting to look far ahead into the future would not have revealed its eventual sale to Google and the loss of the only comprehensive archive of the early Internet community in existence.
- bhouston 6y ago
  I think this is still a good service. It is just a mirror. It is a way of giving back to these open source operating systems and cutting their costs. Thus embrace it but be smart about it so that no one is trapped. Dejanews isn't the same. That was an archive of a dying technology. This isn't the same case. Usenet was a dying technology, ultimately killed by http. Cloudflaree has always done the free thing to raise it's popularity. You are right that as companies get bigger it can be harder to keep revenue growing and they slowly start to cross lines they didn't before though. If that company has built reliance on it's free tools which have reduced diversity, when they start to cross lines it can be sort of a trap. This usually isn't intentional now but it is just how things work... Years from now revenue growth might require crossing a line...
- YorickPeterse 6y ago
bhouston 6y ago
Need Ubuntu and Raspian. When I saw the title I though they were proxying npmjs or docker hub. The install process of editing a file is a bit painful. Means few people will do it. Somehow would be nice to have this offered in the is as a built in optional feature.
- emmelaich 6y ago
  Also Fedora and Centos. Please.
overflow_error 6y ago
How is this supposed to work? (I use Arch BTW) supposed I'm in a country X, does Cloudflare basically caching the request to server that are in country X? But I'm already close to that server and I'm always cautious with mirrors that are in country X, often they were out of date :(
- lsofzz 6y ago
  > How is this supposed to work? (I use Arch BTW) supposed I'm in a country X, does Cloudflare basically caching the request to server that are in country X? But I'm already close to that server and I'm always cautious with mirrors that are in country X, often they were out of date :( According to what I read in their main page https://cloudflaremirrors.com/archlinux https://cloudflaremirrors.com/archlinux, they seem to be caching the packages from mirrors and serving it via their anycast network.
- gravitas 6y ago
  This is my issue with it as well - it's basically their geo-sniffing magic routing you towards the closest officially listed mirrors for that country (glorified proxy); it makes the mistake of presuming that all those mirrors listed in the Arch country-mirrorlist are created equal in terms of reliability and performance, but they're not (we have a whole web page just to track out of date mirrors).
  - capableweb 6y ago
    "geo-sniffing magic routing", Also known as BGP + CloudFlare Argo (https://www.cloudflare.com/learning/performance/routing-vs-smart-routing/ https://www.cloudflare.com/learning/performance/routing-vs-s...)
  - Havoc 6y ago
    Yeah Arch mirrors in particular seem all over the place on quality. In true Arch spirit you def want to configure that yourself
  - iforgotpassword 6y ago
    I'd guess if this gains traction, the caching on Cloudflare's side will usually make up for this, and if it hits the actual mirror behind it, the load on that one should be lower than before. It reads like they will try to proxy multiple local mirrors so time will tell how well they will pick amongst them. I'm switching for now to see how it will perform over a longer period.
cabraca 6y ago
Now every time cloudflare has a hickup, not only is half the internet unreachable, you can't even update your server... nice /s
- YorickPeterse 6y ago
  At least on Arch Linux you can specify multiple servers, ensuring you don't depend on a single one.
  - cabraca 6y ago
    I know, but lets be real for a moment: Cloudflaremirror will probably be the fastest. This will cause people to configure it and because people are generally lazy they will not configure other mirrors.
p00f 6y ago
I don't understand, is Cloudflare hosting their own mirrors or what?
- Yeri 6y ago
  I understand they just proxy (and cache). They are still requesting packages from the actual repository if they don’t have it cached. But I agree — don’t really see the point. Why not just become an official mirror?
  - mritzmann 6y ago
    They have Cloudflare Workers (with limited disk space), but act as a caching proxy is still the main business of Cloudflare. So i think acting as a proxy make sense for Cloudflare.
  - Matthias247 6y ago
    Being a proxy and cache is their business. All it takes to set things up that way is then a little bit of configuration that points to the actual origin. I think if they would be an official mirror there would be a fair amount of more work being involved. Eg you can’t just randomly wipe the cache anymore and need to have higher persistence guarantees, since you are not supposed to refetch from other mirrors all the time.
Pirate-of-SV 6y ago
I ran rankmirrors (from pacman-contrib) on the mirror and the 5 upstream i got on https://cloudflaremirrors.com/archlinux https://cloudflaremirrors.com/archlinux rankmirrors test the time to download core.db from each mirror. Basically this: curl -s -m 1 -w "%{time_total} %{http_code}" -o/dev/null https://cloudflaremirrors.com/archlinux/core/os/x86_64/core.db.tar.gz The results vary a lot sometimes cloudflare mirrors is the fastest, sometimes it's the slowest. Getting anything between 0.07 and 0.30 $ rankmirrors -t ./ml Querying servers. This may take some time... * * * * * * Servers sorted by time (seconds): https://ftp.lysator.liu.se/pub/archlinux/$repo/os/$arch : 0.113249 https://ftp.acc.umu.se/mirror/archlinux/$repo/os/$arch : 0.120371 https://mirror.osbeck.com/archlinux/$repo/os/$arch : 0.127601 https://archlinux.dynamict.se/$repo/os/$arch : 0.180493 https://ftp.myrveln.se/pub/linux/archlinux/$repo/os/$arch : 0.248406 https://cloudflaremirrors.com/archlinux/$repo/os/$arch : 0.302824 $ rankmirrors -t ./ml Querying servers. This may take some time... * * * * * * Servers sorted by time (seconds): https://cloudflaremirrors.com/archlinux/$repo/os/$arch : 0.070069 https://mirror.osbeck.com/archlinux/$repo/os/$arch : 0.081506 https://ftp.lysator.liu.se/pub/archlinux/$repo/os/$arch : 0.098763 https://archlinux.dynamict.se/$repo/os/$arch : 0.125846 https://ftp.acc.umu.se/mirror/archlinux/$repo/os/$arch : 0.127948 https://ftp.myrveln.se/pub/linux/archlinux/$repo/os/$arch : 0.136529
dtf 6y ago
I can't get it to work, packages upgrades show this error. Any ideas? error: failed retrieving file 'XXX' from cloudflaremirrors.com : HTTP server doesn't seem to support byte ranges. Cannot resume.
- p00f 6y ago
  Clear your partial downloads using sudo pacman -Scc, another comment says that it doesn't support byte ranges so it can neither display progress, nor resume partial downloads
rwmj 6y ago
It would be really nice if web proxies supported this as a generic mechanism, which could be applied to Linux packages, ISOs, git commits, docker layers and more. I wrote up a very half-baked proposal for this kind of thing a while back: https://rwmj.wordpress.com/2013/09/09/half-baked-idea-content-addressable-web-proxy/#content https://rwmj.wordpress.com/2013/09/09/half-baked-idea-conten... Then you could set cloudflare as a proxy (or transparent proxy even?) and it could inject the objects without having to visit the endpoints.
- viraptor 6y ago
  Ideally soon repos will be mirrored to ipfs nodes and that will automatically pull the hash-addressed content from the closest proxy, which may or may not be cloudflare. The protocol is already there. Some initial solutions are already there too: https://github.com/victorb/arch-mirror https://github.com/victorb/arch-mirror
Thomaschaaf 6y ago
Before opening the page I hoped that it would be an npm mirror which can be used in case npm goes down.
sneak 6y ago
There are also other mirrors on cloudflare-ipfs.com. If you are interested in other mirrors, perhaps you should IPFS pin some yourself. :)
gravypod 6y ago
I hope this is expanded to other package repositories (Maven, Docker, npm, etc). I'd really like to make it easier for people at edge locations to build my code.
rb2k_ 6y ago
Another set of mirrors with decent network infrastructure can be found at: http://mirror.facebook.net/ http://mirror.facebook.net/ (Not holding my breath for the kind of responses this comment might get ;))

88 points by Foxboron 6y ago

Been looking for something like this for a while. We see if more repos get added soon (Ubuntu and fedora would be nice!)

dennajort 6y ago

Is it me, or it downloads so fast that the progress output doesn't even show in the terminal ? On Archlinux.

deleted 6y ago
[deleted]
husam212 6y ago
I think the header Content-Length is missing.
- mercora 6y ago
  range requests to resume downloads do not work either... but when encountered i can see the progress of downloading that package with the failed resume attempt... quite weird behavior
rwha 6y ago
I saw the same. My best initial guess is that this is somehow caused by cloudflare either setting the content-type header to text/plain or not setting the accept-ranges header. Or maybe both?
- p00f 6y ago
  > not setting the accept-ranges header Mostly that, because it doesn't support resuming partial downloads either (it says http server does not support byte ranges)
- dennajort 6y ago
  Oh ! totally makes sense, nice guess. and very interesting

SXX 6y ago

Is anyone from CloudFlare or Canonical reading? Please parter to make mirrors for Launchpad PPAs. They're so painfully slow around the globe and there is tons of open source projects that depend on PPAs...

adonese 6y ago
Second this. Cloudflare package repo gets me so excited about it and i wish they extend to ubuntu too.
Ayesh 6y ago
I recently moved to Vietnam, and I sometimes get awfully slow connection speeds to the mirrors nearby my city, to a point that switching to US mirrors is faster. (ftp.[kr|sg].debian.org)
- SXX 6y ago
  I also live in Vietnam for 3-6 months every year so it's part of the reason I posted it. PPA is real pain since as far as I understand they're not mirrored at all or at least they haven't been before. In my experience mirrors in Japan work best in Vietnam for whatever reason.
- stingraycharles 6y ago
  How is Docker hub from Vietnam nowadays? I lived in Cambodia from 2014-2016, and it was nearly impossible to pull Docker images which have large single layers. Would be great to have a Cloudflare solution for this.
  - momothereal 6y ago
    Off-topic. I visited Cambodia for 3 weeks in 2017. It's a beautiful country and I enjoyed every minute. My mind has pondered over living there long-term -- it seems so affordable. How was your experience living there for a while, especially as a developer?
    - 6y ago

james412 6y ago

As much as I'd love to use CloudFlare's uniformly fast network for package update, I'd never use this simply because I already depend on them for too much stuff. It's not like letting them gather logs on my computers updating would be terrible somehow, but if it's so easy to avoid donating more logs and more dependence on them, why not do it for the sake of preserving diversity? Growing companies often exhibit a long term pattern that usually seems innocent at first manifestation: - Offer something for free - Become hugely popular at offering that free thing - Alternatives slowly dwindle in popularity and/or begin to die off - User 'choice' increasingly becomes concentrated in that single option - Company introduces surprising policy change, but users can switch to an alternative if they don't like it - Company introduces differentiating functionality setting its option apart from the alternatives. - Users grow additional dependence on the new functionality, alternatives look increasingly obsolete - Company becomes increasingly aggressive at introducing changes, as they now own the primary alternative It's hard to imagine CloudFlare are somehow going to monopolize package management :) But nonetheless, free services like this generally signal the beginning of this pattern At some point I'm sure e.g. DejaNews seemed like an innocent dependency, and that attempting to look far ahead into the future would not have revealed its eventual sale to Google and the loss of the only comprehensive archive of the early Internet community in existence.

bhouston 6y ago
I think this is still a good service. It is just a mirror. It is a way of giving back to these open source operating systems and cutting their costs. Thus embrace it but be smart about it so that no one is trapped. Dejanews isn't the same. That was an archive of a dying technology. This isn't the same case. Usenet was a dying technology, ultimately killed by http. Cloudflaree has always done the free thing to raise it's popularity. You are right that as companies get bigger it can be harder to keep revenue growing and they slowly start to cross lines they didn't before though. If that company has built reliance on it's free tools which have reduced diversity, when they start to cross lines it can be sort of a trap. This usually isn't intentional now but it is just how things work... Years from now revenue growth might require crossing a line...
YorickPeterse 6y ago

bhouston 6y ago

Need Ubuntu and Raspian. When I saw the title I though they were proxying npmjs or docker hub. The install process of editing a file is a bit painful. Means few people will do it. Somehow would be nice to have this offered in the is as a built in optional feature.

emmelaich 6y ago
Also Fedora and Centos. Please.

overflow_error 6y ago

How is this supposed to work? (I use Arch BTW) supposed I'm in a country X, does Cloudflare basically caching the request to server that are in country X? But I'm already close to that server and I'm always cautious with mirrors that are in country X, often they were out of date :(

lsofzz 6y ago
> How is this supposed to work? (I use Arch BTW) supposed I'm in a country X, does Cloudflare basically caching the request to server that are in country X? But I'm already close to that server and I'm always cautious with mirrors that are in country X, often they were out of date :( According to what I read in their main page https://cloudflaremirrors.com/archlinux https://cloudflaremirrors.com/archlinux, they seem to be caching the packages from mirrors and serving it via their anycast network.
gravitas 6y ago
This is my issue with it as well - it's basically their geo-sniffing magic routing you towards the closest officially listed mirrors for that country (glorified proxy); it makes the mistake of presuming that all those mirrors listed in the Arch country-mirrorlist are created equal in terms of reliability and performance, but they're not (we have a whole web page just to track out of date mirrors).
- capableweb 6y ago
  "geo-sniffing magic routing", Also known as BGP + CloudFlare Argo (https://www.cloudflare.com/learning/performance/routing-vs-smart-routing/ https://www.cloudflare.com/learning/performance/routing-vs-s...)
- Havoc 6y ago
  Yeah Arch mirrors in particular seem all over the place on quality. In true Arch spirit you def want to configure that yourself
- iforgotpassword 6y ago
  I'd guess if this gains traction, the caching on Cloudflare's side will usually make up for this, and if it hits the actual mirror behind it, the load on that one should be lower than before. It reads like they will try to proxy multiple local mirrors so time will tell how well they will pick amongst them. I'm switching for now to see how it will perform over a longer period.

cabraca 6y ago

Now every time cloudflare has a hickup, not only is half the internet unreachable, you can't even update your server... nice /s

YorickPeterse 6y ago
At least on Arch Linux you can specify multiple servers, ensuring you don't depend on a single one.
- cabraca 6y ago
  I know, but lets be real for a moment: Cloudflaremirror will probably be the fastest. This will cause people to configure it and because people are generally lazy they will not configure other mirrors.

p00f 6y ago

I don't understand, is Cloudflare hosting their own mirrors or what?

Yeri 6y ago
I understand they just proxy (and cache). They are still requesting packages from the actual repository if they don’t have it cached. But I agree — don’t really see the point. Why not just become an official mirror?
- mritzmann 6y ago
  They have Cloudflare Workers (with limited disk space), but act as a caching proxy is still the main business of Cloudflare. So i think acting as a proxy make sense for Cloudflare.
- Matthias247 6y ago
  Being a proxy and cache is their business. All it takes to set things up that way is then a little bit of configuration that points to the actual origin. I think if they would be an official mirror there would be a fair amount of more work being involved. Eg you can’t just randomly wipe the cache anymore and need to have higher persistence guarantees, since you are not supposed to refetch from other mirrors all the time.

Pirate-of-SV 6y ago

I ran rankmirrors (from pacman-contrib) on the mirror and the 5 upstream i got on https://cloudflaremirrors.com/archlinux https://cloudflaremirrors.com/archlinux rankmirrors test the time to download core.db from each mirror. Basically this: curl -s -m 1 -w "%{time_total} %{http_code}" -o/dev/null https://cloudflaremirrors.com/archlinux/core/os/x86_64/core.db.tar.gz The results vary a lot sometimes cloudflare mirrors is the fastest, sometimes it's the slowest. Getting anything between 0.07 and 0.30 $ rankmirrors -t ./ml Querying servers. This may take some time... * * * * * * Servers sorted by time (seconds): https://ftp.lysator.liu.se/pub/archlinux/$repo/os/$arch : 0.113249 https://ftp.acc.umu.se/mirror/archlinux/$repo/os/$arch : 0.120371 https://mirror.osbeck.com/archlinux/$repo/os/$arch : 0.127601 https://archlinux.dynamict.se/$repo/os/$arch : 0.180493 https://ftp.myrveln.se/pub/linux/archlinux/$repo/os/$arch : 0.248406 https://cloudflaremirrors.com/archlinux/$repo/os/$arch : 0.302824 $ rankmirrors -t ./ml Querying servers. This may take some time... * * * * * * Servers sorted by time (seconds): https://cloudflaremirrors.com/archlinux/$repo/os/$arch : 0.070069 https://mirror.osbeck.com/archlinux/$repo/os/$arch : 0.081506 https://ftp.lysator.liu.se/pub/archlinux/$repo/os/$arch : 0.098763 https://archlinux.dynamict.se/$repo/os/$arch : 0.125846 https://ftp.acc.umu.se/mirror/archlinux/$repo/os/$arch : 0.127948 https://ftp.myrveln.se/pub/linux/archlinux/$repo/os/$arch : 0.136529

dtf 6y ago

I can't get it to work, packages upgrades show this error. Any ideas? error: failed retrieving file 'XXX' from cloudflaremirrors.com : HTTP server doesn't seem to support byte ranges. Cannot resume.

p00f 6y ago
Clear your partial downloads using sudo pacman -Scc, another comment says that it doesn't support byte ranges so it can neither display progress, nor resume partial downloads

rwmj 6y ago

It would be really nice if web proxies supported this as a generic mechanism, which could be applied to Linux packages, ISOs, git commits, docker layers and more. I wrote up a very half-baked proposal for this kind of thing a while back: https://rwmj.wordpress.com/2013/09/09/half-baked-idea-content-addressable-web-proxy/#content https://rwmj.wordpress.com/2013/09/09/half-baked-idea-conten... Then you could set cloudflare as a proxy (or transparent proxy even?) and it could inject the objects without having to visit the endpoints.

viraptor 6y ago
Ideally soon repos will be mirrored to ipfs nodes and that will automatically pull the hash-addressed content from the closest proxy, which may or may not be cloudflare. The protocol is already there. Some initial solutions are already there too: https://github.com/victorb/arch-mirror https://github.com/victorb/arch-mirror

Thomaschaaf 6y ago

Before opening the page I hoped that it would be an npm mirror which can be used in case npm goes down.

sneak 6y ago

There are also other mirrors on cloudflare-ipfs.com. If you are interested in other mirrors, perhaps you should IPFS pin some yourself. :)

gravypod 6y ago

I hope this is expanded to other package repositories (Maven, Docker, npm, etc). I'd really like to make it easier for people at edge locations to build my code.

rb2k_ 6y ago

Another set of mirrors with decent network infrastructure can be found at: http://mirror.facebook.net/ http://mirror.facebook.net/ (Not holding my breath for the kind of responses this comment might get ;))