5 ms·

Rather than listing everything I'm hosting at my home server, I'll just share what saved me the most time, repeatedly: Mirrors of various package repositories

Rather than listing everything I'm hosting at my home server, I'll just share what saved me the most time, repeatedly: Mirrors of various package repositories I use. I'm currently mirroring npm, crates, Arch packages, clojars, maven and some other things, then all my machines (desktop, servers and laptops) point to the mirror rather than directly to upstream. Some of them are mirroring dynamically (basically a cache at that point, do this for npm for example) while others I fetch the entire repository and keep on disk, cleaning out old packages when needed (I do this for Arch packages for example). Best benefit is that downloading stuff and updating my machines takes seconds now, even if there is multi-GB updates to do, and a secondary effect is that I'm not impacted by any downtime from npm et al. Saved my bacon more than once.

nikisweeting 4y ago
Can you share a bit about how you set this up? Also curious if anyone has taken this a step further and MITM Squid proxied their whole home to cache all responses >100mb or something like that.
- capableweb 4y ago
  Basically just a bunch of shellscripts for most part, that is run with systemd timers on a PC-like server (consumer components), served via Caddy. Npm registry is using verdaccio, arch packages is using https://gitlab.archlinux.org/archlinux/infrastructure/-/blob/master/roles/syncrepo/files/syncrepo-template.sh https://gitlab.archlinux.org/archlinux/infrastructure/-/blob... + what is outlined at https://wiki.archlinux.org/title/DeveloperWiki:NewMirrors https://wiki.archlinux.org/title/DeveloperWiki:NewMirrors, clojars is just reading the list of packages and downloading them one by one each day. Not a unified or nicely done setup by any means, just thrown together to solve the problem at hand.
- grepfru_it 4y ago
  I used to do that for my servers. It works quite well but packages update so frequently you don’t see a major benefit
- Nzen 4y ago
  I setup apache archiva [0] to cache maven binaries at work. It was okay. [0] https://archiva.apache.org/ https://archiva.apache.org/
- jamroom 4y ago
  Gitea works really well for this - just choose "new migration" and set the remote up as a mirror - super easy.
Bedon292 4y ago
I would be interested in doing some sort of hybrid mirror / cache of a few repos, if I could do them just in time style. I don't need all of pypi (nor do I want 13.5TB of packages). I probably only ever use a few hundred packages at most. I would like to point all my systems at my server. And if I `pip install pandas` and its not on my server the server grabs it and passes it through and syncs that package locally from then on. Same with yum, npm, docker, or whatever. And I just realized I could use Artifactory as a caching proxy and at least save some time there. However, that doesn't mirror the package it just caches that specific version. I would be very interested in something where the system sees that I use `pandas` now and will mirror it. Or give it a requirements.txt and it flags all those packages and dependencies for mirroring.
- capableweb 4y ago
  I don't know of any particular solution for the python ecosystem, I don't use it often enough to justify dealing with that can of worms. But for npm, there is Verdaccio which does exactly what you want, and is what I'm using.
- pbronez 4y ago
  Maybe DevPi would work? https://devpi.net/docs/devpi/devpi/stable/%2Bd/index.html https://devpi.net/docs/devpi/devpi/stable/%2Bd/index.html https://blog.jcharistech.com/2022/08/13/how-to-run-a-pypi-mirror-locally-on-your-laptop-using-devpi/ https://blog.jcharistech.com/2022/08/13/how-to-run-a-pypi-mi... another option is https://pypi.org/project/python-pypi-mirror/ https://pypi.org/project/python-pypi-mirror/
  - Bedon292 4y ago
    It looks like DevPi may just work as a caching proxy as well. I also found Bandersnatch too. https://github.com/pypa/bandersnatch https://github.com/pypa/bandersnatch which is a configurable mirror with allow / block lists. Essentially want something like DevPi but add the package to the bandersnatch allow list and mirror it from then on. With some extra large packages in the deny list. Probably possible to wire that all up reasonably well, but probably just using a caching proxy is 90%+ of the improvement anyways. So may just stick with that.
cableshaft 4y ago
How much space does it take to mirror npm currently? Tried finding that online but not sure where to look.
- hbn 4y ago
  It sounded like they're essentially caching npm, not mirroring the whole thing > Some of them are mirroring dynamically (basically a cache at that point, do this for npm for example)
- capableweb 4y ago
  I've done it some year ago last time, in order to dig out some statistics, it was around 1.8TB then, just counting the latest available version of each package (not every version of every package). But as another commentator said, I'm only caching packages that already been fetched now, as I have no need for everything in the registry (most of it is junk to be honest).

nikisweeting 4y ago

Can you share a bit about how you set this up? Also curious if anyone has taken this a step further and MITM Squid proxied their whole home to cache all responses >100mb or something like that.

capableweb 4y ago
Basically just a bunch of shellscripts for most part, that is run with systemd timers on a PC-like server (consumer components), served via Caddy. Npm registry is using verdaccio, arch packages is using https://gitlab.archlinux.org/archlinux/infrastructure/-/blob/master/roles/syncrepo/files/syncrepo-template.sh https://gitlab.archlinux.org/archlinux/infrastructure/-/blob... + what is outlined at https://wiki.archlinux.org/title/DeveloperWiki:NewMirrors https://wiki.archlinux.org/title/DeveloperWiki:NewMirrors, clojars is just reading the list of packages and downloading them one by one each day. Not a unified or nicely done setup by any means, just thrown together to solve the problem at hand.
grepfru_it 4y ago
I used to do that for my servers. It works quite well but packages update so frequently you don’t see a major benefit
Nzen 4y ago
I setup apache archiva [0] to cache maven binaries at work. It was okay. [0] https://archiva.apache.org/ https://archiva.apache.org/
jamroom 4y ago
Gitea works really well for this - just choose "new migration" and set the remote up as a mirror - super easy.

Bedon292 4y ago

I would be interested in doing some sort of hybrid mirror / cache of a few repos, if I could do them just in time style. I don't need all of pypi (nor do I want 13.5TB of packages). I probably only ever use a few hundred packages at most. I would like to point all my systems at my server. And if I `pip install pandas` and its not on my server the server grabs it and passes it through and syncs that package locally from then on. Same with yum, npm, docker, or whatever. And I just realized I could use Artifactory as a caching proxy and at least save some time there. However, that doesn't mirror the package it just caches that specific version. I would be very interested in something where the system sees that I use `pandas` now and will mirror it. Or give it a requirements.txt and it flags all those packages and dependencies for mirroring.

capableweb 4y ago
I don't know of any particular solution for the python ecosystem, I don't use it often enough to justify dealing with that can of worms. But for npm, there is Verdaccio which does exactly what you want, and is what I'm using.
pbronez 4y ago
Maybe DevPi would work? https://devpi.net/docs/devpi/devpi/stable/%2Bd/index.html https://devpi.net/docs/devpi/devpi/stable/%2Bd/index.html https://blog.jcharistech.com/2022/08/13/how-to-run-a-pypi-mirror-locally-on-your-laptop-using-devpi/ https://blog.jcharistech.com/2022/08/13/how-to-run-a-pypi-mi... another option is https://pypi.org/project/python-pypi-mirror/ https://pypi.org/project/python-pypi-mirror/
- Bedon292 4y ago
  It looks like DevPi may just work as a caching proxy as well. I also found Bandersnatch too. https://github.com/pypa/bandersnatch https://github.com/pypa/bandersnatch which is a configurable mirror with allow / block lists. Essentially want something like DevPi but add the package to the bandersnatch allow list and mirror it from then on. With some extra large packages in the deny list. Probably possible to wire that all up reasonably well, but probably just using a caching proxy is 90%+ of the improvement anyways. So may just stick with that.

cableshaft 4y ago

How much space does it take to mirror npm currently? Tried finding that online but not sure where to look.

hbn 4y ago
It sounded like they're essentially caching npm, not mirroring the whole thing > Some of them are mirroring dynamically (basically a cache at that point, do this for npm for example)
capableweb 4y ago
I've done it some year ago last time, in order to dig out some statistics, it was around 1.8TB then, just counting the latest available version of each package (not every version of every package). But as another commentator said, I'm only caching packages that already been fetched now, as I have no need for everything in the registry (most of it is junk to be honest).