4 ms·

> I have many thousands of machines running in multiple datacenters and even getting a ~4mb binary distributed onto them without saturating the network (100mbit

> I have many thousands of machines running in multiple datacenters and even getting a ~4mb binary distributed onto them without saturating the network (100mbit) and slowing everything else down, is a bit of a challenge. Maybe I suggest CAS (Content-addressable storage) or something similar for distributing it instead? I've had good success using torrents for distributing large binaries to a large fleet of servers (that were also in close proximity to each other physically, in clusters with some more distance between them) relatively easily.

latchkey 5y ago
Thanks for the suggestion, but as weird as this sounds, we also don't have central servers to use at the data centers for this. The machines don't all need to be running the same version of the binary at the same time, so I took a simpler approach, which is that each machine checks for updates on a random schedule over a configureable amount of time. This distributes the load evenly and everything becomes eventually consistent. After about an hour everything is updated without issues. I use CloudFlare workers to cache at the edge. On a push to master, the binary is built in Github CI and a release is made after all the tests pass. There is a simple json file where I can define release channels for a specific version for a specific CIDR (also on a release ci/cd as well, so I can validate the json with tests). I can upgrade/downgrade and test on subsets of machines. The machines on their random schedule hit the CF worker which checks the cached json file and either returns a 304 or the binary to install depending on the parameters passed in on the query string (current version, ip address, etc.). Binary is downloaded, installed and then it quits and systemd restarts the new version. Works great.
- aspenmayer 5y ago
  I’d seriously look into BitTorrent for this use case, as it sounds ideal. You can even configure your client to run a script after torrents are completed, so you could script the update migration with minimal code. You can also setup upload/download limits easily. I think Resilio Sync might also be a good option; I think it may even use BitTorrent internally. (It was formerly known as BitTorrent Sync, not sure why they changed the name.)
  - latchkey 5y ago
    Over engineering. You're trying to solve a problem that I don't have. What I have now works great with a page of well tested code and doesn't have the complexity of BT. I'll duplicate the comment above that the machines are all on separate vlans and don't really talk to many other machines.
    - aspenmayer 5y ago
      You know your stack best. I appreciate the insights you have into your process and workflow. Do you use anything special to bridge vlans like Tailscale or WireGuard?
      - latchkey 5y ago
        Thanks, indeed. P2P file distribution is an interesting idea, but doesn't really solve any major problems here. ~3mb file to a lot of machines in an hour a few times a week. Simplest solution is that each machine can just download the file itself. The data centers don't need to talk to each other at all. We use wg at each dc for vpn access. I do need to spend a bit more time with my own wg client setup so that I can switch between dc's more easily... right now it is kind of a manual process and that is definitely a pain. One of these days...
        aspenmayer 5y ago
        I’ve heard that there is third party tooling that makes managing wg easier but I haven’t really looked into that space. I need to do it though, as that’s one of the major pain points of wg, that each client needs to be whitelisted serverside essentially. Edit: Maybe something simple like rsync or rclone would be best for the update sync/distribution mechanism rather than BT.
        latchkey 5y ago
        A binary diff (or even plugin) mechanism could potentially optimize the download size even further. Such that only the parts of the app that change need to be downloaded. That said, for now, downloading the binary really is a solved problem for me, with a relatively simple solution. I'd rather work on new features that drive revenue. =)
- capableweb 5y ago
  > Thanks for the suggestion, but as weird as this sounds, we also don't have central servers to use at the data centers for this. Same here, hence the suggestion of using P2P software (BitTorrent) for letting clients fetching the data from each other (together with initial entrypoint for the deployment, obviously), and you'll avoid the congestion issue as clients will fetch the data from whatever node is nearest, and configured properly will only fetch data outside the internal network once, after that it's all local data transfer (within the same data center).
  - latchkey 5y ago
    The machines are all on separate vlans. Only a few of them can talk to each other at a time. There isn't a huge amount of benefit there.
- westurner 5y ago
  Notes re: distributed temporal Data Locality, package mirroring, and CAS such as IPFS: "Draft PEP: PyPI cost solutions: CI, mirrors, containers, and caching to scale" (2020) https://groups.google.com/g/pypa-dev/c/Pdnoi8UeFZ8 https://groups.google.com/g/pypa-dev/c/Pdnoi8UeFZ8 https://discuss.python.org/t/draft-pep-pypi-cost-solutions-ci-mirrors-containers-and-caching-to-scale/3681 https://discuss.python.org/t/draft-pep-pypi-cost-solutions-c... apt-transport-ipfs: https://github.com/JaquerEspeis/apt-transport-ipfs https://github.com/JaquerEspeis/apt-transport-ipfs Gx: https://github.com/whyrusleeping/gx https://github.com/whyrusleeping/gx IPFS: https://wiki.archlinux.org/title/InterPlanetary_File_System https://wiki.archlinux.org/title/InterPlanetary_File_System

latchkey 5y ago

Thanks for the suggestion, but as weird as this sounds, we also don't have central servers to use at the data centers for this. The machines don't all need to be running the same version of the binary at the same time, so I took a simpler approach, which is that each machine checks for updates on a random schedule over a configureable amount of time. This distributes the load evenly and everything becomes eventually consistent. After about an hour everything is updated without issues. I use CloudFlare workers to cache at the edge. On a push to master, the binary is built in Github CI and a release is made after all the tests pass. There is a simple json file where I can define release channels for a specific version for a specific CIDR (also on a release ci/cd as well, so I can validate the json with tests). I can upgrade/downgrade and test on subsets of machines. The machines on their random schedule hit the CF worker which checks the cached json file and either returns a 304 or the binary to install depending on the parameters passed in on the query string (current version, ip address, etc.). Binary is downloaded, installed and then it quits and systemd restarts the new version. Works great.

aspenmayer 5y ago
I’d seriously look into BitTorrent for this use case, as it sounds ideal. You can even configure your client to run a script after torrents are completed, so you could script the update migration with minimal code. You can also setup upload/download limits easily. I think Resilio Sync might also be a good option; I think it may even use BitTorrent internally. (It was formerly known as BitTorrent Sync, not sure why they changed the name.)
- latchkey 5y ago
  Over engineering. You're trying to solve a problem that I don't have. What I have now works great with a page of well tested code and doesn't have the complexity of BT. I'll duplicate the comment above that the machines are all on separate vlans and don't really talk to many other machines.
  - aspenmayer 5y ago
    You know your stack best. I appreciate the insights you have into your process and workflow. Do you use anything special to bridge vlans like Tailscale or WireGuard?
    - latchkey 5y ago
      Thanks, indeed. P2P file distribution is an interesting idea, but doesn't really solve any major problems here. ~3mb file to a lot of machines in an hour a few times a week. Simplest solution is that each machine can just download the file itself. The data centers don't need to talk to each other at all. We use wg at each dc for vpn access. I do need to spend a bit more time with my own wg client setup so that I can switch between dc's more easily... right now it is kind of a manual process and that is definitely a pain. One of these days...
      - aspenmayer 5y ago
        I’ve heard that there is third party tooling that makes managing wg easier but I haven’t really looked into that space. I need to do it though, as that’s one of the major pain points of wg, that each client needs to be whitelisted serverside essentially. Edit: Maybe something simple like rsync or rclone would be best for the update sync/distribution mechanism rather than BT.
        latchkey 5y ago
        A binary diff (or even plugin) mechanism could potentially optimize the download size even further. Such that only the parts of the app that change need to be downloaded. That said, for now, downloading the binary really is a solved problem for me, with a relatively simple solution. I'd rather work on new features that drive revenue. =)
capableweb 5y ago
> Thanks for the suggestion, but as weird as this sounds, we also don't have central servers to use at the data centers for this. Same here, hence the suggestion of using P2P software (BitTorrent) for letting clients fetching the data from each other (together with initial entrypoint for the deployment, obviously), and you'll avoid the congestion issue as clients will fetch the data from whatever node is nearest, and configured properly will only fetch data outside the internal network once, after that it's all local data transfer (within the same data center).
- latchkey 5y ago
  The machines are all on separate vlans. Only a few of them can talk to each other at a time. There isn't a huge amount of benefit there.
westurner 5y ago
Notes re: distributed temporal Data Locality, package mirroring, and CAS such as IPFS: "Draft PEP: PyPI cost solutions: CI, mirrors, containers, and caching to scale" (2020) https://groups.google.com/g/pypa-dev/c/Pdnoi8UeFZ8 https://groups.google.com/g/pypa-dev/c/Pdnoi8UeFZ8 https://discuss.python.org/t/draft-pep-pypi-cost-solutions-ci-mirrors-containers-and-caching-to-scale/3681 https://discuss.python.org/t/draft-pep-pypi-cost-solutions-c... apt-transport-ipfs: https://github.com/JaquerEspeis/apt-transport-ipfs https://github.com/JaquerEspeis/apt-transport-ipfs Gx: https://github.com/whyrusleeping/gx https://github.com/whyrusleeping/gx IPFS: https://wiki.archlinux.org/title/InterPlanetary_File_System https://wiki.archlinux.org/title/InterPlanetary_File_System