4 ms·

Because the 12 GPUS in them are a lot more important than the networking speed. =) They were for mining ETH... we've turned them off though now that PoS has be

Because the 12 GPUS in them are a lot more important than the networking speed. =) They were for mining ETH... we've turned them off though now that PoS has been successful.

justinclift 3y ago
For large-ish scale distributed updates like that, maybe some kind of P2P type of approach would work well? IBM used to use a variant of Bittorrent to internally distribute OS images between machines. That was more than a decade ago though, when I was last working with that stuff.
- latchkey 3y ago
  Answered below. https://news.ycombinator.com/item?id=36052632 https://news.ycombinator.com/item?id=36052632 Another issue with that is that the systems I was running can go offline at any time. P2P, which could work, kind of wants a lot more uptime than what we had. It would just add some complexity to deal with individual downtime.
  - hnlmorg 3y ago
    Interesting stuff. Thanks for the insight
    - latchkey 3y ago
      What I ended up with was really neat. machine <-> cloudflare <-> github CI would run, build a binary that was stored as an asset in github. Since the project is private, I had to build a proxy in front of it to pass the auth token, so I used CF workers. GH also has limitations on number of downloads, so CF also worked as a proxy to reduce the connections to GH. I then had another private repo with a json file in it where I could specify CIDR ranges and version numbers. It also went through a similar CF worker path. Machines regularly/randomly hit a CF worker with their current version and ip address. The worker would grab the json file and then if a new version was needed, in the same response, return the binary (or return a 304 not modified). The binary would download, copy itself into position and then quit. The OS would restart it a minute later. It worked exceptionally well. With CIDR based ranges, I could release a new version and only update a single machine or every machine. It made testing really easy. The initial install process was just a single line bash/curl to request to get the latest version of the app. I also had another 'ping' endpoint, where I could send commands to the machine that would be executed by my golang app (running as root). The machine would ping, and the pong response would be some json that I could use to do anything on the machine. I had a postgres database running in GCP and used GPC functions. I stored machine metrics and other individual worker data in there that just needed to be updated every ping. So, I could just update column and the machine would eventually ping, grab the command out of the column and then erase it. It was all eventually consistent and idempotent. At ~30k workers, we had about 60 requests per second 24/7 and cost us at most about $300 a month total. It worked flawlessly. If anything on the backend went down, the machines would just keep doing their thing.

Because the 12 GPUS in them are a lot more important than the networking speed. =) They were for mining ETH... we've turned them off though now that PoS has been successful.

justinclift 3y ago

For large-ish scale distributed updates like that, maybe some kind of P2P type of approach would work well? IBM used to use a variant of Bittorrent to internally distribute OS images between machines. That was more than a decade ago though, when I was last working with that stuff.

latchkey 3y ago
Answered below. https://news.ycombinator.com/item?id=36052632 https://news.ycombinator.com/item?id=36052632 Another issue with that is that the systems I was running can go offline at any time. P2P, which could work, kind of wants a lot more uptime than what we had. It would just add some complexity to deal with individual downtime.
- hnlmorg 3y ago
  Interesting stuff. Thanks for the insight
  - latchkey 3y ago
    What I ended up with was really neat. machine <-> cloudflare <-> github CI would run, build a binary that was stored as an asset in github. Since the project is private, I had to build a proxy in front of it to pass the auth token, so I used CF workers. GH also has limitations on number of downloads, so CF also worked as a proxy to reduce the connections to GH. I then had another private repo with a json file in it where I could specify CIDR ranges and version numbers. It also went through a similar CF worker path. Machines regularly/randomly hit a CF worker with their current version and ip address. The worker would grab the json file and then if a new version was needed, in the same response, return the binary (or return a 304 not modified). The binary would download, copy itself into position and then quit. The OS would restart it a minute later. It worked exceptionally well. With CIDR based ranges, I could release a new version and only update a single machine or every machine. It made testing really easy. The initial install process was just a single line bash/curl to request to get the latest version of the app. I also had another 'ping' endpoint, where I could send commands to the machine that would be executed by my golang app (running as root). The machine would ping, and the pong response would be some json that I could use to do anything on the machine. I had a postgres database running in GCP and used GPC functions. I stored machine metrics and other individual worker data in there that just needed to be updated every ping. So, I could just update column and the machine would eventually ping, grab the command out of the column and then erase it. It was all eventually consistent and idempotent. At ~30k workers, we had about 60 requests per second 24/7 and cost us at most about $300 a month total. It worked flawlessly. If anything on the backend went down, the machines would just keep doing their thing.