4 ms·
I'm running a fairly large mining farm. This article is spot on. Another point that isn't discussed is that the software solutions to manage large numbers of P
by latchkey 9y ago
I'm running a fairly large mining farm. This article is spot on.
Another point that isn't discussed is that the software solutions to manage large numbers of Pi/Beaglebone class machines is non-existent.
Because of the fast time to market, these machines are unstable... parts burn out all the time, they go offline randomly, a bad PSU can make a machine reboot every 10 minutes (making doing anything with it difficult), etc. Never mind, just keeping track of where an individual machine is in a data center (needle in a haystack).
There is a couple commercial and open source offerings, but the majority of them were built for when farms were only a handful of machines. Scaling to thousands of machines across different data centers has a lot of the same scaling issues I'd have running the same number of EC2 instances.
If you look on Github, you can see that most public development in the mining community stopped around 2014-2015 and hasn't picked up again. Even the largest pools have taken things closed source.
I find it amazing that literally billions of dollars is flowing through this stuff. If it wasn't for math being so factual, I'd really question it all because it is all barely held together with heat sink grease.
- ianai 9y agoSounds like you’re trying to run a data center without the full managerial/software complement.
- latchkey 9y agoThat is exactly what is happening. 2 data centers and growing. As a result, I'm building it for myself as I go, which is a fun sadistic exercise. Prometheus / Grafana (awesome tools), React / TypeScript, golang
- mhluongo 9y agoTalk to the guys at HashRabbit. They've been quietly working on this for years
- latchkey 9y agoI've done a lot of googling on them. Even found the new Gabe Evans... https://github.com/ivy https://github.com/ivy (good for her). Had they invested their $500k in BTC in 2015 instead of HashRabbit... 100m btc @ ~$200 = $1b. It seems like they aren't really in business. Is there something you know that I don't?
- mhluongo 9y agoEh, that's always been a struggle as a crypto startup- BitPay had the opposite issue for years. Consider reaching out, I'm not sure they've published the latest stuff
- sanxiyn 9y ago> Never mind, just keeping track of where an individual machine is in a data center (needle in a haystack). I have a friend who contracted to develop a custom DHCP server to allocate addresses based on Ethernet ports. This is superior to static IP configuration because you invariably make configuration mistakes. Cisco provides this feature, but Cisco is expensive, hence a lucrative side job for my friend. There seems to be lots and lots of opportunities like this to provide custom softwares for mining farm.
- latchkey 9y agoActually, no. DHCP is the common misconception in this industry. The machines are originally come out of the box booting with DHCP and I immediately switch to static when we deploy them. This is based on experience running a large number of these machines originally with DHCP. There is a few reasons: 1. They are physical machines in a physical location. Every DC, Row, Rack, Shelf, Position is defined and set with an IP address before the machine is even installed. It is very important to track which machine is where for maintenance reasons (remote hands, etc). If I don't use IP address, I'm stuck using MAC addresses and now I'm keeping track of two pieces of information instead of one. Oh and MAC address isn't printed on the outside, sigh. 2. DHCP can leave gaps in IP ranges. Unless you deploy your own custom server that is smart about always giving out the next IP, in practice, I see gaps form. You end up wasting private space and having machines cross subnets. I don't run the network in my DC, so if I want more IP space, I have to ask for it. One more thing to do, no thanks. 3. These machines tend to reboot a lot since they are basically hastily made junk from China. Time is money. Every second they are down means lost revenue. DHCP takes a minimum of 5-30 seconds to assign an IP. It adds up when you have thousands of machines. 4. More network load. 100mbit at best. Even updating software on them all can take hours. 5. An extra dependency, what happens if my DHCP server dies? The fewer moving parts, the better. 6. Speeds up the installation process, which uses DHCP to assign the initial IP range. Can't have two DHCP servers on the network conflicting with each other. Installation is as follows: 1. Set the DHCP server to the range I want to use. 2. Unplug ethernet on all machines. 3. Turn them all on and get them warmed up. 4. Plug ethernet in one at a time, in order (to match their location) and watch the leases file. 5. Run a script that bulk sets their IP address as static (same address). When you're turning on 500 new machines and it is an all day ordeal, this is the fastest process I've found so far. In the future, I'd like to have a custom DHCP server (there is a DHCP golang server that I'd like to use) that already knows the ranges necessary, allocates the IP and pings my management system to set the location of the machine. Right now, I just keep the data in a Google Sheet, which works well enough.
- andyway 9y agoWe are using Hive OS to manage ~1500 GPU and ASIC rigs. Switched to it from our custom developed management layer. Saved us a lot time on development. Though it lacks true industrial datacenter management capabilities we are hashing 10-12% more on m-o-m basis.
- latchkey 9y agoAwesome that it works for you. Unfortunately, it doesn't support all of the hardware we use. Probably could be made to support it, but I'm honestly a bit wary of how much it installs onto each machine and their complicated bash code has zero tests [1]. I'd prefer a much simpler and solid solution. [1] https://github.com/minershive/hiveos-asic/blob/master/hive/bin/agent https://github.com/minershive/hiveos-asic/blob/master/hive/b...