3 ms·

> Never mind, just keeping track of where an individual machine is in a data center (needle in a haystack). I have a friend who contracted to develop a custom

> Never mind, just keeping track of where an individual machine is in a data center (needle in a haystack). I have a friend who contracted to develop a custom DHCP server to allocate addresses based on Ethernet ports. This is superior to static IP configuration because you invariably make configuration mistakes. Cisco provides this feature, but Cisco is expensive, hence a lucrative side job for my friend. There seems to be lots and lots of opportunities like this to provide custom softwares for mining farm.

latchkey 9y ago
Actually, no. DHCP is the common misconception in this industry. The machines are originally come out of the box booting with DHCP and I immediately switch to static when we deploy them. This is based on experience running a large number of these machines originally with DHCP. There is a few reasons: 1. They are physical machines in a physical location. Every DC, Row, Rack, Shelf, Position is defined and set with an IP address before the machine is even installed. It is very important to track which machine is where for maintenance reasons (remote hands, etc). If I don't use IP address, I'm stuck using MAC addresses and now I'm keeping track of two pieces of information instead of one. Oh and MAC address isn't printed on the outside, sigh. 2. DHCP can leave gaps in IP ranges. Unless you deploy your own custom server that is smart about always giving out the next IP, in practice, I see gaps form. You end up wasting private space and having machines cross subnets. I don't run the network in my DC, so if I want more IP space, I have to ask for it. One more thing to do, no thanks. 3. These machines tend to reboot a lot since they are basically hastily made junk from China. Time is money. Every second they are down means lost revenue. DHCP takes a minimum of 5-30 seconds to assign an IP. It adds up when you have thousands of machines. 4. More network load. 100mbit at best. Even updating software on them all can take hours. 5. An extra dependency, what happens if my DHCP server dies? The fewer moving parts, the better. 6. Speeds up the installation process, which uses DHCP to assign the initial IP range. Can't have two DHCP servers on the network conflicting with each other. Installation is as follows: 1. Set the DHCP server to the range I want to use. 2. Unplug ethernet on all machines. 3. Turn them all on and get them warmed up. 4. Plug ethernet in one at a time, in order (to match their location) and watch the leases file. 5. Run a script that bulk sets their IP address as static (same address). When you're turning on 500 new machines and it is an all day ordeal, this is the fastest process I've found so far. In the future, I'd like to have a custom DHCP server (there is a DHCP golang server that I'd like to use) that already knows the ranges necessary, allocates the IP and pings my management system to set the location of the machine. Right now, I just keep the data in a Google Sheet, which works well enough.
- sanxiyn 9y ago
  Eh... your conclusion is that you'd like to have a custom DHCP server. I mentioned such custom DHCP server. Why do you start with "Actually, no"? We seem to be in agreement.
  - latchkey 9y ago
    The DHCP server is just to get the machines booted, not to assign IP's on an ongoing basis. I don't keep it turned on (reference to your 'superior to static IP configuration'). In other words, if I could have the machines come out of the box with the correct static IP address (flashed at the factory), I would go that route and not have a DHCP server at all. That would be the absolute optimal solution. We aren't there yet. Some of our hardware has a serial usb port for TTL. For those, we can set the IP address before we even put it on the shelf.
    - sanxiyn 9y ago
      I also mentioned why it is superior to static IP configuration. If you never make mistakes settings static IP addresses static IP is okay. But you will make mistakes.
      - latchkey 9y ago
        I know exactly how many machines I have, where they are located and what IP address they should have. If a single machine isn't doing what I expect, it is immediately obvious in the pretty graphs that I have with Grafana/Prometheus. Any 'mistakes' are easily rectified, but so far, we haven't had a single issue along these lines. I've tried it both ways, the benefits of static IP, for the reasons I mentioned, far outweigh any negatives. DHCP is great for a coffee shop, workplace or college campus, where you don't know anything about your clients or care to. In my case, I need to know exactly who is on my network at all times and exactly where they are located, without having to cross reference switch ports and MAC addresses. In one DC, I don't even have access to the switches.
        9y ago

latchkey 9y ago

Actually, no. DHCP is the common misconception in this industry. The machines are originally come out of the box booting with DHCP and I immediately switch to static when we deploy them. This is based on experience running a large number of these machines originally with DHCP. There is a few reasons: 1. They are physical machines in a physical location. Every DC, Row, Rack, Shelf, Position is defined and set with an IP address before the machine is even installed. It is very important to track which machine is where for maintenance reasons (remote hands, etc). If I don't use IP address, I'm stuck using MAC addresses and now I'm keeping track of two pieces of information instead of one. Oh and MAC address isn't printed on the outside, sigh. 2. DHCP can leave gaps in IP ranges. Unless you deploy your own custom server that is smart about always giving out the next IP, in practice, I see gaps form. You end up wasting private space and having machines cross subnets. I don't run the network in my DC, so if I want more IP space, I have to ask for it. One more thing to do, no thanks. 3. These machines tend to reboot a lot since they are basically hastily made junk from China. Time is money. Every second they are down means lost revenue. DHCP takes a minimum of 5-30 seconds to assign an IP. It adds up when you have thousands of machines. 4. More network load. 100mbit at best. Even updating software on them all can take hours. 5. An extra dependency, what happens if my DHCP server dies? The fewer moving parts, the better. 6. Speeds up the installation process, which uses DHCP to assign the initial IP range. Can't have two DHCP servers on the network conflicting with each other. Installation is as follows: 1. Set the DHCP server to the range I want to use. 2. Unplug ethernet on all machines. 3. Turn them all on and get them warmed up. 4. Plug ethernet in one at a time, in order (to match their location) and watch the leases file. 5. Run a script that bulk sets their IP address as static (same address). When you're turning on 500 new machines and it is an all day ordeal, this is the fastest process I've found so far. In the future, I'd like to have a custom DHCP server (there is a DHCP golang server that I'd like to use) that already knows the ranges necessary, allocates the IP and pings my management system to set the location of the machine. Right now, I just keep the data in a Google Sheet, which works well enough.

sanxiyn 9y ago
Eh... your conclusion is that you'd like to have a custom DHCP server. I mentioned such custom DHCP server. Why do you start with "Actually, no"? We seem to be in agreement.
- latchkey 9y ago
  The DHCP server is just to get the machines booted, not to assign IP's on an ongoing basis. I don't keep it turned on (reference to your 'superior to static IP configuration'). In other words, if I could have the machines come out of the box with the correct static IP address (flashed at the factory), I would go that route and not have a DHCP server at all. That would be the absolute optimal solution. We aren't there yet. Some of our hardware has a serial usb port for TTL. For those, we can set the IP address before we even put it on the shelf.
  - sanxiyn 9y ago
    I also mentioned why it is superior to static IP configuration. If you never make mistakes settings static IP addresses static IP is okay. But you will make mistakes.
    - latchkey 9y ago
      I know exactly how many machines I have, where they are located and what IP address they should have. If a single machine isn't doing what I expect, it is immediately obvious in the pretty graphs that I have with Grafana/Prometheus. Any 'mistakes' are easily rectified, but so far, we haven't had a single issue along these lines. I've tried it both ways, the benefits of static IP, for the reasons I mentioned, far outweigh any negatives. DHCP is great for a coffee shop, workplace or college campus, where you don't know anything about your clients or care to. In my case, I need to know exactly who is on my network at all times and exactly where they are located, without having to cross reference switch ports and MAC addresses. In one DC, I don't even have access to the switches.
      - 9y ago