25 ms·

Did you try Kea[0]? There are a variety of ways to ensure the same MAC addresses get the same IPs even after the initial lease expires. I'm not surprised you h

Did you try Kea[0]? There are a variety of ways to ensure the same MAC addresses get the same IPs even after the initial lease expires. I'm not surprised you had issues with dnsmasq - I wouldn't use it for anything other than a run of the mill /24 (or maybe a few of them on VLANs). Don't get me wrong - it's perfect for what it was originally intended for and I use it in at least half a dozen places. Just not for an application and network like yours. [0] - https://www.isc.org/kea/ https://www.isc.org/kea/

latchkey 4y ago
Kea is what we ended up with. It has a few feature differences from dnsmaq that we had to work around a bit. Not the end of the world, but not fun when you're under pressure. Now that I've studied the source code of dnsmasq a bit more because of the issues we had, I have to agree with you. We had run it with about 3k blades and it was fine. It was fine at about 9k blades too... but once we hit around 10k, it just fell over. The frustrating part was that it just didn't log what was up. We used netdata and discovered the UDP queues were filling up. Tried mucking with tuning the sysctls and that didn't work either. Switched midstream to Kea, which was a giant pain because we had a hard time dealing with duplicate IP addresses (imagine rebooting 10k+ PXE booted blades during a UDP storm).
- kkielhofner 4y ago
  Frankly I'm AMAZED you made it to 9k leases with dnsmasq. Not only from a performance standpoint but the overall logging, lack of instrumentation, limits in configuration, etc. As you say (from reviewing the source) it's really not intended for anything other than small home/SMB networks, memory and CPU constrained devices, etc. It's perfect for that and it's my go-to in those scenarios. A 10k host network is a completely different animal and from the down-votes my comment received it doesn't seem like many here on HN have been in scenarios like these either. Your comment about the logging is spot on as well. I think dnsmasq and I think "small flash device where the logs will never get stored anywhere and no one can or will view them anyway". As you can probably tell I've had issues getting meaningful logging and debug out of dnsmasq as well... I don't envy your situation in the slightest! Note to mods - I'm not complaining about downvotes. I'm using the downvotes as a sign there's little understanding on HN of network ops at scale. People may love dnsmasq on a home router (I do) but plenty of network admins like OP and I have watched it fall over in other scenarios.
  - latchkey 4y ago
    Both myself and my super talented devops coworker hadn't ever run with that many individual machines in a single location. 4-5k was both our maxes. It worked fine before, we didn't expect it to fall over. dnsmasq has some really nice features that kea doesn't have. just finding a single blade in the datacenter, means a naming system, and it was easier with dnsmasq, and the config was easier to create. We ended up having to use ansible to generate the kea config file because we also keep machines subnet separated based on physical location. HN has all sorts of people, I never try to quantify it. We don't know what we don't know.

latchkey 4y ago

Kea is what we ended up with. It has a few feature differences from dnsmaq that we had to work around a bit. Not the end of the world, but not fun when you're under pressure. Now that I've studied the source code of dnsmasq a bit more because of the issues we had, I have to agree with you. We had run it with about 3k blades and it was fine. It was fine at about 9k blades too... but once we hit around 10k, it just fell over. The frustrating part was that it just didn't log what was up. We used netdata and discovered the UDP queues were filling up. Tried mucking with tuning the sysctls and that didn't work either. Switched midstream to Kea, which was a giant pain because we had a hard time dealing with duplicate IP addresses (imagine rebooting 10k+ PXE booted blades during a UDP storm).

kkielhofner 4y ago
Frankly I'm AMAZED you made it to 9k leases with dnsmasq. Not only from a performance standpoint but the overall logging, lack of instrumentation, limits in configuration, etc. As you say (from reviewing the source) it's really not intended for anything other than small home/SMB networks, memory and CPU constrained devices, etc. It's perfect for that and it's my go-to in those scenarios. A 10k host network is a completely different animal and from the down-votes my comment received it doesn't seem like many here on HN have been in scenarios like these either. Your comment about the logging is spot on as well. I think dnsmasq and I think "small flash device where the logs will never get stored anywhere and no one can or will view them anyway". As you can probably tell I've had issues getting meaningful logging and debug out of dnsmasq as well... I don't envy your situation in the slightest! Note to mods - I'm not complaining about downvotes. I'm using the downvotes as a sign there's little understanding on HN of network ops at scale. People may love dnsmasq on a home router (I do) but plenty of network admins like OP and I have watched it fall over in other scenarios.
- latchkey 4y ago
  Both myself and my super talented devops coworker hadn't ever run with that many individual machines in a single location. 4-5k was both our maxes. It worked fine before, we didn't expect it to fall over. dnsmasq has some really nice features that kea doesn't have. just finding a single blade in the datacenter, means a naming system, and it was easier with dnsmasq, and the config was easier to create. We ended up having to use ansible to generate the kea config file because we also keep machines subnet separated based on physical location. HN has all sorts of people, I never try to quantify it. We don't know what we don't know.