5 ms·

I recently completed a "software pilgrimage", as I decided to call it in explaining to friends why I did it, of writing my own DNS recursive resolver. I decided

I recently completed a "software pilgrimage", as I decided to call it in explaining to friends why I did it, of writing my own DNS recursive resolver. I decided to implement it purely from RFC 1034/1035, with no library code aside from TCP and UDP sockets and a JSON config parser, those kinds of things. No library help for anything relating to the RFC. It's part of a homelab-focused DNS solution that I plan to release under the AGPLv3 once I dogfood it for a while. Basically "DNS for impatient homelabbers/SMB's who don't want to have to learn how to write zonefiles and don't want some crazy enterprise omnibus thing either". Anyway, in the process of doing this and putting it in my DHCP config and seeing the traffic from two smart TVs of different brands, my RIPE atlas probe, various phones and laptops, smart thermostat, etc etc, I've noticed that there is a TON of "garbage" dns requests. Like, the chart [the article] shows - where about 70% of queries result in a name error - that totally meshes with what I see on a much smaller scale. Right now, prometheus tells me that since last restart (which was about a week ago at this point?), I've answered 80437 queries with NO_ERROR, 52014 with NAME_ERROR, and 242 with SERVER_FAILURE (funnily enough, when spot checking these, 8.8.8.8 also SERVER_FAILUREs these same requests - usually devices with presumably-buggy dns libraries not correctly specifying lengths of variable-length fields). It really surprised my initial suspicions that so many DNS requests would be coming up with what I was originally considering to be an error condition. But I guess sometimes the absence of a DNS record is just as meaningful as the presence. Incidentally, I also noticed these chromium dns requests, and they had me worried for a bit because I wondered if they were malware trying to exploit some kind of vuln in dns servers. Took a bit of googling to figure it out. I do think they make up a decent % of the name errors I see, though I hadn't gotten around to having prometheus split them out to measure.

fouronnes3 6y ago
That's a very sweet idea. What are some other "software pilgrimage" projects for ambitious hackers with a finite amount of free time? Buying something in bitcoin using only the protocol spec and man pages? Tweeting from a Linux from scratch install? Writing a correct tar command on the first try? Writing a quine on your favorite language without google?
- cashewchoo 6y ago
  Heh. I was inspired by a movement I saw a while ago to read old CS academic research, since e.g. quicksort is still very relevant nowadays but is also still approachable for someone with a B.Sc. in CS. (Whereas more recent papers all require fairly deep knowledge of the research niche). My take on it was that lots of this old software was implemented by candlelight with a magnetized needle, a 512KiB HDD platter and a steady hand; so I should be able to reimplement it in a modern language with modern tooling in a lot less time, while learning a lot about the system. Like, dig command output means a lot more to me now than it did at the start, and I now appreciate what articles like the OP mean now.
- the_imp 6y ago
  I wrote a fully spec-compliant YAML library.
  - zingplex 6y ago
    I didn't think that was possible
- nishanth_v 6y ago
  One of my friends is building a collection of such exercises aimed at experienced & intermediate programmers, where you build Redis / Docker / Git / React from scratch with guidance. https://codecrafters.io/challenges https://codecrafters.io/challenges
  - abhijat 6y ago
    I remember doing a few of these last year. It was quite fun. They even added nim on request. I was able to use it to complete the docker challenge.
  - maxwelljoslyn 6y ago
    See also: Robert Heaton's series of project prompts "Programming Projects for Advanced Beginners." https://robertheaton.com/2018/12/08/programming-projects-for-advanced-beginners/ https://robertheaton.com/2018/12/08/programming-projects-for... The project goals are not as lofty as those proposed by nishanth_v's friend, but Mr. Heaton goes the extra mile to turn each project idea into a step by step mini-curriculum with lots of extension points. Then, he goes another extra mile by allowing readers to email in their buggy projects, and running a companion series where he teaches people how to debug / fix / improve their code by refactoring reader's attempts at the "...Advanced Beginners" projects. Recommended.
justinclift 6y ago
> Basically "DNS for impatient homelabbers/SMB's who don't want to have to learn how to write zonefiles and don't want some crazy enterprise omnibus thing either". Dnsmasq or PowerDNS weren't suitable?
- cashewchoo 6y ago
  Workable but no, not suitable. PowerDNS has a lot of features and isn't terrifically simple to deploy. DNSMasq is easier to deploy, but still has a lot of features and isn't easy to configure. The niche I'm aiming for is for people who just want to map hostnames to IP addresses. And note this isn't a commercial offering, just something I built for myself that I'm going to put up a website for and maybe some other people will find interesting. I do think I have some product-market fit because even among my dev friends, who are all quite talented, many of them have mentioned to me "yeah I threw bind9 on a server but decided I didn't hate typing IPs enough to actually set up and operate it". I'm aiming for zero onboarding time and zero maintenance headache. You run a docker container, map the ports, and then there's a nice web UI that takes you the rest of the way.
  - justinclift 6y ago
    No worries, that does sound useful. :)
- birdiesanders 6y ago
  Try running 15k containers, dnsmasq and powerdns fall over pretty fast. I have to run a BIND cluster of 3 machines to support the days when I am running at-scale testing of software (low load, but high instance count is a better fit for my work these days). I use 4th gen i7 machines as DNS resolver/caching servers, and using dnsmasq on the same machines I start to have issues around 400 clients, massive latency, broken responses, tons of crazy stuff. One BIND instance and I can get to about 7500 clients before it starts to get squeezed. The difference is huge in other ways, though. Configuration, operations, and maintenance of the BIND cluster is 20 fold more involved, and when things break, they can break in really confusing ways. Such is the world of computing though, the better something performs, the more places it has where you can mess it up.

fouronnes3 6y ago

That's a very sweet idea. What are some other "software pilgrimage" projects for ambitious hackers with a finite amount of free time? Buying something in bitcoin using only the protocol spec and man pages? Tweeting from a Linux from scratch install? Writing a correct tar command on the first try? Writing a quine on your favorite language without google?

cashewchoo 6y ago
Heh. I was inspired by a movement I saw a while ago to read old CS academic research, since e.g. quicksort is still very relevant nowadays but is also still approachable for someone with a B.Sc. in CS. (Whereas more recent papers all require fairly deep knowledge of the research niche). My take on it was that lots of this old software was implemented by candlelight with a magnetized needle, a 512KiB HDD platter and a steady hand; so I should be able to reimplement it in a modern language with modern tooling in a lot less time, while learning a lot about the system. Like, dig command output means a lot more to me now than it did at the start, and I now appreciate what articles like the OP mean now.
the_imp 6y ago
I wrote a fully spec-compliant YAML library.
- zingplex 6y ago
  I didn't think that was possible
nishanth_v 6y ago
One of my friends is building a collection of such exercises aimed at experienced & intermediate programmers, where you build Redis / Docker / Git / React from scratch with guidance. https://codecrafters.io/challenges https://codecrafters.io/challenges
- abhijat 6y ago
  I remember doing a few of these last year. It was quite fun. They even added nim on request. I was able to use it to complete the docker challenge.
- maxwelljoslyn 6y ago
  See also: Robert Heaton's series of project prompts "Programming Projects for Advanced Beginners." https://robertheaton.com/2018/12/08/programming-projects-for-advanced-beginners/ https://robertheaton.com/2018/12/08/programming-projects-for... The project goals are not as lofty as those proposed by nishanth_v's friend, but Mr. Heaton goes the extra mile to turn each project idea into a step by step mini-curriculum with lots of extension points. Then, he goes another extra mile by allowing readers to email in their buggy projects, and running a companion series where he teaches people how to debug / fix / improve their code by refactoring reader's attempts at the "...Advanced Beginners" projects. Recommended.

justinclift 6y ago

> Basically "DNS for impatient homelabbers/SMB's who don't want to have to learn how to write zonefiles and don't want some crazy enterprise omnibus thing either". Dnsmasq or PowerDNS weren't suitable?

cashewchoo 6y ago
Workable but no, not suitable. PowerDNS has a lot of features and isn't terrifically simple to deploy. DNSMasq is easier to deploy, but still has a lot of features and isn't easy to configure. The niche I'm aiming for is for people who just want to map hostnames to IP addresses. And note this isn't a commercial offering, just something I built for myself that I'm going to put up a website for and maybe some other people will find interesting. I do think I have some product-market fit because even among my dev friends, who are all quite talented, many of them have mentioned to me "yeah I threw bind9 on a server but decided I didn't hate typing IPs enough to actually set up and operate it". I'm aiming for zero onboarding time and zero maintenance headache. You run a docker container, map the ports, and then there's a nice web UI that takes you the rest of the way.
- justinclift 6y ago
  No worries, that does sound useful. :)
birdiesanders 6y ago
Try running 15k containers, dnsmasq and powerdns fall over pretty fast. I have to run a BIND cluster of 3 machines to support the days when I am running at-scale testing of software (low load, but high instance count is a better fit for my work these days). I use 4th gen i7 machines as DNS resolver/caching servers, and using dnsmasq on the same machines I start to have issues around 400 clients, massive latency, broken responses, tons of crazy stuff. One BIND instance and I can get to about 7500 clients before it starts to get squeezed. The difference is huge in other ways, though. Configuration, operations, and maintenance of the BIND cluster is 20 fold more involved, and when things break, they can break in really confusing ways. Such is the world of computing though, the better something performs, the more places it has where you can mess it up.