6 ms·

Show HN: MassDNS – A high-performance DNS stub resolver in C

35 points by blechschmidt 10y ago

dmlittle 10y ago
What are the benefits of resolving DNS entries in a non-recrusive manner? A recursive resolve means that the final resolution of the domain is cached by the original DNS server that was queried along with all the servers it took to resolve. My understanding is that this behavior was ideal.
- deleted 10y ago
  [deleted]
- blechschmidt 10y ago
  What I mean by non-recursive in this case is that the program does not perform recursion by itself as it relies on a list of DNS resolvers. When querying these resolvers, the recursion bit is actually set. However, I agree that the title might be misleading but I am not sure how I could express that better.
  - nickpsecurity 10y ago
    Yeah, most titles I just brainstormed still suck in some way. Yours is technically true but only locally. Perhaps add something like you just said to the opening paragraph. That reduces confusion on what non-recursive means along with saving reader any wasted time if they were seeking something else.
    - blechschmidt 10y ago
      Well, in order to make it correct, I have implemented a --norecurse option.
audidude 10y ago
// SECURITY IMPROVEMENTS REQUIRED. VULNERABLE TO INTEGER AND BUFFER OVERFLOWS. well at least it's pretty clear up front. No malloc result checks, invalid format in printfs, probably more. (I don't mean to be disparaging, please don't take it that way OP. We clearly need more/better tooling in this area in C). (Edited for clarity)
- blechschmidt 10y ago
  Malloc results are checked by the safe_malloc function in security.h. Could you point out a line that is prone to integer overflows?
  - audidude 10y ago
    stream_buffer_init() calls malloc directly, so it's not protected by safe_malloc(). (Unless there is some funky re-define stuff I didn't see). %x takes unsigned int, not a char. (Although I guess you could just rely on promotion here if you can rely on the unsignedness of char). > Could you point out a line that is prone to integer overflows? Never said that it was an overflow (just C&P that comment from the file). Just said its invalid use. Which I'm totally willing to agree on the compiler mostly getting correct behavior. Just not really something I'd want to see landing in something that we all need a really good, secure implementation of. Best of luck though, I hope you iterate on this to the point we can have a really good DNS implementation in C, in user-space, with optional async processing.
    - blechschmidt 10y ago
      Thank you for pointing this out. This file actually contained some unused code that was not supposed to make it into the repository so I just removed it. The "%x" format specifiers were located in debug comments that have been commented out. They have also been removed now.
    - userbinator 10y ago
      Although I guess you could just rely on promotion here if you can rely on the unsignedness of char If compilers did not do promotion, they would be very obviously not compliant with the C standard and unable to compile correctly most existing code, so I'm willing to bet that is something you can rely on. It's defined by the standard and widely used.
      - audidude 10y ago
        Sure, but in this case it was signed to unsigned promotion.
textmode 10y ago
"A high-performance non-recursive DNS resolver" I believe a better description for this program would be a "stub resolver".
- blechschmidt 10y ago
  This is already discussed below and you are right. It was an issue with the perspective of the term recursion from my side. Unfortunately, I cannot change the HN title anymore. (Maybe some moderator can?) The GitHub project description has been changed. Some of the resolvers might ban/rate-limit you indeed and even send abuse complaints to your ISP. EDIT: You might have a look at the newly implemented --norecurse option.
  - textmode 10y ago
    "--norecurse Useful for DNS cache snooping" Not if the resolver rejects non-recursive queries, as do dnscache and dqcache. There is a way to resolve DNS domainnames to IP addresses using only non-recursive queries. I do it everyday. But I've never seen anyone release any program that did this. Your program does not even attempt to do this -- you need to send the queries to authoritative nameservers not public resolvers. But you used the term "non-recursive" so I thought maybe someone had finally tried. One of the shortcomings of DNS IMO is that the specification allows for the possibility of including more than one name in a query. But no one has ever implemented this, as far as I know. Despite the design of the DNS, most of the information stored in it is more static than dynamic, and much of it is centralized. Most dommainnames do not change IP addresses very often and there are very large numbers of domainnames sharing the same authoritative nameservers.
    - blechschmidt 10y ago
      For DNS cache snooping the usage would of course be different. You would supply the tool with one resolver which does not reject non-recursive queries. Theoretically, one could even perform traffic analyses of DNS resolvers by snooping. Having implemented the --norecurse option, the title is at least not wrong anymore. One can have non-recursive, non-iterative resolver (which is what you use when you want to perform DNS cache snooping) and the title does not suggest that the tool supports iterative lookups. Handling multiple questions within one packet is difficult because response codes such as NXDOMAIN are only included once per packet. AFAIK, bind does not support handling such queries.
otterley 10y ago
For easy comparability, request handling rates of tools like this are best described in terms of requests per second, not requests per hour. Also the hardware on which the benchmarks were performed needs to be described in detail. Are you sure tinydns, which has existed for ages without security vulnerabilities, can't trivially handle 27k requests/second on modern hardware?
- blechschmidt 10y ago
  I have not heard about tindydns before but it seems to be a DNS server, not a client. The tool has mainly been tested on a Hetzner EX41 server. (Ubuntu, Intel® Core™ i7-6700, 32 GB RAM)
  - otterley 10y ago
    Ah, you are correct. There is also dnscache, which is part of the same suite. However, it does not do what you are attempting to do.
    - deleted 10y ago
      [deleted]
    - deleted 10y ago
      [deleted]
    - deleted 10y ago
      [deleted]
  - otterley 10y ago
    Have you tried c-ares? http://c-ares.haxx.se/ http://c-ares.haxx.se/
    - blechschmidt 10y ago
      Not yet. I have had a quick look at ldns (https://www.nlnetlabs.nl/projects/ldns/ https://www.nlnetlabs.nl/projects/ldns/) which supports parsing DNS packets from wire. I will probably replace the DNS implementation with either libldns or c-ares.
known 10y ago
You may want to test it with http://www.verisign.com/en_US/channel-resources/domain-registry-products/zone-file/index.xhtml http://www.verisign.com/en_US/channel-resources/domain-regis...
blumentopf 10y ago
Ugh, so public resolvers are flooded with requests? Wouldn't it make much more sense to set up a local caching resolver like unbound and feed your queries to it? It would be much more considerate towards the public resolvers and also use less of your own bandwidth.
- matt_wulfeck 10y ago
  There are valid use cases for this. For example, web crawling needs to do millions of resolutions one time which completely nullifies the need for a cache.
  - blumentopf 10y ago
    For this to be correct, all records to be resolved would need to have completely disjunct labels from the root down. Which they cannot if they are indeed in the millions.
    - blechschmidt 10y ago
      I have not yet managed to setup a single local recursor, such as PowerDNS recursor, to deliver the same performance as the list consisting of multiple open resolvers, although bandwidth does not seem to be the limiting factor. Testing with dnsperf, the best result I am currently getting is about 6,000 resolves per second with bind, for PDNS the figure is even lower for some reason. I will have to dig a bit deeper in order to find the reason for that cap.
      - blumentopf 10y ago
        I'm not surprised at all that BIND performs poorly, look at those graphs (granted this is for authoritative servers, but says a lot about BIND's performance in general): https://www.nlnetlabs.nl/blog/2013/07/05/nsd4-performance-measurements/ https://www.nlnetlabs.nl/blog/2013/07/05/nsd4-performance-me... I'd stick with Unbound. There are a lot of knobs to fiddle with in the config. Be sure to compile against libevent so that you can use the highly scalable epoll as a backend (assuming you're on Linux). Turn up all the limits for cache size etc. Disable DNSSEC validation if you don't care about spoofed records. Ask on the mailing list if you need help, Wouter and his colleagues are very nice and respond very quickly.
deleted 10y ago
[deleted]
jedisct1 10y ago
if (packet->flags & DNS_RESPONSE_FLAG == 0 || packet->questioncount != 1) I'm not sure that it does what you want.
- blechschmidt 10y ago
  This is the line of code that pre-checks whether an incoming packet should be parsed at all. If it is not a response packet or if the number of questions is not one, it should not be parsed, simply because the packet parsing function expects a reply packet with exactly one question, namely the one that has been queried for.
  - abhorrence 10y ago
    Operator precedence causes it to be interpreted as: if ((packet->flags & (DNS_RESPONSE_FLAG == 0)) || (packet->questioncount != 1)) which at least does not read as intended.
    - blechschmidt 10y ago
      You are correct. Has been fixed.

35 points by blechschmidt 10y ago

What are the benefits of resolving DNS entries in a non-recrusive manner? A recursive resolve means that the final resolution of the domain is cached by the original DNS server that was queried along with all the servers it took to resolve. My understanding is that this behavior was ideal.

deleted 10y ago
[deleted]
blechschmidt 10y ago
What I mean by non-recursive in this case is that the program does not perform recursion by itself as it relies on a list of DNS resolvers. When querying these resolvers, the recursion bit is actually set. However, I agree that the title might be misleading but I am not sure how I could express that better.
- nickpsecurity 10y ago
  Yeah, most titles I just brainstormed still suck in some way. Yours is technically true but only locally. Perhaps add something like you just said to the opening paragraph. That reduces confusion on what non-recursive means along with saving reader any wasted time if they were seeking something else.
  - blechschmidt 10y ago
    Well, in order to make it correct, I have implemented a --norecurse option.

audidude 10y ago

// SECURITY IMPROVEMENTS REQUIRED. VULNERABLE TO INTEGER AND BUFFER OVERFLOWS. well at least it's pretty clear up front. No malloc result checks, invalid format in printfs, probably more. (I don't mean to be disparaging, please don't take it that way OP. We clearly need more/better tooling in this area in C). (Edited for clarity)

blechschmidt 10y ago
Malloc results are checked by the safe_malloc function in security.h. Could you point out a line that is prone to integer overflows?
- audidude 10y ago
  stream_buffer_init() calls malloc directly, so it's not protected by safe_malloc(). (Unless there is some funky re-define stuff I didn't see). %x takes unsigned int, not a char. (Although I guess you could just rely on promotion here if you can rely on the unsignedness of char). > Could you point out a line that is prone to integer overflows? Never said that it was an overflow (just C&P that comment from the file). Just said its invalid use. Which I'm totally willing to agree on the compiler mostly getting correct behavior. Just not really something I'd want to see landing in something that we all need a really good, secure implementation of. Best of luck though, I hope you iterate on this to the point we can have a really good DNS implementation in C, in user-space, with optional async processing.
  - blechschmidt 10y ago
    Thank you for pointing this out. This file actually contained some unused code that was not supposed to make it into the repository so I just removed it. The "%x" format specifiers were located in debug comments that have been commented out. They have also been removed now.
  - userbinator 10y ago
    Although I guess you could just rely on promotion here if you can rely on the unsignedness of char If compilers did not do promotion, they would be very obviously not compliant with the C standard and unable to compile correctly most existing code, so I'm willing to bet that is something you can rely on. It's defined by the standard and widely used.
    - audidude 10y ago
      Sure, but in this case it was signed to unsigned promotion.

textmode 10y ago

"A high-performance non-recursive DNS resolver" I believe a better description for this program would be a "stub resolver".

blechschmidt 10y ago
This is already discussed below and you are right. It was an issue with the perspective of the term recursion from my side. Unfortunately, I cannot change the HN title anymore. (Maybe some moderator can?) The GitHub project description has been changed. Some of the resolvers might ban/rate-limit you indeed and even send abuse complaints to your ISP. EDIT: You might have a look at the newly implemented --norecurse option.
- textmode 10y ago
  "--norecurse Useful for DNS cache snooping" Not if the resolver rejects non-recursive queries, as do dnscache and dqcache. There is a way to resolve DNS domainnames to IP addresses using only non-recursive queries. I do it everyday. But I've never seen anyone release any program that did this. Your program does not even attempt to do this -- you need to send the queries to authoritative nameservers not public resolvers. But you used the term "non-recursive" so I thought maybe someone had finally tried. One of the shortcomings of DNS IMO is that the specification allows for the possibility of including more than one name in a query. But no one has ever implemented this, as far as I know. Despite the design of the DNS, most of the information stored in it is more static than dynamic, and much of it is centralized. Most dommainnames do not change IP addresses very often and there are very large numbers of domainnames sharing the same authoritative nameservers.
  - blechschmidt 10y ago
    For DNS cache snooping the usage would of course be different. You would supply the tool with one resolver which does not reject non-recursive queries. Theoretically, one could even perform traffic analyses of DNS resolvers by snooping. Having implemented the --norecurse option, the title is at least not wrong anymore. One can have non-recursive, non-iterative resolver (which is what you use when you want to perform DNS cache snooping) and the title does not suggest that the tool supports iterative lookups. Handling multiple questions within one packet is difficult because response codes such as NXDOMAIN are only included once per packet. AFAIK, bind does not support handling such queries.

otterley 10y ago

For easy comparability, request handling rates of tools like this are best described in terms of requests per second, not requests per hour. Also the hardware on which the benchmarks were performed needs to be described in detail. Are you sure tinydns, which has existed for ages without security vulnerabilities, can't trivially handle 27k requests/second on modern hardware?

blechschmidt 10y ago
I have not heard about tindydns before but it seems to be a DNS server, not a client. The tool has mainly been tested on a Hetzner EX41 server. (Ubuntu, Intel® Core™ i7-6700, 32 GB RAM)
- otterley 10y ago
  Ah, you are correct. There is also dnscache, which is part of the same suite. However, it does not do what you are attempting to do.
  - deleted 10y ago
    [deleted]
  - deleted 10y ago
    [deleted]
  - deleted 10y ago
    [deleted]
- otterley 10y ago
  Have you tried c-ares? http://c-ares.haxx.se/ http://c-ares.haxx.se/
  - blechschmidt 10y ago
    Not yet. I have had a quick look at ldns (https://www.nlnetlabs.nl/projects/ldns/ https://www.nlnetlabs.nl/projects/ldns/) which supports parsing DNS packets from wire. I will probably replace the DNS implementation with either libldns or c-ares.

known 10y ago

You may want to test it with http://www.verisign.com/en_US/channel-resources/domain-registry-products/zone-file/index.xhtml http://www.verisign.com/en_US/channel-resources/domain-regis...

blumentopf 10y ago

Ugh, so public resolvers are flooded with requests? Wouldn't it make much more sense to set up a local caching resolver like unbound and feed your queries to it? It would be much more considerate towards the public resolvers and also use less of your own bandwidth.

matt_wulfeck 10y ago
There are valid use cases for this. For example, web crawling needs to do millions of resolutions one time which completely nullifies the need for a cache.
- blumentopf 10y ago
  For this to be correct, all records to be resolved would need to have completely disjunct labels from the root down. Which they cannot if they are indeed in the millions.
  - blechschmidt 10y ago
    I have not yet managed to setup a single local recursor, such as PowerDNS recursor, to deliver the same performance as the list consisting of multiple open resolvers, although bandwidth does not seem to be the limiting factor. Testing with dnsperf, the best result I am currently getting is about 6,000 resolves per second with bind, for PDNS the figure is even lower for some reason. I will have to dig a bit deeper in order to find the reason for that cap.
    - blumentopf 10y ago
      I'm not surprised at all that BIND performs poorly, look at those graphs (granted this is for authoritative servers, but says a lot about BIND's performance in general): https://www.nlnetlabs.nl/blog/2013/07/05/nsd4-performance-measurements/ https://www.nlnetlabs.nl/blog/2013/07/05/nsd4-performance-me... I'd stick with Unbound. There are a lot of knobs to fiddle with in the config. Be sure to compile against libevent so that you can use the highly scalable epoll as a backend (assuming you're on Linux). Turn up all the limits for cache size etc. Disable DNSSEC validation if you don't care about spoofed records. Ask on the mailing list if you need help, Wouter and his colleagues are very nice and respond very quickly.

deleted 10y ago

[deleted]

jedisct1 10y ago

if (packet->flags & DNS_RESPONSE_FLAG == 0 || packet->questioncount != 1) I'm not sure that it does what you want.

blechschmidt 10y ago
This is the line of code that pre-checks whether an incoming packet should be parsed at all. If it is not a response packet or if the number of questions is not one, it should not be parsed, simply because the packet parsing function expects a reply packet with exactly one question, namely the one that has been queried for.
- abhorrence 10y ago
  Operator precedence causes it to be interpreted as: if ((packet->flags & (DNS_RESPONSE_FLAG == 0)) || (packet->questioncount != 1)) which at least does not read as intended.
  - blechschmidt 10y ago
    You are correct. Has been fixed.