13 ms·

Systemd-resolved DNS cache poisoning

93 points by comice 12y ago

userbinator 12y ago
Interesting background discussion here for those wondering why systemd has its own DNS resolver: https://news.ycombinator.com/item?id=8203080 https://news.ycombinator.com/item?id=8203080
- marcosdumay 12y ago
  Or, in other words, since they want to integrate the entire OS into systemd, they need a DNS resolver there too.
- digi_owl 12y ago
  As usual, it is all about cloud...
vezzy-fnord 12y ago
The submitter doesn't state affected versions, but judging from systemd's NEWS file, the resolved component was first introduced in 213 and later dubbed as "a pretty complete caching DNS and LLMNR stub resolver" in 216: http://lwn.net/Articles/609740/ http://lwn.net/Articles/609740/ Considering most users run on a backported 208-stable IIRC (I know RHEL 7 settled on that), I'm not sure how widespread this is. Still concerning that they felt to roll their own and not follow RFC guidelines, though. Considering how plagued with vulnerabilities BIND was, I'd assume DNS is a hard thing to do.
- agwa 12y ago
  > "a pretty complete caching DNS and LLMNR stub resolver" That quote is gold. Most of DNS isn't implementing the protocol, but implementing all the protections against cache poisoning, many of which are not intuitive. But as the oss-sec email says: "systemd-resolved does not implement any of the hardening recommendations of rfc5452."
josteink 12y ago
Am I the only one who is getting fed up with everything under the sun having their own DNS resolver and DNS cache? It's border-lining madness right now. Before you had a situation like this. ISP: Server and cache. Router: Client, server and cache. OS: Client and cache. So if you ever encounter DNS failing, you only had 3 possible places to look for errors, or maybe not even errors, just stale data that may or may not correct itself over time. Flush the caches you can and hope for the best. Rinse and repeat. But then came the browsers and decided that 3 levels of caching was good enough. Let's add more which can fail in the name of performance! Combine this with mobile, mobility and data-roaming and now you not only have a DNS-cache (based on already tripple-cached data) which often is guaranteed to be wrong, your browser may actually have cached the wrong DNS-servers, so when you roam from Wifi to cell-data your precious LAN DNS servers are no longer available and your browser just looks stupid. And now systemd adds to this. What the flying fuck? All this nonsense because someone somewhere decided that a nice, shared, global OS level client and cache wasn't good enough. Handling this complexity once would just be too easy. Can someone tell me why? Why wasn't it good enough? What was it that prompted someone to go reimplement all this wrongly, yet again? Yes this is a rant, but if there are underlying technical reasons for this nonsense, I'm very, very interested to hear about it.
- unwind 12y ago
  The commit's comment (http://cgit.freedesktop.org/systemd/systemd/commit/?id=322345fdb9865ef2477fba8e4bdde0e1183ef505 http://cgit.freedesktop.org/systemd/systemd/commit/?id=32234...) says it all: resolved: add DNS cache Yes, that's the full commit message for what I think is the commit that adds the feature. I was trying to dig up an authorative source for the "why" you're asking, but I guess that failed. Perhaps it was discussed/motivated/described/something somewhere else, like a mailing list, I didn't search further.
  - e40 12y ago
    Yes, that's the full commit message for what I think is the commit that adds the feature. If true (that it is a 4 word commit message), that is a disgrace. Seriously, something as important as this should contain a lot more information, if not just a pointer to a place that provides it.
jacquesm 12y ago
Pretty weird for Systemd to have its own cache and resolver. If you need a local resolver and a cache simply run a local DNS server rather than to re-invent the wheel (in a broken way at that). I'm sure they have their reasons but this is not the unix way.
- pjc50 12y ago
  It wouldn't be systemd if it didn't reinvent the wheel.
- kefka 12y ago
  Systemd is just not the Unix Way. It's some shit program that requires to be run as PID 1, and takes over the system in the forms of an octopus. And, you need some gods-awful binary program to even read their logfiles. Yes, it's like the Windows registry. Even Debian has succumbed to Systemd.
  - vezzy-fnord 12y ago
    Yes, it's like the Windows registry. Nope, that would be GSettings/dconf. It's limited to GNOME, thankfully.
    - custardcream 12y ago
      Actually systemd is more like DCOM which is much worse. The registry is fucking marvellous in comparison.
      - talideon 12y ago
        Nah, the equivalent of DCOM is dbus.
        custardcream 12y ago
        Which is what systemd is built on. At least on windows COM stays in user space but no, fuck it, lets use kdbus on Linux...
agwa 12y ago
This is a perfect example of why the systemd approach of putting a bunch of disparate components under a single tightly-coupled umbrella is bad engineering. Generally, projects focused on doing one thing are able to do it well, while projects that attempt to do many things have a hard time doing any of them well. Unfortunately, as the steady stream of cache poisoning vulnerabilities in other DNS servers have demonstrated, implementing DNS securely is really hard. A less-than-perfect attempt at implementing a caching DNS resolver will be insecure. For this reason, it's a task best left to a mature project run by people with DNS-specific experience who can focus on doing that one thing well.
- Someone1234 12y ago
  > This is a perfect example of why the systemd approach of reinventing a bunch of existing components under a single tightly-coupled umbrella is a bad approach. Why? This wasn't caused by systemd's complexity or having too much components interconnect poorly. It was a bug in one of their implementations that could have equally happened had it been a stand-alone DNS cache resolver. > Generally, projects focused on doing one thing are able to do it well, while projects that attempt to do many things have a hard time doing any of them well. I'd totally buy that if BIND didn't have such a checkered history. I mean with BIND 9 they scrapped the entire project and started again because it was so insecure... Yet they're only attempting to do "one thing" so according to you they should have done it well. > Unfortunately, as the steady stream of cache poisoning vulnerabilities in other DNS servers have demonstrated, implementing DNS securely is really hard. This line oddly contradicts the line that directly proceeds it. > For this reason, it's a task best left to a mature project run by people with DNS-specific experience who can focus on doing that one thing well. According to the previous line they aren't doing that one thing well...
  - agwa 12y ago
    The reason why a project with a bunch of tightly-coupled disparate components will have a hard time doing anything well is that an expert who could do a good job isn't going to spend their time working on an implementation that can only be used along with a bunch of unrelated baggage. They're going to work on an independent implementation that can be used by as many people as possible. Take DJB. He's an expert and djbdns has a stellar security record. He's not going to spend his time trying to make systemd's resolver secure. Instead systemd is going to end up repeating all the same mistakes BIND has made. At least BIND has a substantial head start over systemd-resolved.
SixSigma 12y ago
Caches are bugs waiting to happen. Rob Pike ‏@rob_pike 21 Mar 2014 By the same author "There's no such thing as a simple cache bug." "Caches aren't architecture, they're just optimization."
- tedunangst 12y ago
  This isn't really even a cache bug. It's a bug in a cache, but it's a stupid bug to have in any DNS implementation, cache or not.
talideon 12y ago
Of all the components of systemd that doesn't need to exist, resolved is definitely one of them. Why they're reinventing the wheel rather than allowing the user to set up something that works and is battle-tested, such as unbound, is beyond me.
- JeremyNT 12y ago
  I don't necessarily disagree with the heart of your statement, but it's worth mentioning that systemd does not require one to use resolved. The user is still "allowed" to run unbound, dnsmasq, etc., and the preferred solution will be a question for distribution maintainers and users to answer. I understand concerns about systemd's scope, but most of the "questionable" functionality is optional.
  - talideon 12y ago
    Personally, I'd prefer that they concentrate on stuff that's properly within the scope of the project. This is way, way outside what the reasonable scope of the project ought to be.
  - agwa 12y ago
    Small comfort. When a project has the incredibly poor judgment (and, frankly, hubris) to write a caching DNS resolver without even the most basic cache hardening protections, and then declare it "a pretty complete caching DNS and LLMNR stub resolver"[1] instead of just leaving it to people who know what they're doing, it calls into question the rest of the project. This is a major reason why so many of us are wary of systemd. [1] http://lwn.net/Articles/609740/ http://lwn.net/Articles/609740/
amluto 12y ago
I find the design of systemd-resolved to be very strange. It uses dbus to talk to glibc, and it seems to be a new, from-scratch implementation of a DNS resolver. To be clear, I don't really think it matters whether systemd-resolved is under the systemd umbrella, but I do think that the design has a lot of unnecessary NIH syndrome. It turns out that there's a very well-specified protocol by which clients can ask a local cache on their system to answer DNS queries. That protocol is called DNS :) I don't see why routing something DNS-like over dbus makes any sense in contrast to doing it using DNS itself on port 53. Fedora is experimenting with running unbound as a local caching resolver [1]. This gives caching, DNSSEC validation, and all the benefits from the fact that unbound is probably much better hardened than the average libc or application-side DNS client implementation. [1] http://fedoraproject.org/wiki/Features/DNSSEC_on_workstations http://fedoraproject.org/wiki/Features/DNSSEC_on_workstation...
- digi_owl 12y ago
  Poettering and crew seems to have such a hardon for dbus that they want to get their own variant into the Linux kernel (kdbus). Hell, i have recently seen a fedora bug regarding the use of su -. Where Poettering argued that people should not use su because it broke dbus. Instead he seemed to advocate that people ssh into 127.0.0.1 to do their thing with a different account.
4ydx 12y ago
Systemd... a very large mass of c code being written in a short amount of time with huge functionality creep. Yeah, this is going to end/begin really well. I swear all of the people commenting about it being a good thing are not seasoned developers.

93 points by comice 12y ago

Interesting background discussion here for those wondering why systemd has its own DNS resolver: https://news.ycombinator.com/item?id=8203080 https://news.ycombinator.com/item?id=8203080

marcosdumay 12y ago
Or, in other words, since they want to integrate the entire OS into systemd, they need a DNS resolver there too.
digi_owl 12y ago
As usual, it is all about cloud...

vezzy-fnord 12y ago

The submitter doesn't state affected versions, but judging from systemd's NEWS file, the resolved component was first introduced in 213 and later dubbed as "a pretty complete caching DNS and LLMNR stub resolver" in 216: http://lwn.net/Articles/609740/ http://lwn.net/Articles/609740/ Considering most users run on a backported 208-stable IIRC (I know RHEL 7 settled on that), I'm not sure how widespread this is. Still concerning that they felt to roll their own and not follow RFC guidelines, though. Considering how plagued with vulnerabilities BIND was, I'd assume DNS is a hard thing to do.

agwa 12y ago
> "a pretty complete caching DNS and LLMNR stub resolver" That quote is gold. Most of DNS isn't implementing the protocol, but implementing all the protections against cache poisoning, many of which are not intuitive. But as the oss-sec email says: "systemd-resolved does not implement any of the hardening recommendations of rfc5452."

josteink 12y ago

Am I the only one who is getting fed up with everything under the sun having their own DNS resolver and DNS cache? It's border-lining madness right now. Before you had a situation like this. ISP: Server and cache. Router: Client, server and cache. OS: Client and cache. So if you ever encounter DNS failing, you only had 3 possible places to look for errors, or maybe not even errors, just stale data that may or may not correct itself over time. Flush the caches you can and hope for the best. Rinse and repeat. But then came the browsers and decided that 3 levels of caching was good enough. Let's add more which can fail in the name of performance! Combine this with mobile, mobility and data-roaming and now you not only have a DNS-cache (based on already tripple-cached data) which often is guaranteed to be wrong, your browser may actually have cached the wrong DNS-servers, so when you roam from Wifi to cell-data your precious LAN DNS servers are no longer available and your browser just looks stupid. And now systemd adds to this. What the flying fuck? All this nonsense because someone somewhere decided that a nice, shared, global OS level client and cache wasn't good enough. Handling this complexity once would just be too easy. Can someone tell me why? Why wasn't it good enough? What was it that prompted someone to go reimplement all this wrongly, yet again? Yes this is a rant, but if there are underlying technical reasons for this nonsense, I'm very, very interested to hear about it.

unwind 12y ago
The commit's comment (http://cgit.freedesktop.org/systemd/systemd/commit/?id=322345fdb9865ef2477fba8e4bdde0e1183ef505 http://cgit.freedesktop.org/systemd/systemd/commit/?id=32234...) says it all: resolved: add DNS cache Yes, that's the full commit message for what I think is the commit that adds the feature. I was trying to dig up an authorative source for the "why" you're asking, but I guess that failed. Perhaps it was discussed/motivated/described/something somewhere else, like a mailing list, I didn't search further.
- e40 12y ago
  Yes, that's the full commit message for what I think is the commit that adds the feature. If true (that it is a 4 word commit message), that is a disgrace. Seriously, something as important as this should contain a lot more information, if not just a pointer to a place that provides it.

jacquesm 12y ago

Pretty weird for Systemd to have its own cache and resolver. If you need a local resolver and a cache simply run a local DNS server rather than to re-invent the wheel (in a broken way at that). I'm sure they have their reasons but this is not the unix way.

pjc50 12y ago
It wouldn't be systemd if it didn't reinvent the wheel.
kefka 12y ago
Systemd is just not the Unix Way. It's some shit program that requires to be run as PID 1, and takes over the system in the forms of an octopus. And, you need some gods-awful binary program to even read their logfiles. Yes, it's like the Windows registry. Even Debian has succumbed to Systemd.
- vezzy-fnord 12y ago
  Yes, it's like the Windows registry. Nope, that would be GSettings/dconf. It's limited to GNOME, thankfully.
  - custardcream 12y ago
    Actually systemd is more like DCOM which is much worse. The registry is fucking marvellous in comparison.
    - talideon 12y ago
      Nah, the equivalent of DCOM is dbus.
      - custardcream 12y ago
        Which is what systemd is built on. At least on windows COM stays in user space but no, fuck it, lets use kdbus on Linux...

agwa 12y ago

This is a perfect example of why the systemd approach of putting a bunch of disparate components under a single tightly-coupled umbrella is bad engineering. Generally, projects focused on doing one thing are able to do it well, while projects that attempt to do many things have a hard time doing any of them well. Unfortunately, as the steady stream of cache poisoning vulnerabilities in other DNS servers have demonstrated, implementing DNS securely is really hard. A less-than-perfect attempt at implementing a caching DNS resolver will be insecure. For this reason, it's a task best left to a mature project run by people with DNS-specific experience who can focus on doing that one thing well.

Someone1234 12y ago
> This is a perfect example of why the systemd approach of reinventing a bunch of existing components under a single tightly-coupled umbrella is a bad approach. Why? This wasn't caused by systemd's complexity or having too much components interconnect poorly. It was a bug in one of their implementations that could have equally happened had it been a stand-alone DNS cache resolver. > Generally, projects focused on doing one thing are able to do it well, while projects that attempt to do many things have a hard time doing any of them well. I'd totally buy that if BIND didn't have such a checkered history. I mean with BIND 9 they scrapped the entire project and started again because it was so insecure... Yet they're only attempting to do "one thing" so according to you they should have done it well. > Unfortunately, as the steady stream of cache poisoning vulnerabilities in other DNS servers have demonstrated, implementing DNS securely is really hard. This line oddly contradicts the line that directly proceeds it. > For this reason, it's a task best left to a mature project run by people with DNS-specific experience who can focus on doing that one thing well. According to the previous line they aren't doing that one thing well...
- agwa 12y ago
  The reason why a project with a bunch of tightly-coupled disparate components will have a hard time doing anything well is that an expert who could do a good job isn't going to spend their time working on an implementation that can only be used along with a bunch of unrelated baggage. They're going to work on an independent implementation that can be used by as many people as possible. Take DJB. He's an expert and djbdns has a stellar security record. He's not going to spend his time trying to make systemd's resolver secure. Instead systemd is going to end up repeating all the same mistakes BIND has made. At least BIND has a substantial head start over systemd-resolved.

SixSigma 12y ago

Caches are bugs waiting to happen. Rob Pike ‏@rob_pike 21 Mar 2014 By the same author "There's no such thing as a simple cache bug." "Caches aren't architecture, they're just optimization."

tedunangst 12y ago
This isn't really even a cache bug. It's a bug in a cache, but it's a stupid bug to have in any DNS implementation, cache or not.

talideon 12y ago

Of all the components of systemd that doesn't need to exist, resolved is definitely one of them. Why they're reinventing the wheel rather than allowing the user to set up something that works and is battle-tested, such as unbound, is beyond me.

JeremyNT 12y ago
I don't necessarily disagree with the heart of your statement, but it's worth mentioning that systemd does not require one to use resolved. The user is still "allowed" to run unbound, dnsmasq, etc., and the preferred solution will be a question for distribution maintainers and users to answer. I understand concerns about systemd's scope, but most of the "questionable" functionality is optional.
- talideon 12y ago
  Personally, I'd prefer that they concentrate on stuff that's properly within the scope of the project. This is way, way outside what the reasonable scope of the project ought to be.
- agwa 12y ago
  Small comfort. When a project has the incredibly poor judgment (and, frankly, hubris) to write a caching DNS resolver without even the most basic cache hardening protections, and then declare it "a pretty complete caching DNS and LLMNR stub resolver"[1] instead of just leaving it to people who know what they're doing, it calls into question the rest of the project. This is a major reason why so many of us are wary of systemd. [1] http://lwn.net/Articles/609740/ http://lwn.net/Articles/609740/

amluto 12y ago

I find the design of systemd-resolved to be very strange. It uses dbus to talk to glibc, and it seems to be a new, from-scratch implementation of a DNS resolver. To be clear, I don't really think it matters whether systemd-resolved is under the systemd umbrella, but I do think that the design has a lot of unnecessary NIH syndrome. It turns out that there's a very well-specified protocol by which clients can ask a local cache on their system to answer DNS queries. That protocol is called DNS :) I don't see why routing something DNS-like over dbus makes any sense in contrast to doing it using DNS itself on port 53. Fedora is experimenting with running unbound as a local caching resolver [1]. This gives caching, DNSSEC validation, and all the benefits from the fact that unbound is probably much better hardened than the average libc or application-side DNS client implementation. [1] http://fedoraproject.org/wiki/Features/DNSSEC_on_workstations http://fedoraproject.org/wiki/Features/DNSSEC_on_workstation...

digi_owl 12y ago
Poettering and crew seems to have such a hardon for dbus that they want to get their own variant into the Linux kernel (kdbus). Hell, i have recently seen a fedora bug regarding the use of su -. Where Poettering argued that people should not use su because it broke dbus. Instead he seemed to advocate that people ssh into 127.0.0.1 to do their thing with a different account.

4ydx 12y ago

Systemd... a very large mass of c code being written in a short amount of time with huge functionality creep. Yeah, this is going to end/begin really well. I swear all of the people commenting about it being a good thing are not seasoned developers.