5 ms·

Understanding DNS Resolution on Linux and Kubernetes

74 points by fanf2 2y ago

szszrk 2y ago
> The Kubernetes DNS resolves A-B-C-D.N.pod.cluster.local to A.B.C.D, as long as A.B.C.D is a valid IP address and N is an existing namespace. Let’s be honest: I don’t know how this serves any purpose, but if you do, please let me know! You can use that to - test weird dns setups - to issue proper TLS certificates (you can do that technically, but it's less known fact and some services like let's encrypt forbid that as their rule) - to utilize single IP and same port for multiple services (so just a common host/server configuration on typical reverse proxy, optionally with SNI to be used with TLS on top.
- vbezhenar 2y ago
  I don’t think you can issue proper cert for a private IP. So using dns host name is the only option.
  - deleted 2y ago
    [deleted]
  - CableNinja 2y ago
    If you control an internal CA you can make certs for anything. I have one for my homelab, and even have a few certs issued for my homelab, which are not for domains i control as well as certs with IPs. The CA is who says you cant do those things, and yes its generally agreed upon for the public internet, certs shouldnt have IPs in them, but if you are operating internally theres nothing stopping you.
    - znpy 2y ago
      setting up kubernetes typically involves creating a private CA, so most definitely yes, you technically can issue certificates for whatever you want.
    - weinzierl 2y ago
      Let's encrypt public internet certs can have IPs in them. https://letsencrypt.org/2025/01/16/6-day-and-ip-certs/ https://letsencrypt.org/2025/01/16/6-day-and-ip-certs/
      - szszrk 2y ago
        I completely forget about that announcement. Is that already available as GA? Because that blog post was just a teaser for the whole 2025 and I can't see docs about it at first glance.
      - CableNinja 2y ago
        Thats a pretty recent change, only 2 months ago. I wasnt aware of that, and you usually wont find that woth other CAs. Im not sure i like the public internet with ip certs. I do it at home because sometimes dns be down and have some good internal uses. But, shouldnt be public. Imagine firing up a /24 on linode, requesting certs on every ip, then releasing the ips, and saving the certs. Another linode account would later get an ip in that range, and then you can freely mitm the linode site by ip. Im making a number of 'magical' things in between this, of course, but, it seems allowing an IP from a public CA could be a terrible thing. The only saving grace in this case is the short lifetime of the certs, however, im not a fan of that either. As an aside, im starting to get squinty eyes relating to LE, both things they announce in that article, are things that greatly affect the internet at large. I see it as something google would pull to ensure dominance by lock-in. Sorry you can no longer change SSL providers because certs only live a few minutes now, and of course you cant afford to not have a cert or no one will see your site. Im exaggerating slightly, but these changes are not something i think should be allowed, and LE shouldve listened to everyone yelling. Sure, allow down to 6 day certs, but that will surely become the maximum soon.
      - crtasm 2y ago
        Not yet, but coming soon.
    - sciencesama 2y ago
      Which software are you using and what is the process !??
      - CableNinja 2y ago
        Im just using hashicorp vault, with a multi tier CA setup. Real CA is just a docker container that stays off 99.9% of the time, and the furthest branch intermediate lives on the actual vault instance, which is used to issue certs, among other things. As long as you add your CA chain to your system and browser, they get treated as valid. I even have long term certs in some places because i dont want to change them. Public CAs max at 1 year most of the time now. I issue certs for my internal domains, and public subdomains where im only the one to use it. I use LE for the real public things.
        XorNot 2y ago
        If you're running your own CA then really should just set your expiries to the maximum (49 years is practical) and never worry about it. You don't have enough nodes for CRL size to become a problem, and if a node does get compromised you're hardly going to leave it up and running for a year (i.e. you'd obviously kill the node, and the cert is useless without control of the DNS name). EDIT: the other direction to go of course is way shorter. Like do you need a certificate with a lifetime longer then business hours before renewal?
    - szszrk 2y ago
      > its generally agreed upon for the public internet, certs shouldnt have IPs in them That's a bit of a stretch to say anyone agreed on not using IP based certs. Quite the contrary. It is present in RFC 5280 and SAN can contain an IP. It's just very rare to do that, but can be done and is done. Modern browsers and OSs accept it as well. It's nice when you need to do some cert pinning to make sure there is not MITM eavesdropping, or for example on some onprem environments where you can't fully control workstations/DNS of you user endpoints, but still want to have your services behind certs that actually properly validate.
  - deleted 2y ago
    [deleted]
  - szszrk 2y ago
    Private CA's are a thing, it's not even rare in organizations that control their hardware. There are plenty of use cases to go that route.
zokier 2y ago
It's bit curious that traditionally UNIX systems did not run local DNS resolver daemons and instead the resolv.conf (and nsswitch.conf) persisted for so long. In addition to potentially simplifying configuration, having a daemon would allow system-wide dns caching, something I'd imagine would have been especially valuable back in the days of slow networks. Unix has daemons for everything else so that's why it feels odd that name resolution got baked into libc
- tyingq 2y ago
  Some of the early unix systems did have a local daemon. The much-maligned SunOS NIS/YP services as one example.
- bandie91 2y ago
  maybe for most cases nscd was enough. not exactly a dns-cache but a hostname cache on one layer up.
  - sleepydog 2y ago
    I used to work for a huge email sender (constant contact). Our mail servers needed to perform an absurd number of lookups while receiving and validating mail. When I was there, we used dnscache, running locally, on all our mail servers. But even with a local dnscache, the overhead of making DNS requests and handling their responses was high enough that adding nscd made a noticeable improvement in CPU usage.
    - bandie91 2y ago
      i guess this shows that looking up getent hostname database cache is faster than looking up local dns cache because the former is simpler in data structure?
      - sleepydog 2y ago
        I didn't dig into it too deeply at the time, but I think part of it was that you don't need to open and write to a socket, so that's avoiding some system calls (socket(), bind(), sendto(), close()). IIRC we had nscd set up so clients directly read from shared memory that nscd kept updated, rather than getting requests over a socket. There's also probably some savings around not having to convert between the structures used by gethostbyname and DNS questions&answers.
AndyMcConachie 2y ago
People need to stop using .local or .dev for stuff like this. .dev is an actual TLD in the root zone and .local is for multicast DNS. ICANN has said they will never delegate .internal and it should be used for these kinds of private uses. I'm a coauthor on this Internet draft so I'm ofc rather biased. https://datatracker.ietf.org/doc/draft-davies-internal-tld/ https://datatracker.ietf.org/doc/draft-davies-internal-tld/
- rascul 2y ago
  It seems like they are using mDNS, though.
- globular-toast 2y ago
  Yeah, it's quite annoying. foo.bar.svc.cluster.internal even reads better. There is also home.arpa for LAN stuff if you don't own a domain.
- CableNinja 2y ago
  I use .lan at home, which is great, until i enter it in the browser and forget to add a / at the end. Both chrome and firefox just immediately think its a search request
  - sethops1 2y ago
    Same here, feels so natural to have my local machines at <host>.lan
- Joker_vD 2y ago
  > and .local is for multicast DNS. Does reusing it cause any problem for the mDNS, or does mDNS usage cause problem for the internal-domains usage?
  - vel0city 2y ago
    A lot of default configurations won't bother looking up .local hostnames on your DNS server and will only issue an mDNS query. This can often be changed but can be annoying to have to ensure it gets configured correctly everywhere. And then when you reconfigure it, depending on the stack it won't bother querying mDNS at all if a DNS resolver responds.
ChoHag 2y ago
[dead]
deleted 2y ago
[deleted]

74 points by fanf2 2y ago

> The Kubernetes DNS resolves A-B-C-D.N.pod.cluster.local to A.B.C.D, as long as A.B.C.D is a valid IP address and N is an existing namespace. Let’s be honest: I don’t know how this serves any purpose, but if you do, please let me know! You can use that to - test weird dns setups - to issue proper TLS certificates (you can do that technically, but it's less known fact and some services like let's encrypt forbid that as their rule) - to utilize single IP and same port for multiple services (so just a common host/server configuration on typical reverse proxy, optionally with SNI to be used with TLS on top.

vbezhenar 2y ago
I don’t think you can issue proper cert for a private IP. So using dns host name is the only option.
- deleted 2y ago
  [deleted]
- CableNinja 2y ago
  If you control an internal CA you can make certs for anything. I have one for my homelab, and even have a few certs issued for my homelab, which are not for domains i control as well as certs with IPs. The CA is who says you cant do those things, and yes its generally agreed upon for the public internet, certs shouldnt have IPs in them, but if you are operating internally theres nothing stopping you.
  - znpy 2y ago
    setting up kubernetes typically involves creating a private CA, so most definitely yes, you technically can issue certificates for whatever you want.
  - weinzierl 2y ago
    Let's encrypt public internet certs can have IPs in them. https://letsencrypt.org/2025/01/16/6-day-and-ip-certs/ https://letsencrypt.org/2025/01/16/6-day-and-ip-certs/
    - szszrk 2y ago
      I completely forget about that announcement. Is that already available as GA? Because that blog post was just a teaser for the whole 2025 and I can't see docs about it at first glance.
    - CableNinja 2y ago
      Thats a pretty recent change, only 2 months ago. I wasnt aware of that, and you usually wont find that woth other CAs. Im not sure i like the public internet with ip certs. I do it at home because sometimes dns be down and have some good internal uses. But, shouldnt be public. Imagine firing up a /24 on linode, requesting certs on every ip, then releasing the ips, and saving the certs. Another linode account would later get an ip in that range, and then you can freely mitm the linode site by ip. Im making a number of 'magical' things in between this, of course, but, it seems allowing an IP from a public CA could be a terrible thing. The only saving grace in this case is the short lifetime of the certs, however, im not a fan of that either. As an aside, im starting to get squinty eyes relating to LE, both things they announce in that article, are things that greatly affect the internet at large. I see it as something google would pull to ensure dominance by lock-in. Sorry you can no longer change SSL providers because certs only live a few minutes now, and of course you cant afford to not have a cert or no one will see your site. Im exaggerating slightly, but these changes are not something i think should be allowed, and LE shouldve listened to everyone yelling. Sure, allow down to 6 day certs, but that will surely become the maximum soon.
    - crtasm 2y ago
      Not yet, but coming soon.
  - sciencesama 2y ago
    Which software are you using and what is the process !??
    - CableNinja 2y ago
      Im just using hashicorp vault, with a multi tier CA setup. Real CA is just a docker container that stays off 99.9% of the time, and the furthest branch intermediate lives on the actual vault instance, which is used to issue certs, among other things. As long as you add your CA chain to your system and browser, they get treated as valid. I even have long term certs in some places because i dont want to change them. Public CAs max at 1 year most of the time now. I issue certs for my internal domains, and public subdomains where im only the one to use it. I use LE for the real public things.
      - XorNot 2y ago
        If you're running your own CA then really should just set your expiries to the maximum (49 years is practical) and never worry about it. You don't have enough nodes for CRL size to become a problem, and if a node does get compromised you're hardly going to leave it up and running for a year (i.e. you'd obviously kill the node, and the cert is useless without control of the DNS name). EDIT: the other direction to go of course is way shorter. Like do you need a certificate with a lifetime longer then business hours before renewal?
  - szszrk 2y ago
    > its generally agreed upon for the public internet, certs shouldnt have IPs in them That's a bit of a stretch to say anyone agreed on not using IP based certs. Quite the contrary. It is present in RFC 5280 and SAN can contain an IP. It's just very rare to do that, but can be done and is done. Modern browsers and OSs accept it as well. It's nice when you need to do some cert pinning to make sure there is not MITM eavesdropping, or for example on some onprem environments where you can't fully control workstations/DNS of you user endpoints, but still want to have your services behind certs that actually properly validate.
- deleted 2y ago
  [deleted]
- szszrk 2y ago
  Private CA's are a thing, it's not even rare in organizations that control their hardware. There are plenty of use cases to go that route.

zokier 2y ago

It's bit curious that traditionally UNIX systems did not run local DNS resolver daemons and instead the resolv.conf (and nsswitch.conf) persisted for so long. In addition to potentially simplifying configuration, having a daemon would allow system-wide dns caching, something I'd imagine would have been especially valuable back in the days of slow networks. Unix has daemons for everything else so that's why it feels odd that name resolution got baked into libc

tyingq 2y ago
Some of the early unix systems did have a local daemon. The much-maligned SunOS NIS/YP services as one example.
bandie91 2y ago
maybe for most cases nscd was enough. not exactly a dns-cache but a hostname cache on one layer up.
- sleepydog 2y ago
  I used to work for a huge email sender (constant contact). Our mail servers needed to perform an absurd number of lookups while receiving and validating mail. When I was there, we used dnscache, running locally, on all our mail servers. But even with a local dnscache, the overhead of making DNS requests and handling their responses was high enough that adding nscd made a noticeable improvement in CPU usage.
  - bandie91 2y ago
    i guess this shows that looking up getent hostname database cache is faster than looking up local dns cache because the former is simpler in data structure?
    - sleepydog 2y ago
      I didn't dig into it too deeply at the time, but I think part of it was that you don't need to open and write to a socket, so that's avoiding some system calls (socket(), bind(), sendto(), close()). IIRC we had nscd set up so clients directly read from shared memory that nscd kept updated, rather than getting requests over a socket. There's also probably some savings around not having to convert between the structures used by gethostbyname and DNS questions&answers.

AndyMcConachie 2y ago

People need to stop using .local or .dev for stuff like this. .dev is an actual TLD in the root zone and .local is for multicast DNS. ICANN has said they will never delegate .internal and it should be used for these kinds of private uses. I'm a coauthor on this Internet draft so I'm ofc rather biased. https://datatracker.ietf.org/doc/draft-davies-internal-tld/ https://datatracker.ietf.org/doc/draft-davies-internal-tld/

rascul 2y ago
It seems like they are using mDNS, though.
globular-toast 2y ago
Yeah, it's quite annoying. foo.bar.svc.cluster.internal even reads better. There is also home.arpa for LAN stuff if you don't own a domain.
CableNinja 2y ago
I use .lan at home, which is great, until i enter it in the browser and forget to add a / at the end. Both chrome and firefox just immediately think its a search request
- sethops1 2y ago
  Same here, feels so natural to have my local machines at <host>.lan
Joker_vD 2y ago
> and .local is for multicast DNS. Does reusing it cause any problem for the mDNS, or does mDNS usage cause problem for the internal-domains usage?
- vel0city 2y ago
  A lot of default configurations won't bother looking up .local hostnames on your DNS server and will only issue an mDNS query. This can often be changed but can be annoying to have to ensure it gets configured correctly everywhere. And then when you reconfigure it, depending on the stack it won't bother querying mDNS at all if a DNS resolver responds.

ChoHag 2y ago

[dead]

deleted 2y ago

[deleted]