4 ms·

So I am using Alpine and generally like it. One question that perhaps someone on this thread will know... why does it cache DNS differently from Ubuntu, and is

So I am using Alpine and generally like it. One question that perhaps someone on this thread will know... why does it cache DNS differently from Ubuntu, and is there a way to fix it? My app has a domain name for the database sever that I pass to Psycopg2.. say foobar.something.google-cloud.internal. It resolves to the private IP of my PostgreSQL sever. The problem, is EVERY SINGLE database query is querying the DNS. I can sometimes push 10,000 DNS queries per second. 9,990 of those are for the same stupid database server. Kubernetes runs it's own DNS server which takes the load, but sometimes they get overwhelmed a bit. I am obviously doing something wrong, but not sure the path to explore down. If I can fix this, then alpine is working pretty amazing for me :)

tbrock 9y ago
You can fix this by installing and running a local dns server like dnsmasq to cache for you: apk install dnsmasq See this: https://github.com/gliderlabs/docker-alpine/blob/master/docs/caveats.md#dns https://github.com/gliderlabs/docker-alpine/blob/master/docs...
- brianwawok 9y ago
  Fantastic will try this edit - so far no results, must need some other change to turn it on...
  - mrighele 9y ago
    If you're still in search for a solution: dnsmasq acts as a caching dns server listening on your loopback device, so you probably need to change the dns server of your lan/wifi to 127.0.0.1 (or whatever address dnsmasq is configured to listen at), and also set what external dns servers should be used by dnsmasq. Those are usually not setup properly by default.
tsuraan 9y ago
tbrock gave the work-around, but as for the "why", it's probably because Debian uses glibc for its libc, while Alpine uses musl. musl aims to be a much more compliant libc than glibc, so when you tell it to look up a name, it probably always does so. Like tbrock said, installing a local dns cache should help (or, failing that, an /etc/hosts entry?).
- mitchty 9y ago
  Its probably just nscd that is caching dns requests on debian not glibc. Though I thought nscd dns ttl caching was broken there. I could be wrong. Musl libc's dns strategy should actually be faster than glibc when its used. https://wiki.musl-libc.org/functional-differences-from-glibc.html#Name-Resolver/DNS https://wiki.musl-libc.org/functional-differences-from-glibc... That said a local cache would be ideal.
- brianwawok 9y ago
  Ah ok.. My apk list looks something like gcc \ make \ libc-dev \ musl-dev \ linux-headers \ pcre-dev \ postgresql-dev \ zlib-dev \ jpeg-dev \ libxslt-dev \ libxml2-dev \ git \ so I see musl but glibc in there.... I do not think I want to do the /etc/hosts thing, as the IP can change for some things.. I would rather just cache it locally for ? 1 minute or something
gtirloni 9y ago
If your app is opening a new connection for every query, that might be your problem. I would look into having a pool of connections. Alpine uses musl instead of glibc. Maybe you can find a way to run nscd on it but it doesn't seem very idiomatic for a container platform.
- paulfurtado 9y ago
  This and a couple of other comments mention nscd, however, unless it's different on other distributions, nscd works via a unix socket. To make it work in containers, you need to explicitly mount it in from the host or run the nscd daemon inside the container. I'd be surprised if @brianwawok was doing that with their non-alpine containers and didn't realize it since it's so explicit That said, I believe Alpine does have a glibc package which only adds a few MB. So @brianwawok can at least still run Alpine for the image size benefits if musl is unimportant to them.
- brianwawok 9y ago
  These are persistent connections, it's not that.
  - gtirloni 9y ago
    That's weird. I would fire up strace and check what's really happening. musl/glibc shouldn't have visibility into the queries you're making. My suspicion is that something at the app level is doing something that it shouldn't, not musl.
- ifbizo 9y ago
  Thanks @gtirloni! I'm new to SQL and was working on a small internal tool. This change, an obvious one in hindsight, sped up my tool by ~2-3x. So for anyone else, check this!
  - deleted 9y ago
    [deleted]
the_duke 9y ago
A) If you are opening 10000 new db connections per second you are doing something very wrong. Use connection pooling! Psycopg2 actually has one built in. (http://initd.org/psycopg/docs/pool.html http://initd.org/psycopg/docs/pool.html) B) If DNS actually becomes a legitimate bottleneck, you can just manually resolve the domain name and cache it. If the db sits behind a load balancer or does DNS round robing you can keep the cache time low and re-resolve on connection failures.
- brianwawok 9y ago
  a) No they are persistent. I set the timeout to be something like 600 seconds. They are wanting to do a DNS query every time they are used for ???reasons????. b) I think dnsmasq is a better answer. My DNS is already load balanced in k8s, but that is getting hit pretty hard.

tbrock 9y ago

You can fix this by installing and running a local dns server like dnsmasq to cache for you: apk install dnsmasq See this: https://github.com/gliderlabs/docker-alpine/blob/master/docs/caveats.md#dns https://github.com/gliderlabs/docker-alpine/blob/master/docs...

brianwawok 9y ago
Fantastic will try this edit - so far no results, must need some other change to turn it on...
- mrighele 9y ago
  If you're still in search for a solution: dnsmasq acts as a caching dns server listening on your loopback device, so you probably need to change the dns server of your lan/wifi to 127.0.0.1 (or whatever address dnsmasq is configured to listen at), and also set what external dns servers should be used by dnsmasq. Those are usually not setup properly by default.

tsuraan 9y ago

tbrock gave the work-around, but as for the "why", it's probably because Debian uses glibc for its libc, while Alpine uses musl. musl aims to be a much more compliant libc than glibc, so when you tell it to look up a name, it probably always does so. Like tbrock said, installing a local dns cache should help (or, failing that, an /etc/hosts entry?).

mitchty 9y ago
Its probably just nscd that is caching dns requests on debian not glibc. Though I thought nscd dns ttl caching was broken there. I could be wrong. Musl libc's dns strategy should actually be faster than glibc when its used. https://wiki.musl-libc.org/functional-differences-from-glibc.html#Name-Resolver/DNS https://wiki.musl-libc.org/functional-differences-from-glibc... That said a local cache would be ideal.
brianwawok 9y ago
Ah ok.. My apk list looks something like gcc \ make \ libc-dev \ musl-dev \ linux-headers \ pcre-dev \ postgresql-dev \ zlib-dev \ jpeg-dev \ libxslt-dev \ libxml2-dev \ git \ so I see musl but glibc in there.... I do not think I want to do the /etc/hosts thing, as the IP can change for some things.. I would rather just cache it locally for ? 1 minute or something

gtirloni 9y ago

If your app is opening a new connection for every query, that might be your problem. I would look into having a pool of connections. Alpine uses musl instead of glibc. Maybe you can find a way to run nscd on it but it doesn't seem very idiomatic for a container platform.

paulfurtado 9y ago
This and a couple of other comments mention nscd, however, unless it's different on other distributions, nscd works via a unix socket. To make it work in containers, you need to explicitly mount it in from the host or run the nscd daemon inside the container. I'd be surprised if @brianwawok was doing that with their non-alpine containers and didn't realize it since it's so explicit That said, I believe Alpine does have a glibc package which only adds a few MB. So @brianwawok can at least still run Alpine for the image size benefits if musl is unimportant to them.
brianwawok 9y ago
These are persistent connections, it's not that.
- gtirloni 9y ago
  That's weird. I would fire up strace and check what's really happening. musl/glibc shouldn't have visibility into the queries you're making. My suspicion is that something at the app level is doing something that it shouldn't, not musl.
ifbizo 9y ago
Thanks @gtirloni! I'm new to SQL and was working on a small internal tool. This change, an obvious one in hindsight, sped up my tool by ~2-3x. So for anyone else, check this!
- deleted 9y ago
  [deleted]

the_duke 9y ago

A) If you are opening 10000 new db connections per second you are doing something very wrong. Use connection pooling! Psycopg2 actually has one built in. (http://initd.org/psycopg/docs/pool.html http://initd.org/psycopg/docs/pool.html) B) If DNS actually becomes a legitimate bottleneck, you can just manually resolve the domain name and cache it. If the db sits behind a load balancer or does DNS round robing you can keep the cache time low and re-resolve on connection failures.

brianwawok 9y ago
a) No they are persistent. I set the timeout to be something like 600 seconds. They are wanting to do a DNS query every time they are used for ???reasons????. b) I think dnsmasq is a better answer. My DNS is already load balanced in k8s, but that is getting hit pretty hard.