7 ms·

First, the lack of confidence that "localhost" actually resolves to the loopback interface encourages application developers to hard-code IP addresses lik

First, the lack of confidence that "localhost" actually resolves to the loopback interface encourages application developers to hard-code IP addresses like "127.0.0.1" in order to obtain certainty regarding routing. This causes problems in the transition from IPv4 to IPv6 (see problem 8 in [draft-ietf-sunset4-gapanalysis]). That does remind me of the times I was dealing with weird connection issues in some critical services. It turned to be related to the use of "localhost" in the configuration. It resolves to ipv6 on some systems and that breaks everything because the target app is only listening to the ipv4 address. Went as far as removing all references to localhost and added lint errors in the configuration system so that noone could ever be able to give localhost as a setting in anything.

rmc 9y ago
The weirdest was a co-worker who had some simple webserver, which was listening on only IPv4 or IPv6 (but not both). When he went to "localhost" on Firefox it used IPv4 and he was able to see it. On Chrome "localhost" was IPv6 (or the other way around), and he got "Could not connect" error. It confused him no end how this simple web server worked on FF but not Chrome. :)
tootie 9y ago
I always have local.my company.com DNS that resolves to 127.0.0.1. I can get a valid cert that way too.
- rblatz 9y ago
  I hope you don't publish that record to the world.
  - icebraining 9y ago
    Why?
    - brazzledazzle 9y ago
      Just a guess: CORS-related attacks
      - icebraining 9y ago
        How would that work?
        deleted 9y ago
        [deleted]
    - IshKebab 9y ago
      I guess anyone on 127.0.0.1 can pretend to that address. Very unlikely to matter.
    - jwilk 9y ago
      Same-site scripting: http://seclists.org/bugtraq/2008/Jan/270 http://seclists.org/bugtraq/2008/Jan/270
      - icebraining 9y ago
        Interesting. Still, that requires the attacker to be already running a process on the victim's machine, even if with reduced privileges. Nowadays that's rare, since there's no reason not to give each user its own network namespace, at the very least.
  - piscisaureus 9y ago
    It's already been done: https://git.daplie.com/Daplie/localhost.daplie.me-certificates#https-certs-for-localhost-development https://git.daplie.com/Daplie/localhost.daplie.me-certificat... Which I find to be a very practical solution for connecting to localhost over https, it frees you from having to install a self-signed certificates/CAs on your machine.
    - 0x0 9y ago
      Not a great idea to publish private keys for valid certificates. Anyone could probably submit a certificate revocation request to the CA, as the key would be considered compromised.
    - jaas 9y ago
      Publishing private keys is a violation of the Let's Encrypt terms of service. We are revoking these certificates. https://letsencrypt.org/documents/LE-SA-v1.1.1-August-1-2016.pdf https://letsencrypt.org/documents/LE-SA-v1.1.1-August-1-2016...
  - Spivak 9y ago
    Lots of sites seem to be doing it now. Off the top of my head I know that Box and Spotify both do it.
tootie 9y ago
I always have local.my company.com DNS that resolves to 127.0.0.1. I can get a valid cert that way too.
tedunangst 9y ago
Hello me from last week. Had exactly this bug, sometimes nginx couldn't connect to the backend (but very rarely, and not reproducible on demand), which I eventually tracked to the fact that localhost sometimes resolved to ::1 instead of 127, which is what the backend was listening on. Still don't understand why it was only like 1 in 1000 requests, and not every or every other request. Just one more slice of ipv6 mystery.
- jacquesm 9y ago
  I've had weird errors like that where two DNS servers were giving answers to my query rather than just the one that I intended. This will never happen when using TCP but when using UDP it may happen. Every now and then the packets would receive in a different order and then I'd be paged because some app fell over. Fun times.
- inopinatus 9y ago
  Is the Nginx client using the happy eyeballs algorithm? https://en.m.wikipedia.org/wiki/Happy_Eyeballs https://en.m.wikipedia.org/wiki/Happy_Eyeballs Can be a source of race conditions.
- jsz0 9y ago
  I had a very similar problem recently with docker + nginx. Best I could figure out was the randomness of the problem was being caused by keep alive connection limits. If the connection was opened as IPv4 it would work until it hit the keep alive limit but the new connection might run into the IPv4/IPv6 lookup problem and fail. Never really figured it out for sure. It's definitely thrown some cold water on my plans to go dual stack everywhere all the time. Not sure it's worth the risk of running into these stupid bugs.
deathanatos 9y ago
> and that breaks everything If you're on a POSIX system, I'd argue that this is a bug in the client. Typically, the client should call getaddrinfo(3); as part of that, the application would either specify directly that it's only interested in AF_INET results, or just filter out non-AF_INET results. (Further, if you support IPv6 in the client, and thus request such results from getaddrinfo, you should skip to the next result if the connection fails.) On the server, you can also bind to both the IPv4 and the IPv6 addresses. If you listen to ::, you should get IPv4 connections too. (Through this[1] mechanism.) [1]: https://en.wikipedia.org/wiki/IPv6_address#Transition_from_IPv4 https://en.wikipedia.org/wiki/IPv6_address#Transition_from_I...
- wahern 9y ago
  The mechanism you alluded to (dual binding--receiving IPv4-mapped IPv6 addresses on an IPv6 socket) requires explicitly disabling the IPV6_V6ONLY option on each socket. Some systems have IPV6_V6ONLY as the default; I think modern FreeBSD releases do this out-of-the-box. I don't think many Linux distributions enable IPV6_V6ONLY by default, but administrators can enable it globally, necessitating a per-socket reversion. Some systems, like OpenBSD, don't even support disabling IPV6_V6ONLY and therefore don't support dual binding at all. OpenBSD contentiously argues that dual binding is likely to lead to security exploits as applications that naively bind to "::" might not expect to de-queue IPv4-mapped IPv6 addresses, consequentially possibly breaking their local access control logic. For example, they may setup firewalls rules that restrict access to the IPv6 port but forget to set the same restrictions on IPv4 ports. I'm not sure I agree with OpenBSD's approach, but in any event applications should explicitly disable the IPV6_V6ONLY socket option if they're relying on dual binding. Ideally they should use two different sockets; one for each address family. If the application stack doesn't make it easy to listen on multiple sockets, that's a strong hint that the design is broken.
- kbutler 9y ago
  Does it matter if it's a bug in the client? Make it work like expected. Especially when changing infrastructure like IPV4->IPV6 - don't break existing userbase code! (This is a fundamental precept of Linux development.)
p1mrx 9y ago
Ideally, a server binding to "localhost" should create a listening socket for each of its IP addresses (e.g. 127.0.0.1 and ::1), and a client connecting to "localhost" should try each IP address in order, until one succeeds. But a lot of standard libraries (including parts of Java and Go) get this wrong, and pick exactly one IP address arbitrarily. When you combine a buggy client with a buggy server, and their preferences for IPv4/IPv6 disagree, then all hell breaks loose. These are the open bugs I'm aware of: https://bugs.openjdk.java.net/browse/JDK-8170568 https://bugs.openjdk.java.net/browse/JDK-8170568 https://github.com/golang/go/issues/9334 https://github.com/golang/go/issues/9334
- fulafel 9y ago
  Or use a single socket for both protocols (set the IPV6_V6ONLY socket option to false).
  - p1mrx 9y ago
    That works when binding to :: (all interfaces), but it's irrelevant when binding to "localhost", because you can only bind one address per socket.
    - fulafel 9y ago
      Yes, I thought it worked for ::1 too but you're right.
- user5994461 9y ago
  An interface on linux has only one ip address. The "localhost" interface will either designate the ipv4 127.0.0.1 interface or the ipv6 ::1 interface. That's the realm of undefined behavior and system specifics. This whole IETF draft looks like a mess. They should reserve the names localhost4 and localhost6.
  - pilif 9y ago
    > An interface on linux has only one ip address. absolutely not true. Neither for IPv4 nor IPv6, where it's even the default to have a multitude of addresses on an interface. > The "localhost" interface will either designate the ipv4 127.0.0.1 interface or the ipv6 ::1 interface. That's the realm of undefined behavior and system specifics. no it won't. My loopback (lo) interface has both 127.0.0.1 and ::1 as its addresses.
DominoTree 9y ago
Some genius at my company decided that ~180,000 Windows endpoints needed "localhost" removed from their hosts file, which has resulted in millions of requests per minute for localhost hitting our resolvers just to return 127.0.0.1. My guess is that it was some hack they tried to disable IPv6, but aside from the insane load it added to the DNS infrastructure, the other result is that if these machines talk to a malicious resolver, their traffic destined for the loopback interface could end up going anywhere and being captured by anyone. Great job!
- JdeBP 9y ago
  If the machines talk to a malicious resolving proxy DNS server, then more than traffic destined for loopback is at risk. I suspect that removing the "localhost." record was nothing to do with IPv6 and everything to do with a corporate policy to not have anything other than the Microsoft default contents in hosts files, possibly because of concerns relating to malware prevention. The problem is possibly the result of the default hosts content changing in Windows NT 6.1. * https://support.microsoft.com/en-gb/help/972034/ https://support.microsoft.com/en-gb/help/972034/ As of Windows NT 6.1, lookups of "localhost." are handled internally within (as I understand) the DNS Client, and never require inspecting a hosts file or sending a query to a DNS server. So the new default hosts file content no longer contains a "localhost." record. But use the Windows NT 6.1 or later default hosts file content on earlier versions of Windows NT, and one will see "localhost." queries being sent by the DNS Client to a server. Handling "localhost." within the DNS Client is -- reportedly -- so that the DNS Client can inspect the local machine's protocol support and only return non-empty AAAA and A resource record sets if IPv6 or IPv4 is actually enabled on the machine.
okket 9y ago
> It turned to be related to the use of "localhost" in the configuration. It resolves to ipv6 on some systems and that breaks everything because the target app is only listening to the ipv4 address. You just found a major bug in the application and should complain to the developer. Applications that do not support IPv6 are simply broken and should be avoided at all cost by now.
fundabulousrIII 9y ago
Got to chuckle about this. The new generation is fearless and naked. Break it all, admit nothing, make it 'better'. Standards, practices, tradition, culture. They mean nothing when a devops lead has commit rights to the ansible playbook and a will to deliver a fix in 5 seconds flat.

rmc 9y ago

The weirdest was a co-worker who had some simple webserver, which was listening on only IPv4 or IPv6 (but not both). When he went to "localhost" on Firefox it used IPv4 and he was able to see it. On Chrome "localhost" was IPv6 (or the other way around), and he got "Could not connect" error. It confused him no end how this simple web server worked on FF but not Chrome. :)

tootie 9y ago

I always have local.my company.com DNS that resolves to 127.0.0.1. I can get a valid cert that way too.

rblatz 9y ago
I hope you don't publish that record to the world.
- icebraining 9y ago
  Why?
  - brazzledazzle 9y ago
    Just a guess: CORS-related attacks
    - icebraining 9y ago
      How would that work?
      - deleted 9y ago
        [deleted]
  - IshKebab 9y ago
    I guess anyone on 127.0.0.1 can pretend to that address. Very unlikely to matter.
  - jwilk 9y ago
    Same-site scripting: http://seclists.org/bugtraq/2008/Jan/270 http://seclists.org/bugtraq/2008/Jan/270
    - icebraining 9y ago
      Interesting. Still, that requires the attacker to be already running a process on the victim's machine, even if with reduced privileges. Nowadays that's rare, since there's no reason not to give each user its own network namespace, at the very least.
- piscisaureus 9y ago
  It's already been done: https://git.daplie.com/Daplie/localhost.daplie.me-certificates#https-certs-for-localhost-development https://git.daplie.com/Daplie/localhost.daplie.me-certificat... Which I find to be a very practical solution for connecting to localhost over https, it frees you from having to install a self-signed certificates/CAs on your machine.
  - 0x0 9y ago
    Not a great idea to publish private keys for valid certificates. Anyone could probably submit a certificate revocation request to the CA, as the key would be considered compromised.
  - jaas 9y ago
    Publishing private keys is a violation of the Let's Encrypt terms of service. We are revoking these certificates. https://letsencrypt.org/documents/LE-SA-v1.1.1-August-1-2016.pdf https://letsencrypt.org/documents/LE-SA-v1.1.1-August-1-2016...
- Spivak 9y ago
  Lots of sites seem to be doing it now. Off the top of my head I know that Box and Spotify both do it.

tootie 9y ago

I always have local.my company.com DNS that resolves to 127.0.0.1. I can get a valid cert that way too.

tedunangst 9y ago

Hello me from last week. Had exactly this bug, sometimes nginx couldn't connect to the backend (but very rarely, and not reproducible on demand), which I eventually tracked to the fact that localhost sometimes resolved to ::1 instead of 127, which is what the backend was listening on. Still don't understand why it was only like 1 in 1000 requests, and not every or every other request. Just one more slice of ipv6 mystery.

jacquesm 9y ago
I've had weird errors like that where two DNS servers were giving answers to my query rather than just the one that I intended. This will never happen when using TCP but when using UDP it may happen. Every now and then the packets would receive in a different order and then I'd be paged because some app fell over. Fun times.
inopinatus 9y ago
Is the Nginx client using the happy eyeballs algorithm? https://en.m.wikipedia.org/wiki/Happy_Eyeballs https://en.m.wikipedia.org/wiki/Happy_Eyeballs Can be a source of race conditions.
jsz0 9y ago
I had a very similar problem recently with docker + nginx. Best I could figure out was the randomness of the problem was being caused by keep alive connection limits. If the connection was opened as IPv4 it would work until it hit the keep alive limit but the new connection might run into the IPv4/IPv6 lookup problem and fail. Never really figured it out for sure. It's definitely thrown some cold water on my plans to go dual stack everywhere all the time. Not sure it's worth the risk of running into these stupid bugs.

deathanatos 9y ago

> and that breaks everything If you're on a POSIX system, I'd argue that this is a bug in the client. Typically, the client should call getaddrinfo(3); as part of that, the application would either specify directly that it's only interested in AF_INET results, or just filter out non-AF_INET results. (Further, if you support IPv6 in the client, and thus request such results from getaddrinfo, you should skip to the next result if the connection fails.) On the server, you can also bind to both the IPv4 and the IPv6 addresses. If you listen to ::, you should get IPv4 connections too. (Through this[1] mechanism.) [1]: https://en.wikipedia.org/wiki/IPv6_address#Transition_from_IPv4 https://en.wikipedia.org/wiki/IPv6_address#Transition_from_I...

wahern 9y ago
The mechanism you alluded to (dual binding--receiving IPv4-mapped IPv6 addresses on an IPv6 socket) requires explicitly disabling the IPV6_V6ONLY option on each socket. Some systems have IPV6_V6ONLY as the default; I think modern FreeBSD releases do this out-of-the-box. I don't think many Linux distributions enable IPV6_V6ONLY by default, but administrators can enable it globally, necessitating a per-socket reversion. Some systems, like OpenBSD, don't even support disabling IPV6_V6ONLY and therefore don't support dual binding at all. OpenBSD contentiously argues that dual binding is likely to lead to security exploits as applications that naively bind to "::" might not expect to de-queue IPv4-mapped IPv6 addresses, consequentially possibly breaking their local access control logic. For example, they may setup firewalls rules that restrict access to the IPv6 port but forget to set the same restrictions on IPv4 ports. I'm not sure I agree with OpenBSD's approach, but in any event applications should explicitly disable the IPV6_V6ONLY socket option if they're relying on dual binding. Ideally they should use two different sockets; one for each address family. If the application stack doesn't make it easy to listen on multiple sockets, that's a strong hint that the design is broken.
kbutler 9y ago
Does it matter if it's a bug in the client? Make it work like expected. Especially when changing infrastructure like IPV4->IPV6 - don't break existing userbase code! (This is a fundamental precept of Linux development.)

p1mrx 9y ago

Ideally, a server binding to "localhost" should create a listening socket for each of its IP addresses (e.g. 127.0.0.1 and ::1), and a client connecting to "localhost" should try each IP address in order, until one succeeds. But a lot of standard libraries (including parts of Java and Go) get this wrong, and pick exactly one IP address arbitrarily. When you combine a buggy client with a buggy server, and their preferences for IPv4/IPv6 disagree, then all hell breaks loose. These are the open bugs I'm aware of: https://bugs.openjdk.java.net/browse/JDK-8170568 https://bugs.openjdk.java.net/browse/JDK-8170568 https://github.com/golang/go/issues/9334 https://github.com/golang/go/issues/9334

fulafel 9y ago
Or use a single socket for both protocols (set the IPV6_V6ONLY socket option to false).
- p1mrx 9y ago
  That works when binding to :: (all interfaces), but it's irrelevant when binding to "localhost", because you can only bind one address per socket.
  - fulafel 9y ago
    Yes, I thought it worked for ::1 too but you're right.
user5994461 9y ago
An interface on linux has only one ip address. The "localhost" interface will either designate the ipv4 127.0.0.1 interface or the ipv6 ::1 interface. That's the realm of undefined behavior and system specifics. This whole IETF draft looks like a mess. They should reserve the names localhost4 and localhost6.
- pilif 9y ago
  > An interface on linux has only one ip address. absolutely not true. Neither for IPv4 nor IPv6, where it's even the default to have a multitude of addresses on an interface. > The "localhost" interface will either designate the ipv4 127.0.0.1 interface or the ipv6 ::1 interface. That's the realm of undefined behavior and system specifics. no it won't. My loopback (lo) interface has both 127.0.0.1 and ::1 as its addresses.

DominoTree 9y ago

Some genius at my company decided that ~180,000 Windows endpoints needed "localhost" removed from their hosts file, which has resulted in millions of requests per minute for localhost hitting our resolvers just to return 127.0.0.1. My guess is that it was some hack they tried to disable IPv6, but aside from the insane load it added to the DNS infrastructure, the other result is that if these machines talk to a malicious resolver, their traffic destined for the loopback interface could end up going anywhere and being captured by anyone. Great job!

JdeBP 9y ago
If the machines talk to a malicious resolving proxy DNS server, then more than traffic destined for loopback is at risk. I suspect that removing the "localhost." record was nothing to do with IPv6 and everything to do with a corporate policy to not have anything other than the Microsoft default contents in hosts files, possibly because of concerns relating to malware prevention. The problem is possibly the result of the default hosts content changing in Windows NT 6.1. * https://support.microsoft.com/en-gb/help/972034/ https://support.microsoft.com/en-gb/help/972034/ As of Windows NT 6.1, lookups of "localhost." are handled internally within (as I understand) the DNS Client, and never require inspecting a hosts file or sending a query to a DNS server. So the new default hosts file content no longer contains a "localhost." record. But use the Windows NT 6.1 or later default hosts file content on earlier versions of Windows NT, and one will see "localhost." queries being sent by the DNS Client to a server. Handling "localhost." within the DNS Client is -- reportedly -- so that the DNS Client can inspect the local machine's protocol support and only return non-empty AAAA and A resource record sets if IPv6 or IPv4 is actually enabled on the machine.

okket 9y ago

> It turned to be related to the use of "localhost" in the configuration. It resolves to ipv6 on some systems and that breaks everything because the target app is only listening to the ipv4 address. You just found a major bug in the application and should complain to the developer. Applications that do not support IPv6 are simply broken and should be avoided at all cost by now.

fundabulousrIII 9y ago

Got to chuckle about this. The new generation is fearless and naked. Break it all, admit nothing, make it 'better'. Standards, practices, tradition, culture. They mean nothing when a devops lead has commit rights to the ansible playbook and a will to deliver a fix in 5 seconds flat.