6 ms·

I want to say something else about IPv6 adoption! (Apart from how you can't necessarily unilaterally stop using IPv4, even if you can unilaterally start using I

by schoen 2y ago

I want to say something else about IPv6 adoption! (Apart from how you can't necessarily unilaterally stop using IPv4, even if you can unilaterally start using IPv6.) There is a great level of IPv6 awareness and enthusiasm in sort of technically elite spaces. Like at Ben's talk at RIPE that we're talking about, the audience was kind of laughing together at the non-IPv6 outgroup. Meanwhile, I keep helping people use Let's Encrypt over on the Let's Encrypt forum. One of the most common reasons that people are failing to get their certificates is ... misconfigured IPv6 on their servers or in their DNS! And people in that situation often (1) didn't think of that as a possibility, (2) didn't notice (!!), (3) didn't necessarily care about it. Even worse, when people are told something like "your certificate issuance isn't working because your AAAA record is pointing at your old hosting provider rather than your current hosting provider" their usual response is to delete the AAAA record rather than updating it. Other forum members may also advocate this as a solution. "Oh, just delete your AAAA record, you don't need it." I'm not making this up, I've seen these things happen about 100 times on the Let's Encrypt forum. Now, these are primarily sites in the long tail of Internet popularity, the proverbial Joe's Lamp Store rather than Gmail or Facebook or something. But the level of IPv6 awareness and interest among small-scale site operators and people doing small-scale IT stuff is very weak. And I've also seen that a surprisingly high fraction of web sites are IPv4-only rather than dual-stack. I'm working on 240/4 usability, but I certainly don't want people to delete their AAAA records. I want them to know what an AAAA record is, and want to have their AAAA records be correct. But there's quite a gap, in a way, between the elite and the non-elite tech spaces on this.

beefnugs 2y ago
wait... what about ip6 and lets encrypt? I thought I had it working with no ip6 anything or AAAA records??
- saghm 2y ago
  It sounded like they were saying that people had badly configured IPv6 and/or AAAA records that prevented stuff from working; the fact that you don't technically need it is why the "solution" of deleting AAAA records would make sense in this context. If my understanding is correct, then part of the problem of IPv6 adoption is people not really understanding what it is or why it might be useful, instead seeing it as a nuisance that they should avoid. Crucially, they're not exactly _wrong_ with this viewpoint because IPv6 doesn't really provide any benefits at the individual adoption level; the benefits are only realized if others adopt it as well, and that's not likely to be a convincing argument to someone having a concrete, immediate issue with their servers that they can address (pun intended) by not supporting IPv6.
  - schoen 2y ago
    Yes, exactly. If you do have AAAA records, Let's Encrypt (not unlike other software and services!) will prefer the AAAA records when doing domain validation for your domain. So if you have a correct A record and an incorrect AAAA record, your domain validation process will fail. You can then "fix" this by deleting the AAAA record, but it isn't a fix in the larger scheme of things, but indeed it makes plenty of users happy right away.
redox99 2y ago
I intentionally disable IPv6. It provides no advantage whatsoever (every user will be able to connect via IPv6) and it adds complexity and security risks, because you need separate firewall rules, dns records, etc for each stack. In addition, rate limiting with IPv6 is much harder (because IPs aren't scarce), so you risk DoS or other kind of abuse.
- pests 2y ago
  I think you mean every user will connect via IPv4. Users aren't getting single IPv6 addresses though. Most assign a /48 which is at least a bit better.
  - redox99 2y ago
    Yes, I meant IPv4. And rate limiting is very uncertain. If I limit per /56, users that have an entire /48 can outright bypass the rate limit. If I limit per /48, maybe I'll erroneously limit thousands of users whose ISP has all of them on the same /48.
    - pests 2y ago
      Don't you have those same issues with banning/rate limiting IV46 though?
      - ta1243 2y ago
        The difference between a /48 (like my residential handoff) and a /64 (like some people get) is the same as a difference between blocking a single /32 ipv4 address and blocking an entire /16 - which is a very large allocation If you wanted to block me, you'd have to block the whole /48. If you blocked a /48, you could be blocking 64,000 people.
        pests 2y ago
        I meant IPv4 - one IP might represent thousands of users. Looks like crote brought the issue up in a sibling thread.
viraptor 2y ago
A similar issue is documentation. I wish more companies defaulted to IPv6-and-also-IPv4 examples rather than other way around. Or at least make them same importance rather than ignore/hide the v6 part.
- jiggawatts 2y ago
  This is one of my tricks when evaluating products going through some sort of transition from a legacy to a modern standard: I mentally swap the labels[1] when reading their marketing or release notes. If it says things like "No longer crashes when using IPv6", then I flip that and read it as "No longer crashes when using IPv4". That latter statement is absolutely insane and would have you abandoning that vendor as fast as you can tear up the contract. Nobody bats an eye with the first statement! Why not!? Azure literally had this scenario, where merely enabling IPv6 on one network would crash their managed PostegreSQL service on another peered network with no recourse other than rebuilding everything form scratch without IPv6 to roll the change back! [1] Or a generic placeholder than encompasses both. E.g.: "Fix: Enabling the internet protocol no longer causes irreversible outages with your database service."
  - simiones 2y ago
    The reality today on the Internet is that IPv4 is still "the modern IP standard," and IPv6 is the next/future standard for early adopters.
    - jiggawatts 2y ago
      IPv6 is over a quarter of a century old at this point.
      - tsimionescu 2y ago
        Yes, but its install base is still less than half of that of IPv4, and plenty of things that claim they support IPv6 don't do it properly. There are countries and industries where it is really entrenched, but in the majority of the networking world, IPv4 is the 100% guaranteed to work solution that all new applications and deployments first target, and IPv6 support is an after-thought at best. Even Google, one of the champions of IPv6 adoption on the Internet, only started adding IPv6 support to GCP in the 2020s. In all public clouds, there are still numerous services that don't support it at all. And I can 100% guarantee that the next public cloud that anyone creates will start off with IPv4 support; maybe it will have IPv6 as well on day one, maybe not.
yjftsjthsd-h 2y ago
> Apart from how you can't necessarily unilaterally stop using IPv4, even if you can unilaterally start using IPv6. I thought it was possible to make a pure v6 network and use NAT64 as needed? I mean, we can argue if that counts, but I argue yes:)
- schoen 2y ago
  How about in the other direction (running public services)?
  - yjftsjthsd-h 2y ago
    Oh yeah, that definitely needs dual stack on the outside. You could do v6-only internally, but you should support everything you reasonably can for users, which 1. applies to a lot of the stack, ex. TLS and HTTP versions, and 2. can be more or less a freebie if you're using a CDN or the like (ex. cloudflare will do it for you). So yes, I was only thinking of client-side.
    - schoen 2y ago
      Thanks for the clarification. I was mainly thinking of server-side! :-) On the client side there are apparently some subtleties about the DNS interaction if you want to literally have IPv6-only on the clients (as opposed to IPv4 CGN). If the clients do anything to not trust their ISP's view of DNS (like DNSSEC enforcement, or using DoH and an outside resolver), the ISP can't fool them into thinking that they should connect to an IPv6 address for a service that, from the point of view of the Internet as a whole, is actually IPv4-only.
- ta1243 2y ago
  > I thought it was possible to make a pure v6 network and use NAT64 as needed? I mean, we can argue if that counts, but I argue yes:) Can I do "ssh 1.2.3.4" on a pure ipv6 stack yet? Last time I looked at ipv6 I couldn't even ping an ipv6 host by doing "ping 1234::5678", I had to run a separate tool (ping6) The nat64 and vice versa should have been the play from the start. Want to host a sevice? Put in ipv6 only network, host it on 1234:5678::1, have your firewall with a public ipv4 address do the port forwarding, taking your public 12.34.56.78:1234 and natting it to 1234:5678::1. Just the same as hosting your service on 192.168.0.1 and port forwarding. But 20 years of dual stack and horrible cludges around changing public IPs later and many people still prefer ipv4.
braiamp 2y ago
This is something that without some pain inflicted on host and network operators, the needle will not move. The CA/B Forum could start asking that it's a requirement for CA's to issue a certificate that the validation will try AAAA records first, and warn if they can't validate, then try A. Then later require validation through both AAAA and A records. This will prepare most of the hosts to make sure every page is accessible through both networks. Then require AAAA records only, and not bothering to check A records.
- schoen 2y ago
  Let's Encrypt does try AAAA records first and then A. That's what leads to the phenomenon that I described where many people end up deciding to delete their AAAA records. :-( I think Let's Encrypt's behavior is correct, and I don't think anyone inside Let's Encrypt or CA/B Forum would like to inflict "pain [...] on host and network operators" beyond the current practice. Let's Encrypt's current behavior could be described as cooperating with IPv6 adoption rather than compelling IPv6 adoption, which seems like a reasonable place for a certificate authority to be. There are cases where Let's Encrypt has made choices that potentially slightly reduce compatibility in order to encourage what it considers technically correct behavior. The first one that comes to mind is https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417 https://community.letsencrypt.org/t/adding-random-entries-to... but I know there are others. On the other hand, all of those decisions can be justified in terms of improving the correctness of PKI software (like not hard-coding things that should not be hard-coded). I can't think of any example that's like "we wish this other technical thing would happen outside of the PKI, so we'll try to force it along", and I don't foresee the broader community getting comfortable with that.
  - braiamp 2y ago
    The power that Lets Encrypt has over domain operators is limited. If it becomes a requirement by the CA/B Forum, then every CA would have instructions on how to make sure you have AAAA valid records and it won't be a matter that "LE doesn't work because they fail on this scenario". I'm a behavioral scientist (aka economist), and if you want the public to change behavior you have to motivate such change and here "pain" is defined as what is necessary to effect that change, by increasing the cognitive load. There was a comment below that in the LE forums instead of asking people to fix their AAAA records, they ask them to remove it. That's exactly the behavior you don't want to reward, so you must cause pain.
polski-g 2y ago
I would love to use IPv6. Verizon's page on IPv6 has literally said "check back later" for a decade. At this point there needs to be taxation penalties applied to carriers that don't support it. https://www.verizon.com/support/residential/internet/getting-started/learn-about-ipv6 https://www.verizon.com/support/residential/internet/getting...
- roywashere 2y ago
  Odido (formerly T-Mobile) in The Netherlands is rolling out a huge glass fiber network, also to my house. They have an IPv4-only internet offering w/CGNAT. I'm really looking forward to fiber instead of my current DSL but really _not_ looking forward to CGNAT and no IPv6
  - sambazi 2y ago
    > CGNAT and no IPv6 there should be an itu mandate to label this kind of service distinctly different (e.g internet lite/basic)

by schoen 2y ago

beefnugs 2y ago

wait... what about ip6 and lets encrypt? I thought I had it working with no ip6 anything or AAAA records??

saghm 2y ago
It sounded like they were saying that people had badly configured IPv6 and/or AAAA records that prevented stuff from working; the fact that you don't technically need it is why the "solution" of deleting AAAA records would make sense in this context. If my understanding is correct, then part of the problem of IPv6 adoption is people not really understanding what it is or why it might be useful, instead seeing it as a nuisance that they should avoid. Crucially, they're not exactly _wrong_ with this viewpoint because IPv6 doesn't really provide any benefits at the individual adoption level; the benefits are only realized if others adopt it as well, and that's not likely to be a convincing argument to someone having a concrete, immediate issue with their servers that they can address (pun intended) by not supporting IPv6.
- schoen 2y ago
  Yes, exactly. If you do have AAAA records, Let's Encrypt (not unlike other software and services!) will prefer the AAAA records when doing domain validation for your domain. So if you have a correct A record and an incorrect AAAA record, your domain validation process will fail. You can then "fix" this by deleting the AAAA record, but it isn't a fix in the larger scheme of things, but indeed it makes plenty of users happy right away.

redox99 2y ago

I intentionally disable IPv6. It provides no advantage whatsoever (every user will be able to connect via IPv6) and it adds complexity and security risks, because you need separate firewall rules, dns records, etc for each stack. In addition, rate limiting with IPv6 is much harder (because IPs aren't scarce), so you risk DoS or other kind of abuse.

pests 2y ago
I think you mean every user will connect via IPv4. Users aren't getting single IPv6 addresses though. Most assign a /48 which is at least a bit better.
- redox99 2y ago
  Yes, I meant IPv4. And rate limiting is very uncertain. If I limit per /56, users that have an entire /48 can outright bypass the rate limit. If I limit per /48, maybe I'll erroneously limit thousands of users whose ISP has all of them on the same /48.
  - pests 2y ago
    Don't you have those same issues with banning/rate limiting IV46 though?
    - ta1243 2y ago
      The difference between a /48 (like my residential handoff) and a /64 (like some people get) is the same as a difference between blocking a single /32 ipv4 address and blocking an entire /16 - which is a very large allocation If you wanted to block me, you'd have to block the whole /48. If you blocked a /48, you could be blocking 64,000 people.
      - pests 2y ago
        I meant IPv4 - one IP might represent thousands of users. Looks like crote brought the issue up in a sibling thread.

viraptor 2y ago

A similar issue is documentation. I wish more companies defaulted to IPv6-and-also-IPv4 examples rather than other way around. Or at least make them same importance rather than ignore/hide the v6 part.

jiggawatts 2y ago
This is one of my tricks when evaluating products going through some sort of transition from a legacy to a modern standard: I mentally swap the labels[1] when reading their marketing or release notes. If it says things like "No longer crashes when using IPv6", then I flip that and read it as "No longer crashes when using IPv4". That latter statement is absolutely insane and would have you abandoning that vendor as fast as you can tear up the contract. Nobody bats an eye with the first statement! Why not!? Azure literally had this scenario, where merely enabling IPv6 on one network would crash their managed PostegreSQL service on another peered network with no recourse other than rebuilding everything form scratch without IPv6 to roll the change back! [1] Or a generic placeholder than encompasses both. E.g.: "Fix: Enabling the internet protocol no longer causes irreversible outages with your database service."
- simiones 2y ago
  The reality today on the Internet is that IPv4 is still "the modern IP standard," and IPv6 is the next/future standard for early adopters.
  - jiggawatts 2y ago
    IPv6 is over a quarter of a century old at this point.
    - tsimionescu 2y ago
      Yes, but its install base is still less than half of that of IPv4, and plenty of things that claim they support IPv6 don't do it properly. There are countries and industries where it is really entrenched, but in the majority of the networking world, IPv4 is the 100% guaranteed to work solution that all new applications and deployments first target, and IPv6 support is an after-thought at best. Even Google, one of the champions of IPv6 adoption on the Internet, only started adding IPv6 support to GCP in the 2020s. In all public clouds, there are still numerous services that don't support it at all. And I can 100% guarantee that the next public cloud that anyone creates will start off with IPv4 support; maybe it will have IPv6 as well on day one, maybe not.

yjftsjthsd-h 2y ago

> Apart from how you can't necessarily unilaterally stop using IPv4, even if you can unilaterally start using IPv6. I thought it was possible to make a pure v6 network and use NAT64 as needed? I mean, we can argue if that counts, but I argue yes:)

schoen 2y ago
How about in the other direction (running public services)?
- yjftsjthsd-h 2y ago
  Oh yeah, that definitely needs dual stack on the outside. You could do v6-only internally, but you should support everything you reasonably can for users, which 1. applies to a lot of the stack, ex. TLS and HTTP versions, and 2. can be more or less a freebie if you're using a CDN or the like (ex. cloudflare will do it for you). So yes, I was only thinking of client-side.
  - schoen 2y ago
    Thanks for the clarification. I was mainly thinking of server-side! :-) On the client side there are apparently some subtleties about the DNS interaction if you want to literally have IPv6-only on the clients (as opposed to IPv4 CGN). If the clients do anything to not trust their ISP's view of DNS (like DNSSEC enforcement, or using DoH and an outside resolver), the ISP can't fool them into thinking that they should connect to an IPv6 address for a service that, from the point of view of the Internet as a whole, is actually IPv4-only.
ta1243 2y ago
> I thought it was possible to make a pure v6 network and use NAT64 as needed? I mean, we can argue if that counts, but I argue yes:) Can I do "ssh 1.2.3.4" on a pure ipv6 stack yet? Last time I looked at ipv6 I couldn't even ping an ipv6 host by doing "ping 1234::5678", I had to run a separate tool (ping6) The nat64 and vice versa should have been the play from the start. Want to host a sevice? Put in ipv6 only network, host it on 1234:5678::1, have your firewall with a public ipv4 address do the port forwarding, taking your public 12.34.56.78:1234 and natting it to 1234:5678::1. Just the same as hosting your service on 192.168.0.1 and port forwarding. But 20 years of dual stack and horrible cludges around changing public IPs later and many people still prefer ipv4.

braiamp 2y ago

This is something that without some pain inflicted on host and network operators, the needle will not move. The CA/B Forum could start asking that it's a requirement for CA's to issue a certificate that the validation will try AAAA records first, and warn if they can't validate, then try A. Then later require validation through both AAAA and A records. This will prepare most of the hosts to make sure every page is accessible through both networks. Then require AAAA records only, and not bothering to check A records.

schoen 2y ago
Let's Encrypt does try AAAA records first and then A. That's what leads to the phenomenon that I described where many people end up deciding to delete their AAAA records. :-( I think Let's Encrypt's behavior is correct, and I don't think anyone inside Let's Encrypt or CA/B Forum would like to inflict "pain [...] on host and network operators" beyond the current practice. Let's Encrypt's current behavior could be described as cooperating with IPv6 adoption rather than compelling IPv6 adoption, which seems like a reasonable place for a certificate authority to be. There are cases where Let's Encrypt has made choices that potentially slightly reduce compatibility in order to encourage what it considers technically correct behavior. The first one that comes to mind is https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417 https://community.letsencrypt.org/t/adding-random-entries-to... but I know there are others. On the other hand, all of those decisions can be justified in terms of improving the correctness of PKI software (like not hard-coding things that should not be hard-coded). I can't think of any example that's like "we wish this other technical thing would happen outside of the PKI, so we'll try to force it along", and I don't foresee the broader community getting comfortable with that.
- braiamp 2y ago
  The power that Lets Encrypt has over domain operators is limited. If it becomes a requirement by the CA/B Forum, then every CA would have instructions on how to make sure you have AAAA valid records and it won't be a matter that "LE doesn't work because they fail on this scenario". I'm a behavioral scientist (aka economist), and if you want the public to change behavior you have to motivate such change and here "pain" is defined as what is necessary to effect that change, by increasing the cognitive load. There was a comment below that in the LE forums instead of asking people to fix their AAAA records, they ask them to remove it. That's exactly the behavior you don't want to reward, so you must cause pain.

polski-g 2y ago

I would love to use IPv6. Verizon's page on IPv6 has literally said "check back later" for a decade. At this point there needs to be taxation penalties applied to carriers that don't support it. https://www.verizon.com/support/residential/internet/getting-started/learn-about-ipv6 https://www.verizon.com/support/residential/internet/getting...

roywashere 2y ago
Odido (formerly T-Mobile) in The Netherlands is rolling out a huge glass fiber network, also to my house. They have an IPv4-only internet offering w/CGNAT. I'm really looking forward to fiber instead of my current DSL but really _not_ looking forward to CGNAT and no IPv6
- sambazi 2y ago
  > CGNAT and no IPv6 there should be an itu mandate to label this kind of service distinctly different (e.g internet lite/basic)