9 ms·

OK so — I just migrated my personal website to use IPv6 only. It runs on AWS EC2, and my motivation was purely to save on paying the IPv4 fee that AWS is introd

OK so — I just migrated my personal website to use IPv6 only. It runs on AWS EC2, and my motivation was purely to save on paying the IPv4 fee that AWS is introducing next week. I put it behind Cloudflare which can provide an IPv4 proxy address for site visitors. But the thing is, even though IPv6 is such well established tech, it was not a great migration experience. I'm not a network engineer, and you have to figure out how to update a bunch of networking configuration to make this switch on an existing instance, with lots of potential footguns. Even after I got the IPv6 address provisioned and the virtual networking correctly configured, the connectivity was still broken until I found an obscure security group setting that was still set to allow only IPv4 traffic. A lot of basic debugging tools like curl are designed to use IPv4 by default. You have to figure out how to get your webserver (nginx in my case) to listen on the IPv6 interface, which is fiddly. And it's easy to inadvertently break ssh connectivity, which turns out to also be set up to expect IPv4 traffic only... In the end, it took about 90 minutes of fiddling, including migrating to Cloudflare, fixing DNS, etc. I mean, it's fine. I learned some things. But if we're all going to switch to an IPv6 world, it would really be nice if the systems and tooling made it slightly easier, somehow.

jeroenhd 3y ago
The thing is, with my mediocre $5 VPS, all I needed to do was copy four lines of config files (as documented by the VPS provider) and open a port in the firewall on the server itself, just like on IPv4. Amazon is horribly behind on a lot of things (just look how long it took them to get DNSSEC working, and it took them a couple of tries to not break entire domains for people using it!). They're ahead of Azure, but that's about the best you can say about that. I suppose it makes sense, IPv4 addressing is now an additional method to make money for Amazon, so why make it easy for customers to migrate?
- decasia 3y ago
  Yep, it crossed my mind that AWS has no incentive to optimize this particular process since they have planned to monetize it :/
  - Animats 3y ago
    Yeah. That's a problem. A big problem. If AT&T and AWS went IPv6 by default, most of the US would convert over quickly.
    - joecool1029 3y ago
      > If AT&T and AWS went IPv6 by default, most of the US would convert over quickly. Comcast and the mobile carriers went ipv6 years ago, that's already something like over half the US endpoints? It hasn't sped adoption (Github STILL isn't ipv6 for git). Another example, I've been trying to get Georgia Tech to fix their ipv6 linux mirrors for over half a year, they advertise they support ipv6 and publish AAAA records that don't work. http://rsync.gtlib.gatech.edu/ http://rsync.gtlib.gatech.edu/ < check it out. So yeah, even when it's deployed nobody checks to see if it's still working when ipv4 is fine.
      - alright2565 3y ago
        That GTlib issue is frustrating. I've made a ticket -- maybe they'll prioritize a member of the GT community over a random person.
        joecool1029 3y ago
        They already had one of the main Gentoo devs email them. Here's my bug report to Gentoo that prompted it: https://bugs.gentoo.org/911751 https://bugs.gentoo.org/911751 It's not the only issue it has continued to have.... https://bugs.gentoo.org/585524 https://bugs.gentoo.org/585524
        alright2565 3y ago
        > Thank you for contacting PACE. We appreciate your patience, as we are experiencing a larger volume of incidents and work-related activity. Please be mindful that we will begin our Maintenance Period starting at 6:00 AM on Tuesday, 01/23/2024, and tentatively plan to conclude by 11:59 PM on Friday, 01/26/2024. I'll continue to follow up here, but it seems like an actual human has read the message this time.
- dotancohen 3y ago
  Where is that VPS hosted?
  - jeroenhd 3y ago
    Contabo. They've supported IPv6 for a decade now based on the date on their instructions (https://contabo.com/blog/adding-ipv6-connectivity-to-your-server/ https://contabo.com/blog/adding-ipv6-connectivity-to-your-se...). Enabling IPv6 comes down to copying "gateway6: fe80::1" to your netplan file, adding the /64 listed in the control panel, and optionally adding an IPv6 DNS server. After a reboot (or sudo netplan apply) you'll have full IPv6 connectivity. Well, their default images come with a script that'll allow you to `sudo -H enable_ipv6`, but I removed their customisations so I had to do it manually. They have raised their prices since I last ordered a server there, it looks like €5,45 is the cheapest server you can get these days (includes VAT of course).
    - dotancohen 3y ago
      Terrific, thank you.
krupan 3y ago
"even though IPv6 is such well established tech" It's actually not, and I think you learned that, and why it's not, in this process. Sorry.
- newsclues 3y ago
  Introduction December 1995; 28 years ago IPv4 is 43 years old
  - kragen 3y ago
    in december 01995 zero nsps and zero isps supported ipv6. you had to be on the 6bone, which didn't even exist until march 01996. also you probably had to write your own protocol stack because there wasn't an off-the-shelf one for your platform. in december 01995 what you had was a paper spec
  - o11c 3y ago
    June 6, 2012 is a better date; that was the "World IPv6 Launch Day". But even that was negligible since nobody actually had IPv6 connectivity. Even now I still don't have a home wifi router that supports it, though at least my ISP does since last year. The real driver of IPv6 was the rise of smartphones.
    - apearson 3y ago
      What router do you use that doesn't support IPv6?
      - o11c 3y ago
        It's a tp-link Archer, and since it supports Wi-Fi 6 it must have been from 2019 or later. Hmm, I just went deep into the settings and there's an IPv6 toggle (disabled by default, and requires playing with menus to make it actually work) now. I don't remember seeing that (and I explicitly checked) when we switched the ISP-side last April. But the firmware is dated after that, so I choose to trust my memory. Actually trying to connect to ipv6 stuff besides the router itself (in fd00::/8, that's good) just gives "Destination unreachable: Beyond scope of source address" though (probably because I only have an fe80 link-local address though?). If I play with the settings some more I get a ::/64 address, but that just gives me a black hole ... wait, is that even a legitimate address? Do I need to fill in the prefix manually or something? I'll play with this some more in the morning I guess. But I certainly wouldn't expect a normal user to get IPv6 working under these circumstances if I haven't figured it out yet ...
nine_k 3y ago
In my book, 90 minutes of fiddling in a case like this is pretty fast.' Ideally it'd be 5 minutes (1 minute to locate and click a checkbox, 4 minutes to test all services from a few locations), but this is still fast. It includes zero tech support tickets, for instance.
- morelisp 3y ago
  Also my thought - only 90m for a single non-SME to migrate a full stack across two hosting providers? Sounds pretty good for something so fundamental!
wg0 3y ago
And if I'm not wrong, AWS VPCs only support IPv4.
- dangus 3y ago
  You can do dual stack with IPV6-only subnets: https://aws.amazon.com/blogs/networking-and-content-delivery/introducing-ipv6-only-subnets-and-ec2-instances/ https://aws.amazon.com/blogs/networking-and-content-delivery...
amarshall 3y ago
curl, and likely many other tools, do not use IPv4 by default, but rather quickly and seamlessly fallback (“happy eyeballs”).
doctorpangloss 3y ago
IPv4 is to AWS what iMessage is to Apple. Actually, why doesn’t Apple mandate the IPv6 transition for iOS?
- JimDabell 3y ago
  They have required that iOS apps must work in an IPv6-only environment for years.
  - p1mrx 3y ago
    That was an interesting decision. At roughly the same time, Android added 'clatd' so existing apps would just work in an IPv6-only environment via NAT64. Apple refused to implement clatd, instead forcing app developers to fix their own code. More work upfront, but now iOS apps should have an easier time transitioning their server-side components to IPv6, compared to Android apps.
kiririn 3y ago
This is an AWS/cloud issue rather than an IPv6 issue. With any decent VPS provider ipv6 just works. AWS needs better defaults
shedside 3y ago
FWIW, I'd love to see a blog post or similar that would lead me through the steps to do precisely this – including the nginx config.
vladvasiliu 3y ago
> an obscure security group setting that was still set to allow only IPv4 traffic Which setting was that? I've had 0 issues with IPv6 on EC2 for a while now. Not all services support v6, though, but that's a separate issue.
miohtama 3y ago
> I found an obscure security group setting that was still set to allow only IPv4 traffic This has more to do with AWS, less with IPv6.
xinayder 3y ago
I have to appreciate the Brazilian effort for promoting IPv6. I attended a talk on a university about 5 years ago where members of the Brazilian Internet Steering Committee (CGI.br) talked about IPv6 and why it's the future. It was very informative, and I also found out they offer free courses about IPv6 networking, with certificates included. I think CZ.NIC offers something similar, but they should be promoted more often so sysadmins can prepare themselves for the eventual shutdown of public IPv4 addresses.
lazyeye 3y ago
I think Tailscale does IPv6 out of the box so installing a Tailscale client might be an option for ssh etc.
eirikbakke 3y ago
For some AWS services, such as Elastic Beanstalk, there appears to be no way to avoid consuming a public IPv4 address on the EC2 instances, except by introducing a NAT gateway. (And "NAT gateways are an extremely cost prohibitive device in AWS, costing at least as much per day as medium sized EC2 instances!") https://github.com/aws/elastic-beanstalk-roadmap/issues/208 https://github.com/aws/elastic-beanstalk-roadmap/issues/208
dotancohen 3y ago
I've got to make the same migration this coming weekend. Any information or links would be appreciated! I'm sure that many HNers would appreciate the information here, but if you prefer my Gmail username is the same as my HN username. Thank you!
- decasia 3y ago
  One of the comments in the thread links to a useful writeup for the AWS side: https://www.knowmatic.app/how-to-migrate-aws-ec2-instance-from-ipv4-to-ipv6 https://www.knowmatic.app/how-to-migrate-aws-ec2-instance-fr... I didn't find such great docs on the Cloudflare part, but some of these were useful: https://www.reddit.com/r/selfhosted/comments/14d16lt/if_i_only_have_an_ipv6_and_a_domain_can_i_use/ https://www.reddit.com/r/selfhosted/comments/14d16lt/if_i_on... https://community.cloudflare.com/t/the-page-isn-t-redirecting-properly/151245/4?__cf_chl_rt_tk=1Z2j5SMzeI89NmLkdESh9rIpYu0OjHOSLc7sdSYz_3E-1705804219-0-gaNycGzNEGU https://community.cloudflare.com/t/the-page-isn-t-redirectin... https://stackoverflow.com/questions/4616521/nginx-configuration-leads-to-endless-redirect-loop https://stackoverflow.com/questions/4616521/nginx-configurat... https://developers.cloudflare.com/support/troubleshooting/cloudflare-errors/troubleshooting-cloudflare-5xx-errors/#error-522-connection-timed-out https://developers.cloudflare.com/support/troubleshooting/cl...
geek_at 3y ago
people will do anything except for renting a 5€/mo vps
- decasia 3y ago
  I mean, I used to rent the $5/month VPS, and it worked well. In fact, it had much nicer developer experience than EC2 for many reasons. But my provider (digitalocean) raised the price, and I don't really need a lot of CPU resources to serve mostly static files plus a couple of tiny server processes that get very very low traffic. So I downsized to a tiny EC2 instance. I would go back to the 5/mo VPS if it became clearly better than the alternative. I don't really need an IPv4 for what I use it for, though, as long as someone else wants to provide an inexpensive (or free tier) proxy service.
jiannengli 3y ago
Reading this reminded me of my recent pains of migrating my EC2 to IPv6. I ended up writing a guide on it. Take a look if it helps! https://news.ycombinator.com/item?id=39111148 https://news.ycombinator.com/item?id=39111148

jeroenhd 3y ago

The thing is, with my mediocre $5 VPS, all I needed to do was copy four lines of config files (as documented by the VPS provider) and open a port in the firewall on the server itself, just like on IPv4. Amazon is horribly behind on a lot of things (just look how long it took them to get DNSSEC working, and it took them a couple of tries to not break entire domains for people using it!). They're ahead of Azure, but that's about the best you can say about that. I suppose it makes sense, IPv4 addressing is now an additional method to make money for Amazon, so why make it easy for customers to migrate?

decasia 3y ago
Yep, it crossed my mind that AWS has no incentive to optimize this particular process since they have planned to monetize it :/
- Animats 3y ago
  Yeah. That's a problem. A big problem. If AT&T and AWS went IPv6 by default, most of the US would convert over quickly.
  - joecool1029 3y ago
    > If AT&T and AWS went IPv6 by default, most of the US would convert over quickly. Comcast and the mobile carriers went ipv6 years ago, that's already something like over half the US endpoints? It hasn't sped adoption (Github STILL isn't ipv6 for git). Another example, I've been trying to get Georgia Tech to fix their ipv6 linux mirrors for over half a year, they advertise they support ipv6 and publish AAAA records that don't work. http://rsync.gtlib.gatech.edu/ http://rsync.gtlib.gatech.edu/ < check it out. So yeah, even when it's deployed nobody checks to see if it's still working when ipv4 is fine.
    - alright2565 3y ago
      That GTlib issue is frustrating. I've made a ticket -- maybe they'll prioritize a member of the GT community over a random person.
      - joecool1029 3y ago
        They already had one of the main Gentoo devs email them. Here's my bug report to Gentoo that prompted it: https://bugs.gentoo.org/911751 https://bugs.gentoo.org/911751 It's not the only issue it has continued to have.... https://bugs.gentoo.org/585524 https://bugs.gentoo.org/585524
        alright2565 3y ago
        > Thank you for contacting PACE. We appreciate your patience, as we are experiencing a larger volume of incidents and work-related activity. Please be mindful that we will begin our Maintenance Period starting at 6:00 AM on Tuesday, 01/23/2024, and tentatively plan to conclude by 11:59 PM on Friday, 01/26/2024. I'll continue to follow up here, but it seems like an actual human has read the message this time.
dotancohen 3y ago
Where is that VPS hosted?
- jeroenhd 3y ago
  Contabo. They've supported IPv6 for a decade now based on the date on their instructions (https://contabo.com/blog/adding-ipv6-connectivity-to-your-server/ https://contabo.com/blog/adding-ipv6-connectivity-to-your-se...). Enabling IPv6 comes down to copying "gateway6: fe80::1" to your netplan file, adding the /64 listed in the control panel, and optionally adding an IPv6 DNS server. After a reboot (or sudo netplan apply) you'll have full IPv6 connectivity. Well, their default images come with a script that'll allow you to `sudo -H enable_ipv6`, but I removed their customisations so I had to do it manually. They have raised their prices since I last ordered a server there, it looks like €5,45 is the cheapest server you can get these days (includes VAT of course).
  - dotancohen 3y ago
    Terrific, thank you.

krupan 3y ago

"even though IPv6 is such well established tech" It's actually not, and I think you learned that, and why it's not, in this process. Sorry.

newsclues 3y ago
Introduction December 1995; 28 years ago IPv4 is 43 years old
- kragen 3y ago
  in december 01995 zero nsps and zero isps supported ipv6. you had to be on the 6bone, which didn't even exist until march 01996. also you probably had to write your own protocol stack because there wasn't an off-the-shelf one for your platform. in december 01995 what you had was a paper spec
- o11c 3y ago
  June 6, 2012 is a better date; that was the "World IPv6 Launch Day". But even that was negligible since nobody actually had IPv6 connectivity. Even now I still don't have a home wifi router that supports it, though at least my ISP does since last year. The real driver of IPv6 was the rise of smartphones.
  - apearson 3y ago
    What router do you use that doesn't support IPv6?
    - o11c 3y ago
      It's a tp-link Archer, and since it supports Wi-Fi 6 it must have been from 2019 or later. Hmm, I just went deep into the settings and there's an IPv6 toggle (disabled by default, and requires playing with menus to make it actually work) now. I don't remember seeing that (and I explicitly checked) when we switched the ISP-side last April. But the firmware is dated after that, so I choose to trust my memory. Actually trying to connect to ipv6 stuff besides the router itself (in fd00::/8, that's good) just gives "Destination unreachable: Beyond scope of source address" though (probably because I only have an fe80 link-local address though?). If I play with the settings some more I get a ::/64 address, but that just gives me a black hole ... wait, is that even a legitimate address? Do I need to fill in the prefix manually or something? I'll play with this some more in the morning I guess. But I certainly wouldn't expect a normal user to get IPv6 working under these circumstances if I haven't figured it out yet ...

nine_k 3y ago

In my book, 90 minutes of fiddling in a case like this is pretty fast.' Ideally it'd be 5 minutes (1 minute to locate and click a checkbox, 4 minutes to test all services from a few locations), but this is still fast. It includes zero tech support tickets, for instance.

morelisp 3y ago
Also my thought - only 90m for a single non-SME to migrate a full stack across two hosting providers? Sounds pretty good for something so fundamental!

wg0 3y ago

And if I'm not wrong, AWS VPCs only support IPv4.

dangus 3y ago
You can do dual stack with IPV6-only subnets: https://aws.amazon.com/blogs/networking-and-content-delivery/introducing-ipv6-only-subnets-and-ec2-instances/ https://aws.amazon.com/blogs/networking-and-content-delivery...

amarshall 3y ago

curl, and likely many other tools, do not use IPv4 by default, but rather quickly and seamlessly fallback (“happy eyeballs”).

doctorpangloss 3y ago

IPv4 is to AWS what iMessage is to Apple. Actually, why doesn’t Apple mandate the IPv6 transition for iOS?

JimDabell 3y ago
They have required that iOS apps must work in an IPv6-only environment for years.
- p1mrx 3y ago
  That was an interesting decision. At roughly the same time, Android added 'clatd' so existing apps would just work in an IPv6-only environment via NAT64. Apple refused to implement clatd, instead forcing app developers to fix their own code. More work upfront, but now iOS apps should have an easier time transitioning their server-side components to IPv6, compared to Android apps.

kiririn 3y ago

This is an AWS/cloud issue rather than an IPv6 issue. With any decent VPS provider ipv6 just works. AWS needs better defaults

shedside 3y ago

FWIW, I'd love to see a blog post or similar that would lead me through the steps to do precisely this – including the nginx config.

vladvasiliu 3y ago

> an obscure security group setting that was still set to allow only IPv4 traffic Which setting was that? I've had 0 issues with IPv6 on EC2 for a while now. Not all services support v6, though, but that's a separate issue.

miohtama 3y ago

> I found an obscure security group setting that was still set to allow only IPv4 traffic This has more to do with AWS, less with IPv6.

xinayder 3y ago

I have to appreciate the Brazilian effort for promoting IPv6. I attended a talk on a university about 5 years ago where members of the Brazilian Internet Steering Committee (CGI.br) talked about IPv6 and why it's the future. It was very informative, and I also found out they offer free courses about IPv6 networking, with certificates included. I think CZ.NIC offers something similar, but they should be promoted more often so sysadmins can prepare themselves for the eventual shutdown of public IPv4 addresses.

lazyeye 3y ago

I think Tailscale does IPv6 out of the box so installing a Tailscale client might be an option for ssh etc.

eirikbakke 3y ago

For some AWS services, such as Elastic Beanstalk, there appears to be no way to avoid consuming a public IPv4 address on the EC2 instances, except by introducing a NAT gateway. (And "NAT gateways are an extremely cost prohibitive device in AWS, costing at least as much per day as medium sized EC2 instances!") https://github.com/aws/elastic-beanstalk-roadmap/issues/208 https://github.com/aws/elastic-beanstalk-roadmap/issues/208

dotancohen 3y ago

I've got to make the same migration this coming weekend. Any information or links would be appreciated! I'm sure that many HNers would appreciate the information here, but if you prefer my Gmail username is the same as my HN username. Thank you!

decasia 3y ago
One of the comments in the thread links to a useful writeup for the AWS side: https://www.knowmatic.app/how-to-migrate-aws-ec2-instance-from-ipv4-to-ipv6 https://www.knowmatic.app/how-to-migrate-aws-ec2-instance-fr... I didn't find such great docs on the Cloudflare part, but some of these were useful: https://www.reddit.com/r/selfhosted/comments/14d16lt/if_i_only_have_an_ipv6_and_a_domain_can_i_use/ https://www.reddit.com/r/selfhosted/comments/14d16lt/if_i_on... https://community.cloudflare.com/t/the-page-isn-t-redirecting-properly/151245/4?__cf_chl_rt_tk=1Z2j5SMzeI89NmLkdESh9rIpYu0OjHOSLc7sdSYz_3E-1705804219-0-gaNycGzNEGU https://community.cloudflare.com/t/the-page-isn-t-redirectin... https://stackoverflow.com/questions/4616521/nginx-configuration-leads-to-endless-redirect-loop https://stackoverflow.com/questions/4616521/nginx-configurat... https://developers.cloudflare.com/support/troubleshooting/cloudflare-errors/troubleshooting-cloudflare-5xx-errors/#error-522-connection-timed-out https://developers.cloudflare.com/support/troubleshooting/cl...

geek_at 3y ago

people will do anything except for renting a 5€/mo vps

decasia 3y ago
I mean, I used to rent the $5/month VPS, and it worked well. In fact, it had much nicer developer experience than EC2 for many reasons. But my provider (digitalocean) raised the price, and I don't really need a lot of CPU resources to serve mostly static files plus a couple of tiny server processes that get very very low traffic. So I downsized to a tiny EC2 instance. I would go back to the 5/mo VPS if it became clearly better than the alternative. I don't really need an IPv4 for what I use it for, though, as long as someone else wants to provide an inexpensive (or free tier) proxy service.

jiannengli 3y ago

Reading this reminded me of my recent pains of migrating my EC2 to IPv6. I ended up writing a guide on it. Take a look if it helps! https://news.ycombinator.com/item?id=39111148 https://news.ycombinator.com/item?id=39111148