24 ms·

AWS Customers Cannot Escape IPv4

306 points by apparentorder 3y ago

pnpnp 3y ago
AWS has a way to go with their IPV6 services, but IPV6-only is very doable right now in EC2. I have a WireGuard server that was dual-stack. I turned off IPV4 to see what would happen, and it kept chugging along very nicely.
- mike_d 3y ago
  ...if you are running one service and not really using AWS in any meaningful way other than as a VPS provider.
  - pnpnp 3y ago
    You’re not wrong, that’s why I prefaced it saying their services are lacking. I was just pleasantly surprised that EC2 supports it.
mjevans 3y ago
It would really help if there were real ISP competition in the USA. There's only one actually broadband ISP provider where I rent, which is in the suburbs near Seattle. It's NOT a rural area by any definition, and yet Comcast is my only option. Their price and service reflect that reality...
- gnopgnip 3y ago
  Comcast supports ipv6 though, in 2016 they hit 98% ipv6 support.
  - 1ncorrect 3y ago
    And Comcast were the biggest proponents of building IPv6 support into the DOCSIS specs, because they exhausted 10/8 in the management network for their modem fleet.
- chungy 3y ago
  > yet Comcast is my only option. Thank endless lobbying that make legally mandated monopolies a thing in this arena. They did it with phone companies too. It's not "just Comcast" by happenstance. It's "just Comcast" by legal regulation.
  - paulddraper 3y ago
    AFAIK, most areas in the U.S. have wireless ISPs available.
- mike_d 3y ago
  IPv6 adoption is only going to further the consolidation of customers onto the big monopoly providers. They will be the only ones who can afford to add the dedicated network engineering staff to make it work reliably. Most people don't realize there are two IPv6 internets right now, the Cogent side and the Hurricane Electric side. Both are equally sized and refuse to connect to each other, so you need to know that and either buy transit from both or buy transit from a network that buys from both. At least one major provider I know of is still running v6 over tunnels. In many places your v6 traffic is taking suboptimal routes, whereas an enterprise network may have v4 connectivity at each datacenter, v6 all gets sent to that one box under Dave's desk. But we continue to measure v6 adoption at places like Google and Cloudflare where dedicated teams make sure packets arrive and pat ourselves on the back.
  - greyface- 3y ago
    > two IPv6 internets right now, the Cogent side and the Hurricane Electric side Cogent engages in peering spats on IPv4 too; this dynamic is not new with or unique to IPv6, or limited to Cogent/HE. The lesson here is to not go singlehomed under Cogent, not to reject IPv6.
    - mike_d 3y ago
      The takeaway here was that you need to be aware of it to make IPv6 work. Again, your average operator of a small regional WISP may try to deploy v6 because they lack v4 space and face customer complaints because they single home behind HE and can't reach the other half of the internet. Currently v6 is like connecting to the late 90s internet. It dosen't work as well as people think.
      - greyface- 3y ago
        You need to be aware of it to make v4 work, too. I once saw an issue where a network similar to your example WISP (singlehomed behind HE) was unable to reach a network that was advertising its v4 prefixes NO_EXPORT to HE at a distant IX. Adding a second upstream would have fixed (and indeed, did later fix) the issue. It's been advisable since the early days to have multiple upstreams because of routing table holes created by situations like this. Again, not unique to v6, and the HE/Cogent schism is not the only one (though admittedly it is likely the largest).
- huijzer 3y ago
  > Comcast is my only option Maybe Starlink is another option. It has some drawbacks like reduced performance during heavy rain, but I've seen some positive reviews as well.
  - ninkendo 3y ago
    Starlink doesn’t have capacity for a large urban area. There’s only so many satellites available per square mile, and if you have a sizable fraction of a major city using starlink, they’d all be bottlenecked on a handful of satellites at best, even with the huge number of satellites. Starlink only makes sense for low-geographic-density deployments, where the number of customers on the same satellite is (relatively) low.
- luhn 3y ago
  Consumers don't know or care about IPv6, so competition won't incentivize ISPs to implement it. Funny that you complain about Comcast, when I was with Comcast I actually had native IPv6. With my current provider I only have IPv6 through NAT46.
fulafel 3y ago
It seems obviously against AWS incentives to offer working v6 - all their influencing tools ("well architected" criteria, certificates) strongly herd you towards building mazes of ambigously addressed 10.x RFC1918 networks, and not internet style architectures with end-to-end addressing. In the world of their recommendations, even the concept of a "public ip address" is a red flag, and AWS even recommends (for an added cost of course) tooling to flag and "mitigate" them. These provide a strong lock-in effect when customers spend effort to build the complex infrastructure for them in the name of security, even though in reality they hurt security through unnecessary complexity, addressing ambiguity, etc.
- beoberha 3y ago
  I work in Azure, but my experience is that customers want this - and for good reason. Customers want their own private network to prevent intrusions and exfiltrations, just on machines they don’t own. Or even better, put the nice fancy batteries included PaaS services in these networks too.
  - jojobas 3y ago
    Right, cause customers are too stupid to manage their own IPv6 firewalls, it's for their own good! /s
    - rad_gruchalski 3y ago
      I’m pretty confident that this statement is mostly true without sarcasm, and that you are in the minority.
      - pantalaimon 3y ago
        How is managing a NAT easier than managing a firewall?
        ninkendo 3y ago
        I’m one typo away from accidentally allowing IPv6 access to every machine in my network with my pf config on my home router. (I know this because I’ve done it one time, and didn’t notice for about a week.) There is no such typo i could make with my single shared public ipv4 address because it’s just one address. Saying “allow” by accident isn’t enough, I’d have to somehow accidentally configure the particular ingress port to NAT to a particular internal machine, and even then it would only affect that machine and no other. (Full disclosure, i actually like IPv6 and am in full favor of everything moving to it. This is in spite of the above, but i at least recognize that the above is the case.)
donor20 3y ago
IPv6 is such a massive headache it’s kind of mind boggling. I used to be super enthused - but it is absolutely less useful and more annoying than it’s worth.
- 9dev 3y ago
  It’s really not a headache in and of itself. The technology is beautiful and enables lots of cool things - just look at all the awesome stuff the fly.io team makes it do. The headache are vendors that still refuse to properly implement IPv6, in 2023.
- jeroenhd 3y ago
  This is one example where it's clear IPv6 isn't the problem, actually. A lot of problems with AWS would disappear if they would just support IPv6 like your average budget ISP does. IPv6 just works. Amazon, Github, and Azure don't. That's not really a problem in most cases (very few people go IPv6 only because it's just not necessary with CGNAT, and even then network translation tricks can put up IPv6<->IPv4 bridges easily). In Amazon's case, they don't even need to bother setting up a real network, they could abuse an fd00::/8 network to mimic their 10.0.0.0/8 network if they wanted to. Amazon is terrible at implementing modern standards. Just look at how long it took them to support DNSSEC on their domains, and even that didn't exactly roll out great the first time.
  - donor20 3y ago
    WAN failover is not fun with ipv6 - npt doesn’t solve things because prefix lengths are variable. You end up back with NAT with basic networks again but with ridiculously large address space. Firewall rules a trickier- you need to both let ICMP through but be careful because some can drive network reconfig. DHCP is second class, and the network can do weird things when port isolations are on. The number of (rotating) ipv6 addresses per host gets silly and makes logging / accountability/ trace back systems more convoluted. Then you’ve got neighbor discovery threats, header extension manipulation stuff. And if multicast isn’t working because of a security configuration that breaks assumptions but you also have multicast amplification stuff. There is a reason well resourced companies like google cloud have been slow w IPv6 - and it can be even more hair pulling in smaller settings.
solatic 3y ago
Half the reason AWS has leading IPv6 support in the first place is due to mandates from the US government to start migrating. Author is correct that, from a cost perspective, the new costs are immaterial to large customers, but I wouldn't discount the power of policy mandates from the largest customers, where the threat of building an in-house alternative to comply with policy might be sufficient to force AWS to finally prioritize support.
- apparentorder 3y ago
  Indeed. I really don't like the thought, but I more and more believe that there is no other way to incentivize IPv6 at the "server side". The client (end user) side seems to do well, considering that Google reports IPv6 end user traffic of almost 50% these days.
  - biohax2015 3y ago
    > The client (end user) side seems to do well, considering that Google reports IPv6 end user traffic of almost 50% these days. That's because all mobile data connections are on IPv6
hkt 3y ago
There needs to be a body of law relating to technical matters like this (and interoperability etc) that is adjacent to competition law. Some things we just need everyone to be on the same page about. It is manifestly the case that ipv6 is never going to be that, because the incentives to invest simply don't exist for companies like AWS. This distorts the market in eyeball networks and hosting - the former are under little pressure to offer v6, and new entrants to the latter can only offer v6. Competition law in the EU works (I think?) on the principles of consumer benefit and market fairness. On that basis, I'm left wondering why this has never been pursued by the EU's competition authorities.
- robertlagrant 3y ago
  Lots of AWS customers want IPv4 because that's what they know, and that's what they benefit from. To me, the question is: what stops me today from spinning up an IPv6-only website and having 99% of the world's browsers use it? If the answer is "nothing", then AWS shouldn't be forced to offer IPv6 (or only IPv6) - IPv4 is just part of what they offer customers. If the answer is "these 7 things" then those 7 things need to be fixed[0] before we pay civil servants to try and force companies to do things that they barely understand. [0] E.g. in the UK, it's some of the big ISPs that don't do IPv6, so there's no point forcing someone way upstream (and way more optional) in the process to do something https://www.ispreview.co.uk/index.php/2021/11/update-on-ipv6-plans-for-virgin-media-talktalk-plusnet-and-vodafone.html https://www.ispreview.co.uk/index.php/2021/11/update-on-ipv6...
- jeroenhd 3y ago
  The EU did have a mandate for government services to use IPv6, but the programme it was part of got replaced by another that didn't include IPv6. The European Commission did advocate for IPv6 use, but, the EU being the EU, motivated their recommendation by complaining that law enforcement had issues tracking down people behind CGNAT, and made clear that they wanted every IP address to point to a specific person for law enforcement reasons. So, yeah, I don't think we should let the EU deal with the specifics of network infrastructure just yet. I think it's hard to make an economic argument for IPv6. Yes, it's obviously a superior technology, but ISPs can CGNAT for cheap, consumers can still access every server, and the €40 per year a business needs to pay for an IPv4 address isn't exactly breaking the bank either. Perhaps the EU should force the issue, but I think countries like Lithuania ,where there is practically no IPv6 available (0.58%, according to https://stats.labs.apnic.net/ipv6-zoom https://stats.labs.apnic.net/ipv6-zoom, but who knows how accurate that is), will protest any mandate that will force their ISPs to buy new networking equipment.
peter_retief 3y ago
I am struggling to move to IPv6, from ISP blocking to unhelpful cloud services. Who is a good provider of ipv6?
- supriyo-biswas 3y ago
  At the ISP level, you have better chances of having IPv6 connectivity if you’re based out of a developing country, whose ISPs don’t have the means to pay for too many IPv4 ranges. For servers, there are plenty; AWS Lightsail, Hetzner and Vultr both provide IPv6 out of the box. If you don’t have an ISP which provides IPv6, you could use a server and set up a wireguard tunnel for IPv6 connectivity.
  - tyingq 3y ago
    >AWS Lightsail Read the burst/throttling page carefully before you choose this product. https://lightsail.aws.amazon.com/ls/docs/en_us/articles/amazon-lightsail-viewing-instance-burst-capacity https://lightsail.aws.amazon.com/ls/docs/en_us/articles/amaz...
  - ninkendo 3y ago
    > you have better chances of having IPv6 connectivity if you’re based out of a developing country, whose ISPs don’t have the means to pay for too many IPv4 ranges I don’t know that this is true based on google’s IPv6 adoption data: https://www.google.com/intl/en/ipv6/statistics.html#tab=per-country-ipv6-adoption https://www.google.com/intl/en/ipv6/statistics.html#tab=per-... It seems like developing nations have the worst IPv6 adoption, at least by a cursory look of how there’s very little green in Africa, for instance. I think ISP’s in countries with small IPv4 blocks just use CGNAT.
    - supriyo-biswas 3y ago
      Thank you for the link. I was taking about India for the most part, but it seems France, Germany, India, and Saudi Arabia are the leaders in IPv6 deployment, a weird mix of countries that I honestly didn’t expect.
    - 3y ago
alias_neo 3y ago
Off topic: Does anyone know if this page is generated from a Static-Site generator starting from Markdown? I currently use Hugo and my blog is in Markdown in git, but the theme is pretty heavy-weight, and I like this look of the page in OP; Looking at the source, it's so minimal!
- apparentorder 3y ago
  Yes, it's Markdown and I use https://jekyllrb.com https://jekyllrb.com with the theme "jekyll-theme-hacker" to generate the site. I quite like how simple it is.
  - alias_neo 3y ago
    Amazing, thank you!
iopq 3y ago
I migrate my proxy's IPv4 address from time to time to avoid blocks This is not so easy to do with a IPv6 address, AWS tends to want to keep it the same
- kaliszad 3y ago
  If you get a /48 you can probably evade the problem by assigning a /64 for your proxy at a time. You will have another ~65,500 such blocks for use. Yes, some might just block the /56 (you would still have another ~250 chances) or /48 but nothing is perfect.
  - iopq 3y ago
    but how do I change the IPv6 address in AWS? I can just assign a new Static IP for IPv4
    - xnyanta 3y ago
      Same thing for IPv6, but instead of assigning a new singular IPv6, you assign a new subnet with millions of IPv6 addresses.
supriyo-biswas 3y ago
IMO services like Lambda and S3 not supporting IPv6 is the real issue, and AWS shouldn’t have made the pricing change without first making them dual-stacked, accessible over IPv6. (Technically S3 does have a separate dual stack endpoint, however it doesn’t really help as I have to change application configuration anyway to deal with this change.)
Hikikomori 3y ago
There's one viable solution to be able to run IPv6 only subnets in AWS, their (or your own) NAT gateways support v6->v4 NAT. So it allows you to create large IPv6 only subnets for your compute services (ec2, ecs, k8s, elb, all supports that), allowing your containers to scale without worrying about IP addresses. Then you use dual stack subnets for other AWS services that may not support IPv6 and your compute services can access them through the NAT gateway.
- apparentorder 3y ago
  ECS, ELB and most other services do not support IPv6-only subnets, as mentioned in the article. ECS does support dual-stack IPv6, but most other services do not support IPv6 at all.
wheybags 3y ago
neveragain.de is... a strange choice of domain for a tech blog. It sounds like a holocaust memorial site.
- apparentorder 3y ago
  This domain is very, very old and at that time, the phrase wasn't usually associated like it is today. Not sure what to do about that.
kennu 3y ago
Worst for me is CloudFront not supporting IPv6 for custom origins. If you happen to run a lot of separate Fargate containers as origins, you have to enable public IPv4 addresses for them, and that will soon double the price of small confainers. Amazon needs to make their infrastructure actually support IPv6 before starting to charge extra for legacy IPv4 usage.
- paraxial_0 3y ago
  Yeah, somehow this one hurts the most. I know Amazon has built a giant beast here and rolling out IPv6 across all of their million services is a huge undertaking, but I can't see any reason for CloudFront not supporting IPv6 origins, like yesterday. It doesn't seem like it should be that hard relatively speaking, and would provide a good tool for working around other limitations. Honestly I've always felt that Amazon's decision-making ultimately had the best interests of their customers in mind, until now. I think this is a bad sign for things to come.
dopylitty 3y ago
As an AWS customer I want to escape IP entirely. It's a waste of time managing these complex networking systems with their archaic protocols (IP, BGP, DNS, etc) Just let me strongly associate identities with my workloads and apply policy indicating which workloads should be able to send data with which other workloads. How data gets from one workload to another should not even be my concern, just make it happen.
- drpossum 3y ago
  While your sentiment is valid, this is the type of argument people make on low code solutions. Which has never worked in reality and never will. There's just too much nuance and detail that needs to be considered when you have to do and optimize real workloads.
  - Spivak 3y ago
    Of course but it pushes the abstraction forward to what we really want. It turns out that most applications don't actually want to mess with IP except as an implementation detail and optimization. Which is why most of the time you don't and you just get some application layer HTTP payload, RPC thing or WSGIish type call on the incoming side and let the ops people deal with the networking bits and it works well enough that people mostly don't complain. The request is that more outgoing services adopt this model where you had it off to your application server and it does the work and just gives you the data you want back. I don't think it's that crazy, it's just formally standardizing where we're already going.
- pragma_x 3y ago
  I agree completely. FWIW, this sentiment is why we're seeing a lot of cloud "platforms" crop up that do exactly what you're talking about. Rather than get mired in the component-zoo of virtualized datacenter (read: pretend) networking, just abstract all of it away.
- flerchin 3y ago
  IME "big picture folks" suddenly care about the details when their big picture fails to solve for the details.
Bluecobra 3y ago
> There is no concept of private addresses in IPv6, which means farewell to the Managed NAT Gateway and its magnificent pricing. Maybe not in AWS, but there are Unique Local IPv6 addresses in fc00::/7 and NAT66 if you really love NAT!
- apparentorder 3y ago
  Yeah, I was referring to AWS; I should have made that clear. ULA is frequently discouraged though, and NAT66, well ... just no. I just recently heard that MS apparently has built everything IPv6 on Azure around NAT. This is so weird.
  - klysm 3y ago
    It’s the most Microsoft thing possible. That way they can charge you! How else are they supposed to rent seek?
digitalsushi 3y ago
My ISP, Fidium, does not have the word IPv6 on its entire website. And definitely has no support of it on my WAN. They should. I want to connect to IPv6 services using their connection. I actually keep a cheap Comcast connection as a second WAN just to get IPv6 enabled on my home network. (And also because I live in the woods in New Hampshire and having two ISPs means I have fairly ok uptime)
mabbo 3y ago
Many years ago when I was a junior dev at Amazon, there was a massive project internally to split up every internal system into regional versions with limited gateways allowing calls between regions. The reason? We had run out of internal IPv4 addresses. The Principal PM in charge of the "regionalization" effort was asked in a Q&A "why didn't we just switch to IPv6?". Her answer was something along the lines of "The number of internal networking devices we currently have that cannot support IPv6 is so large that to replace them we would have needed to buy nearly the entire world's yearly output of those devices, and then install them all."[0] It's easy to presume malicious intent on the IPv4 front from Amazon, but with so many AWS systems being on the scale they are at, I find it easy to believe that replacing all of the old network hardware may just be a project too large to do on a short timescale. [0] - At least, that's my memory of it. I'm sure that's not an entirely accurate quotation.
- tinix 3y ago
  this doesn't forgo v6 phase-in though, can't kick that can down the road forever. surely they started the process... right? i cannot imagine AWS just sticking head in the ground and ignoring this...
  - mtnGoat 3y ago
    Yes they are working on it. A number of services already support v6, more to come.
    - AdamJacobMuller 3y ago
      1 is a number. 0 is also a number.
  - Twirrim 3y ago
    No one is ignoring it, and the US Government has done everyone another favour on this score. Years ago in the late Bush / early Obama administration, NIST required that all federal government agencies have IPv6 at the border. Federal government money is not to be sniffed at, and that had the effect of forcing a number of vendors to add IPv6 support. A few years after that, it became that the federal agencies needed to have dual-stack IPv4/IPv6. About 18 months ago, the requirement came that federal agencies are required to be IPv6 Only, dropping the dual stack. IIRC they have until 2025 to do that. This has the neat effect of forcing all vendors to make IPv6 a first class citizen. The extra little fun from this is that it applies to the military JWCC contract that all the major clouds have been trying to land. The timescales of JWCC meant that initial offerings are pretty bare, but that won't be allowed to last.
welder 3y ago
I use DigitalOcean... almost all their products support IPv6. Only floating IPs are IPv4, but can work around that by not destroying droplets so the IPv6 address doesn't change.
- kevincox 3y ago
  One annoyance I have is that IPv6 is disabled on VMs (droplets) by default. So every time you upgrade your Kubernetes cluster and it recreates every node they all have no IPv6 again with no way to change this. In the end I couldn't be bothered and just stopped rebooting every node after every update.
mkl95 3y ago
> The first pattern is having multiple Load Balancers (per VPC); this is often the result of using several readily available Cloudformation templates / Terraform modules, or somehow using Kubernetes ingress controllers that create a Load Balancer for every service. This is fixed by not doing that! A single Load Balancer can handle many URLs and services. This is the definition of cloud bloat. The fact there are tons of systems abusing that kind of architecture probably justifies charging for IPv4.
- Spivak 3y ago
  I think it's an impedance mismatch between the feature people want -- "logical load balancers" and the feature they're offered "physical load balancers." How nice it would be if you could just create a bunch of load balancers and all that actually meant was that it was just adding config profiles to a single physical load balancer and kept them truly isolated? Right now it's really annoying because load balancer config is global state and everyone has to either be kind neighbors when adding themselves to it or manage them top-down.
- doublet00th 3y ago
  I do believe the AWS Load Balancer Controller on Kubernetes allows for sharing a single "physical" load balancer. You have to set a load-balancer-name annotation https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.4/guide/ingress/annotations/#load-balancer-name https://kubernetes-sigs.github.io/aws-load-balancer-controll... to tie everything together to one load balancer. There is a downside where you have to have a few other annotations be the same value across your ingresses, but once you work around that, you're good to go.
  - renewiltord 3y ago
    I couldn't find it easily specified in docs. This is a common use-case and part of why I avoid EKS for HTTP workloads is that I have tiny services I want to just make available and I don't want to have another full ELB sitting there. It isn't a cost thing primarily. It's that now I have another significant resource. I want all of these things on a misc LB.
josephcsible 3y ago
I feel like there's now a perverse incentive here for Amazon to drag their feet at implementing full feature parity with IPv6. I wish these new charges only applied for IPv4 addresses used with services that already do have that.
tormeh 3y ago
It's not that people dislike IPv6 or like IPv4, it's that network people are comfortable with IPv4 and all the extra tech surrounding it. They know it works, so there's no technological risk. There's nothing new to learn. It's cheap. There's nothing your average business wants to do that can't be done on IPv4 that can on IPv6. The ROI of just paying for IPv4 addresses and associated tech/services is undeniable.
- talent_deprived 3y ago
  > It's not that people dislike IPv6 Yes, for some who understand and have tested it, we do not like or want IPv6, it has no privacy when the device's IP is public on the Internet 24/7. No privacy extensions fix this. Test it, it's not difficult, disable IPv6 in your home router, wait a few hours, the kids will be complaining their search results are messed up, that's just the start of the indication that the advertisers now have a veil where with IPv6 they had clear fully trackable results.
  - ianhawes 3y ago
    What’s funny is IPv6 commonly triggers captcha on Google.
tormeh 3y ago
Is there a cloud provider that is IPv6-first? Or do they all have the same problems?
ruiseal 3y ago
The "cannot escape IPv4" is apt because while you can setup an IPv6 only VPC so many things break; from various AWS services to package repositories [0]. So then you're stuck either enabling IPv4 or running a NAT64 gateway (or trusting someone to run one for you [1]). [0] https://blog.devopstom.com/ipv6-only-ec2/ https://blog.devopstom.com/ipv6-only-ec2/ [1] https://nat64.net https://nat64.net and http://v4-frontend.netiter.com http://v4-frontend.netiter.com
michaelteter 3y ago
Sorry for the aside, but I hope the neveragain.de author will make a blog post about their site theme. I _really_ like it, and of course I would like to mostly copy it for my own personal site. That said, until the cost of IPv4 becomes really huge, few organizations are going to suffer the effort-cost of embracing IPv6. I would argue that the IPv6 sales story is unmemorable|unclear|weak. Also, it is arguable that most IPv4 addresses are wasted.
- chandlerswift 3y ago
  Looks like the theme might be a lightly modified version of the GitHub Pages Hacker theme[0]? [0]: https://github.com/pages-themes/hacker https://github.com/pages-themes/hacker
  - apparentorder 3y ago
    It's a Jekyll theme, https://news.ycombinator.com/item?id=37609543 https://news.ycombinator.com/item?id=37609543
mannyv 3y ago
One major weirdness with ipv6 is that it occasionally works with ipv4 and it's unclear why. Example: we run a bunch of endpoints on ipv4, but get ipv6 IPs in our logs. How? Are there 6-to-4 translators out there at ISP edges? Unknowns in networking are bad.
- p1mrx 3y ago
  It's impossible for an IPv4 endpoint to accept an IPv6 connection. Perhaps you have a dual-stack CDN with an unpublished IPv6 address that some users have found? Or your service is accepting third-party 'Forwarded' headers, which would allow HTTP clients to spoof their IP address.
- mschuster91 3y ago
  > Example: we run a bunch of endpoints on ipv4, but get ipv6 IPs in our logs. How? Are there 6-to-4 translators out there at ISP edges? Proxies. Your logs are too trusting of X-Forwarded-For headers.
NovemberWhiskey 3y ago
>almost no AWS API can be used from a VPC without public IPv4 addresses Virtually every single application at the company I work at deploys into VPCs without public IPv4 addresses - this seems like a ridiculous claim.
- ericpauley 3y ago
  Do they use NAT gateways? These require public IPs.
  - NovemberWhiskey 3y ago
    No; we use AWS Direct Connect for access from our on-premise networks.
- apparentorder 3y ago
  As mentioned in the footnote, this can be done by using PrivateLink; it costs a few bucks too, but it is the way to go if your VPC does not (or must not, for Compliance™ reasons) have internet connectivity. If your target VPC has neither PrivateLink nor public IPv4 connectivity somewhere, I'm not sure how that would work; I'd love to learn how that was built.
  - NovemberWhiskey 3y ago
    Yeah, sure, we use PrivateLink. In my opinion, it's clickbait to say "almost no AWS API can be used from a VPC without public IPv4 addresses" with a footnote "actually most can if you use the service that enables that".
dangus 3y ago
The moment you have a customer with crappy legacy infrastructure who refuses to allowlist anything but static IPV4 addresses, you have to support IPV4.
- klysm 3y ago
  This shit boggles my mind. Whitelisting static IPs as a security measure has a terrible pain / added security ratio
remram 3y ago
Which cloud providers does have good IPv6 support?
mschuster91 3y ago
> RDS (nine in ten customers have public IP on RDS by accident) AWS does make it easy to fuck up with its default settings. Subnets that auto-assign EIPs for every instance attached to them should not exist, period. And neither should RDS instances or anything else be reachable from the public Internet by default.
dpc_01234 3y ago
In my experience the biggest issue for being IPv6 only in AWS is that github still can't IPv6! Tons of software expects to be able to reach out to github for something. One can use some public NAT64 services, but that's not very reliable for anything serious. https://nat64.xyz/ https://nat64.xyz/ . AWS chargers arm and leg for NAT gateways traffic, and I don't think it's possible to configure them so that they only intercept traffic to ipv4-only hosts (please let me know if i'm wrong). Other than this being IPv6-only in AWS works flawlessly and is cheaper (free egress gateways for private networks). As long as you don't care about IPv4 of course - that's given.
- supertrope 3y ago
  It's too bad GitHub was not able to follow its parent company Microsoft's lead in deploying IPv6.
- klysm 3y ago
  If you have a dual stack network in AWS, wouldn’t it prefer IPv6 from DNS but IPv4 would still work? This is how I have most of my things that need general internet egress configured
joshstrange 3y ago
Honestly IPv6 is a clusterfuck. From the horrible addresses (why are they impossible to memorize? Who thought that was a smart idea? At least I can wrap my brain around IPv4) to the need for specific support in literally every layer of the network stack. If you are going to mention gateways or other methods make it work please just stop. No end-user is going to do that, or rather no appreciable amount of end users are going to do it. If your fix starts with “why don’t you just…” then please stop living in a fantasy world. I was excited for IPv6 when it was announced, I was excited years later, I was excited a decade later, now I’m just tired of it. 2024, year of IPv6 and and the Linux desktop, ok sure. My ISP, literally the best available in my area and fairly cutting edge in every other aspect, has zero IPv6 support. While the idea of every device having its own public IP address was attractive to a younger me, I look at it with a bit of horror now. The privacy/security aspects alone are staggering and you rarely want your device to be publicly available by default. I’m not going to exceed the 16M+ limit of 10.0.0.0/8 so I don’t see why I would ever want to use anything but IPv4 internally for my sanity. Are STUN/TURN servers fun? Is needing some central server ideal? No but the alternative (everyone can talk to everyone directly) makes my head hurt with the implications and footguns. At the end of the day I’ve started disabling IPv6 as a matter of course. Leaving it on is a landmine I’m laying for my future self. I’ve dealt with too many issues directly myself or for clients/customers which end with “let’s try disabling IPv6, oh it’s working now?” (on my end or theirs) that I’m done. Something drastic would have the happen to get me to change that thinking and seeing how it’s been over 2 decades and major websites I use daily still don’t support IPv6 I’m not holding my breath.
- orangeboats 3y ago
  >From the horrible addresses (why are they impossible to memorize? Who thought that was a smart idea? At least I can wrap my brain around IPv4) Every time someone brings up this point, I have to assume that they know nothing about IPv6 but the superficial things. If you work with IPv6 long enough you will remember the addresses, we all remember 192.168.0.* through years of typing it repeatedly and looking at it. Not because it is easily to remember. I can already recall 2606:4700:4700::1111 or 64:ff9b::101:101 from memory. >My ISP, literally the best available in my area and fairly cutting edge in every other aspect, has zero IPv6 support. This is almost exclusively an Euro-American phenomenon. I am not sure why you are lashing out on IPv6 when it's the ISPs' fault. In most East or Southeast Asian countries we are looking at double-digit % of IPv6 deployment, the moment you click on the IPv6 checkbox you get IPv6 connectivity here. >The privacy/security aspects alone are staggering and you rarely want your device to be publicly available by default. Another one who mistakes "having a globally unique address" with "public accessibility". Boo. >“let’s try disabling IPv6, oh it’s working now?” (on my end or theirs) that I’m done Just say that you are lazy in fixing IPv6 problems. I have found that lots of old networking guys would say "it's defo my fault somewhere" when IPv4 fails but when it comes to IPv6 it's always IPv6's fault somehow. Protip: most of time it isn't.
rswail 3y ago
AWS Lambda doesn't support IPv6 in VPCs. This means that everything else in your VPC has to be dual stack if the lambdas talk to them. AWS Lambdas don't actually run in your VPC, there is an ENI that is exposed to your VPC that belongs to the VPC-equivalent in the Lambda space. If you have to run dual-stack just to accomodate lambdas, then you may as well run IPv4 anyway.

306 points by apparentorder 3y ago

AWS has a way to go with their IPV6 services, but IPV6-only is very doable right now in EC2. I have a WireGuard server that was dual-stack. I turned off IPV4 to see what would happen, and it kept chugging along very nicely.

mike_d 3y ago
...if you are running one service and not really using AWS in any meaningful way other than as a VPS provider.
- pnpnp 3y ago
  You’re not wrong, that’s why I prefaced it saying their services are lacking. I was just pleasantly surprised that EC2 supports it.

mjevans 3y ago

It would really help if there were real ISP competition in the USA. There's only one actually broadband ISP provider where I rent, which is in the suburbs near Seattle. It's NOT a rural area by any definition, and yet Comcast is my only option. Their price and service reflect that reality...

gnopgnip 3y ago
Comcast supports ipv6 though, in 2016 they hit 98% ipv6 support.
- 1ncorrect 3y ago
  And Comcast were the biggest proponents of building IPv6 support into the DOCSIS specs, because they exhausted 10/8 in the management network for their modem fleet.
chungy 3y ago
> yet Comcast is my only option. Thank endless lobbying that make legally mandated monopolies a thing in this arena. They did it with phone companies too. It's not "just Comcast" by happenstance. It's "just Comcast" by legal regulation.
- paulddraper 3y ago
  AFAIK, most areas in the U.S. have wireless ISPs available.
mike_d 3y ago
IPv6 adoption is only going to further the consolidation of customers onto the big monopoly providers. They will be the only ones who can afford to add the dedicated network engineering staff to make it work reliably. Most people don't realize there are two IPv6 internets right now, the Cogent side and the Hurricane Electric side. Both are equally sized and refuse to connect to each other, so you need to know that and either buy transit from both or buy transit from a network that buys from both. At least one major provider I know of is still running v6 over tunnels. In many places your v6 traffic is taking suboptimal routes, whereas an enterprise network may have v4 connectivity at each datacenter, v6 all gets sent to that one box under Dave's desk. But we continue to measure v6 adoption at places like Google and Cloudflare where dedicated teams make sure packets arrive and pat ourselves on the back.
- greyface- 3y ago
  > two IPv6 internets right now, the Cogent side and the Hurricane Electric side Cogent engages in peering spats on IPv4 too; this dynamic is not new with or unique to IPv6, or limited to Cogent/HE. The lesson here is to not go singlehomed under Cogent, not to reject IPv6.
  - mike_d 3y ago
    The takeaway here was that you need to be aware of it to make IPv6 work. Again, your average operator of a small regional WISP may try to deploy v6 because they lack v4 space and face customer complaints because they single home behind HE and can't reach the other half of the internet. Currently v6 is like connecting to the late 90s internet. It dosen't work as well as people think.
    - greyface- 3y ago
      You need to be aware of it to make v4 work, too. I once saw an issue where a network similar to your example WISP (singlehomed behind HE) was unable to reach a network that was advertising its v4 prefixes NO_EXPORT to HE at a distant IX. Adding a second upstream would have fixed (and indeed, did later fix) the issue. It's been advisable since the early days to have multiple upstreams because of routing table holes created by situations like this. Again, not unique to v6, and the HE/Cogent schism is not the only one (though admittedly it is likely the largest).
huijzer 3y ago
> Comcast is my only option Maybe Starlink is another option. It has some drawbacks like reduced performance during heavy rain, but I've seen some positive reviews as well.
- ninkendo 3y ago
  Starlink doesn’t have capacity for a large urban area. There’s only so many satellites available per square mile, and if you have a sizable fraction of a major city using starlink, they’d all be bottlenecked on a handful of satellites at best, even with the huge number of satellites. Starlink only makes sense for low-geographic-density deployments, where the number of customers on the same satellite is (relatively) low.
luhn 3y ago
Consumers don't know or care about IPv6, so competition won't incentivize ISPs to implement it. Funny that you complain about Comcast, when I was with Comcast I actually had native IPv6. With my current provider I only have IPv6 through NAT46.

fulafel 3y ago

It seems obviously against AWS incentives to offer working v6 - all their influencing tools ("well architected" criteria, certificates) strongly herd you towards building mazes of ambigously addressed 10.x RFC1918 networks, and not internet style architectures with end-to-end addressing. In the world of their recommendations, even the concept of a "public ip address" is a red flag, and AWS even recommends (for an added cost of course) tooling to flag and "mitigate" them. These provide a strong lock-in effect when customers spend effort to build the complex infrastructure for them in the name of security, even though in reality they hurt security through unnecessary complexity, addressing ambiguity, etc.

beoberha 3y ago
I work in Azure, but my experience is that customers want this - and for good reason. Customers want their own private network to prevent intrusions and exfiltrations, just on machines they don’t own. Or even better, put the nice fancy batteries included PaaS services in these networks too.
- jojobas 3y ago
  Right, cause customers are too stupid to manage their own IPv6 firewalls, it's for their own good! /s
  - rad_gruchalski 3y ago
    I’m pretty confident that this statement is mostly true without sarcasm, and that you are in the minority.
    - pantalaimon 3y ago
      How is managing a NAT easier than managing a firewall?
      - ninkendo 3y ago
        I’m one typo away from accidentally allowing IPv6 access to every machine in my network with my pf config on my home router. (I know this because I’ve done it one time, and didn’t notice for about a week.) There is no such typo i could make with my single shared public ipv4 address because it’s just one address. Saying “allow” by accident isn’t enough, I’d have to somehow accidentally configure the particular ingress port to NAT to a particular internal machine, and even then it would only affect that machine and no other. (Full disclosure, i actually like IPv6 and am in full favor of everything moving to it. This is in spite of the above, but i at least recognize that the above is the case.)

donor20 3y ago

IPv6 is such a massive headache it’s kind of mind boggling. I used to be super enthused - but it is absolutely less useful and more annoying than it’s worth.

9dev 3y ago
It’s really not a headache in and of itself. The technology is beautiful and enables lots of cool things - just look at all the awesome stuff the fly.io team makes it do. The headache are vendors that still refuse to properly implement IPv6, in 2023.
jeroenhd 3y ago
This is one example where it's clear IPv6 isn't the problem, actually. A lot of problems with AWS would disappear if they would just support IPv6 like your average budget ISP does. IPv6 just works. Amazon, Github, and Azure don't. That's not really a problem in most cases (very few people go IPv6 only because it's just not necessary with CGNAT, and even then network translation tricks can put up IPv6<->IPv4 bridges easily). In Amazon's case, they don't even need to bother setting up a real network, they could abuse an fd00::/8 network to mimic their 10.0.0.0/8 network if they wanted to. Amazon is terrible at implementing modern standards. Just look at how long it took them to support DNSSEC on their domains, and even that didn't exactly roll out great the first time.
- donor20 3y ago
  WAN failover is not fun with ipv6 - npt doesn’t solve things because prefix lengths are variable. You end up back with NAT with basic networks again but with ridiculously large address space. Firewall rules a trickier- you need to both let ICMP through but be careful because some can drive network reconfig. DHCP is second class, and the network can do weird things when port isolations are on. The number of (rotating) ipv6 addresses per host gets silly and makes logging / accountability/ trace back systems more convoluted. Then you’ve got neighbor discovery threats, header extension manipulation stuff. And if multicast isn’t working because of a security configuration that breaks assumptions but you also have multicast amplification stuff. There is a reason well resourced companies like google cloud have been slow w IPv6 - and it can be even more hair pulling in smaller settings.

solatic 3y ago

Half the reason AWS has leading IPv6 support in the first place is due to mandates from the US government to start migrating. Author is correct that, from a cost perspective, the new costs are immaterial to large customers, but I wouldn't discount the power of policy mandates from the largest customers, where the threat of building an in-house alternative to comply with policy might be sufficient to force AWS to finally prioritize support.

apparentorder 3y ago
Indeed. I really don't like the thought, but I more and more believe that there is no other way to incentivize IPv6 at the "server side". The client (end user) side seems to do well, considering that Google reports IPv6 end user traffic of almost 50% these days.
- biohax2015 3y ago
  > The client (end user) side seems to do well, considering that Google reports IPv6 end user traffic of almost 50% these days. That's because all mobile data connections are on IPv6

hkt 3y ago

There needs to be a body of law relating to technical matters like this (and interoperability etc) that is adjacent to competition law. Some things we just need everyone to be on the same page about. It is manifestly the case that ipv6 is never going to be that, because the incentives to invest simply don't exist for companies like AWS. This distorts the market in eyeball networks and hosting - the former are under little pressure to offer v6, and new entrants to the latter can only offer v6. Competition law in the EU works (I think?) on the principles of consumer benefit and market fairness. On that basis, I'm left wondering why this has never been pursued by the EU's competition authorities.

robertlagrant 3y ago
Lots of AWS customers want IPv4 because that's what they know, and that's what they benefit from. To me, the question is: what stops me today from spinning up an IPv6-only website and having 99% of the world's browsers use it? If the answer is "nothing", then AWS shouldn't be forced to offer IPv6 (or only IPv6) - IPv4 is just part of what they offer customers. If the answer is "these 7 things" then those 7 things need to be fixed[0] before we pay civil servants to try and force companies to do things that they barely understand. [0] E.g. in the UK, it's some of the big ISPs that don't do IPv6, so there's no point forcing someone way upstream (and way more optional) in the process to do something https://www.ispreview.co.uk/index.php/2021/11/update-on-ipv6-plans-for-virgin-media-talktalk-plusnet-and-vodafone.html https://www.ispreview.co.uk/index.php/2021/11/update-on-ipv6...
jeroenhd 3y ago
The EU did have a mandate for government services to use IPv6, but the programme it was part of got replaced by another that didn't include IPv6. The European Commission did advocate for IPv6 use, but, the EU being the EU, motivated their recommendation by complaining that law enforcement had issues tracking down people behind CGNAT, and made clear that they wanted every IP address to point to a specific person for law enforcement reasons. So, yeah, I don't think we should let the EU deal with the specifics of network infrastructure just yet. I think it's hard to make an economic argument for IPv6. Yes, it's obviously a superior technology, but ISPs can CGNAT for cheap, consumers can still access every server, and the €40 per year a business needs to pay for an IPv4 address isn't exactly breaking the bank either. Perhaps the EU should force the issue, but I think countries like Lithuania ,where there is practically no IPv6 available (0.58%, according to https://stats.labs.apnic.net/ipv6-zoom https://stats.labs.apnic.net/ipv6-zoom, but who knows how accurate that is), will protest any mandate that will force their ISPs to buy new networking equipment.

peter_retief 3y ago

I am struggling to move to IPv6, from ISP blocking to unhelpful cloud services. Who is a good provider of ipv6?

supriyo-biswas 3y ago
At the ISP level, you have better chances of having IPv6 connectivity if you’re based out of a developing country, whose ISPs don’t have the means to pay for too many IPv4 ranges. For servers, there are plenty; AWS Lightsail, Hetzner and Vultr both provide IPv6 out of the box. If you don’t have an ISP which provides IPv6, you could use a server and set up a wireguard tunnel for IPv6 connectivity.
- tyingq 3y ago
  >AWS Lightsail Read the burst/throttling page carefully before you choose this product. https://lightsail.aws.amazon.com/ls/docs/en_us/articles/amazon-lightsail-viewing-instance-burst-capacity https://lightsail.aws.amazon.com/ls/docs/en_us/articles/amaz...
- ninkendo 3y ago
  > you have better chances of having IPv6 connectivity if you’re based out of a developing country, whose ISPs don’t have the means to pay for too many IPv4 ranges I don’t know that this is true based on google’s IPv6 adoption data: https://www.google.com/intl/en/ipv6/statistics.html#tab=per-country-ipv6-adoption https://www.google.com/intl/en/ipv6/statistics.html#tab=per-... It seems like developing nations have the worst IPv6 adoption, at least by a cursory look of how there’s very little green in Africa, for instance. I think ISP’s in countries with small IPv4 blocks just use CGNAT.
  - supriyo-biswas 3y ago
    Thank you for the link. I was taking about India for the most part, but it seems France, Germany, India, and Saudi Arabia are the leaders in IPv6 deployment, a weird mix of countries that I honestly didn’t expect.
  - 3y ago

alias_neo 3y ago

Off topic: Does anyone know if this page is generated from a Static-Site generator starting from Markdown? I currently use Hugo and my blog is in Markdown in git, but the theme is pretty heavy-weight, and I like this look of the page in OP; Looking at the source, it's so minimal!

apparentorder 3y ago
Yes, it's Markdown and I use https://jekyllrb.com https://jekyllrb.com with the theme "jekyll-theme-hacker" to generate the site. I quite like how simple it is.
- alias_neo 3y ago
  Amazing, thank you!

iopq 3y ago

I migrate my proxy's IPv4 address from time to time to avoid blocks This is not so easy to do with a IPv6 address, AWS tends to want to keep it the same

kaliszad 3y ago
If you get a /48 you can probably evade the problem by assigning a /64 for your proxy at a time. You will have another ~65,500 such blocks for use. Yes, some might just block the /56 (you would still have another ~250 chances) or /48 but nothing is perfect.
- iopq 3y ago
  but how do I change the IPv6 address in AWS? I can just assign a new Static IP for IPv4
  - xnyanta 3y ago
    Same thing for IPv6, but instead of assigning a new singular IPv6, you assign a new subnet with millions of IPv6 addresses.

supriyo-biswas 3y ago

IMO services like Lambda and S3 not supporting IPv6 is the real issue, and AWS shouldn’t have made the pricing change without first making them dual-stacked, accessible over IPv6. (Technically S3 does have a separate dual stack endpoint, however it doesn’t really help as I have to change application configuration anyway to deal with this change.)

Hikikomori 3y ago

There's one viable solution to be able to run IPv6 only subnets in AWS, their (or your own) NAT gateways support v6->v4 NAT. So it allows you to create large IPv6 only subnets for your compute services (ec2, ecs, k8s, elb, all supports that), allowing your containers to scale without worrying about IP addresses. Then you use dual stack subnets for other AWS services that may not support IPv6 and your compute services can access them through the NAT gateway.

apparentorder 3y ago
ECS, ELB and most other services do not support IPv6-only subnets, as mentioned in the article. ECS does support dual-stack IPv6, but most other services do not support IPv6 at all.

wheybags 3y ago

neveragain.de is... a strange choice of domain for a tech blog. It sounds like a holocaust memorial site.

apparentorder 3y ago
This domain is very, very old and at that time, the phrase wasn't usually associated like it is today. Not sure what to do about that.

kennu 3y ago

Worst for me is CloudFront not supporting IPv6 for custom origins. If you happen to run a lot of separate Fargate containers as origins, you have to enable public IPv4 addresses for them, and that will soon double the price of small confainers. Amazon needs to make their infrastructure actually support IPv6 before starting to charge extra for legacy IPv4 usage.

paraxial_0 3y ago
Yeah, somehow this one hurts the most. I know Amazon has built a giant beast here and rolling out IPv6 across all of their million services is a huge undertaking, but I can't see any reason for CloudFront not supporting IPv6 origins, like yesterday. It doesn't seem like it should be that hard relatively speaking, and would provide a good tool for working around other limitations. Honestly I've always felt that Amazon's decision-making ultimately had the best interests of their customers in mind, until now. I think this is a bad sign for things to come.

dopylitty 3y ago

As an AWS customer I want to escape IP entirely. It's a waste of time managing these complex networking systems with their archaic protocols (IP, BGP, DNS, etc) Just let me strongly associate identities with my workloads and apply policy indicating which workloads should be able to send data with which other workloads. How data gets from one workload to another should not even be my concern, just make it happen.

drpossum 3y ago
While your sentiment is valid, this is the type of argument people make on low code solutions. Which has never worked in reality and never will. There's just too much nuance and detail that needs to be considered when you have to do and optimize real workloads.
- Spivak 3y ago
  Of course but it pushes the abstraction forward to what we really want. It turns out that most applications don't actually want to mess with IP except as an implementation detail and optimization. Which is why most of the time you don't and you just get some application layer HTTP payload, RPC thing or WSGIish type call on the incoming side and let the ops people deal with the networking bits and it works well enough that people mostly don't complain. The request is that more outgoing services adopt this model where you had it off to your application server and it does the work and just gives you the data you want back. I don't think it's that crazy, it's just formally standardizing where we're already going.
pragma_x 3y ago
I agree completely. FWIW, this sentiment is why we're seeing a lot of cloud "platforms" crop up that do exactly what you're talking about. Rather than get mired in the component-zoo of virtualized datacenter (read: pretend) networking, just abstract all of it away.
flerchin 3y ago
IME "big picture folks" suddenly care about the details when their big picture fails to solve for the details.

Bluecobra 3y ago

> There is no concept of private addresses in IPv6, which means farewell to the Managed NAT Gateway and its magnificent pricing. Maybe not in AWS, but there are Unique Local IPv6 addresses in fc00::/7 and NAT66 if you really love NAT!

apparentorder 3y ago
Yeah, I was referring to AWS; I should have made that clear. ULA is frequently discouraged though, and NAT66, well ... just no. I just recently heard that MS apparently has built everything IPv6 on Azure around NAT. This is so weird.
- klysm 3y ago
  It’s the most Microsoft thing possible. That way they can charge you! How else are they supposed to rent seek?

digitalsushi 3y ago

My ISP, Fidium, does not have the word IPv6 on its entire website. And definitely has no support of it on my WAN. They should. I want to connect to IPv6 services using their connection. I actually keep a cheap Comcast connection as a second WAN just to get IPv6 enabled on my home network. (And also because I live in the woods in New Hampshire and having two ISPs means I have fairly ok uptime)

mabbo 3y ago

Many years ago when I was a junior dev at Amazon, there was a massive project internally to split up every internal system into regional versions with limited gateways allowing calls between regions. The reason? We had run out of internal IPv4 addresses. The Principal PM in charge of the "regionalization" effort was asked in a Q&A "why didn't we just switch to IPv6?". Her answer was something along the lines of "The number of internal networking devices we currently have that cannot support IPv6 is so large that to replace them we would have needed to buy nearly the entire world's yearly output of those devices, and then install them all."[0] It's easy to presume malicious intent on the IPv4 front from Amazon, but with so many AWS systems being on the scale they are at, I find it easy to believe that replacing all of the old network hardware may just be a project too large to do on a short timescale. [0] - At least, that's my memory of it. I'm sure that's not an entirely accurate quotation.

tinix 3y ago
this doesn't forgo v6 phase-in though, can't kick that can down the road forever. surely they started the process... right? i cannot imagine AWS just sticking head in the ground and ignoring this...
- mtnGoat 3y ago
  Yes they are working on it. A number of services already support v6, more to come.
  - AdamJacobMuller 3y ago
    1 is a number. 0 is also a number.
- Twirrim 3y ago
  No one is ignoring it, and the US Government has done everyone another favour on this score. Years ago in the late Bush / early Obama administration, NIST required that all federal government agencies have IPv6 at the border. Federal government money is not to be sniffed at, and that had the effect of forcing a number of vendors to add IPv6 support. A few years after that, it became that the federal agencies needed to have dual-stack IPv4/IPv6. About 18 months ago, the requirement came that federal agencies are required to be IPv6 Only, dropping the dual stack. IIRC they have until 2025 to do that. This has the neat effect of forcing all vendors to make IPv6 a first class citizen. The extra little fun from this is that it applies to the military JWCC contract that all the major clouds have been trying to land. The timescales of JWCC meant that initial offerings are pretty bare, but that won't be allowed to last.

welder 3y ago

I use DigitalOcean... almost all their products support IPv6. Only floating IPs are IPv4, but can work around that by not destroying droplets so the IPv6 address doesn't change.

kevincox 3y ago
One annoyance I have is that IPv6 is disabled on VMs (droplets) by default. So every time you upgrade your Kubernetes cluster and it recreates every node they all have no IPv6 again with no way to change this. In the end I couldn't be bothered and just stopped rebooting every node after every update.

mkl95 3y ago

> The first pattern is having multiple Load Balancers (per VPC); this is often the result of using several readily available Cloudformation templates / Terraform modules, or somehow using Kubernetes ingress controllers that create a Load Balancer for every service. This is fixed by not doing that! A single Load Balancer can handle many URLs and services. This is the definition of cloud bloat. The fact there are tons of systems abusing that kind of architecture probably justifies charging for IPv4.

Spivak 3y ago
I think it's an impedance mismatch between the feature people want -- "logical load balancers" and the feature they're offered "physical load balancers." How nice it would be if you could just create a bunch of load balancers and all that actually meant was that it was just adding config profiles to a single physical load balancer and kept them truly isolated? Right now it's really annoying because load balancer config is global state and everyone has to either be kind neighbors when adding themselves to it or manage them top-down.
doublet00th 3y ago
I do believe the AWS Load Balancer Controller on Kubernetes allows for sharing a single "physical" load balancer. You have to set a load-balancer-name annotation https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.4/guide/ingress/annotations/#load-balancer-name https://kubernetes-sigs.github.io/aws-load-balancer-controll... to tie everything together to one load balancer. There is a downside where you have to have a few other annotations be the same value across your ingresses, but once you work around that, you're good to go.
- renewiltord 3y ago
  I couldn't find it easily specified in docs. This is a common use-case and part of why I avoid EKS for HTTP workloads is that I have tiny services I want to just make available and I don't want to have another full ELB sitting there. It isn't a cost thing primarily. It's that now I have another significant resource. I want all of these things on a misc LB.

josephcsible 3y ago

I feel like there's now a perverse incentive here for Amazon to drag their feet at implementing full feature parity with IPv6. I wish these new charges only applied for IPv4 addresses used with services that already do have that.

tormeh 3y ago

It's not that people dislike IPv6 or like IPv4, it's that network people are comfortable with IPv4 and all the extra tech surrounding it. They know it works, so there's no technological risk. There's nothing new to learn. It's cheap. There's nothing your average business wants to do that can't be done on IPv4 that can on IPv6. The ROI of just paying for IPv4 addresses and associated tech/services is undeniable.

talent_deprived 3y ago
> It's not that people dislike IPv6 Yes, for some who understand and have tested it, we do not like or want IPv6, it has no privacy when the device's IP is public on the Internet 24/7. No privacy extensions fix this. Test it, it's not difficult, disable IPv6 in your home router, wait a few hours, the kids will be complaining their search results are messed up, that's just the start of the indication that the advertisers now have a veil where with IPv6 they had clear fully trackable results.
- ianhawes 3y ago
  What’s funny is IPv6 commonly triggers captcha on Google.

tormeh 3y ago

Is there a cloud provider that is IPv6-first? Or do they all have the same problems?

ruiseal 3y ago

The "cannot escape IPv4" is apt because while you can setup an IPv6 only VPC so many things break; from various AWS services to package repositories [0]. So then you're stuck either enabling IPv4 or running a NAT64 gateway (or trusting someone to run one for you [1]). [0] https://blog.devopstom.com/ipv6-only-ec2/ https://blog.devopstom.com/ipv6-only-ec2/ [1] https://nat64.net https://nat64.net and http://v4-frontend.netiter.com http://v4-frontend.netiter.com

michaelteter 3y ago

Sorry for the aside, but I hope the neveragain.de author will make a blog post about their site theme. I _really_ like it, and of course I would like to mostly copy it for my own personal site. That said, until the cost of IPv4 becomes really huge, few organizations are going to suffer the effort-cost of embracing IPv6. I would argue that the IPv6 sales story is unmemorable|unclear|weak. Also, it is arguable that most IPv4 addresses are wasted.

chandlerswift 3y ago
Looks like the theme might be a lightly modified version of the GitHub Pages Hacker theme[0]? [0]: https://github.com/pages-themes/hacker https://github.com/pages-themes/hacker
- apparentorder 3y ago
  It's a Jekyll theme, https://news.ycombinator.com/item?id=37609543 https://news.ycombinator.com/item?id=37609543

mannyv 3y ago

One major weirdness with ipv6 is that it occasionally works with ipv4 and it's unclear why. Example: we run a bunch of endpoints on ipv4, but get ipv6 IPs in our logs. How? Are there 6-to-4 translators out there at ISP edges? Unknowns in networking are bad.

p1mrx 3y ago
It's impossible for an IPv4 endpoint to accept an IPv6 connection. Perhaps you have a dual-stack CDN with an unpublished IPv6 address that some users have found? Or your service is accepting third-party 'Forwarded' headers, which would allow HTTP clients to spoof their IP address.
mschuster91 3y ago
> Example: we run a bunch of endpoints on ipv4, but get ipv6 IPs in our logs. How? Are there 6-to-4 translators out there at ISP edges? Proxies. Your logs are too trusting of X-Forwarded-For headers.

NovemberWhiskey 3y ago

>almost no AWS API can be used from a VPC without public IPv4 addresses Virtually every single application at the company I work at deploys into VPCs without public IPv4 addresses - this seems like a ridiculous claim.

ericpauley 3y ago
Do they use NAT gateways? These require public IPs.
- NovemberWhiskey 3y ago
  No; we use AWS Direct Connect for access from our on-premise networks.
apparentorder 3y ago
As mentioned in the footnote, this can be done by using PrivateLink; it costs a few bucks too, but it is the way to go if your VPC does not (or must not, for Compliance™ reasons) have internet connectivity. If your target VPC has neither PrivateLink nor public IPv4 connectivity somewhere, I'm not sure how that would work; I'd love to learn how that was built.
- NovemberWhiskey 3y ago
  Yeah, sure, we use PrivateLink. In my opinion, it's clickbait to say "almost no AWS API can be used from a VPC without public IPv4 addresses" with a footnote "actually most can if you use the service that enables that".

dangus 3y ago

The moment you have a customer with crappy legacy infrastructure who refuses to allowlist anything but static IPV4 addresses, you have to support IPV4.

klysm 3y ago
This shit boggles my mind. Whitelisting static IPs as a security measure has a terrible pain / added security ratio

remram 3y ago

Which cloud providers does have good IPv6 support?

mschuster91 3y ago

> RDS (nine in ten customers have public IP on RDS by accident) AWS does make it easy to fuck up with its default settings. Subnets that auto-assign EIPs for every instance attached to them should not exist, period. And neither should RDS instances or anything else be reachable from the public Internet by default.

dpc_01234 3y ago

In my experience the biggest issue for being IPv6 only in AWS is that github still can't IPv6! Tons of software expects to be able to reach out to github for something. One can use some public NAT64 services, but that's not very reliable for anything serious. https://nat64.xyz/ https://nat64.xyz/ . AWS chargers arm and leg for NAT gateways traffic, and I don't think it's possible to configure them so that they only intercept traffic to ipv4-only hosts (please let me know if i'm wrong). Other than this being IPv6-only in AWS works flawlessly and is cheaper (free egress gateways for private networks). As long as you don't care about IPv4 of course - that's given.

supertrope 3y ago
It's too bad GitHub was not able to follow its parent company Microsoft's lead in deploying IPv6.
klysm 3y ago
If you have a dual stack network in AWS, wouldn’t it prefer IPv6 from DNS but IPv4 would still work? This is how I have most of my things that need general internet egress configured

joshstrange 3y ago

Honestly IPv6 is a clusterfuck. From the horrible addresses (why are they impossible to memorize? Who thought that was a smart idea? At least I can wrap my brain around IPv4) to the need for specific support in literally every layer of the network stack. If you are going to mention gateways or other methods make it work please just stop. No end-user is going to do that, or rather no appreciable amount of end users are going to do it. If your fix starts with “why don’t you just…” then please stop living in a fantasy world. I was excited for IPv6 when it was announced, I was excited years later, I was excited a decade later, now I’m just tired of it. 2024, year of IPv6 and and the Linux desktop, ok sure. My ISP, literally the best available in my area and fairly cutting edge in every other aspect, has zero IPv6 support. While the idea of every device having its own public IP address was attractive to a younger me, I look at it with a bit of horror now. The privacy/security aspects alone are staggering and you rarely want your device to be publicly available by default. I’m not going to exceed the 16M+ limit of 10.0.0.0/8 so I don’t see why I would ever want to use anything but IPv4 internally for my sanity. Are STUN/TURN servers fun? Is needing some central server ideal? No but the alternative (everyone can talk to everyone directly) makes my head hurt with the implications and footguns. At the end of the day I’ve started disabling IPv6 as a matter of course. Leaving it on is a landmine I’m laying for my future self. I’ve dealt with too many issues directly myself or for clients/customers which end with “let’s try disabling IPv6, oh it’s working now?” (on my end or theirs) that I’m done. Something drastic would have the happen to get me to change that thinking and seeing how it’s been over 2 decades and major websites I use daily still don’t support IPv6 I’m not holding my breath.

orangeboats 3y ago
>From the horrible addresses (why are they impossible to memorize? Who thought that was a smart idea? At least I can wrap my brain around IPv4) Every time someone brings up this point, I have to assume that they know nothing about IPv6 but the superficial things. If you work with IPv6 long enough you will remember the addresses, we all remember 192.168.0.* through years of typing it repeatedly and looking at it. Not because it is easily to remember. I can already recall 2606:4700:4700::1111 or 64:ff9b::101:101 from memory. >My ISP, literally the best available in my area and fairly cutting edge in every other aspect, has zero IPv6 support. This is almost exclusively an Euro-American phenomenon. I am not sure why you are lashing out on IPv6 when it's the ISPs' fault. In most East or Southeast Asian countries we are looking at double-digit % of IPv6 deployment, the moment you click on the IPv6 checkbox you get IPv6 connectivity here. >The privacy/security aspects alone are staggering and you rarely want your device to be publicly available by default. Another one who mistakes "having a globally unique address" with "public accessibility". Boo. >“let’s try disabling IPv6, oh it’s working now?” (on my end or theirs) that I’m done Just say that you are lazy in fixing IPv6 problems. I have found that lots of old networking guys would say "it's defo my fault somewhere" when IPv4 fails but when it comes to IPv6 it's always IPv6's fault somehow. Protip: most of time it isn't.

rswail 3y ago

AWS Lambda doesn't support IPv6 in VPCs. This means that everything else in your VPC has to be dual stack if the lambdas talk to them. AWS Lambdas don't actually run in your VPC, there is an ENI that is exposed to your VPC that belongs to the VPC-equivalent in the Lambda space. If you have to run dual-stack just to accomodate lambdas, then you may as well run IPv4 anyway.