8 ms·

Azure is copying this wholesale. I've lost track of all of the "Private Endpoints", "Private Links", "Service Endpoints", "Private Resolvers" and "Virtual WAN"

Azure is copying this wholesale. I've lost track of all of the "Private Endpoints", "Private Links", "Service Endpoints", "Private Resolvers" and "Virtual WAN" products they've introduced... all to make IPv4 work at scale. Literally none of those products would be required if they had just made IPv6 work properly. Instead, they NAT IPv6, so you can't even use it to avoid the NAT forced upon you by IPv4. They also release new products -- in 2023 -- that don't support IPv6 and likely never will. There is still an entire page[1] of listed limitations in the docs: https://learn.microsoft.com/en-us/azure/virtual-network/ip-services/ipv6-overview#limitations https://learn.microsoft.com/en-us/azure/virtual-network/ip-s... Think about how insane it would be if this was the IPX -> IPv4 transition. Imagine if this page said "IPv4 limitations: All VMs must include at least one IPX address, etc..." Sounds nuts, right? I was migrating customers to IPv4 from IPX in 1999, and IPv6 support materialised in about the 2001-2003 timeframe. It's been decades, but it still feels like 1999 and migrating Novell NetWare where we had to have IPX+IPv4 because "not everything supports IPv4 yet". [1] This is definitely not all of the limitations. Most of their PaaS products don't support IPv6. Hence, any IaaS+PaaS solutions must use (mostly) IPv4.

benterix 3y ago
Wait a second. A private link means that a service endpoint is public so a part of your traffic goes through the Internet, which is supposed to be insecure (even if you have encryption in transit?), so you don't want to do that and they will happily route your traffic internally so it is not exposed to the bad Internet - for a fee. All these VPC Endpoints etc. cost money and you are charged by the our. I don't think IPv6 would change anything here, they would find a way to charge customers for "security" anyway.
- jiggawatts 3y ago
  That's what I thought at first, but that's not quite right. Service Endpoint: Allows a PaaS service (that itself uses public addresses) have firewall rules for overlapping private vnet addresses. E.g.: You can have have two VMs both on 10.0.0.123 addresses (in separate VNets) using individual "Allow" rules to the target service. Essentially Azure tags the traffic at the VXLAN level with the source network ID on top of the IP address, making it a "fat IP address" that is unique within Azure and can be used in firewall rules. Private Endpoint: Makes a PaaS service appear on a private network address range instead of the default public range. This allows your on-prem firewalls to isolate your specific PaaS instance from other customers -- otherwise the traffic gets "blended in" with everyone else in the same public service tag ranges. This also allows you to use your ExpressRoute fibre links to route traffic from on-prem to the public service. In all scenarios, the traffic goes over Azure networks and/or Microsoft's private backbone. You have to go out of your way to route traffic "via the Internet". Remember: Network addresses are just numbers! Routing rules determine how they flow, and public addresses can be used on private networks. Fundamentally, all this exists just to enable the ability to firewall things. With overlapping IPv4 addresses and small shared blocks of IPv4 addresses with NAT behind them, it would be impossible otherwise. With IPv6, using firewalls would be much simpler because overlapping addresses aren't needed any more. Similarly, PaaS services could trivially allocate IPv6 addresses per customer instance, so that customers could apply selective firewall rules.
  - eddythompson80 3y ago
    What a shit show. Seriously I can never rant enough about how awful Azure networking and their bullshit concepts is. Like they don’t know how to do networking, so they’re gonna introduce a bunch of shit and pretend that nonesense makes perfect sense because of their own ineptitude.
    - jiggawatts 3y ago
      Most of the complexity is a side-effect of having to do IPv4 far past the scale where the ~17 million private addresses might be sufficient.
      - WorldMaker 3y ago
        Some of the complexity seems to be the security theater of the lowest common denominator of customer demands. Companies invest too much money into incredibly expensive Palo Alto firewalls then demand Azure route through those too so that their cloud operations are as theatrically "secure" as their main network because look at all those amazing sunk costs invested in it.
  - rewmie 3y ago
    > In all scenarios, the traffic goes over Azure networks and/or Microsoft's private backbone. That's the whole selling point of private links. How could you possibly missed that? That's exactly why companies onboard onto the service. They say exactly this exactly on the marketing brochure. That's why customers line up to pay for it: to get their traffic flow only through private networks instead of through the wild. What kind of confusion reigns in your mind to come to the conclusion this was some obscure conspiratorial gotcha? It boggles the mind how you felt the need to come up with absurd conspiracies involving IPv4 to arrive at a claim that the marketing pamphlets show front and center as their whole reason of existence: avoid traffic to go through the internet, and instead pay extra to go through private pipes they own. It's as if the name actually means something!
    - d-z-m 3y ago
thaumasiotes 3y ago
Comcast at some point stopped letting you administrate your own router. You can log in to it, but port forwarding is no longer available through the administration interface. If you want port forwarding, they recommend that you do... something. It's not clear what; what you can find on the internet is mostly just people complaining that they insisted to customer support that they needed port forwarding, customer support said they'd do that, and port forwarding still doesn't work. But, intriguingly, it turns out that a Comcast router will also assign every device on your network at least one public ipv6 address. They also firewall all incoming ipv6 traffic, but unlike the situation with port forwarding, you can disable that firewall on the router admin page. (You can't put up your own firewall.)
- lloeki 3y ago
  > port forwarding is no longer available through the administration interface. Sounds like the CGNAT experience, possibly blanket applied even if you have a globally routable public IPv4 in order to have consistent behaviour and reduce network management / support case complexity.
- supertrope 3y ago
  One cannot unplug the Comcast issued router, power cycle the modem, and plug in a customer owned router?
  - StuffMaster 3y ago
    All I see are combo units these days.
    - iudqnolq 3y ago
      You have to bring your own modem. Then you bring your own combo unit or bring your own modem that you can plug a router into.
    - spookthesunset 3y ago
      All the modems I see support bridge mode. When enabled the Comcast device doesn’t do any of the routing at all. Your own device gets to do that instead.
rewmie 3y ago
> I've lost track of all of the "Private Endpoints", "Private Links", "Service Endpoints", "Private Resolvers" and "Virtual WAN" products they've introduced... all to make IPv4 work at scale. I'm not sure what you've been reading, but the concept of a private link has absolutely nothing to do with IPv4 vs IPv6. In fact, practically all your remarks don't involve the issue at all. The most charitable interpretation of your comment is that you're making the mistake of conflating any application involving a virtual network as something caused or involing IPv4.
- jiggawatts 3y ago
  I replied to someone else's similar misconception of what problem Private Link solves here: https://news.ycombinator.com/edit?id=37609614 https://news.ycombinator.com/edit?id=37609614 Hint: it's a complex workaround for insufficient IPv4 addresses.
  - rewmie 3y ago
    I think your comment shows a high dose of ignorance and complete lack of research on the topic. The whole point of private link is to not route traffic over the internet, and instead flow traffic between private networks through private pipes. One of the primary usecases and design requirements for this service is regulatory compliance. They say right on the tin that the service is designed to send traffic over private networks, including AWS's own global network. The whole point of private link is to ship data through the pipes you own, instead of routing it through the wild. I don't know how you could have missed that. More importantly, you really need to want to use private link connections. This is a value-added service. You need to want to go out of your way to avoid your traffic to go through the internet to onboard both ends of your services to private link. Not only are your conspiratorial hypothesis completely out of base, even your baseless assumptions have absolutely no relation with what version of the IP protocol is in place. I'm the first to join in on any good old fashioned AWS/big cloud provider bashing, but these should be grounded on reality.
    - jiggawatts 3y ago
devoutsalsa 3y ago
If you're not using token ring, it's not a real network.
- anyoneamous 3y ago
  I read that as "Tolkien ring" - which would also be appropriate, since "...and in the darkness bind them" is a good summary of cloud vendor lock-in.
  - IntelMiner 3y ago
    "Nine /8's were gifted to the race of cloud providers, who above all else, desire power"
pantalaimon 3y ago
It's almost as if those big cloud providers want to prevent a transition to IPv6 as relying on IPv4 will solidify their oligopoly. Any new competitor would have to cough up a huge upfront cost for IPv4 addresses, if they could even get them at that scale.
icedchai 3y ago
A lot of IT folks are still fearful of IPv6. I've been on calls where people disable IPv6 as a matter of "best practice." It's sad. People will gladly learn the latest flavor of the month web framework but won't take time to gain experience with a fundamental protocol.
- Varriount 3y ago
  Why? What's the (supposed) fear?
  - icedchai 3y ago
    It's simple fear of the unknown. Many folks haven't learned it, so they just turn it off... Ignore it, kick the can down the road.
    - nerdponx 3y ago
      It's individuals responding to the incentives before them. If something goes wrong because they're using IPv6, it's their fault. If you never upgrade anything until you're forced to, then you can never break anything by upgrading, and you're never seen as a "stuff breaker".
      - tormeh 3y ago
        This is it. Having broken stuff by upgrading myself, I can say it does not make you popular. This is especially the case if what the upgrade did was tighten up some bug that it turns hid a bug in your team's code...
  - LinuxBender 3y ago
    Why? What's the (supposed) fear? Disclaimer: I have never worked for Amazon but I can add some input from the assorted medium to large companies I have been employed with. I won't mention any names. This is for someone I know reads my comments wink wink I am not justifying it, just adding some of the bits I experienced. There are many security devices that do not have the same capabilities on IPv6 as IPv4 yet. Some enterprise IoT devices only support IPv4. Adding to this some network engineers don't want to step outside of their comfort zone and tooling/scripts to generate configurations automagically do not yet support IPv6. As the company grows they hire less Sr. Network Engineers and some of the new people depend on but do not understand the automation. e.g. someone wrote some API and they retired or changed companies. Some tooling may remain stagnant for some time. It's also a heavy lift to retrofit some enterprise environments for smaller changes so people fear the outages they will induce implementing IPv6. In some companies it is a major change just to migrate customers to a new load balancer endpoint. There may also be hundreds of undocumented things due to employee churn and lack of change control running that when broken will cause extended outages. And then there is internal politics and finger pointing... Again, not justifying it, rather I think there are too many moving bits and complexity that people have added over the decades and they are paralyzed by fear and risking the loss of their paycheck. And then there is the embarrassment that comes from having to acknowledge that nobody knows the current state of an environment and that embarrassment can go all the way up the organizational chain. That is based on my experience of being brought into companies with the speicifc task of, "Hey, make this simpler, reduce outages." It's rarely strictly a technical challenge but rather having to navigate politics, personality types and individual sub-org leaders that have had independent control of their environment for a long time. The more I think about it this could be a topic in and of itself how companies induce self inflicted bloat as they grow.

benterix 3y ago

Wait a second. A private link means that a service endpoint is public so a part of your traffic goes through the Internet, which is supposed to be insecure (even if you have encryption in transit?), so you don't want to do that and they will happily route your traffic internally so it is not exposed to the bad Internet - for a fee. All these VPC Endpoints etc. cost money and you are charged by the our. I don't think IPv6 would change anything here, they would find a way to charge customers for "security" anyway.

jiggawatts 3y ago
That's what I thought at first, but that's not quite right. Service Endpoint: Allows a PaaS service (that itself uses public addresses) have firewall rules for overlapping private vnet addresses. E.g.: You can have have two VMs both on 10.0.0.123 addresses (in separate VNets) using individual "Allow" rules to the target service. Essentially Azure tags the traffic at the VXLAN level with the source network ID on top of the IP address, making it a "fat IP address" that is unique within Azure and can be used in firewall rules. Private Endpoint: Makes a PaaS service appear on a private network address range instead of the default public range. This allows your on-prem firewalls to isolate your specific PaaS instance from other customers -- otherwise the traffic gets "blended in" with everyone else in the same public service tag ranges. This also allows you to use your ExpressRoute fibre links to route traffic from on-prem to the public service. In all scenarios, the traffic goes over Azure networks and/or Microsoft's private backbone. You have to go out of your way to route traffic "via the Internet". Remember: Network addresses are just numbers! Routing rules determine how they flow, and public addresses can be used on private networks. Fundamentally, all this exists just to enable the ability to firewall things. With overlapping IPv4 addresses and small shared blocks of IPv4 addresses with NAT behind them, it would be impossible otherwise. With IPv6, using firewalls would be much simpler because overlapping addresses aren't needed any more. Similarly, PaaS services could trivially allocate IPv6 addresses per customer instance, so that customers could apply selective firewall rules.
- eddythompson80 3y ago
  What a shit show. Seriously I can never rant enough about how awful Azure networking and their bullshit concepts is. Like they don’t know how to do networking, so they’re gonna introduce a bunch of shit and pretend that nonesense makes perfect sense because of their own ineptitude.
  - jiggawatts 3y ago
    Most of the complexity is a side-effect of having to do IPv4 far past the scale where the ~17 million private addresses might be sufficient.
    - WorldMaker 3y ago
      Some of the complexity seems to be the security theater of the lowest common denominator of customer demands. Companies invest too much money into incredibly expensive Palo Alto firewalls then demand Azure route through those too so that their cloud operations are as theatrically "secure" as their main network because look at all those amazing sunk costs invested in it.
- rewmie 3y ago
  > In all scenarios, the traffic goes over Azure networks and/or Microsoft's private backbone. That's the whole selling point of private links. How could you possibly missed that? That's exactly why companies onboard onto the service. They say exactly this exactly on the marketing brochure. That's why customers line up to pay for it: to get their traffic flow only through private networks instead of through the wild. What kind of confusion reigns in your mind to come to the conclusion this was some obscure conspiratorial gotcha? It boggles the mind how you felt the need to come up with absurd conspiracies involving IPv4 to arrive at a claim that the marketing pamphlets show front and center as their whole reason of existence: avoid traffic to go through the internet, and instead pay extra to go through private pipes they own. It's as if the name actually means something!
  - d-z-m 3y ago

thaumasiotes 3y ago

Comcast at some point stopped letting you administrate your own router. You can log in to it, but port forwarding is no longer available through the administration interface. If you want port forwarding, they recommend that you do... something. It's not clear what; what you can find on the internet is mostly just people complaining that they insisted to customer support that they needed port forwarding, customer support said they'd do that, and port forwarding still doesn't work. But, intriguingly, it turns out that a Comcast router will also assign every device on your network at least one public ipv6 address. They also firewall all incoming ipv6 traffic, but unlike the situation with port forwarding, you can disable that firewall on the router admin page. (You can't put up your own firewall.)

lloeki 3y ago
> port forwarding is no longer available through the administration interface. Sounds like the CGNAT experience, possibly blanket applied even if you have a globally routable public IPv4 in order to have consistent behaviour and reduce network management / support case complexity.
supertrope 3y ago
One cannot unplug the Comcast issued router, power cycle the modem, and plug in a customer owned router?
- StuffMaster 3y ago
  All I see are combo units these days.
  - iudqnolq 3y ago
    You have to bring your own modem. Then you bring your own combo unit or bring your own modem that you can plug a router into.
  - spookthesunset 3y ago
    All the modems I see support bridge mode. When enabled the Comcast device doesn’t do any of the routing at all. Your own device gets to do that instead.

rewmie 3y ago

> I've lost track of all of the "Private Endpoints", "Private Links", "Service Endpoints", "Private Resolvers" and "Virtual WAN" products they've introduced... all to make IPv4 work at scale. I'm not sure what you've been reading, but the concept of a private link has absolutely nothing to do with IPv4 vs IPv6. In fact, practically all your remarks don't involve the issue at all. The most charitable interpretation of your comment is that you're making the mistake of conflating any application involving a virtual network as something caused or involing IPv4.

jiggawatts 3y ago
I replied to someone else's similar misconception of what problem Private Link solves here: https://news.ycombinator.com/edit?id=37609614 https://news.ycombinator.com/edit?id=37609614 Hint: it's a complex workaround for insufficient IPv4 addresses.
- rewmie 3y ago
  I think your comment shows a high dose of ignorance and complete lack of research on the topic. The whole point of private link is to not route traffic over the internet, and instead flow traffic between private networks through private pipes. One of the primary usecases and design requirements for this service is regulatory compliance. They say right on the tin that the service is designed to send traffic over private networks, including AWS's own global network. The whole point of private link is to ship data through the pipes you own, instead of routing it through the wild. I don't know how you could have missed that. More importantly, you really need to want to use private link connections. This is a value-added service. You need to want to go out of your way to avoid your traffic to go through the internet to onboard both ends of your services to private link. Not only are your conspiratorial hypothesis completely out of base, even your baseless assumptions have absolutely no relation with what version of the IP protocol is in place. I'm the first to join in on any good old fashioned AWS/big cloud provider bashing, but these should be grounded on reality.
  - jiggawatts 3y ago

devoutsalsa 3y ago

If you're not using token ring, it's not a real network.

anyoneamous 3y ago
I read that as "Tolkien ring" - which would also be appropriate, since "...and in the darkness bind them" is a good summary of cloud vendor lock-in.
- IntelMiner 3y ago
  "Nine /8's were gifted to the race of cloud providers, who above all else, desire power"

pantalaimon 3y ago

It's almost as if those big cloud providers want to prevent a transition to IPv6 as relying on IPv4 will solidify their oligopoly. Any new competitor would have to cough up a huge upfront cost for IPv4 addresses, if they could even get them at that scale.

icedchai 3y ago

A lot of IT folks are still fearful of IPv6. I've been on calls where people disable IPv6 as a matter of "best practice." It's sad. People will gladly learn the latest flavor of the month web framework but won't take time to gain experience with a fundamental protocol.

Varriount 3y ago
Why? What's the (supposed) fear?
- icedchai 3y ago
  It's simple fear of the unknown. Many folks haven't learned it, so they just turn it off... Ignore it, kick the can down the road.
  - nerdponx 3y ago
    It's individuals responding to the incentives before them. If something goes wrong because they're using IPv6, it's their fault. If you never upgrade anything until you're forced to, then you can never break anything by upgrading, and you're never seen as a "stuff breaker".
    - tormeh 3y ago
      This is it. Having broken stuff by upgrading myself, I can say it does not make you popular. This is especially the case if what the upgrade did was tighten up some bug that it turns hid a bug in your team's code...
- LinuxBender 3y ago
  Why? What's the (supposed) fear? Disclaimer: I have never worked for Amazon but I can add some input from the assorted medium to large companies I have been employed with. I won't mention any names. This is for someone I know reads my comments wink wink I am not justifying it, just adding some of the bits I experienced. There are many security devices that do not have the same capabilities on IPv6 as IPv4 yet. Some enterprise IoT devices only support IPv4. Adding to this some network engineers don't want to step outside of their comfort zone and tooling/scripts to generate configurations automagically do not yet support IPv6. As the company grows they hire less Sr. Network Engineers and some of the new people depend on but do not understand the automation. e.g. someone wrote some API and they retired or changed companies. Some tooling may remain stagnant for some time. It's also a heavy lift to retrofit some enterprise environments for smaller changes so people fear the outages they will induce implementing IPv6. In some companies it is a major change just to migrate customers to a new load balancer endpoint. There may also be hundreds of undocumented things due to employee churn and lack of change control running that when broken will cause extended outages. And then there is internal politics and finger pointing... Again, not justifying it, rather I think there are too many moving bits and complexity that people have added over the decades and they are paralyzed by fear and risking the loss of their paycheck. And then there is the embarrassment that comes from having to acknowledge that nobody knows the current state of an environment and that embarrassment can go all the way up the organizational chain. That is based on my experience of being brought into companies with the speicifc task of, "Hey, make this simpler, reduce outages." It's rarely strictly a technical challenge but rather having to navigate politics, personality types and individual sub-org leaders that have had independent control of their environment for a long time. The more I think about it this could be a topic in and of itself how companies induce self inflicted bloat as they grow.