7 ms·

Why? What's the (supposed) fear?

icedchai 3y ago
It's simple fear of the unknown. Many folks haven't learned it, so they just turn it off... Ignore it, kick the can down the road.
- nerdponx 3y ago
  It's individuals responding to the incentives before them. If something goes wrong because they're using IPv6, it's their fault. If you never upgrade anything until you're forced to, then you can never break anything by upgrading, and you're never seen as a "stuff breaker".
  - tormeh 3y ago
    This is it. Having broken stuff by upgrading myself, I can say it does not make you popular. This is especially the case if what the upgrade did was tighten up some bug that it turns hid a bug in your team's code...
LinuxBender 3y ago
Why? What's the (supposed) fear? Disclaimer: I have never worked for Amazon but I can add some input from the assorted medium to large companies I have been employed with. I won't mention any names. This is for someone I know reads my comments wink wink I am not justifying it, just adding some of the bits I experienced. There are many security devices that do not have the same capabilities on IPv6 as IPv4 yet. Some enterprise IoT devices only support IPv4. Adding to this some network engineers don't want to step outside of their comfort zone and tooling/scripts to generate configurations automagically do not yet support IPv6. As the company grows they hire less Sr. Network Engineers and some of the new people depend on but do not understand the automation. e.g. someone wrote some API and they retired or changed companies. Some tooling may remain stagnant for some time. It's also a heavy lift to retrofit some enterprise environments for smaller changes so people fear the outages they will induce implementing IPv6. In some companies it is a major change just to migrate customers to a new load balancer endpoint. There may also be hundreds of undocumented things due to employee churn and lack of change control running that when broken will cause extended outages. And then there is internal politics and finger pointing... Again, not justifying it, rather I think there are too many moving bits and complexity that people have added over the decades and they are paralyzed by fear and risking the loss of their paycheck. And then there is the embarrassment that comes from having to acknowledge that nobody knows the current state of an environment and that embarrassment can go all the way up the organizational chain. That is based on my experience of being brought into companies with the speicifc task of, "Hey, make this simpler, reduce outages." It's rarely strictly a technical challenge but rather having to navigate politics, personality types and individual sub-org leaders that have had independent control of their environment for a long time. The more I think about it this could be a topic in and of itself how companies induce self inflicted bloat as they grow.
- talent_deprived 3y ago
  The biggest issue is IPv6 is a privacy, wide open wild west, there is no privacy on IPv6. Every device's IP is literally public, on the public Internet, 24/7. All so called privacy extensions or improvements do not change the lack of privacy of IPv6 and one more thing, the address structure sucks.
  - LinuxBender 3y ago
    FWIW this does not have to be true for companies that do not wish to expose internal nodes. I'm not even talking about the privacy extensions. I realize that people beat the drums that one must not NAT IPv6 but it can absolutely be a NAT just like IPv4. I would actually expect in most companies that they don't even add IPv6 inside their datacenters, rather they just put a block of IPv6 addresses on some load balancers at the edge and then point their edge devices L4 and L7 mapping to IPv4 nodes in their datacenter. The same could apply to VPN/WAN configurations in some cases much like how mobile networks are configured nowadays. There would need to be some of this regardless due to ACL's required between companies that don't use network VPN's for restricted end-points. e.g. PCI to PCI environments. In the early days of IPv4 many big companies did not NAT IPv4. I was at a company that did this. Our workstations all had routable public IPv4 addresses. There are still some companies that do this. One of these was a company that had 20 managers and VP's on a call when I just wanted to give their network engineer a CIDR block and a pre-shared secret for a network VPN. I suspect they will be using public IPv4 addresses internally forever. And I doubt they will ever sell their /8's unless people comment on their Vogon poetry. Amazon is probably one of the exceptions as they have so many geographically disperse configurations it would be harder to continue using RFC1918 address space. Not impossible, just difficult. I can think of a few other big companies that do manufacturing that are spread out around the world that would probably run into walls with RFC1918 at some point. I've seen some of them take over public address space for internal routing which then breaks access to some things on the internet thus requiring double/triple NATs. 1/8 assorted, 25/8 MoD, 26/8 DISA are a few I've seen.
    - SoftTalker 3y ago
baby_souffle 3y ago
It’s a big scary number and the letters just break understanding. There are some annoying operational issues around it as well, common one: DNS hostnames for devices that only do SLAAC
- WirelessGigabit 3y ago
  You wouldn't want to do SLAAC and hostnames. If your network card breaks you switch it out, and you make sure your IPv4 settings apply to the new card. If you fully rely on IPv6 you'll get a new address. And if your devices self-update DNS then you have to make sure they pick the right address, as there can be many due to privacy extensions. Lastly, combining privacy extensions plus hosting stuff is hard, as you don't know on which address a certain port is bound.
mr_mitm 3y ago
It's definitely "best practice" to turn off features you don't need. Who knows, maybe in five years someone will find a bad bug in the implementation of the IPv6 stack, then you'll be glad you decreased your attack surface. Not saying this is what you should do, just a common rationalization.
- icedchai 3y ago
  It's also "best practice" to learn a new, foundational technology (like IPv6) sooner than later, perhaps less than 20 years after it was first available.
  - dilyevsky 3y ago
    Right and what is better way to learn than deploying random features to prod!
    - icedchai 3y ago
      Did I say they had to deploy it to production? If they deployed it in dev and test environments even a few years ago, they might have some experience deploying it in prod today.
      - IntelMiner 3y ago
        It's only been ratified since 1998 and available as far back as Windows NT4 Pretty sure that's enough time to test it in development...
  - taway1237 3y ago
    I know IPv6, just don't feel the need to use it. I prefer to have one firewall and public network interface to worry about. I can't disable IPv4 yet, so the logical solution is to disable IPv6.
forty 3y ago
I have a reason: we do per IP rate limiting. It's easy enough for IPv4 when the number of IPs is necessarily not too big to fit in a small redis for example, but for IPv6 everyone have at least a /64. I'm curious how people do it btw, if you have tips to share, I'm all hear. Do you simply rate limit IP ranges? Even limiting per /64, it's still potentially quite a lot of /64 to track.
- Dylan16807 3y ago
  When a bunch of households or cell phones are on the same IPv4 do you have any measures to compensate? > Do you simply rate limit IP ranges? Even limiting per /64, it's still potentially quite a lot of /64 to track. Yes you'd limit by /64 or slightly larger. The live set of IPs shouldn't be very big.
  - forty 3y ago
    We put limits high enough that it's far enough for any expected usage, including a bunch of users on a single IP. If we see rate limiting happening in practice and it doesn't seem to be an attack, we revisit.
    - Dylan16807 3y ago
      Well it sounds like you'd do fine tracking the IPv6 blocks that are currently very active, without needing any significant amount of resources. If you go the extra mile and simultaneously track /64, /56, and /48 with moderately increasing thresholds, you'll probably end up causing less collateral damage when you block someone than with IPv4.
- tepmoc 3y ago
  You treat ipv6 /64 just like /32 in ipv4
- forty 3y ago
  Answering myself: I found this interesting article https://adam-p.ca/blog/2022/02/ipv6-rate-limiting/ https://adam-p.ca/blog/2022/02/ipv6-rate-limiting/
mschuster91 3y ago
Because there's a lot of shit that still doesn't work well with IPv4. Logging is one good example - a lot of software that uses its database for event logs has the database column for remote_ip defined as VARCHAR(15), you can guess the rest of what happens when deploying that with IPv6 enabled.
- icedchai 3y ago
  These things take time. I mean, it's only been 25+ years since the first IPv6 RFC was released...
aidenn0 3y ago
Mainly that everything is publicly routable. People see a 10.x and instantly know it can't be reached from the public internet. IPv6 is much harder. For internal-only stuff there is the fd00::/8 block, which AWS actually does use, but there is no equivalent range for outgoing-only connections.
Aeolun 3y ago
In the case of my corporate VPN, Teams doesn’t work with IPv6 enabled. Something on either the MS side, or the SSO one.

It's simple fear of the unknown. Many folks haven't learned it, so they just turn it off... Ignore it, kick the can down the road.

nerdponx 3y ago
It's individuals responding to the incentives before them. If something goes wrong because they're using IPv6, it's their fault. If you never upgrade anything until you're forced to, then you can never break anything by upgrading, and you're never seen as a "stuff breaker".
- tormeh 3y ago
  This is it. Having broken stuff by upgrading myself, I can say it does not make you popular. This is especially the case if what the upgrade did was tighten up some bug that it turns hid a bug in your team's code...

LinuxBender 3y ago

Why? What's the (supposed) fear? Disclaimer: I have never worked for Amazon but I can add some input from the assorted medium to large companies I have been employed with. I won't mention any names. This is for someone I know reads my comments wink wink I am not justifying it, just adding some of the bits I experienced. There are many security devices that do not have the same capabilities on IPv6 as IPv4 yet. Some enterprise IoT devices only support IPv4. Adding to this some network engineers don't want to step outside of their comfort zone and tooling/scripts to generate configurations automagically do not yet support IPv6. As the company grows they hire less Sr. Network Engineers and some of the new people depend on but do not understand the automation. e.g. someone wrote some API and they retired or changed companies. Some tooling may remain stagnant for some time. It's also a heavy lift to retrofit some enterprise environments for smaller changes so people fear the outages they will induce implementing IPv6. In some companies it is a major change just to migrate customers to a new load balancer endpoint. There may also be hundreds of undocumented things due to employee churn and lack of change control running that when broken will cause extended outages. And then there is internal politics and finger pointing... Again, not justifying it, rather I think there are too many moving bits and complexity that people have added over the decades and they are paralyzed by fear and risking the loss of their paycheck. And then there is the embarrassment that comes from having to acknowledge that nobody knows the current state of an environment and that embarrassment can go all the way up the organizational chain. That is based on my experience of being brought into companies with the speicifc task of, "Hey, make this simpler, reduce outages." It's rarely strictly a technical challenge but rather having to navigate politics, personality types and individual sub-org leaders that have had independent control of their environment for a long time. The more I think about it this could be a topic in and of itself how companies induce self inflicted bloat as they grow.

talent_deprived 3y ago
The biggest issue is IPv6 is a privacy, wide open wild west, there is no privacy on IPv6. Every device's IP is literally public, on the public Internet, 24/7. All so called privacy extensions or improvements do not change the lack of privacy of IPv6 and one more thing, the address structure sucks.
- LinuxBender 3y ago
  FWIW this does not have to be true for companies that do not wish to expose internal nodes. I'm not even talking about the privacy extensions. I realize that people beat the drums that one must not NAT IPv6 but it can absolutely be a NAT just like IPv4. I would actually expect in most companies that they don't even add IPv6 inside their datacenters, rather they just put a block of IPv6 addresses on some load balancers at the edge and then point their edge devices L4 and L7 mapping to IPv4 nodes in their datacenter. The same could apply to VPN/WAN configurations in some cases much like how mobile networks are configured nowadays. There would need to be some of this regardless due to ACL's required between companies that don't use network VPN's for restricted end-points. e.g. PCI to PCI environments. In the early days of IPv4 many big companies did not NAT IPv4. I was at a company that did this. Our workstations all had routable public IPv4 addresses. There are still some companies that do this. One of these was a company that had 20 managers and VP's on a call when I just wanted to give their network engineer a CIDR block and a pre-shared secret for a network VPN. I suspect they will be using public IPv4 addresses internally forever. And I doubt they will ever sell their /8's unless people comment on their Vogon poetry. Amazon is probably one of the exceptions as they have so many geographically disperse configurations it would be harder to continue using RFC1918 address space. Not impossible, just difficult. I can think of a few other big companies that do manufacturing that are spread out around the world that would probably run into walls with RFC1918 at some point. I've seen some of them take over public address space for internal routing which then breaks access to some things on the internet thus requiring double/triple NATs. 1/8 assorted, 25/8 MoD, 26/8 DISA are a few I've seen.
  - SoftTalker 3y ago

baby_souffle 3y ago

It’s a big scary number and the letters just break understanding. There are some annoying operational issues around it as well, common one: DNS hostnames for devices that only do SLAAC

WirelessGigabit 3y ago
You wouldn't want to do SLAAC and hostnames. If your network card breaks you switch it out, and you make sure your IPv4 settings apply to the new card. If you fully rely on IPv6 you'll get a new address. And if your devices self-update DNS then you have to make sure they pick the right address, as there can be many due to privacy extensions. Lastly, combining privacy extensions plus hosting stuff is hard, as you don't know on which address a certain port is bound.

mr_mitm 3y ago

It's definitely "best practice" to turn off features you don't need. Who knows, maybe in five years someone will find a bad bug in the implementation of the IPv6 stack, then you'll be glad you decreased your attack surface. Not saying this is what you should do, just a common rationalization.

icedchai 3y ago
It's also "best practice" to learn a new, foundational technology (like IPv6) sooner than later, perhaps less than 20 years after it was first available.
- dilyevsky 3y ago
  Right and what is better way to learn than deploying random features to prod!
  - icedchai 3y ago
    Did I say they had to deploy it to production? If they deployed it in dev and test environments even a few years ago, they might have some experience deploying it in prod today.
    - IntelMiner 3y ago
      It's only been ratified since 1998 and available as far back as Windows NT4 Pretty sure that's enough time to test it in development...
- taway1237 3y ago
  I know IPv6, just don't feel the need to use it. I prefer to have one firewall and public network interface to worry about. I can't disable IPv4 yet, so the logical solution is to disable IPv6.

forty 3y ago

I have a reason: we do per IP rate limiting. It's easy enough for IPv4 when the number of IPs is necessarily not too big to fit in a small redis for example, but for IPv6 everyone have at least a /64. I'm curious how people do it btw, if you have tips to share, I'm all hear. Do you simply rate limit IP ranges? Even limiting per /64, it's still potentially quite a lot of /64 to track.

Dylan16807 3y ago
When a bunch of households or cell phones are on the same IPv4 do you have any measures to compensate? > Do you simply rate limit IP ranges? Even limiting per /64, it's still potentially quite a lot of /64 to track. Yes you'd limit by /64 or slightly larger. The live set of IPs shouldn't be very big.
- forty 3y ago
  We put limits high enough that it's far enough for any expected usage, including a bunch of users on a single IP. If we see rate limiting happening in practice and it doesn't seem to be an attack, we revisit.
  - Dylan16807 3y ago
    Well it sounds like you'd do fine tracking the IPv6 blocks that are currently very active, without needing any significant amount of resources. If you go the extra mile and simultaneously track /64, /56, and /48 with moderately increasing thresholds, you'll probably end up causing less collateral damage when you block someone than with IPv4.
tepmoc 3y ago
You treat ipv6 /64 just like /32 in ipv4
forty 3y ago
Answering myself: I found this interesting article https://adam-p.ca/blog/2022/02/ipv6-rate-limiting/ https://adam-p.ca/blog/2022/02/ipv6-rate-limiting/

mschuster91 3y ago

Because there's a lot of shit that still doesn't work well with IPv4. Logging is one good example - a lot of software that uses its database for event logs has the database column for remote_ip defined as VARCHAR(15), you can guess the rest of what happens when deploying that with IPv6 enabled.

icedchai 3y ago
These things take time. I mean, it's only been 25+ years since the first IPv6 RFC was released...

aidenn0 3y ago

Mainly that everything is publicly routable. People see a 10.x and instantly know it can't be reached from the public internet. IPv6 is much harder. For internal-only stuff there is the fd00::/8 block, which AWS actually does use, but there is no equivalent range for outgoing-only connections.

Aeolun 3y ago

In the case of my corporate VPN, Teams doesn’t work with IPv6 enabled. Something on either the MS side, or the SSO one.