5 ms·

GitHub is preparing for IPv6 support for Github.com

111 points by tonymet 3y ago

tonymet 3y ago
"This incident was the result of an infrastructure change that was made to our load balancers to prepare us for IPv6 enablement of GitHub.com. This change was deployed to a subset of our global edge sites."
kaetemi 3y ago
Seems like this is a common error, systems that treat IPv4 and IPv6 addresses as entirely distinct things. The mapping is easily missed in new implementations. Wonder how many vulnerabilities there are out there with this same mistake.
- superjan 3y ago
  Yes, and what other vulnerabilities would be out there? IP6 is something that nearly all machines have, and have enabled, but it is rarely used.
  - aranelsurion 3y ago
    One potential scenario is bypassing rate limits, enabling abuse, scraping etc. because of the large IP blocks each customer is granted, and seemingly the lack of standardization there. The assumption of there being a finite amount of IPs to cycle for an attack doesn’t entirely hold with IPv6, and seems like not all software is configured to take that into account. Explained here better: https://adam-p.ca/blog/2022/02/ipv6-rate-limiting/ https://adam-p.ca/blog/2022/02/ipv6-rate-limiting/
    - hot_gril 3y ago
      Unsavory but true, it's nice in a way that ip addrs have a cost. Kinda like phone numbers. The naive suggestion for ipv6 in that article is what I first thought, block a /64, but since that might still be too cheap, it also quotes a more robust algo of dynamically expanding the blocking scope.
- arkadiyt 3y ago
  It's a common bypass of server side request forgery filtering. Backends will try to validate that a user-submitted url doesn't resolve to an internal IPv4 address, but they'll happily allow an IPv6 mapped version for the same IPv4 address.
roboben 3y ago
I can just imagine that moving to IPv6 is as messy on Azure than it is on AWS. Don’t get me wrong, actually using it is simpler, you can just have your range and then be free to do whatever you want. But the fact that it was added later to a stack never made for it shows everywhere, which makes UX tough. Currently struggling with metadata service weirdness and slowness in IPv6 land on AWS :’)
- jeroenhd 3y ago
  With how weirdly complicated Azure was about IPv6 last time I looked into it, I wouldn't be surprised if it'd be more complicated than on AWS. I'm not convinced it ever improved. Looking up a quick guide brings up https://learn.microsoft.com/en-us/azure/virtual-network/ip-services/create-vm-dual-stack-ipv6-portal https://learn.microsoft.com/en-us/azure/virtual-network/ip-s... which tells you to just... assign a random network from 2404:f800::? What even is this network? Are they using a routable IPv6 address as a substitute for an ULA for their NAT'ing load balancers? Why 2404:f800:8000:122::/64 specifically?
  - WirelessGigabit 3y ago
    Microsoft owns 2404:f800::/30. I just did the tutorial and I noticed that in the Azure portal it shows a public IPv6 address and a private IPv6 address. From my machine I connect to the public one and magically end up on the private one. Curling what is my ip6 from the machine yield the public IPv6 address. I suppose all of this is needed to ensure LB can be done? And it's easier to do with a range like this than a ULA which by default isn't routable.
    - jeroenhd 3y ago
      They do, but they tell you to use a specific /64, which confuses me. I would've expected them to say something like "the address starting with 2404:f800 in your dashboard". The "private" IPv6 address can be a ULA without any issues if the network is designed to be fully NAT'ed (i.e. for load balancing, maybe failover I guess). If you're not using the global address on your local machine and translate the public address into a private one, your local network doesn't need to have a routable IP address. I suppose it works just as well, but it makes using IPv6 more confusing for now reason. It's as if Microsoft decide to use 20.64.0.0/10 for private networking on Azure, which they can do (they own that space after all, they can decide not to use it), but just doesn't make much sense.
everfrustrated 3y ago
Seriously what is going on at GitHub engineering? How was this not caught in a pre-production environment?
KingOfCoders 3y ago
Funny, around 25 years ago as a university project I added IPv6 support to the Java JDK. Today we're still not there.
- echelon 3y ago
  On the scale of migrations, this is worse than Python 2 to 3 (which was really bad), but better than Perl to Raku (the relevancy is entirely gone). Perhaps a smaller migration should have been considered first.
  - jowea 3y ago
    At least py3 runs on the same hardware.
    - hot_gril 3y ago
      Most hardware supports ipv6 nowadays (I think), but the problem is something else.
  - hot_gril 3y ago
    I think the problem with Py2 to Py3 was that it was too big of a migration to be easy but also too small of a migration to be game-changing. Like, switching for seriously improved packaging, parallelism, or performance would've been more compelling. And yeah ipv6 could've been a much smaller change.
  - unixhero 3y ago
    Yeah did anyone make the switch to Raku??? I don't really understand it.
- p1mrx 3y ago
  Java's not there either: https://bugs.openjdk.org/browse/JDK-8170568 https://bugs.openjdk.org/browse/JDK-8170568
zokier 3y ago
But but all the people on the interwebs told me that ipv6 is ugly and stupid and it's never going to happen!
- gertop 3y ago
  > But but all the people on the interwebs told me that ipv6 is ugly and stupid and it's never going to happen! And that article proves them right.
  - hot_gril 3y ago
    This article isn't proof, or even a very good example of ipv6's issues.
  - orangeboats 3y ago
    Terrible reasoning. If the article has truly proven them right, GitHub will have not experimented with enabling IPv6 in the first place.
  - AndrewDucker 3y ago
    There was a 42 minute downtime for 0.23% of their user base. Hardly "never going to happen".
barkingcat 3y ago
Sad that even for experts ipv6 is still a no go filled with landmines.
- colonwqbang 3y ago
  Any tool can be used incorrectly. IPv4 mapped addresses are a part of the protocol as any "expert" would know. How many bugs occur due to misconfigured HTTP servers every day? Do you conclude that HTTP is also a "no go"?
  - josefx 3y ago
    Which http are we talking about? HTTP3 that people wanted to use before the security portion of the spec. was even done? HTTP2 that had an entire section dedicated to possible exploits that where later found on major sites? Or HTTP 1 which enabled some of the exploits seen with 2 since implementations are neither consistent nor strict when parsing parameters? Sometimes it feels as if the Internets security was supported by paper walls with a "please punch here to exploit" written all over the place.
    - hot_gril 3y ago
      Was gonna say, if this is about ipv4 vs ipv6, better also specify which version of HTTP we're talking about.
      - orangeboats 3y ago
        Does it matter? Every HTTP version from HTTP/1 to HTTP/3 has each experienced its own set of vulnerabilities. Never stopped us from using HTTP. IPv4 has had multiple vulnerabilities in its stack as well, never stopped the internet from growing. (But the exhaustion of IPv4 _is_ hindering further expansion of the internet!)
        hot_gril 3y ago
        I mean it's not a good comparison. IPv4->6 is a migration, with a lot of difficulty having to do with the fact that v4 is already in use. HTTP wasn't a replacement for something, but there is HTTP1->2.
artsi0m 3y ago
I am not gonna read this postmortem, but post link to this HN thread as a proof of mass IPv6 adoption.
- artsi0m 3y ago
  Sorry, I thought that it would sound like a fine joke. Don't want to offend someone
supriyo-biswas 3y ago
I’m curious how this wasn’t caught in testing. When you set up a dual-stacked listener which is common when you omit the binding IP, you already get IPv4-mapped IPv6 address, and the request logs would have also shown the ::ffff:x.x.x.x addresses. Also curious as to why they are working with text representations of IP addresses instead of binary representations, although the incident GitHub described here could be very well a firewall deciding that IPv6 datagrams were invalid traffic. However, to make this thread complete, I guess someone will again complain about the address format, without realizing that shoving in extra address bits on an IPv4 datagram is already a new protocol, the networking equivalent of Godwin’s law[1]. [1] https://news.ycombinator.com/item?id=39101523 https://news.ycombinator.com/item?id=39101523
- hot_gril 3y ago
  I'll be that guy. Glad to know I'm not the only one who just wants ipv4 with extra octets. Edit: An aside, these addresses are different in both text or binary representation, but the article doesn't provide many details, so I can't entirely follow.
  - dessimus 3y ago
    The other big problem I see to getting buy-in from non-networking people is the lack of good sources on how to prepare and roll out ipv6. It takes forever scrolling through pages of documentation that go through the history of ipv6 and the ipv6 numbering format. As though the world needs yet another primer on how long groups of 0's can be dropped to make ::1 be a valid address. Just link to Wikipedia's page explaining ipv6 addressing and move on.
israrkhan 3y ago
NAT is the main reason slowing down (or preventing) IPV6 adoption.
- hot_gril 3y ago
  NAT is a nice feature for home and office. Safe default-deny behavior, clearly separated LAN. I'd even want it with ipv6.
  - miyuru 3y ago
    Don't you have any firewalls? My ISPs router blocks inbound connections by default on both address types.
    - hot_gril 3y ago
      I have firewalls but don't trust them to have safe defaults. Or more like, I don't trust every random router, including ones that each member of my extended family will inevitably call me about, to have safe firewall defaults until I test it myself. I reasonably trust, not 100% but pretty well, that my boss's or grandma's PC with ip 192.168.1.3 on NAT (not even routable from the WAN) will not receive inbound TCP/UDP. That would take effort.
      - orangeboats 3y ago
        Following this logic, don't trust your grandma's router to have UPnP and NAT-PMP disabled by default too. NAT becomes effectively unsecure with them. This argument is so overused we might as well add it to ipv6-excuses.txt.
        hot_gril 3y ago
        Upnp is where the shred of doubt comes from. Only a shred because the PC very likely won't use it, and nowadays few routers have it by default. And the solution is, upnp shouldn't exist.
omeid2 3y ago
IPv6 will never replace IPv4 for good. It is a non-solution that we will have to contend with along with IPv4 until something proper is worked out.
- orangeboats 3y ago
  Oh no, is this yet another proposal of IPv4 with extra octets?
  - omeid2 3y ago
    No. The notion that it must be IPv6 or extra octets is naive. IPv6 is not worth a complete departure, and it won't happen for that reason.
rodlette 3y ago
> The change had the unintended consequence of causing IPv4 addresses to start being passed as an IPv4-mapped IPv6-compatible address to our IP Allow List functionality. Sounds like this bug: "Unable to reliably distinguish IPv4-mapped-IPv6 addresses from regular IPv4 addresses" https://github.com/golang/go/issues/37921 https://github.com/golang/go/issues/37921 . Use https://pkg.go.dev/net/netip https://pkg.go.dev/net/netip instead.
- p1mrx 3y ago
  Sounds like the opposite of this bug. GitHub's system represents 0.0.0.0/0 and ::ffff:0.0.0.0/96 as distinct values, when they should be merged/normalized. I think it's best to treat IPv4-mapped addresses as an implementation detail of the socket API, and not leak them into general purpose code. Though things get interesting when an external user provides "::ffff:1.2.3.4" as text.
tonymet 3y ago
I created a realtime app for us to track https://isgithubipv6.live/ https://isgithubipv6.live/

111 points by tonymet 3y ago

"This incident was the result of an infrastructure change that was made to our load balancers to prepare us for IPv6 enablement of GitHub.com. This change was deployed to a subset of our global edge sites."

kaetemi 3y ago

Seems like this is a common error, systems that treat IPv4 and IPv6 addresses as entirely distinct things. The mapping is easily missed in new implementations. Wonder how many vulnerabilities there are out there with this same mistake.

superjan 3y ago
Yes, and what other vulnerabilities would be out there? IP6 is something that nearly all machines have, and have enabled, but it is rarely used.
- aranelsurion 3y ago
  One potential scenario is bypassing rate limits, enabling abuse, scraping etc. because of the large IP blocks each customer is granted, and seemingly the lack of standardization there. The assumption of there being a finite amount of IPs to cycle for an attack doesn’t entirely hold with IPv6, and seems like not all software is configured to take that into account. Explained here better: https://adam-p.ca/blog/2022/02/ipv6-rate-limiting/ https://adam-p.ca/blog/2022/02/ipv6-rate-limiting/
  - hot_gril 3y ago
    Unsavory but true, it's nice in a way that ip addrs have a cost. Kinda like phone numbers. The naive suggestion for ipv6 in that article is what I first thought, block a /64, but since that might still be too cheap, it also quotes a more robust algo of dynamically expanding the blocking scope.
arkadiyt 3y ago
It's a common bypass of server side request forgery filtering. Backends will try to validate that a user-submitted url doesn't resolve to an internal IPv4 address, but they'll happily allow an IPv6 mapped version for the same IPv4 address.

roboben 3y ago

I can just imagine that moving to IPv6 is as messy on Azure than it is on AWS. Don’t get me wrong, actually using it is simpler, you can just have your range and then be free to do whatever you want. But the fact that it was added later to a stack never made for it shows everywhere, which makes UX tough. Currently struggling with metadata service weirdness and slowness in IPv6 land on AWS :’)

jeroenhd 3y ago
With how weirdly complicated Azure was about IPv6 last time I looked into it, I wouldn't be surprised if it'd be more complicated than on AWS. I'm not convinced it ever improved. Looking up a quick guide brings up https://learn.microsoft.com/en-us/azure/virtual-network/ip-services/create-vm-dual-stack-ipv6-portal https://learn.microsoft.com/en-us/azure/virtual-network/ip-s... which tells you to just... assign a random network from 2404:f800::? What even is this network? Are they using a routable IPv6 address as a substitute for an ULA for their NAT'ing load balancers? Why 2404:f800:8000:122::/64 specifically?
- WirelessGigabit 3y ago
  Microsoft owns 2404:f800::/30. I just did the tutorial and I noticed that in the Azure portal it shows a public IPv6 address and a private IPv6 address. From my machine I connect to the public one and magically end up on the private one. Curling what is my ip6 from the machine yield the public IPv6 address. I suppose all of this is needed to ensure LB can be done? And it's easier to do with a range like this than a ULA which by default isn't routable.
  - jeroenhd 3y ago
    They do, but they tell you to use a specific /64, which confuses me. I would've expected them to say something like "the address starting with 2404:f800 in your dashboard". The "private" IPv6 address can be a ULA without any issues if the network is designed to be fully NAT'ed (i.e. for load balancing, maybe failover I guess). If you're not using the global address on your local machine and translate the public address into a private one, your local network doesn't need to have a routable IP address. I suppose it works just as well, but it makes using IPv6 more confusing for now reason. It's as if Microsoft decide to use 20.64.0.0/10 for private networking on Azure, which they can do (they own that space after all, they can decide not to use it), but just doesn't make much sense.

everfrustrated 3y ago

Seriously what is going on at GitHub engineering? How was this not caught in a pre-production environment?

KingOfCoders 3y ago

Funny, around 25 years ago as a university project I added IPv6 support to the Java JDK. Today we're still not there.

echelon 3y ago
On the scale of migrations, this is worse than Python 2 to 3 (which was really bad), but better than Perl to Raku (the relevancy is entirely gone). Perhaps a smaller migration should have been considered first.
- jowea 3y ago
  At least py3 runs on the same hardware.
  - hot_gril 3y ago
    Most hardware supports ipv6 nowadays (I think), but the problem is something else.
- hot_gril 3y ago
  I think the problem with Py2 to Py3 was that it was too big of a migration to be easy but also too small of a migration to be game-changing. Like, switching for seriously improved packaging, parallelism, or performance would've been more compelling. And yeah ipv6 could've been a much smaller change.
- unixhero 3y ago
  Yeah did anyone make the switch to Raku??? I don't really understand it.
p1mrx 3y ago
Java's not there either: https://bugs.openjdk.org/browse/JDK-8170568 https://bugs.openjdk.org/browse/JDK-8170568

zokier 3y ago

But but all the people on the interwebs told me that ipv6 is ugly and stupid and it's never going to happen!

gertop 3y ago
> But but all the people on the interwebs told me that ipv6 is ugly and stupid and it's never going to happen! And that article proves them right.
- hot_gril 3y ago
  This article isn't proof, or even a very good example of ipv6's issues.
- orangeboats 3y ago
  Terrible reasoning. If the article has truly proven them right, GitHub will have not experimented with enabling IPv6 in the first place.
- AndrewDucker 3y ago
  There was a 42 minute downtime for 0.23% of their user base. Hardly "never going to happen".

barkingcat 3y ago

Sad that even for experts ipv6 is still a no go filled with landmines.

colonwqbang 3y ago
Any tool can be used incorrectly. IPv4 mapped addresses are a part of the protocol as any "expert" would know. How many bugs occur due to misconfigured HTTP servers every day? Do you conclude that HTTP is also a "no go"?
- josefx 3y ago
  Which http are we talking about? HTTP3 that people wanted to use before the security portion of the spec. was even done? HTTP2 that had an entire section dedicated to possible exploits that where later found on major sites? Or HTTP 1 which enabled some of the exploits seen with 2 since implementations are neither consistent nor strict when parsing parameters? Sometimes it feels as if the Internets security was supported by paper walls with a "please punch here to exploit" written all over the place.
  - hot_gril 3y ago
    Was gonna say, if this is about ipv4 vs ipv6, better also specify which version of HTTP we're talking about.
    - orangeboats 3y ago
      Does it matter? Every HTTP version from HTTP/1 to HTTP/3 has each experienced its own set of vulnerabilities. Never stopped us from using HTTP. IPv4 has had multiple vulnerabilities in its stack as well, never stopped the internet from growing. (But the exhaustion of IPv4 _is_ hindering further expansion of the internet!)
      - hot_gril 3y ago
        I mean it's not a good comparison. IPv4->6 is a migration, with a lot of difficulty having to do with the fact that v4 is already in use. HTTP wasn't a replacement for something, but there is HTTP1->2.

artsi0m 3y ago

I am not gonna read this postmortem, but post link to this HN thread as a proof of mass IPv6 adoption.

artsi0m 3y ago
Sorry, I thought that it would sound like a fine joke. Don't want to offend someone

supriyo-biswas 3y ago

I’m curious how this wasn’t caught in testing. When you set up a dual-stacked listener which is common when you omit the binding IP, you already get IPv4-mapped IPv6 address, and the request logs would have also shown the ::ffff:x.x.x.x addresses. Also curious as to why they are working with text representations of IP addresses instead of binary representations, although the incident GitHub described here could be very well a firewall deciding that IPv6 datagrams were invalid traffic. However, to make this thread complete, I guess someone will again complain about the address format, without realizing that shoving in extra address bits on an IPv4 datagram is already a new protocol, the networking equivalent of Godwin’s law[1]. [1] https://news.ycombinator.com/item?id=39101523 https://news.ycombinator.com/item?id=39101523

hot_gril 3y ago
I'll be that guy. Glad to know I'm not the only one who just wants ipv4 with extra octets. Edit: An aside, these addresses are different in both text or binary representation, but the article doesn't provide many details, so I can't entirely follow.
- dessimus 3y ago
  The other big problem I see to getting buy-in from non-networking people is the lack of good sources on how to prepare and roll out ipv6. It takes forever scrolling through pages of documentation that go through the history of ipv6 and the ipv6 numbering format. As though the world needs yet another primer on how long groups of 0's can be dropped to make ::1 be a valid address. Just link to Wikipedia's page explaining ipv6 addressing and move on.

israrkhan 3y ago

NAT is the main reason slowing down (or preventing) IPV6 adoption.

hot_gril 3y ago
NAT is a nice feature for home and office. Safe default-deny behavior, clearly separated LAN. I'd even want it with ipv6.
- miyuru 3y ago
  Don't you have any firewalls? My ISPs router blocks inbound connections by default on both address types.
  - hot_gril 3y ago
    I have firewalls but don't trust them to have safe defaults. Or more like, I don't trust every random router, including ones that each member of my extended family will inevitably call me about, to have safe firewall defaults until I test it myself. I reasonably trust, not 100% but pretty well, that my boss's or grandma's PC with ip 192.168.1.3 on NAT (not even routable from the WAN) will not receive inbound TCP/UDP. That would take effort.
    - orangeboats 3y ago
      Following this logic, don't trust your grandma's router to have UPnP and NAT-PMP disabled by default too. NAT becomes effectively unsecure with them. This argument is so overused we might as well add it to ipv6-excuses.txt.
      - hot_gril 3y ago
        Upnp is where the shred of doubt comes from. Only a shred because the PC very likely won't use it, and nowadays few routers have it by default. And the solution is, upnp shouldn't exist.

omeid2 3y ago

IPv6 will never replace IPv4 for good. It is a non-solution that we will have to contend with along with IPv4 until something proper is worked out.

orangeboats 3y ago
Oh no, is this yet another proposal of IPv4 with extra octets?
- omeid2 3y ago
  No. The notion that it must be IPv6 or extra octets is naive. IPv6 is not worth a complete departure, and it won't happen for that reason.

rodlette 3y ago

> The change had the unintended consequence of causing IPv4 addresses to start being passed as an IPv4-mapped IPv6-compatible address to our IP Allow List functionality. Sounds like this bug: "Unable to reliably distinguish IPv4-mapped-IPv6 addresses from regular IPv4 addresses" https://github.com/golang/go/issues/37921 https://github.com/golang/go/issues/37921 . Use https://pkg.go.dev/net/netip https://pkg.go.dev/net/netip instead.

p1mrx 3y ago
Sounds like the opposite of this bug. GitHub's system represents 0.0.0.0/0 and ::ffff:0.0.0.0/96 as distinct values, when they should be merged/normalized. I think it's best to treat IPv4-mapped addresses as an implementation detail of the socket API, and not leak them into general purpose code. Though things get interesting when an external user provides "::ffff:1.2.3.4" as text.

tonymet 3y ago

I created a realtime app for us to track https://isgithubipv6.live/ https://isgithubipv6.live/