4 ms·

I have started working for a startup recently. My main responsibility is to develop networking features for our cloud on bare metal. We started ipv6 by default

I have started working for a startup recently. My main responsibility is to develop networking features for our cloud on bare metal. We started ipv6 by default but soon we discovered that the biggest issue is "not" the setup side. Ipv6 setup is actually quite straightforward, if you're starting from scratch. The biggest problem of ipv6 is that the ecosystem is not ready for it, at all. You cannot even use github without a proxy! Hence, we had to start implementing ipv4 support immediately, because VMs for developers that only has ipv6 is almost useless.

GolDDranks 3y ago
Yeah, the situation is pretty awful for something so big and not lacking of technical talent as GitHub :(
brk 3y ago
Yeah, it's a real shit show when you get down to actually trying to utilize IPv6 in any scenario that needs legacy IPv4 access in a straight-forward way. I'm somewhat happy in that I've moved away from being way down at the low-level ISP/network side of things, so I may be missing something, but I don't see how we are ever going to elegantly transition away from IPv4 addresses. Everything just seems hacky and fragile in terms of trying to run a "pure" IPv6 environment, and be connected to the rest of the Internet.
- GolDDranks 3y ago
  I think that ISP-wide single-stack IPv6 deployments are the key. They throw the legacy IPv4 internet behind a huge NAT, while letting IPv6 internet to function natively. There is an IPv6 address range that represents the totality of the legacy IPv4 addresses, and the IPv4 addresses are translated into that, so from the IPv6 side of the NAT, every IPv6 service looks like it has an IPv6 address. From the IPv4 side, it looks like the your standard carrier-grade NAT, with huge numbers of users sharing IPv4 addresses. That should simplify the network over dual-stack deploys, plus it makes providing services in "native" IPv6 the more attractive choice over NATted-to-death IPv4. There are already some ISPs doing this. In Japan, where I live, one of the big three, NTT Docomo transitioned into a deployment like this just last year.
  - toast0 3y ago
    Exactly this. As more ISPs go through this process, service providers are incentivised to provide v6 services or get poor service. As things continue in that pattern, v6 becomes dominant and v4 becomes the afterthought to be neglected on the sideline (like v6 is currently for many ISPs with lots of v4 resources)
  - ianburrell 3y ago
    The protocol is called MAP and it is nearly stateless, cause the IPv4 address is in the IPv6 packet, and more efficient than carrier-grade NAT. The downside is that requires special gateway on customer side. The advantage is that ISP can have IPv6 internal network with IPv4 on the edges.
    - GolDDranks 3y ago
      Oh, I was talking about NAT64/DNS64 which is what NTT Docomo is doing. Now I got to read about MAP... But how can the mapping be almost stateless? When packets are sent from IPv6 to IPv4, if there is significant exhaustion, multiple IPv6 address are going to map into a single IPv4 address, unless the ISP has enough for everybody, in which case I don't see the motivation to hasten moving to IPv6? Edit: Ah, uses port numbers as extra bytes to encode the original senders address information. Smart!
hinata08 3y ago
I would be surprised if gitlab didn't support ipv6. Some of the ecosystem must be ready for it, and ipv6 support can be just another requirement to choose among solutions. Also, you can have a reverse proxy and a cloud behind NAT64 to run servers on ipv4, but access them with ipv6.
- pjc50 3y ago
  Do "dig -t AAAA github.com": you'll see they have no AAAA records. No github for you. Why then should anyone else bother unless forced by cost?
  - JimDabell 3y ago
    They were talking about GitLab in comparison to GitHub, not talking about GitHub. GitLab has AAAA records.
- Habgdnv 3y ago
  GitLab supports ipv6, but just not out of the box. My private gitlab instance is v6 only. I had problems with the updates in the beginning because because the official repo was not on V6 but I think they fixed it some years ago. (I am not gitlab expert, but i think my type of install is called Omnibus)
throw0101b 3y ago
Is/was DNS64+NAT64 a viable way option for your use case?
crote 3y ago
> You cannot even use github without a proxy! Luckily that does not seem to be an issue here. You only have to pay for a public IPv4 address, you still have a full IPv4 stack and are able to make outbound connections via NAT.
- ArchOversight 3y ago
  AWS won't give you a NAT without you paying for it (NAT Gateway) which requires a public IPv4 address (which you will soon have to pay for). There ain't no such thing as a free lunch.
  - jeroenhd 3y ago
    I don't think you need that based on https://aws.amazon.com/blogs/aws/let-your-ipv6-only-workloads-connect-to-ipv4-services/ https://aws.amazon.com/blogs/aws/let-your-ipv6-only-workload... It's possible that the business model has changed, but at least on this blog they say it's available free of cost.
    - yorwba 3y ago
      Not free of cost, free of additional cost if you were already using a NAT Gateway before they added NAT64 capability.
    - tedivm 3y ago
      That article makes it clear that the NAT Gateway needs IPv4 > The NAT gateway recognizes the IPv6 address prefix, extracts the IPv4 address from it, and initiates an IPv4 connection to the destination. As usual, the source IPv4 address is the IPv4 address of the NAT gateway itself. If the NAT Gateway doesn't have an IPv4 address it won't work.
    - ArchOversight 3y ago
      > Pricing and Availability > These two new capabilities to the VPC NAT gateway and Route 53 are available today in all AWS Regions at no additional costs. Regular NAT gateway charges may apply. Specifically: > Regular NAT gateway charges may apply. Which should say: "Regular NAT gateway charges apply" Since you still pay for the traffic that is processed by the NAT gateway (per GiB).
jeroenhd 3y ago
Github is one of the most idiotic IPv4 exclusive services. Microsoft and Azure has all the knowledge and equipment to make IPv6 available to practically any site, but Github seems afraid to ask. They had IPv6 for a short while and turned it off later. https://github.com/orgs/community/discussions/10539 https://github.com/orgs/community/discussions/10539 is full of people voicing their grievances but I don't think Github is paying this issue any attention anymore. Luckily almost all providers or IPv6-only networks also offer NAT64 or similar NAT mechanisms to make IPv4 addresses reachable.
- zamalek 3y ago
  ZScaler is worse on a practical level. I have to disable IPv6 system-wide else I can't access internal services (it only routes IPv4). The crazy thing is that they call out VPNs for being archaic, but force users to use an even more archaic technology.
  - MandieD 3y ago
    Bless you for mentioning that ZScaler doesn’t play nice with IPv6 - this gives me another possible cause for why a certain MSFT distributed app sometimes, but not always, installs on our users’ PCs when ZScaler is enabled, but works with it off. Thanks so much!
- jiggawatts 3y ago
  Microsoft used to “get” IPv6 and they added support to Windows back in the very early 2000s. All those engineers left or retired and have been replaced by outsourcers and H1Bs. Nobody at Azure can even spell IPv6. Turning it on is difficult and then it breaks everything, including unrelated services in other peered vnets. It’s just shameful how much the engineering skill has degraded in Redmond.
  - nine_zeros 3y ago
    As long as the stock goes up, no-fin-body cares.
Habgdnv 3y ago
I recently tried to deploy GitLab from scratch on an IPv6-only network, and the initial experience was anything but smooth. I was met with an exception right in the console during the initial setup. GitLab attempted to obtain a Let's Encrypt certificate and immediately failed, as it doesn't listen to IPv6 addresses by default. A year ago, we (at work) faced similar issues when trying to deploy GlusterFS on an IPv6-only network, and it also failed. (I pushed for V6 only, my manager was not happy) It's evident that while IPv6 may be the future, the present ecosystem doesn't seem fully prepared to support it. For years, I have wanted to use Docker with IPv6 only, and I am really thinking about learning Go so I can write my own IPv6-only driver.
- probably_a_gpt 3y ago
  I’m getting python 2 v 3 flashbacks
arianvanp 3y ago
AWS NAT Gateway comes with native NAT64 support so GitHub just works over ipv6. No issues whatsoever.

GolDDranks 3y ago

Yeah, the situation is pretty awful for something so big and not lacking of technical talent as GitHub :(

brk 3y ago

Yeah, it's a real shit show when you get down to actually trying to utilize IPv6 in any scenario that needs legacy IPv4 access in a straight-forward way. I'm somewhat happy in that I've moved away from being way down at the low-level ISP/network side of things, so I may be missing something, but I don't see how we are ever going to elegantly transition away from IPv4 addresses. Everything just seems hacky and fragile in terms of trying to run a "pure" IPv6 environment, and be connected to the rest of the Internet.

GolDDranks 3y ago
I think that ISP-wide single-stack IPv6 deployments are the key. They throw the legacy IPv4 internet behind a huge NAT, while letting IPv6 internet to function natively. There is an IPv6 address range that represents the totality of the legacy IPv4 addresses, and the IPv4 addresses are translated into that, so from the IPv6 side of the NAT, every IPv6 service looks like it has an IPv6 address. From the IPv4 side, it looks like the your standard carrier-grade NAT, with huge numbers of users sharing IPv4 addresses. That should simplify the network over dual-stack deploys, plus it makes providing services in "native" IPv6 the more attractive choice over NATted-to-death IPv4. There are already some ISPs doing this. In Japan, where I live, one of the big three, NTT Docomo transitioned into a deployment like this just last year.
- toast0 3y ago
  Exactly this. As more ISPs go through this process, service providers are incentivised to provide v6 services or get poor service. As things continue in that pattern, v6 becomes dominant and v4 becomes the afterthought to be neglected on the sideline (like v6 is currently for many ISPs with lots of v4 resources)
- ianburrell 3y ago
  The protocol is called MAP and it is nearly stateless, cause the IPv4 address is in the IPv6 packet, and more efficient than carrier-grade NAT. The downside is that requires special gateway on customer side. The advantage is that ISP can have IPv6 internal network with IPv4 on the edges.
  - GolDDranks 3y ago
    Oh, I was talking about NAT64/DNS64 which is what NTT Docomo is doing. Now I got to read about MAP... But how can the mapping be almost stateless? When packets are sent from IPv6 to IPv4, if there is significant exhaustion, multiple IPv6 address are going to map into a single IPv4 address, unless the ISP has enough for everybody, in which case I don't see the motivation to hasten moving to IPv6? Edit: Ah, uses port numbers as extra bytes to encode the original senders address information. Smart!

hinata08 3y ago

I would be surprised if gitlab didn't support ipv6. Some of the ecosystem must be ready for it, and ipv6 support can be just another requirement to choose among solutions. Also, you can have a reverse proxy and a cloud behind NAT64 to run servers on ipv4, but access them with ipv6.

pjc50 3y ago
Do "dig -t AAAA github.com": you'll see they have no AAAA records. No github for you. Why then should anyone else bother unless forced by cost?
- JimDabell 3y ago
  They were talking about GitLab in comparison to GitHub, not talking about GitHub. GitLab has AAAA records.
Habgdnv 3y ago
GitLab supports ipv6, but just not out of the box. My private gitlab instance is v6 only. I had problems with the updates in the beginning because because the official repo was not on V6 but I think they fixed it some years ago. (I am not gitlab expert, but i think my type of install is called Omnibus)

throw0101b 3y ago

Is/was DNS64+NAT64 a viable way option for your use case?

crote 3y ago

> You cannot even use github without a proxy! Luckily that does not seem to be an issue here. You only have to pay for a public IPv4 address, you still have a full IPv4 stack and are able to make outbound connections via NAT.

ArchOversight 3y ago
AWS won't give you a NAT without you paying for it (NAT Gateway) which requires a public IPv4 address (which you will soon have to pay for). There ain't no such thing as a free lunch.
- jeroenhd 3y ago
  I don't think you need that based on https://aws.amazon.com/blogs/aws/let-your-ipv6-only-workloads-connect-to-ipv4-services/ https://aws.amazon.com/blogs/aws/let-your-ipv6-only-workload... It's possible that the business model has changed, but at least on this blog they say it's available free of cost.
  - yorwba 3y ago
    Not free of cost, free of additional cost if you were already using a NAT Gateway before they added NAT64 capability.
  - tedivm 3y ago
    That article makes it clear that the NAT Gateway needs IPv4 > The NAT gateway recognizes the IPv6 address prefix, extracts the IPv4 address from it, and initiates an IPv4 connection to the destination. As usual, the source IPv4 address is the IPv4 address of the NAT gateway itself. If the NAT Gateway doesn't have an IPv4 address it won't work.
  - ArchOversight 3y ago
    > Pricing and Availability > These two new capabilities to the VPC NAT gateway and Route 53 are available today in all AWS Regions at no additional costs. Regular NAT gateway charges may apply. Specifically: > Regular NAT gateway charges may apply. Which should say: "Regular NAT gateway charges apply" Since you still pay for the traffic that is processed by the NAT gateway (per GiB).

jeroenhd 3y ago

Github is one of the most idiotic IPv4 exclusive services. Microsoft and Azure has all the knowledge and equipment to make IPv6 available to practically any site, but Github seems afraid to ask. They had IPv6 for a short while and turned it off later. https://github.com/orgs/community/discussions/10539 https://github.com/orgs/community/discussions/10539 is full of people voicing their grievances but I don't think Github is paying this issue any attention anymore. Luckily almost all providers or IPv6-only networks also offer NAT64 or similar NAT mechanisms to make IPv4 addresses reachable.

zamalek 3y ago
ZScaler is worse on a practical level. I have to disable IPv6 system-wide else I can't access internal services (it only routes IPv4). The crazy thing is that they call out VPNs for being archaic, but force users to use an even more archaic technology.
- MandieD 3y ago
  Bless you for mentioning that ZScaler doesn’t play nice with IPv6 - this gives me another possible cause for why a certain MSFT distributed app sometimes, but not always, installs on our users’ PCs when ZScaler is enabled, but works with it off. Thanks so much!
jiggawatts 3y ago
Microsoft used to “get” IPv6 and they added support to Windows back in the very early 2000s. All those engineers left or retired and have been replaced by outsourcers and H1Bs. Nobody at Azure can even spell IPv6. Turning it on is difficult and then it breaks everything, including unrelated services in other peered vnets. It’s just shameful how much the engineering skill has degraded in Redmond.
- nine_zeros 3y ago
  As long as the stock goes up, no-fin-body cares.

Habgdnv 3y ago

I recently tried to deploy GitLab from scratch on an IPv6-only network, and the initial experience was anything but smooth. I was met with an exception right in the console during the initial setup. GitLab attempted to obtain a Let's Encrypt certificate and immediately failed, as it doesn't listen to IPv6 addresses by default. A year ago, we (at work) faced similar issues when trying to deploy GlusterFS on an IPv6-only network, and it also failed. (I pushed for V6 only, my manager was not happy) It's evident that while IPv6 may be the future, the present ecosystem doesn't seem fully prepared to support it. For years, I have wanted to use Docker with IPv6 only, and I am really thinking about learning Go so I can write my own IPv6-only driver.

probably_a_gpt 3y ago
I’m getting python 2 v 3 flashbacks

arianvanp 3y ago

AWS NAT Gateway comes with native NAT64 support so GitHub just works over ipv6. No issues whatsoever.