7 ms·

Dealing with IPv6 fragmentation in the DNS

87 points by jsnell 9y ago

anc84 9y ago
> The initial connection between Cloudflare's network and the origin web server timed out. As a result, the web page can not be displayed. Sad to see even a NIC letting itself be MITMed by Cloudflare :(
dom0 9y ago
Maybe we should bow to the inevitable and recognise that IPv6 is an unfixable problem.
- pilif 9y ago
  if it's unfixable, why is 40% of our in-office internet traffic running over IPv6? Why is 100% of our LAN traffic over IPv6? Why is 30% of the traffic to our sites running over IPv6?
  - majke 9y ago
    The article suggests "Fragmentation in IPv6 is unfixable", not ipv6 itself. Try running this to see if fragments get delivered to you: http://icmpcheckv6.popcount.org http://icmpcheckv6.popcount.org http://icmpcheck.popcount.org http://icmpcheck.popcount.org (IPv4 version) via: https://blog.cloudflare.com/ip-fragmentation-is-broken/ https://blog.cloudflare.com/ip-fragmentation-is-broken/
    - heinrich5991 9y ago
      Seems to work for me (?).
    - vidarh 9y ago
      Works great for me on my laptop. I don't doubt for a second that there are problems (my phone doesn't get ipv6 at all and one of the ipv4 tests failed... but mobile networks are notorious when it comes to filtering traffic), but "unfixable"?
    - pilif 9y ago
      The comment I was responding to asserted that IPv6 as a whole is unfixable which is not true. With regards to fragmentation, I would say "fragmentation in general is unfixable", no matter the protocol. Thankfully with IPv6, the guaranteed minimal packet size is relatively reasonable (1280 bytes), so maybe it's now time to finally go there. v6 already changed the rules with fragmentation a bit in that routers won't have to fragment any more but are allowed to just drop packets smaller than the MTU and sending back an ICMPv6 packet announcing that they have done so. Unfortunately, ICMP is still dropped at many firewalls even though I would argue that it's absolutely essential for IPv6 functionality (when it was more optional in v4). As it stands now, hosts are still allowed to fragment packets themselves (which is what's happening in this article), but the solution is probably to not do that either in the future. DNSSec over UDP is a problem in that regards, but people aren't convinced that DNSSec is the way to go anyways (just ask tptacek here on this site). (edit: with regards to the fragment checking site you pointed me to: In our case, all is well because I do recognise the necessity of ICMPv6 and thus have configured the firewalls accordingly)
    - joeseeder 9y ago
      Network I am on atm, IPv6 give me both green, IPv4 MTU fails ... It means my IPv6 vpn is better than IPv4 delivered by some wannabe corpo network masters.
      - pilif 9y ago
        the firewall is probably blocking ICMP. For a long time, it was considered best-practice to blanket-block ICMP and for a long time you could mostly get away with it. But over time some really useful stuff has been added to ICMPv4 (early congestion notification for example) and IPv6 mostly won't work at all without ICMP. I would argue that it's time for firewall administrators to reconsider this decision. The time where ICMP packets could do harm are long past (remember the ping-of-death on Windows 98?) and I would argue that the time when a network could work at peak efficiency (or at all) without ICMP are also mostly past.
        joeseeder 9y ago
        Totally agree, ICMP is needed. When it's not blocked, you can debug and have healthy network, instead of a frankensteins monster.
        oasisbob 9y ago
        > For a long time, it was considered best-practice to blanket-block ICMP I agree with your point, but think this needs some clarification about who considered it a best practice - among enterprise firewall admins and financial-industry compliance audits, perhaps. Large scale network operators have seen the problems associated with entirely blocking ICMP for at least a decade. (v4 has blackhole problems too) A lot of the hatred for ICMP came from simple things like easy endpoint discoverability. nmap blew that argument out of the water pretty quick.
  - IshKebab 9y ago
    For the reasons outlined in the article. Most DNS responses aren't fragmented, and IPv4 covers up failures where they are.
    - pilif 9y ago
      The comment I was responding to asserted that IPv6 in general is unfixable whereas the article was only talking about IPv6 fragmentation where I'd say it's totally correct. But in general, fragmentation is a bad idea and we should not have gone there in the first place. DNSSec should only ever have been specified over TCP (or not at all if you ask some people like tptacek here on this site)
- deleted 9y ago
  [deleted]
the8472 9y ago
> IPv6 Extension Headers require any transport protocol-sensitive functions in network switches to unravel the packet header’s extension header chain. This takes a variable number of cycles for the device and furthermore requires that the switch should recognise all the extension headers encountered on the header chain. This is anathema to a switch, in so far as it entails a variable amount of time to process. If it is so much anathema to their nature then maybe they shouldn't insist on violating network layers?
- rocqua 9y ago
  I agree that switches shouldn't care about TCP or UDP ports. That is a job for a firewall. However, that does not mean it is a good idea to have variable length headers in IPv6. In fact it seems like a singularly bad idea to me. Regarding firewalls, I imagine it still sucks a firewall cannot judge each IP-packet separately but needs to deal with possible fragmented packets. Then again, that is the cost of being a firewall. You need to function at the application level and yet decide on the IP-packet level.
  - colmmacc 9y ago
    Switches use L4 ports to implement flow-switching. When multiple paths for a packet are available, it is convenient (it leads to less re-ordering) if the related packets from the same flow all choose the same path.
    - avasylev 9y ago
      It's not just convenience for anycast addresses, with multiple pathes and anycast if your round robin packets across different routes, you may end up receiving them in two different locations (ok for UDP, but TCP broken completely). Anycast is widely deployed now and imho ECMP support is now considered "standard". I'm not sure what are other alternative solution to ECMP that don't require looking into TCP headers.
  - Avernar 9y ago
    I don't see an issue with extension headers in IPv6. The RFCs have rules on who should look at what header and what to do when an unrecognized one is encountered. Most firewalls I've dealt with have some option to defragment the inbound packet at the firewall so that the rest of the rules can be processed properly and to protect nodes from malicious fragments.
stephengillie 9y ago
Having a variable header complicates Cut-Through switching[0], which forwards the entire frame just after the switch gets the destination field of the header. This sounds like it would force a return to Store-And-Forward[1], where the switch waits for the entire frame to load into memory, before forwarding to its next hop. Waiting these few milliseconds doesn't sound bad, until you consider that's added to each packet, reducing (e.g.) application performance. [0] https://en.wikipedia.org/wiki/Cut-through_switching https://en.wikipedia.org/wiki/Cut-through_switching [1] https://en.wikipedia.org/wiki/Store_and_forward https://en.wikipedia.org/wiki/Store_and_forward
- squeed 9y ago
  The real problem is that you can no longer look at a fixed offset for your parameters, which complicates filtering silicon. In other words, if you want to have a firewall rule of "drop tcp 135", in ipv4 you can just look at bytes 0x24-25 (for Ethernet). IPv6's extension-header mechanism means that the header has a non-fixed length, so you need to do work for the same effect. Given that firewalls are critical in a world without NAT, this can be a scary prospect. Part of the blame falls on vendors who go "well, IPv6 is fringe so we'll just do all firewalling in software." The protocol is 20 years old, it's time to design the silicon.
  - tssva 9y ago
    Most enterprise firewalls these days do some level of layer 7 packet inspection which requires going beyond looking at fixed offsets. The silicon to support this already exists. Even the $50 Ubiquiti ER-X router I use at home can do deep packet inspection assisted by silicon.
  - verri 9y ago
    But IPv4 headers could have a variable length too, it's just that we don't encounter that much in the wild. And what about the AH header, GRE and IPIP tunnel headers, 6rd/6in4 tunnel headers, etc. Filtering at fixed offsets sounds very brittle to me. Isn't the real problem here that network providers and administrators appropriate the right to filter on OSI layers they shouldn't be touching? This problem sounds like the exact reason why Google insisted on having its QUIC headers ciphered: so network equipment can't pull of this kind of misbehaviour.
tinus_hn 9y ago
> another painful issue in today’s IPv6 Internet, namely that of network filters discarding ICMPv6 Packet Too Big messages. How about we just let the people who configure these filters solve their own problems?
- kevin_nisbet 9y ago
  I've actually had this conversation before, and it get's a bit tricky. I'd say it has a few main facets: - ICMP is dangerous, and needs to be blocked. This is partly true, as there is historically information leakage with ICMP under certain codes. So blocking all of it is easier than trying to figure out what should be allowed and what should not. - It's not perceived to be a problem. Most services just work, so TCP/IP with mss clamping will avoid the fragmentation issue for many services. So the services that have an issue, might be something like 2%, and the question that always lingers is if everything else works, and this one thing doesn't, it's not the filters problem. - ICMP doesn't necessarily work with multiple layers of encapsulation anyways I encountered this on mobile wireless networks, where doing multiple rounds of encapsulation caused multiple packet size reductions. Basically, if we encapsulate in tunnels used by the mobile network, then encapsulate again in say IPSec, the IPSec node will send it's icmp packet too big to the tunnel endpoint, not the originator of the traffic. While I suppose it could be argued that the tunnel endpoint should maintain it's own PMTU, and pro-actively send it's own icmp packet too big based on this, it can be difficult to guarantee these components work. I've seen lots of telco equipment struggle with handling this properly, and it's not just filtering, it's bugs. So unfortunately, we're in a situation where many networks are likely miss-configured, but lots of other services for various reasons just work. So unless you can drive the networks with problems to actually change, realistically you just lose a portion of your own customer base if you don't handle the situation while your competitors just work.
  - wogna 9y ago
    > ICMP is dangerous, and needs to be blocked I've fought this argument countless times and lost. Not being able to ping a server, check for "destination unreachable" messages, etc. leads to frustrating troubleshooting sessions.
deleted 9y ago
[deleted]
tveita 9y ago
It sounds like IPv6 acknowledges that most protocols try hard to avoid fragmentation anyway, so fragmented packets are the exception. And it's not like fragmentation hasn't caused problems with firewalls in IPv4 either. E.g. http://all.net/Analyst/netsec/1995-09.html http://all.net/Analyst/netsec/1995-09.html The fix is for equipment to handle the fragmented packets properly, just like for IPv4. How is forward fragmentation, which involves creating multiple new packets + headers with recomputed checksums, easier than simply parsing a header, which is apparently "anathema to a switch"?
- skywhopper 9y ago
  The problem is that the fragmentation header in IPv6 is not deterministic, so it can't be parsed in constant time, which is what makes it "anathema to a switch" which needs to be able to guarantee certain levels of packet throughput. IPv4-style fragmentation can be done in constant time.
  - pas 9y ago
    It is still bounded time, which is still O(1). Yes, the standard doesn't limit the number of extension headers, but looking at the fixed header, and at most 5-6 extenion headers is the maximum time required to process a packet. And that'd be enough for 100% of traffic. The exact limit is the number of header types known to the processing node. Because encountering an unknown header should result in Parameter Error, and only one header can be used multiple times (the Destination Options), but that should be after the Frag one anyway.

87 points by jsnell 9y ago

> The initial connection between Cloudflare's network and the origin web server timed out. As a result, the web page can not be displayed. Sad to see even a NIC letting itself be MITMed by Cloudflare :(

dom0 9y ago

Maybe we should bow to the inevitable and recognise that IPv6 is an unfixable problem.

pilif 9y ago
if it's unfixable, why is 40% of our in-office internet traffic running over IPv6? Why is 100% of our LAN traffic over IPv6? Why is 30% of the traffic to our sites running over IPv6?
- majke 9y ago
  The article suggests "Fragmentation in IPv6 is unfixable", not ipv6 itself. Try running this to see if fragments get delivered to you: http://icmpcheckv6.popcount.org http://icmpcheckv6.popcount.org http://icmpcheck.popcount.org http://icmpcheck.popcount.org (IPv4 version) via: https://blog.cloudflare.com/ip-fragmentation-is-broken/ https://blog.cloudflare.com/ip-fragmentation-is-broken/
  - heinrich5991 9y ago
    Seems to work for me (?).
  - vidarh 9y ago
    Works great for me on my laptop. I don't doubt for a second that there are problems (my phone doesn't get ipv6 at all and one of the ipv4 tests failed... but mobile networks are notorious when it comes to filtering traffic), but "unfixable"?
  - pilif 9y ago
    The comment I was responding to asserted that IPv6 as a whole is unfixable which is not true. With regards to fragmentation, I would say "fragmentation in general is unfixable", no matter the protocol. Thankfully with IPv6, the guaranteed minimal packet size is relatively reasonable (1280 bytes), so maybe it's now time to finally go there. v6 already changed the rules with fragmentation a bit in that routers won't have to fragment any more but are allowed to just drop packets smaller than the MTU and sending back an ICMPv6 packet announcing that they have done so. Unfortunately, ICMP is still dropped at many firewalls even though I would argue that it's absolutely essential for IPv6 functionality (when it was more optional in v4). As it stands now, hosts are still allowed to fragment packets themselves (which is what's happening in this article), but the solution is probably to not do that either in the future. DNSSec over UDP is a problem in that regards, but people aren't convinced that DNSSec is the way to go anyways (just ask tptacek here on this site). (edit: with regards to the fragment checking site you pointed me to: In our case, all is well because I do recognise the necessity of ICMPv6 and thus have configured the firewalls accordingly)
  - joeseeder 9y ago
    Network I am on atm, IPv6 give me both green, IPv4 MTU fails ... It means my IPv6 vpn is better than IPv4 delivered by some wannabe corpo network masters.
    - pilif 9y ago
      the firewall is probably blocking ICMP. For a long time, it was considered best-practice to blanket-block ICMP and for a long time you could mostly get away with it. But over time some really useful stuff has been added to ICMPv4 (early congestion notification for example) and IPv6 mostly won't work at all without ICMP. I would argue that it's time for firewall administrators to reconsider this decision. The time where ICMP packets could do harm are long past (remember the ping-of-death on Windows 98?) and I would argue that the time when a network could work at peak efficiency (or at all) without ICMP are also mostly past.
      - joeseeder 9y ago
        Totally agree, ICMP is needed. When it's not blocked, you can debug and have healthy network, instead of a frankensteins monster.
      - oasisbob 9y ago
        > For a long time, it was considered best-practice to blanket-block ICMP I agree with your point, but think this needs some clarification about who considered it a best practice - among enterprise firewall admins and financial-industry compliance audits, perhaps. Large scale network operators have seen the problems associated with entirely blocking ICMP for at least a decade. (v4 has blackhole problems too) A lot of the hatred for ICMP came from simple things like easy endpoint discoverability. nmap blew that argument out of the water pretty quick.
- IshKebab 9y ago
  For the reasons outlined in the article. Most DNS responses aren't fragmented, and IPv4 covers up failures where they are.
  - pilif 9y ago
    The comment I was responding to asserted that IPv6 in general is unfixable whereas the article was only talking about IPv6 fragmentation where I'd say it's totally correct. But in general, fragmentation is a bad idea and we should not have gone there in the first place. DNSSec should only ever have been specified over TCP (or not at all if you ask some people like tptacek here on this site)
deleted 9y ago
[deleted]

the8472 9y ago

> IPv6 Extension Headers require any transport protocol-sensitive functions in network switches to unravel the packet header’s extension header chain. This takes a variable number of cycles for the device and furthermore requires that the switch should recognise all the extension headers encountered on the header chain. This is anathema to a switch, in so far as it entails a variable amount of time to process. If it is so much anathema to their nature then maybe they shouldn't insist on violating network layers?

rocqua 9y ago
I agree that switches shouldn't care about TCP or UDP ports. That is a job for a firewall. However, that does not mean it is a good idea to have variable length headers in IPv6. In fact it seems like a singularly bad idea to me. Regarding firewalls, I imagine it still sucks a firewall cannot judge each IP-packet separately but needs to deal with possible fragmented packets. Then again, that is the cost of being a firewall. You need to function at the application level and yet decide on the IP-packet level.
- colmmacc 9y ago
  Switches use L4 ports to implement flow-switching. When multiple paths for a packet are available, it is convenient (it leads to less re-ordering) if the related packets from the same flow all choose the same path.
  - avasylev 9y ago
    It's not just convenience for anycast addresses, with multiple pathes and anycast if your round robin packets across different routes, you may end up receiving them in two different locations (ok for UDP, but TCP broken completely). Anycast is widely deployed now and imho ECMP support is now considered "standard". I'm not sure what are other alternative solution to ECMP that don't require looking into TCP headers.
- Avernar 9y ago
  I don't see an issue with extension headers in IPv6. The RFCs have rules on who should look at what header and what to do when an unrecognized one is encountered. Most firewalls I've dealt with have some option to defragment the inbound packet at the firewall so that the rest of the rules can be processed properly and to protect nodes from malicious fragments.

stephengillie 9y ago

Having a variable header complicates Cut-Through switching[0], which forwards the entire frame just after the switch gets the destination field of the header. This sounds like it would force a return to Store-And-Forward[1], where the switch waits for the entire frame to load into memory, before forwarding to its next hop. Waiting these few milliseconds doesn't sound bad, until you consider that's added to each packet, reducing (e.g.) application performance. [0] https://en.wikipedia.org/wiki/Cut-through_switching https://en.wikipedia.org/wiki/Cut-through_switching [1] https://en.wikipedia.org/wiki/Store_and_forward https://en.wikipedia.org/wiki/Store_and_forward

squeed 9y ago
The real problem is that you can no longer look at a fixed offset for your parameters, which complicates filtering silicon. In other words, if you want to have a firewall rule of "drop tcp 135", in ipv4 you can just look at bytes 0x24-25 (for Ethernet). IPv6's extension-header mechanism means that the header has a non-fixed length, so you need to do work for the same effect. Given that firewalls are critical in a world without NAT, this can be a scary prospect. Part of the blame falls on vendors who go "well, IPv6 is fringe so we'll just do all firewalling in software." The protocol is 20 years old, it's time to design the silicon.
- tssva 9y ago
  Most enterprise firewalls these days do some level of layer 7 packet inspection which requires going beyond looking at fixed offsets. The silicon to support this already exists. Even the $50 Ubiquiti ER-X router I use at home can do deep packet inspection assisted by silicon.
- verri 9y ago
  But IPv4 headers could have a variable length too, it's just that we don't encounter that much in the wild. And what about the AH header, GRE and IPIP tunnel headers, 6rd/6in4 tunnel headers, etc. Filtering at fixed offsets sounds very brittle to me. Isn't the real problem here that network providers and administrators appropriate the right to filter on OSI layers they shouldn't be touching? This problem sounds like the exact reason why Google insisted on having its QUIC headers ciphered: so network equipment can't pull of this kind of misbehaviour.

tinus_hn 9y ago

> another painful issue in today’s IPv6 Internet, namely that of network filters discarding ICMPv6 Packet Too Big messages. How about we just let the people who configure these filters solve their own problems?

kevin_nisbet 9y ago
I've actually had this conversation before, and it get's a bit tricky. I'd say it has a few main facets: - ICMP is dangerous, and needs to be blocked. This is partly true, as there is historically information leakage with ICMP under certain codes. So blocking all of it is easier than trying to figure out what should be allowed and what should not. - It's not perceived to be a problem. Most services just work, so TCP/IP with mss clamping will avoid the fragmentation issue for many services. So the services that have an issue, might be something like 2%, and the question that always lingers is if everything else works, and this one thing doesn't, it's not the filters problem. - ICMP doesn't necessarily work with multiple layers of encapsulation anyways I encountered this on mobile wireless networks, where doing multiple rounds of encapsulation caused multiple packet size reductions. Basically, if we encapsulate in tunnels used by the mobile network, then encapsulate again in say IPSec, the IPSec node will send it's icmp packet too big to the tunnel endpoint, not the originator of the traffic. While I suppose it could be argued that the tunnel endpoint should maintain it's own PMTU, and pro-actively send it's own icmp packet too big based on this, it can be difficult to guarantee these components work. I've seen lots of telco equipment struggle with handling this properly, and it's not just filtering, it's bugs. So unfortunately, we're in a situation where many networks are likely miss-configured, but lots of other services for various reasons just work. So unless you can drive the networks with problems to actually change, realistically you just lose a portion of your own customer base if you don't handle the situation while your competitors just work.
- wogna 9y ago
  > ICMP is dangerous, and needs to be blocked I've fought this argument countless times and lost. Not being able to ping a server, check for "destination unreachable" messages, etc. leads to frustrating troubleshooting sessions.

deleted 9y ago

[deleted]

tveita 9y ago

It sounds like IPv6 acknowledges that most protocols try hard to avoid fragmentation anyway, so fragmented packets are the exception. And it's not like fragmentation hasn't caused problems with firewalls in IPv4 either. E.g. http://all.net/Analyst/netsec/1995-09.html http://all.net/Analyst/netsec/1995-09.html The fix is for equipment to handle the fragmented packets properly, just like for IPv4. How is forward fragmentation, which involves creating multiple new packets + headers with recomputed checksums, easier than simply parsing a header, which is apparently "anathema to a switch"?

skywhopper 9y ago
The problem is that the fragmentation header in IPv6 is not deterministic, so it can't be parsed in constant time, which is what makes it "anathema to a switch" which needs to be able to guarantee certain levels of packet throughput. IPv4-style fragmentation can be done in constant time.
- pas 9y ago
  It is still bounded time, which is still O(1). Yes, the standard doesn't limit the number of extension headers, but looking at the fixed header, and at most 5-6 extenion headers is the maximum time required to process a packet. And that'd be enough for 100% of traffic. The exact limit is the number of header types known to the processing node. Because encountering an unknown header should result in Parameter Error, and only one header can be used multiple times (the Destination Options), but that should be after the Frag one anyway.