9 ms·

The "issues with IPv6" are with education, operation, configuration. I personally ran WiFi networks with 8000+ wireless clients on a single /64 subnet (my empl

by ay 12y ago

The "issues with IPv6" are with education, operation, configuration. I personally ran WiFi networks with 8000+ wireless clients on a single /64 subnet (my employer's CiscoLive conference), and assisted/consulted in running the networks with more than 25000 clients on a single /64 subnet (Mobile World Congress). The default kinda suck, and the bugs may happen. but the statement "IPv6 is not ready for production" is wrong. I'd be happy to volunteer a reasonable amount of time to work with the OP or others having a network of >1000 hosts, to debug the issues like this, time permitting, vendor independent. (glass houses and all that). There are bazillion knobs in IPv6 and a lot of things can be fixed by just tweaking the defaults (which kinda suck). Network of <500-700 nodes generally don't need to bother much. It's not optimal a lot of times with the defaults but it will work. EDIT: the seeming "charity" of volunteering the time isn't. I want to understand what is broken, so we can bring it up back to IETF and get it fixed + make better educational publicity to prevent folks shooting themselves in the foot. It'll make it to the stacks in another decade, but it will. IPv6 is powering nontrivial millions of hosts now - so the correct words to use "needs tweaking for my network", not "not ready for production". Let's see what the tweaks are and if we can incorporate them into the protocol, if necessary.

MichaelGG 12y ago
Why add such knobs, if they weren't there in the first place, haven't been tested, and obviously are causing nontrivial problems?
- ay 12y ago
  Everything unknown of is untrivial - and I'll leave it at that.
- calinet6 12y ago
  It seems to me that the presence of a "bazillion knobs" is a bug in itself. A stable system can not depend on puny humans for its operation—nor should a design be able to blame puny humans for its failure.
  - ay 12y ago
    Millions of people are using it today, one and a half decades after it was conceived. It's not perfect but nothing is. If you have examples of tech to compare with - please bring them in! EDIT: and yeah, a lot of knobs does sound like a bit of a poor design, but on the other hand, today, 1.5 decades later, I am grateful I can tweak some of the defaults! Again, if you know something knob-less that'd get flawlessly deployed in 15 years after it was designed - shout, would be very useful to try and learn from that success!
    - tedunangst 12y ago
      The nearest comparison to IPv6 is DNSSEC. IPv6 has done a little better, but that's not saying much. Both suffer from massive over engineering, and lots of false starts and back pedaling over the years. Trying to solve too many problems, and therefore failing to solve the one problem that's of immediate concern.
      - ay 12y ago
        Oh, "same as" cases are indeed all over, e.g. the EUI-64 https://standards.ieee.org/develop/regauth/tut/eui64.pdf https://standards.ieee.org/develop/regauth/tut/eui64.pdf I was looking for the example of the sparkly unicorn protocol from 2000 the people seem to evoke, which does not need any knobs and is available on every device to update some other protocol from 1970s. (I suppose the hard meta-question to protocol designers is: how do you make sure your children will not hate you for the protocol you have imposed on them today ?)
        wbl 12y ago
        UTF-8 updating ASCII is probably the best example of this going well. And even then there was a terrible mistake made in the design of Korean encodings which had to be fixed.
        taejo 12y ago
        What was the problem with Korean encodings?
- walshemj 12y ago
  Absolutely that's why all network pros advise against tuning the timings for a v4 network even though in some edge cases it can help unless you really really know what you are doing.
- GregoryLS 12y ago
  couldn't agree with ya more my friend !
- ay 12y ago
  If you are genuinely interested in answer to this question, you will like the below eBook that answers that question fabulously. http://www.internetsociety.org/deploy360/blog/2013/12/free-ebook-ipv6-for-ipv4-experts-in-english-and-russian/ http://www.internetsociety.org/deploy360/blog/2013/12/free-e... It's free, very high quality content, and it is written as if you co-design the new IP protocol together with the author - you start from "why" and arrive to "how".
x0x0 12y ago
And somehow juniper (and juniper support engineers for large routers) is unaware of these things? I'm skeptical. Particularly for ongoing problems where second and third level support get involved.
- ploxiln 12y ago
  Don't be so surprised. There can be corporate-political problems to engineers solving the problem. For example, the official marketing line is "we support feature X!", and while the engineer wants to say "that doesn't work right, turn that shit off" he can't really do that. Or maybe the solution is to not use Juniper for ipv6. I really don't know. It's possible that Juniper's development process and goals prevented solid ipv6 support - another case where even if their engineers were aware, they couldn't do anything to help this guy.
- jauer 12y ago
  I wouldn't be surprised if this turns out to be a MIT-environment-specific issue. E.g. Juniper support wasn't aware of it because they assumed no one would try to use stateless configuration in what would generally be a managed environment using dhcpv6.
  - ay 12y ago
    Yes and no. Quite a few large environments use SLAAC and they work just fine (though I work for cisco so a bias is quite possible ;-). But there's certainly something specific to MIT's setup that makes it melt down in such a spectacular fashion with v6. "Forwarding loops" (preexisting?) from the blog raise some suspicion. The local environment frequently has a huge impact - and for the most extreme/"obvious" cases it took me to recreate the setup in the lab and show folks how it works fine with their config and traffic to make them believe that it was not just stupidly broken - because in their environment they could reproduce the bug every time!
- res0nat0r 12y ago
  Normally when the big dogs for a product get called in it is to find out why their product has an issue, they are the code experts for that. This doesn't mean they are inherently experts in every protocol their product happens to be using.
haberman 12y ago
How can you be so certain that the author's issue is not real? If you have enough information to be sure that the author's situation isn't a genuine problem with IPv6, why don't you also have enough information to say what the administrator's mistake is, and its solution? I don't mean to be snarky considering your generous offer to help, but your certainty that the problem isn't with IPv6 seems odd, given the specificity of the author's description and analysis.
- ay 12y ago
  That's not what I said. There are plenty of very real and very painful ways to shoot oneself in the foot with IPv6, especially in a larger kind of network that I can imagine the folks at MIT have. But based on my experience with running the real-world IPv6 networks of large scale, there has to be a better solution than just "turning off IPv6" - even if it is a great interim band-aid after being up for 40 hours! (and I emphasize with that!) The blog entry says: "The most serious issue is that there appear to be bridge loops occasionally being created". IS this something related to IPv6 ? Specific to IPv6 ? What's the root cause ? IPv6 with default settings in a larger net has very nonlinear behavior in the case of unreliable connectivity, and can make the situation worse. The other issue, multicast group overload, should be solvable by making the switches blind to MLD. This will make it as "bad" as IPv4, but in todays' network size it will work fine because IPv4 works, mostly. Meantime they would need to kick their vendor's representative to have it fixed. Also, a nit about the whole issue at hand: the "worst offender" is iOS, which will create a new privacy address everytime you disconnect and reconnect to WiFi. And if you were to "fix" that, imagine the wrath of the "privacy advocates" about this. Privacy addresses in IPv6 are very valuable for a reason that due to a fast refresh cycle they help to uncover a lot of corner cases that would otherwise pop up with a kind of "shrug, something weird" type of RCA. And fix them before the IPv6 usage hits 50% (cf my comments on the other thread). (From the privacy standpoint, privacy addresses are a pretty entertainingly weird construct.) So to summarize this somewhat long-winded post: no question there's a problem. Whether it's in "IPv6 the protocol" or "IPv6 the vendor implementation" or "IPv6 the configuration" is another story. And that'd be something to understand and fix - but needs more info/context/legwork IMO.
  - 12y ago
cesarb 12y ago
Playing devil's advocate: your experiences of "CiscoLive conference" and "Mobile World Congress" sound like conferences, which are the kind of scenario where most hosts are connected to the network for less than a day each time. If I understood it correctly, his privacy addresses issue wouldn't show up in that scenario, since it needs several days using the same network connection continuously for each host to accumulate enough privacy addresses.
- ay 12y ago
  University campus typically have similar, although a bit less dynamic pattern of usage, than conference type scenarios. We actually observed on different (not just conference, but also campus) that wireless networks in general seem to have the same curve of connection duration distribution, though with different parameters. (If you have a large WiFi net, and would like to collaborate on re-confirming or disproving this hypothesis, please let me know). The privacy addresses refresh depends on 2 factors: 1) preferred lifetime. http://tools.ietf.org/html/rfc4941#section-3.5 http://tools.ietf.org/html/rfc4941#section-3.5 for more info. 2) frequency of reconnection. Some platforms (iDevices in particular) decide to make a new the IPv6 address every time they reconnect. Great for privacy (And that's exactly how I ran the abovementioned networks, also take a look at the http://tools.ietf.org/html/draft-desmouceaux-ipv6-mcast-wifi-power-usage-00 http://tools.ietf.org/html/draft-desmouceaux-ipv6-mcast-wifi... for the observed leave/join stats), but if the infra does not have enough leeway to accomodate the extra addresses. In any case even then not all is lost - clear the "A" bit in the RA to turn off the SLAAC, and use DHCPv6. Poor Android clients will be off the IPv6 grid because they don't bother to implement IPv6 for the reasons left unnamed here, but the rest of the gear will work just fine, with a single, predictable address, that an admin controls. So, again - it all depends on the local situation and there's no unsolvable problems. There are existing and production-proven solutions.
  - ysleepy 12y ago
    You imply that DHCPv6 allows admin control, but as I understand it, it is not guaranteed that the request (DUID) can be mapped to a known device (MAC) (Type 1 DUID). Especially with installation media, it is just not feasible to custom script them to force Type1 usage. This makes bringup and configuration mgmt just rely on IPv4 to get an identity. I'm running into difficulties with this in our campus deployment. - I need static addresses in DNS (DDNS is just dirty, especially with DNSSEC) and also reverse records for kerberos. IPv6 just bums me out.
cm2187 12y ago
Shooting themselves or shooting an innocent and necessary technology. I personally see IPv6 adoption going to a full stall if users are told that advertisers found a way to tattoo cookies on their machines. I am worried by the hostile view on privacy addresses in that article and many of the comments.
- simoncion 12y ago
  > I personally see IPv6 adoption going to a full stall if users are told that advertisers found a way to tattoo cookies on their machines. Heh. Better not tell them about https://panopticlick.eff.org/ https://panopticlick.eff.org/ . Seriously, tracking via your v6 address is a minor concern. Browsers already provide more than enough information to an advertiser to do a really good job of tracking a person as he changes his attach point to the Internet. Let me ask you a question. (Please address this question, and not some hypothetical situation that isn't covered by this question.) If the combination of my User Agent string, screen resolution, and installed browser plugins uniquely identifies my PC, why should I be concerned that I also have a stable, globally unique IP address?
  - cm2187 12y ago
    I agree. This is not great. Though I believe that if this start being used, browser vendors will crack down on this (it is shocking that they give away so much information). But I read that this technique still leaves thousands (if not hundred of thousands) of collisions (particularly on more standardised mobile platforms) so I don't expect that to ever be widely used. But the fact that there is a leak somewhere else in the house isn't an excuse to create another leak. Also, browsers evolve and react relatively quickly. Not the IP protocol. When you see how hard it is to switch from IPv4 to IPv6, I don't think I will live to see IPv8! So making a disastrous decision from a privacy point of view in IPv6 will have long and lasting consequences. It's not because Chrome v30something is broken that we should leave a flaw in the IP design.
    - ay 12y ago
      On "still leaves thousands (if not hundred of thousands) of collisions", here's some research: "In 2010, Eckersley demonstrated that benign characteris- tics of a browser's environment, like the screen dimensions and list of installed fonts, could be combined to create a unique device-specic ngerprint [7]. Of the half million users who participated in Eckersley's experiment, 94.2% of those using Flash or Java had a unique device ngerprint, and could thus be identied and tracked without the need for stateful client-side technologies, such as browser or Flash cookies." Source: http://www.cosic.esat.kuleuven.be/publications/article-2334.pdf http://www.cosic.esat.kuleuven.be/publications/article-2334....

by ay 12y ago

MichaelGG 12y ago

Why add such knobs, if they weren't there in the first place, haven't been tested, and obviously are causing nontrivial problems?

ay 12y ago
Everything unknown of is untrivial - and I'll leave it at that.
calinet6 12y ago
It seems to me that the presence of a "bazillion knobs" is a bug in itself. A stable system can not depend on puny humans for its operation—nor should a design be able to blame puny humans for its failure.
- ay 12y ago
  Millions of people are using it today, one and a half decades after it was conceived. It's not perfect but nothing is. If you have examples of tech to compare with - please bring them in! EDIT: and yeah, a lot of knobs does sound like a bit of a poor design, but on the other hand, today, 1.5 decades later, I am grateful I can tweak some of the defaults! Again, if you know something knob-less that'd get flawlessly deployed in 15 years after it was designed - shout, would be very useful to try and learn from that success!
  - tedunangst 12y ago
    The nearest comparison to IPv6 is DNSSEC. IPv6 has done a little better, but that's not saying much. Both suffer from massive over engineering, and lots of false starts and back pedaling over the years. Trying to solve too many problems, and therefore failing to solve the one problem that's of immediate concern.
    - ay 12y ago
      Oh, "same as" cases are indeed all over, e.g. the EUI-64 https://standards.ieee.org/develop/regauth/tut/eui64.pdf https://standards.ieee.org/develop/regauth/tut/eui64.pdf I was looking for the example of the sparkly unicorn protocol from 2000 the people seem to evoke, which does not need any knobs and is available on every device to update some other protocol from 1970s. (I suppose the hard meta-question to protocol designers is: how do you make sure your children will not hate you for the protocol you have imposed on them today ?)
      - wbl 12y ago
        UTF-8 updating ASCII is probably the best example of this going well. And even then there was a terrible mistake made in the design of Korean encodings which had to be fixed.
        taejo 12y ago
        What was the problem with Korean encodings?
walshemj 12y ago
Absolutely that's why all network pros advise against tuning the timings for a v4 network even though in some edge cases it can help unless you really really know what you are doing.
GregoryLS 12y ago
couldn't agree with ya more my friend !
ay 12y ago
If you are genuinely interested in answer to this question, you will like the below eBook that answers that question fabulously. http://www.internetsociety.org/deploy360/blog/2013/12/free-ebook-ipv6-for-ipv4-experts-in-english-and-russian/ http://www.internetsociety.org/deploy360/blog/2013/12/free-e... It's free, very high quality content, and it is written as if you co-design the new IP protocol together with the author - you start from "why" and arrive to "how".

x0x0 12y ago

And somehow juniper (and juniper support engineers for large routers) is unaware of these things? I'm skeptical. Particularly for ongoing problems where second and third level support get involved.

ploxiln 12y ago
Don't be so surprised. There can be corporate-political problems to engineers solving the problem. For example, the official marketing line is "we support feature X!", and while the engineer wants to say "that doesn't work right, turn that shit off" he can't really do that. Or maybe the solution is to not use Juniper for ipv6. I really don't know. It's possible that Juniper's development process and goals prevented solid ipv6 support - another case where even if their engineers were aware, they couldn't do anything to help this guy.
jauer 12y ago
I wouldn't be surprised if this turns out to be a MIT-environment-specific issue. E.g. Juniper support wasn't aware of it because they assumed no one would try to use stateless configuration in what would generally be a managed environment using dhcpv6.
- ay 12y ago
  Yes and no. Quite a few large environments use SLAAC and they work just fine (though I work for cisco so a bias is quite possible ;-). But there's certainly something specific to MIT's setup that makes it melt down in such a spectacular fashion with v6. "Forwarding loops" (preexisting?) from the blog raise some suspicion. The local environment frequently has a huge impact - and for the most extreme/"obvious" cases it took me to recreate the setup in the lab and show folks how it works fine with their config and traffic to make them believe that it was not just stupidly broken - because in their environment they could reproduce the bug every time!
res0nat0r 12y ago
Normally when the big dogs for a product get called in it is to find out why their product has an issue, they are the code experts for that. This doesn't mean they are inherently experts in every protocol their product happens to be using.

haberman 12y ago

How can you be so certain that the author's issue is not real? If you have enough information to be sure that the author's situation isn't a genuine problem with IPv6, why don't you also have enough information to say what the administrator's mistake is, and its solution? I don't mean to be snarky considering your generous offer to help, but your certainty that the problem isn't with IPv6 seems odd, given the specificity of the author's description and analysis.

ay 12y ago
That's not what I said. There are plenty of very real and very painful ways to shoot oneself in the foot with IPv6, especially in a larger kind of network that I can imagine the folks at MIT have. But based on my experience with running the real-world IPv6 networks of large scale, there has to be a better solution than just "turning off IPv6" - even if it is a great interim band-aid after being up for 40 hours! (and I emphasize with that!) The blog entry says: "The most serious issue is that there appear to be bridge loops occasionally being created". IS this something related to IPv6 ? Specific to IPv6 ? What's the root cause ? IPv6 with default settings in a larger net has very nonlinear behavior in the case of unreliable connectivity, and can make the situation worse. The other issue, multicast group overload, should be solvable by making the switches blind to MLD. This will make it as "bad" as IPv4, but in todays' network size it will work fine because IPv4 works, mostly. Meantime they would need to kick their vendor's representative to have it fixed. Also, a nit about the whole issue at hand: the "worst offender" is iOS, which will create a new privacy address everytime you disconnect and reconnect to WiFi. And if you were to "fix" that, imagine the wrath of the "privacy advocates" about this. Privacy addresses in IPv6 are very valuable for a reason that due to a fast refresh cycle they help to uncover a lot of corner cases that would otherwise pop up with a kind of "shrug, something weird" type of RCA. And fix them before the IPv6 usage hits 50% (cf my comments on the other thread). (From the privacy standpoint, privacy addresses are a pretty entertainingly weird construct.) So to summarize this somewhat long-winded post: no question there's a problem. Whether it's in "IPv6 the protocol" or "IPv6 the vendor implementation" or "IPv6 the configuration" is another story. And that'd be something to understand and fix - but needs more info/context/legwork IMO.
- 12y ago

cesarb 12y ago

Playing devil's advocate: your experiences of "CiscoLive conference" and "Mobile World Congress" sound like conferences, which are the kind of scenario where most hosts are connected to the network for less than a day each time. If I understood it correctly, his privacy addresses issue wouldn't show up in that scenario, since it needs several days using the same network connection continuously for each host to accumulate enough privacy addresses.

ay 12y ago
University campus typically have similar, although a bit less dynamic pattern of usage, than conference type scenarios. We actually observed on different (not just conference, but also campus) that wireless networks in general seem to have the same curve of connection duration distribution, though with different parameters. (If you have a large WiFi net, and would like to collaborate on re-confirming or disproving this hypothesis, please let me know). The privacy addresses refresh depends on 2 factors: 1) preferred lifetime. http://tools.ietf.org/html/rfc4941#section-3.5 http://tools.ietf.org/html/rfc4941#section-3.5 for more info. 2) frequency of reconnection. Some platforms (iDevices in particular) decide to make a new the IPv6 address every time they reconnect. Great for privacy (And that's exactly how I ran the abovementioned networks, also take a look at the http://tools.ietf.org/html/draft-desmouceaux-ipv6-mcast-wifi-power-usage-00 http://tools.ietf.org/html/draft-desmouceaux-ipv6-mcast-wifi... for the observed leave/join stats), but if the infra does not have enough leeway to accomodate the extra addresses. In any case even then not all is lost - clear the "A" bit in the RA to turn off the SLAAC, and use DHCPv6. Poor Android clients will be off the IPv6 grid because they don't bother to implement IPv6 for the reasons left unnamed here, but the rest of the gear will work just fine, with a single, predictable address, that an admin controls. So, again - it all depends on the local situation and there's no unsolvable problems. There are existing and production-proven solutions.
- ysleepy 12y ago
  You imply that DHCPv6 allows admin control, but as I understand it, it is not guaranteed that the request (DUID) can be mapped to a known device (MAC) (Type 1 DUID). Especially with installation media, it is just not feasible to custom script them to force Type1 usage. This makes bringup and configuration mgmt just rely on IPv4 to get an identity. I'm running into difficulties with this in our campus deployment. - I need static addresses in DNS (DDNS is just dirty, especially with DNSSEC) and also reverse records for kerberos. IPv6 just bums me out.

cm2187 12y ago

Shooting themselves or shooting an innocent and necessary technology. I personally see IPv6 adoption going to a full stall if users are told that advertisers found a way to tattoo cookies on their machines. I am worried by the hostile view on privacy addresses in that article and many of the comments.

simoncion 12y ago
> I personally see IPv6 adoption going to a full stall if users are told that advertisers found a way to tattoo cookies on their machines. Heh. Better not tell them about https://panopticlick.eff.org/ https://panopticlick.eff.org/ . Seriously, tracking via your v6 address is a minor concern. Browsers already provide more than enough information to an advertiser to do a really good job of tracking a person as he changes his attach point to the Internet. Let me ask you a question. (Please address this question, and not some hypothetical situation that isn't covered by this question.) If the combination of my User Agent string, screen resolution, and installed browser plugins uniquely identifies my PC, why should I be concerned that I also have a stable, globally unique IP address?
- cm2187 12y ago
  I agree. This is not great. Though I believe that if this start being used, browser vendors will crack down on this (it is shocking that they give away so much information). But I read that this technique still leaves thousands (if not hundred of thousands) of collisions (particularly on more standardised mobile platforms) so I don't expect that to ever be widely used. But the fact that there is a leak somewhere else in the house isn't an excuse to create another leak. Also, browsers evolve and react relatively quickly. Not the IP protocol. When you see how hard it is to switch from IPv4 to IPv6, I don't think I will live to see IPv8! So making a disastrous decision from a privacy point of view in IPv6 will have long and lasting consequences. It's not because Chrome v30something is broken that we should leave a flaw in the IP design.
  - ay 12y ago
    On "still leaves thousands (if not hundred of thousands) of collisions", here's some research: "In 2010, Eckersley demonstrated that benign characteris- tics of a browser's environment, like the screen dimensions and list of installed fonts, could be combined to create a unique device-specic ngerprint [7]. Of the half million users who participated in Eckersley's experiment, 94.2% of those using Flash or Java had a unique device ngerprint, and could thus be identied and tracked without the need for stateful client-side technologies, such as browser or Flash cookies." Source: http://www.cosic.esat.kuleuven.be/publications/article-2334.pdf http://www.cosic.esat.kuleuven.be/publications/article-2334....