3 ms·

That's not what I said. There are plenty of very real and very painful ways to shoot oneself in the foot with IPv6, especially in a larger kind of network that

by ay 12y ago

That's not what I said. There are plenty of very real and very painful ways to shoot oneself in the foot with IPv6, especially in a larger kind of network that I can imagine the folks at MIT have. But based on my experience with running the real-world IPv6 networks of large scale, there has to be a better solution than just "turning off IPv6" - even if it is a great interim band-aid after being up for 40 hours! (and I emphasize with that!) The blog entry says: "The most serious issue is that there appear to be bridge loops occasionally being created". IS this something related to IPv6 ? Specific to IPv6 ? What's the root cause ? IPv6 with default settings in a larger net has very nonlinear behavior in the case of unreliable connectivity, and can make the situation worse. The other issue, multicast group overload, should be solvable by making the switches blind to MLD. This will make it as "bad" as IPv4, but in todays' network size it will work fine because IPv4 works, mostly. Meantime they would need to kick their vendor's representative to have it fixed. Also, a nit about the whole issue at hand: the "worst offender" is iOS, which will create a new privacy address everytime you disconnect and reconnect to WiFi. And if you were to "fix" that, imagine the wrath of the "privacy advocates" about this. Privacy addresses in IPv6 are very valuable for a reason that due to a fast refresh cycle they help to uncover a lot of corner cases that would otherwise pop up with a kind of "shrug, something weird" type of RCA. And fix them before the IPv6 usage hits 50% (cf my comments on the other thread). (From the privacy standpoint, privacy addresses are a pretty entertainingly weird construct.) So to summarize this somewhat long-winded post: no question there's a problem. Whether it's in "IPv6 the protocol" or "IPv6 the vendor implementation" or "IPv6 the configuration" is another story. And that'd be something to understand and fix - but needs more info/context/legwork IMO.

kashkhan 12y ago
I have dealt with MIT network admins who while not infallible are pretty darn competent. If people like this don't know or cannot fix the problem then there is a very real issue with the process by a which ipv6 is taught. It is not reasonable to have a super duper airplane if the pilots you have cannot fly it safely. The complexity and general human unfriendliness is a major downgrade from ip4 to ip6.
- ay 12y ago
  By no means there's any doubt in MIT admins' competency. Just that by reading the encounter, I feel sorry for the cards they've been dealt with, so to say: 1) a s/w hang for which there was no way to advance in a getting a solid RCA. This alone is a very hard one, and I know the ugly feeling both from the customer's side and from the vendor's side when all you have as an action plan is to try to reduce the potential exposure to what you think the problem is. I've been there myself multiple times, both as a customer, and as a vendor support. 2) Bridge loops. You can fix an active loop by disconnecting links, but root-causing such a problem is not a piece of cake, by far. 3) A buggy client firmware. This particular nastiness has been unfortunately a bane of quite a few networks already - see http://packetpushers.net/good-nics-bad-things-blast-ipv6-multicast-listener-discovery-queries/ http://packetpushers.net/good-nics-bad-things-blast-ipv6-mul... for the details about it, or https://www.mail-archive.com/cisco-nsp@puck.nether.net/msg55556.html https://www.mail-archive.com/cisco-nsp@puck.nether.net/msg55... for the first-hand encounters. 4) TCAM overload (?) due to privacy addresses rotation. "TCAM space dedicated to IPv6 multicast is only 3,000 entries. That would be just barely enough to support all of CSAIL". The network gear frequently allows to re-carve the TCAM space - maybe they might have taken some space from IPv4 which is by default is provisioned much more generously (for various reasons outside the scope for this thread). Besides just brute-forcing, there might be other strategies in dealing with the temporary addresses - actually shortening the valid lifetime and bringing it closer to the preferred lifetime should work. This gives me some useful material to update my IPv6 troubleshooting talk with. There are other scenarios in other setups that one can go out of TCAM, see e.g. http://lists.cluenet.de/pipermail/ipv6-ops/2013-October/009416.html http://lists.cluenet.de/pipermail/ipv6-ops/2013-October/0094... So, no, MIT folks are by far not stupid - it's the problem that is complicated - and, after staying up for 40 hours, whatever helps to reduce the pain is good. But I'm confident there's more learning here to be done from this unfortunate incident.

by ay 12y ago

kashkhan 12y ago

I have dealt with MIT network admins who while not infallible are pretty darn competent. If people like this don't know or cannot fix the problem then there is a very real issue with the process by a which ipv6 is taught. It is not reasonable to have a super duper airplane if the pilots you have cannot fly it safely. The complexity and general human unfriendliness is a major downgrade from ip4 to ip6.

ay 12y ago
By no means there's any doubt in MIT admins' competency. Just that by reading the encounter, I feel sorry for the cards they've been dealt with, so to say: 1) a s/w hang for which there was no way to advance in a getting a solid RCA. This alone is a very hard one, and I know the ugly feeling both from the customer's side and from the vendor's side when all you have as an action plan is to try to reduce the potential exposure to what you think the problem is. I've been there myself multiple times, both as a customer, and as a vendor support. 2) Bridge loops. You can fix an active loop by disconnecting links, but root-causing such a problem is not a piece of cake, by far. 3) A buggy client firmware. This particular nastiness has been unfortunately a bane of quite a few networks already - see http://packetpushers.net/good-nics-bad-things-blast-ipv6-multicast-listener-discovery-queries/ http://packetpushers.net/good-nics-bad-things-blast-ipv6-mul... for the details about it, or https://www.mail-archive.com/cisco-nsp@puck.nether.net/msg55556.html https://www.mail-archive.com/cisco-nsp@puck.nether.net/msg55... for the first-hand encounters. 4) TCAM overload (?) due to privacy addresses rotation. "TCAM space dedicated to IPv6 multicast is only 3,000 entries. That would be just barely enough to support all of CSAIL". The network gear frequently allows to re-carve the TCAM space - maybe they might have taken some space from IPv4 which is by default is provisioned much more generously (for various reasons outside the scope for this thread). Besides just brute-forcing, there might be other strategies in dealing with the temporary addresses - actually shortening the valid lifetime and bringing it closer to the preferred lifetime should work. This gives me some useful material to update my IPv6 troubleshooting talk with. There are other scenarios in other setups that one can go out of TCAM, see e.g. http://lists.cluenet.de/pipermail/ipv6-ops/2013-October/009416.html http://lists.cluenet.de/pipermail/ipv6-ops/2013-October/0094... So, no, MIT folks are by far not stupid - it's the problem that is complicated - and, after staying up for 40 hours, whatever helps to reduce the pain is good. But I'm confident there's more learning here to be done from this unfortunate incident.