3 ms·
By no means there's any doubt in MIT admins' competency. Just that by reading the encounter, I feel sorry for the cards they've been dealt with, so to say: 1)
by ay 12y ago
By no means there's any doubt in MIT admins' competency. Just that by reading the encounter, I feel sorry for the cards they've been dealt with, so to say:
1) a s/w hang for which there was no way to advance in a getting a solid RCA. This alone is a very hard one, and I know the ugly feeling both from the customer's side and from the vendor's side when all you have as an action plan is to try to reduce the potential exposure to what you think the problem is. I've been there myself multiple times, both as a customer, and as a vendor support.
2) Bridge loops. You can fix an active loop by disconnecting links, but root-causing such a problem is not a piece of cake, by far.
3) A buggy client firmware. This particular nastiness has been unfortunately a bane of quite a few networks already - see http://packetpushers.net/good-nics-bad-things-blast-ipv6-multicast-listener-discovery-queries/ http://packetpushers.net/good-nics-bad-things-blast-ipv6-mul... for the details about it, or https://www.mail-archive.com/cisco-nsp@puck.nether.net/msg55556.html https://www.mail-archive.com/cisco-nsp@puck.nether.net/msg55... for the first-hand encounters.
4) TCAM overload (?) due to privacy addresses rotation. "TCAM space dedicated to IPv6 multicast is only 3,000 entries. That would be just barely enough to support all of CSAIL". The network gear frequently allows to re-carve the TCAM space - maybe they might have taken some space from IPv4 which is by default is provisioned much more generously (for various reasons outside the scope for this thread).
Besides just brute-forcing, there might be other strategies in dealing with the temporary addresses - actually shortening the valid lifetime and bringing it closer to the preferred lifetime should work. This gives me some useful material to update my IPv6 troubleshooting talk with.
There are other scenarios in other setups that one can go out of TCAM, see e.g. http://lists.cluenet.de/pipermail/ipv6-ops/2013-October/009416.html http://lists.cluenet.de/pipermail/ipv6-ops/2013-October/0094...
So, no, MIT folks are by far not stupid - it's the problem that is complicated - and, after staying up for 40 hours, whatever helps to reduce the pain is good.
But I'm confident there's more learning here to be done from this unfortunate incident.