10 ms·
IPv6 privacy addresses crashed the MIT CSAIL network
- acd 12y agoAdvocate the engineering principle fault domains to isolate the problem with L2 broadcasts with L3 routers between the L2.
- tgflynn 12y agoAs someone watching from the sidelines I had no idea there were such major issues with IPv6. It seems like IPv6 has been out there for a long time (about 10 years) in terms of being supported by OS's and networking hardware, if not ISP's. So I would have thought that cutting edge institutions (like MIT) would already have years of experience with it and have worked out most of the kinks by now. If this is not the case what does it mean for more widespread IPv6 adoption ? If such adoption is significantly delayed or stalled what will the consequences be, both for current Internet growth in the face of IPv4 address depletion and for new technologies like IoT ?
- tedunangst 12y agoIPv6 has been sold as "just like IPv4, but bigger" but it's not. It's much more than that, leading to the schism between the "adopt or die" and "not on my watch" people.
- p1mrx 12y agoThat depends which part of the stack you're working on. If you're writing web apps that receive a REMOTE_ADDR field, then the protocols look very similar; you just need to parse and store bigger values, and perhaps account for the fact that users tend to control a prefix instead of a single address. At the BGP and packet forwarding layers, everything's the same, with bigger numbers. The linked article relates to IPv6 over Ethernet, which is the area where IPv6 added the most new features and quirks.
- tedunangst 12y agoSure. The people who got "128-bit IPv4" aren't the ones complaining. The network admins who deal with all the rest of ipv6 are the ones who don't like the imposition.
- wmf 12y agoAlmost all large organizations decided to delay adopting IPv6 until the last possible moment because this reduces various costs. But this meant that many deployment problems were not discovered until very late.
- IvyMike 12y ago> what does it mean for more widespread IPv6 adoption? If I were a security hacker (whatever shade), I'd be spending all of my time looking at IPv6, from device drivers through kernels and routers up to user-level programs. IPv6 is simply not hardened in the way IPv4 is. It will be some day, but it's going to have to earn it the hard way, same as IPv4 did.
- spydum 12y agoIndeed, IPV6 is a possible entry point for pen testers, as plenty of admins will leave ipv6 enabled, but not bother to setup ip6tables. Plenty of software is setup to bind on "all interfaces", which includes your link local address. Fortunately, takes some effort to scan the link local address space.. But if they are based on MAC, and you can just ask for the MAC of the host via ARP...
- danielweber 12y agoI've got a bit of a headache right now, but IIRC the second remotely exploitable OpenBSD bug was because they had IPv6 enabled by default. There are a lot of grues lurking in there.
- hueving 12y agoThe parent wasn't just implying it being enabled, but also likely buggy in the implementation as well. E.g. Maybe an overflow with the next header implementation.
- ay 12y agoDepending on software, you might just listen for the DAD upon bootup or the service advertisements, of which any host today emits plenty... On the topic of the IPv6 implementation testing per se: https://www.thc.org/thc-ipv6/ https://www.thc.org/thc-ipv6/ plug-and-play type set of tools, allows you to try out some preexisting weaknesses/attacks. http://www.si6networks.com/tools/ipv6toolkit/ http://www.si6networks.com/tools/ipv6toolkit/ Not an "attack toolkit" per se but rather a manipulation toolkit - much more powerful but requires learning IPv6 quite in depth.
- 12y ago
- ChuckMcM 12y agoI haven't been on the front lines of new protocol deployment for a long time now, but the pattern then (and it appears unchanged) was that larger deployments brought out 2nd and 3rd order issues with the protocol. The old joke was "How can you tell someone is a pioneer?" answer, "Count the number of arrows in their back." which expressed that folks who adopted new protocols bore much of the burden of their failure and revision. Sounds like CSAIL has made some great progress in this respect.
- ay 12y agoA very astute observation! But... There are already quite large both enterprise and service provider networks already using IPv6. It's more in-between the "pioneers did not document the thorns they hit, so the others would not" and "the future is not evenly distributed yet, so we don't know about it" territory. I've volunteered myself to understand which of the two and to do whatever is actionable.
- ay 12y agoDon't mix the "mad professors" and those who has to ensure the lights are on. The criteria are very different. The consequences for the widespread IPv6 adoption is that one needs to do more RTFM and know vendor's escalation paths.
- spindritf 12y agothe random address is changed regularly, typically daily, but the old random addresses are kept around for a fairly long time I don't understand this part. I have Ubuntu machines in a network which is technically /48 but only one /64 prefix is announced by radvd and they all have only two addresses, one derived from MAC and one private/random changing over time. They certainly never have eight. Are those previous addresses not visable in ifconfig or ip -6 addr show?
- justincormack 12y agoI think this may have changed (in 14.04?) as I remember having more addresses in the past but don't now.
- spindritf 12y agoI have 14.04s and OP talks about 14.04s. Thus, a typical machine — say, an Ubuntu 14.04 workstation with the default configuration
- colmmacc 12y agoA significant factor is that privacy addresses won't be released if there are active connections using that address. So if the machine is running something with long lived connections; big downloads, push notifications, IM clients, ssh sessions and so on, then it can end up a growing stack of unreleased addresses. A typical admin who ssh's to a few boxes every so often, but then leaves those connections open in a tmux/screen/terminal-tab session can cause havoc.
- X-Istence 12y agoI haven't seen my Ubuntu servers use privacy addresses, however Ubuntu desktop does use privacy addresses, and I've seen my Ubuntu desktop grow more IPv6 privacy addresses over time on my local network. And they are visible in ifconfig or ip addr.
- AaronFriel 12y agoI used Ubuntu as an example, but it is hardly the worst offender. We have seen Windows machines with more than 300 IPv6 addresses Wow! I don't operate a very large network, but I do operate an IPv6 network and I've never seen one of our machines use more than 2 addresses. I feel like they've got some other configuration option or oddity going on that's causing a lot of these problems, but I am guessing they're much smarter than I am, so I don't know what to say. Could someone elaborate on this? I've never seen this behavior on an IPv6 network, and I'm just running a server or two with radvd and no custom switch configuration.
- rjsw 12y agoDo you use privacy addresses ? I don't, the /64 is identifiable as "my house" so there is no reason to hide which particular machine is doing something.
- mgbmtl 12y agoA house may have multiple people living there, multiple devices, visitors, etc. You could track a person connecting through multiple locations by using their MAC address used to assign their IP. i.e. if I am connecting with my laptop from various places, my IP address will be roughly the same, only the network prefix will change.
- rjsw 12y agoA house could have multiple users, mine doesn't. I would maybe see turning on privacy addresses as part of a general "leaving home" script that also turned on other stuff like the firewall.
- RRRA 12y agoTime to participate in open wifi or a mesh network :-)
- ay 12y agoYou are completely right. Privacy IPv6 address are an illusion when used in a household that has effectively an active /64 per subscriber. (Those who say "but IPv4..." last time my IPv4 address changed, was half a year ago). Anyway this is mostly a political give-in, that happens to also help ruggedize the rest of the stack (the relatively rapid change of privacy addresses uncovers more bugs than we'd otherwise find - so from that standpoint they're to be advocated). But privacy.. huh. Last IETF there was precisely this discussion that one might want to force a new /64 on themselves, and privacy addresses have nothing to do with that. (DHC WG, FWIW).
- p1mrx 12y agoHere's a proposed algorithm for making privacy addresses more manageable: http://tools.ietf.org/html/rfc7217 http://tools.ietf.org/html/rfc7217 Essentially, the suffix is hash(secret | prefix), so your address is stable on a given network, but changes as you roam between networks.
- cm2187 12y agoAs I mentioned above, this misses the point. All you are getting is that instead of having a unique identifier (your MAC) you have 4 or 5 (the number of networks you roam between within a day). This will be trivial to defeat with a browser cookie. The identifier are still static which means that it only takes one observation to establish a permanent link between these IPs.
- nly 12y agoA small domain encryption scheme, rather than a hash, would make more sense... that way there'd be no collisions. SDE isn't that much harder to put together in hardware.
- p1mrx 12y agoThe subnet space is 64 bits long. Collisions between pseudo-random values only become likely when you have billions of nodes, so it's not worth the effort to preemptively avoid them. This would also require all nodes to implement the same algorithm, which means it's not incrementally deployable. Edit: The actual RFC7217 algorithm contains a DAD_Counter field in the hash input. In the event of a collision, the counter increments, generating a new address.
- MichaelGG 12y agoWhat's the reasoning for dropping ARP? It seemed like a simple architecture. The post seems to indicate IPv6 requires a ton more hardware resources. And if Juniper doesn't have a basic feature like MLD snooping after all this time, uh? Shouldn't practically designing a high-volume switch be part of creating such a fundamental protocol? (I know designing 2 elegant implementations of other protocols would have fixed a ton of things in nasty protocols like HTTP - dumbass things like line folding and comments-in-headers.) Is this a case of idiocy seeping through the IETF because they can? It's pretty easy to write something down on paper if you don't have to implement engineering and product management on the result. Or because you're out of touch with reality, like the source routing feature which was kept in IPv6 despite it only ever being a problem in IPv4? Or is this a case of the protocol being superior and vendors just being very lazy?
- akira2501 12y agoIt seems like they really wanted to push multicast as a "first-class" mode of the protocol, and by weaving it into the core of IPv6 they've forced everyone to have a somewhat exercised implementation of it in their stack.
- ChuckMcM 12y agoThis was my take on it as well. Multicast was a very under deployed system, one way to fix that is to make it mandatory :-)
- MichaelGG 12y agoSo pushing something no one uses, and has significantly narrow uses, and still isn't deployed publicly, into the core... How is that anything but self-serving protocol designers? How do they get away with that?
- teach 12y agoMulticast has much more than "significantly narrow uses". When we were studying IPv6 in my network protocols course 20 years ago(!) I remember thinking how useful that would be if widely deployed.
- walshemj 12y agoIt would have been interesting to see a network diagram.
- ay 12y agoThe "issues with IPv6" are with education, operation, configuration. I personally ran WiFi networks with 8000+ wireless clients on a single /64 subnet (my employer's CiscoLive conference), and assisted/consulted in running the networks with more than 25000 clients on a single /64 subnet (Mobile World Congress). The default kinda suck, and the bugs may happen. but the statement "IPv6 is not ready for production" is wrong. I'd be happy to volunteer a reasonable amount of time to work with the OP or others having a network of >1000 hosts, to debug the issues like this, time permitting, vendor independent. (glass houses and all that). There are bazillion knobs in IPv6 and a lot of things can be fixed by just tweaking the defaults (which kinda suck). Network of <500-700 nodes generally don't need to bother much. It's not optimal a lot of times with the defaults but it will work. EDIT: the seeming "charity" of volunteering the time isn't. I want to understand what is broken, so we can bring it up back to IETF and get it fixed + make better educational publicity to prevent folks shooting themselves in the foot. It'll make it to the stacks in another decade, but it will. IPv6 is powering nontrivial millions of hosts now - so the correct words to use "needs tweaking for my network", not "not ready for production". Let's see what the tweaks are and if we can incorporate them into the protocol, if necessary.
- MichaelGG 12y agoWhy add such knobs, if they weren't there in the first place, haven't been tested, and obviously are causing nontrivial problems?
- ay 12y agoEverything unknown of is untrivial - and I'll leave it at that.
- calinet6 12y agoIt seems to me that the presence of a "bazillion knobs" is a bug in itself. A stable system can not depend on puny humans for its operation—nor should a design be able to blame puny humans for its failure.
- 12y ago
- praseodym 12y agoWe've also hit this Intel Ethernet driver bug, even though we don't have IPv6 deployed. Linux will send MLD packets on bridged ports by default, triggering the Intel driver bug on Windows machines. With only two Windows machines saturating their Gigabit Ethernet connection whenever they went into standby, we managed to crash the university's switches big time (we're a group with our own VLAN within the university's network, so we make use of their network equipment). Naturally, because the issue only occurs during standby, and usually users don't log off thus preventing Windows from sleeping, we first hit the bug during the Christmas holidays (2013). The culprit hosts were all in use for just a couple of months. In the end, it took a couple of hours to reproduce this bug during working hours! We fixed it by using different NICs (we didn't want to rely on the Intel driver to be updated after a clean install; Windows Update doesn't have the fixed version), and by disabling MLD snooping on the Linux hosts, since we aren't yet using IPv6 anyways. This prevents the Intel bug from being triggered in our environment.
- s_q_b 12y agoThis is now the second major network that I've heard ran into this exact problem. The update to Windows caused the end-user nodes to send out lots of IPv6 packets The access layer switches went to full CPU utilization, and you ended up with packet storms across the network. There really should be an advisory about this.
- mrb 12y agoAn issue glossed over by people is: "the entire TCAM space dedicated to IPv6 multicast is only 3,000 entries" Some mid-level switches normally have TCAM space for hundreds of thousands, or millions of entries, IPv4 or IPv6. Maybe their vendor artificially crippled their line of switches, or maybe the switches were deployed with a configuration error. It is probably the former though. Network vendors like to make you believe some features cost a lot to implement and that you really need their highest-level gear, when in fact even the biggest TCAM in silicon cost a few tens of dollars, at most.
- smutticus 12y agoThe title should probably read,"Bugs in JunOS caused network downtime." This isn't really news. There are bugs in all routing and switching OS's. That's why they hire support people. This isn't me trying to rag on Juniper. I know lots of people who work for JTAC and they're incredibly smart folks. I'm sure they'll get this sorted out and fixed in JunOS and, like any bug, merged into upstream releases. This isn't me trying to point out that IPv6 is infallible. There might be some design choices that the IETF made with IPv6 that were stupid, but mostly they got it right, and it's too late now to change most of them anyways. This is just reality with new software. New software has bugs. You can blame it on MLD, but really MLD is no more complicated than IGMP. You can blame it on NDP, but really NDP isn't much more complicated than ARP. At a minimum IPv4 required ARP to function. However, in reality it also required atleast IGMPv2. Since without IGMP, or some way to manage multicast, how are you going to get something like VRRP to work. Link-layer multicast is not new to IPv6.
- ay 12y ago++. A small nit to add: IPv4 did not have 1+ multicast groups per every host. That's a dramatic difference in terms of capacity on the middle-gear which escapes if one thinks of IPv6 as "IPv4 with bigger addresses".
- smutticus 12y agoThanks for the upvote :) And you're right about the additional mcast addresses required by IPv6 NDP. It's called the 'solicited-node multicast address' that every host must join. My historical guess about why multicast was chosen over broadcast for NDP was due to NBMA networks. In the 1990's NBMA networks(frame-relay) were much more common than they are today. And NDP just makes more sense than ARP over NBMA networks. This is just a guess. Someone else suggested DOCSIS was the reason that multicast was chosen instead of broadcast. I doubt this. I'm not that familiar with DOCSIS, but I think it has a broadcast type link-layer. Also, IPv6 predates DOCSIS. It could well turn out that multicast was the right choice for NDP once we get over the inevitable roll out problems. NBMA networks could return in 20-30 years. You never know.
- windexh8er 12y agoWow. I have no idea where to even start - this article was written by someone who has no large scale IPv6 deployment experience. There are errors upon, back-to-back, errors in what's assumed and the expected results and assertions with the vendor (Juniper) and the protocol operation (IPv6). I'm not surprised that it's towards the top of HN but it shows the relative understanding of the HN crowd with regard to complex network related topics.
- ghshephard 12y agoI'm pretty familiar with "complex network related topics" - and I think this is the most interesting IPv6 related post on HN in 2+ years (there may have been others that got by me). I"m curious - are you someone with "large scale IPv6 deployment experience?" - in particularly, how would you have approached their issues regarding MLD snooping and TCAM exhaustion?
- windexh8er 12y agoIt may be interesting, but my point was that there is nothing in this article to learn from with the exception of problematic code from a network vendor. Yes - I am. I deployed a 4 state IPv6 overlay servicing 250k subscribers on one of the very first (fully rolled out) DOCSIS 3 HFC networks in the US. I was responsible for the security and performance of the architecture. This rollout started in 2010. The TCAM exhaustion was self inflicted based on the hints of design throughout the article and the original authors understand of MLD is, just generally, incorrect unfortunately.
- dang 12y agoThis comment would be better if instead of insulting its audience, it taught them something. Many people here would appreciate a chance to improve their "relative understanding". As it is, although it alludes to questions of substance, it doesn't actually say anything about them, and therefore is just noise.
- windexh8er 12y agoI think that is a very telling point in and of itself. I've sat on the boundary between the devops / security and networking world my entire career. I know many people who are great at *nix and tooling of backend infrastructure but choose to ignore the plumbing, and complexity contained within, the planning and architecture that can make the pieces above it that much more successful. When I read an article like this there are blatant points that showcase how green the OP is with regard to networking. I happen to have an advantage in this space and the point being is that I don't claim any expertise in compiler design, never will. However - it pays to bring people in to help. While you only see the story from the OPs perspective there's an entirely different side from the vendor most likely (probably around fundamental design if I had to throw a dart). Sure, I knew I'd get downvoted for the post - but sometimes that's a calculated position to show a point, and my point is - sometimes you need help, this article was very one-sided and rife with flaws. Not something I generally expect to read on HN - but I know there is likely a lot of that out there that I "just miss" because I know that I don't know.
- ghshephard 12y agoNobody on this thread really seems to be talking about one of the issues brought up in the IPv6 analysis (though I'm not sure if it caused their outages - any time I read a post mortem with the phrase, "bridge loops" I usually don't look any further - that alone is enough to bring a network down) If I read the post correctly, one of the roots of their problems seems to be that either (A) Traffic flooding causing excess traffic as a result of multicast packets being flooded over their network, or (B) If they used MLD snooping to reduce the flooding, the switches they have only support 3,000 entries for multicast groups - which are quickly exceeded with the privacy IPv6 addresses that are generated by the hosts (each of which creates it's own multicast entry, and some of their hosts had 10+ addresses) Other than turning off privacy based IPv6 addresses, and moving to something like RFC 7217, is there a solution? Increasing the number of multicast entries on the switch to something larger, say, around 30,000 entries combined with reducing the length of time in which a privacy address is valid (and therefore requiring a group) from one day, to say, one hour?
- deleted 12y ago[deleted]
- rogerrabbitt 12y agomy guess is NSA or some other TLA was attacking Roger Dingledine's directory server on the Tor network, which runs on MIT CSAIL's network: https://atlas.torproject.org/#details/9695DFC35FFEB861329B9F1AB04C46397020CE31 https://atlas.torproject.org/#details/9695DFC35FFEB861329B9F... Does this JunOS/IPv6 issue look more like a bugdoor or backdoor scenario? I believe it's been reported that NSA had 0days on JunOS...