6 ms·

I wonder if this is a problem for any cloud provider, I also wonder if ipv6 could help mitigate? Then the IP collusions would be rarer.

schimmy_changa 12y ago
I think you are definitely right that IPv6 would mitigate, but I think that the real way to avoid this issue is for the kernel to make the tradeoff for you - it's so cheap to do an arp ping, you might as well on maybe every 100th packet or so. Or similar: timing out and clearing the arp entries more frequently. As for other cloud providers, I'd be really interested to hear from Digital Ocean, Rackspace, and Google!
- pyvpx 12y ago
  cheap for one server...definitely not cheap at AWS scale.
- jsolson 12y ago
  Answered parallel to you in this thread: https://news.ycombinator.com/item?id=8732356 https://news.ycombinator.com/item?id=8732356
pquerna 12y ago
It's common on these provider "isolated" networks to use a small /24 subnet by default. I'm more surprised that AWS isn't sending a gratuitous ARP when IPs are re-cycled to mitigate issues like this.
- windexh8er 12y ago
  That would mean that one of two things would need to occur: 1) Amazon sends GARP on behalf of the guest OS, from outside the kernel 2) Amazon has hooks into the guest OS to instruct the GARP to occur Both are very bad things with regard to customer / Amazon separation of privileges and control. I think every network engineer was probably yelling ARP at the screen early on in this write up and is likely one of the first things those seasoned in L2/L3 would look at first. Also, as others have stated IPv6 removes this problem altogether. With the complete gutting of the concept of a broadcast and replaced by announcements within multicast groups which are much more efficient when you're talking about "small" prefixes that house 18,446,744,073,709,552,000 hosts (/64). ARP tables wouldn't be able to scale or work efficiently. Overall this makes me think that it would be interesting to build a best practices guide around IPv4 and IPv6 networking within cloud provider environments. I think the gratuitous ARP on boot is a relatively safe practice, especially with regard to environments that are in continual flux. $0.02.
justincormack 12y ago
ipv6 doesnt actually use arp, so it is not going to be exactly the same anyway.
jsolson 12y ago
This particular problem would not crop up on Google Compute Engine. See this cryptic snippet from /sbin/ifconfig on a production GCE VM: eth0 Link encap:Ethernet HWaddr 42:01:0a:f0:40:dd inet addr:10.240.64.221 Bcast:10.240.64.221 Mask:255.255.255.255 This isn't an intentional feature, it's just a property of how our Andromeda SDN[0] is wired up. In particular, we lift the business end of figuring out where the other hosts on your LAN are up into the SDN rather than relying on the guest to cache ARP entries (hence the /32 netmask). [0]: http://googlecloudplatform.blogspot.com/2014/04/enter-andromeda-zone-google-cloud-platforms-latest-networking-stack.html http://googlecloudplatform.blogspot.com/2014/04/enter-androm...
- jbeda 12y ago
  Actually, this is intentional. Pre-andromeda, we hit ARP caching issues when testing scale for GCE. The model to let the SDN figure out the other end (and stub out things like ARP) was carried forward with Andromeda. (I'm the original TL for GCE)
  - jsolson 12y ago
    Oh, interesting. I didn't know it was actually in response to ARP issues. By the time I showed up it was just how bits were wired. Apparently I should've gone to the source :)

schimmy_changa 12y ago

I think you are definitely right that IPv6 would mitigate, but I think that the real way to avoid this issue is for the kernel to make the tradeoff for you - it's so cheap to do an arp ping, you might as well on maybe every 100th packet or so. Or similar: timing out and clearing the arp entries more frequently. As for other cloud providers, I'd be really interested to hear from Digital Ocean, Rackspace, and Google!

pyvpx 12y ago
cheap for one server...definitely not cheap at AWS scale.
jsolson 12y ago
Answered parallel to you in this thread: https://news.ycombinator.com/item?id=8732356 https://news.ycombinator.com/item?id=8732356

pquerna 12y ago

It's common on these provider "isolated" networks to use a small /24 subnet by default. I'm more surprised that AWS isn't sending a gratuitous ARP when IPs are re-cycled to mitigate issues like this.

windexh8er 12y ago
That would mean that one of two things would need to occur: 1) Amazon sends GARP on behalf of the guest OS, from outside the kernel 2) Amazon has hooks into the guest OS to instruct the GARP to occur Both are very bad things with regard to customer / Amazon separation of privileges and control. I think every network engineer was probably yelling ARP at the screen early on in this write up and is likely one of the first things those seasoned in L2/L3 would look at first. Also, as others have stated IPv6 removes this problem altogether. With the complete gutting of the concept of a broadcast and replaced by announcements within multicast groups which are much more efficient when you're talking about "small" prefixes that house 18,446,744,073,709,552,000 hosts (/64). ARP tables wouldn't be able to scale or work efficiently. Overall this makes me think that it would be interesting to build a best practices guide around IPv4 and IPv6 networking within cloud provider environments. I think the gratuitous ARP on boot is a relatively safe practice, especially with regard to environments that are in continual flux. $0.02.

justincormack 12y ago

ipv6 doesnt actually use arp, so it is not going to be exactly the same anyway.

jsolson 12y ago

This particular problem would not crop up on Google Compute Engine. See this cryptic snippet from /sbin/ifconfig on a production GCE VM: eth0 Link encap:Ethernet HWaddr 42:01:0a:f0:40:dd inet addr:10.240.64.221 Bcast:10.240.64.221 Mask:255.255.255.255 This isn't an intentional feature, it's just a property of how our Andromeda SDN[0] is wired up. In particular, we lift the business end of figuring out where the other hosts on your LAN are up into the SDN rather than relying on the guest to cache ARP entries (hence the /32 netmask). [0]: http://googlecloudplatform.blogspot.com/2014/04/enter-andromeda-zone-google-cloud-platforms-latest-networking-stack.html http://googlecloudplatform.blogspot.com/2014/04/enter-androm...

jbeda 12y ago
Actually, this is intentional. Pre-andromeda, we hit ARP caching issues when testing scale for GCE. The model to let the SDN figure out the other end (and stub out things like ARP) was carried forward with Andromeda. (I'm the original TL for GCE)
- jsolson 12y ago
  Oh, interesting. I didn't know it was actually in response to ARP issues. By the time I showed up it was just how bits were wired. Apparently I should've gone to the source :)