16 ms·

When your IP traffic in AWS disappears into a black hole

185 points by schimmy_changa 12y ago

maerF0x0 12y ago
I wonder if this is a problem for any cloud provider, I also wonder if ipv6 could help mitigate? Then the IP collusions would be rarer.
- schimmy_changa 12y ago
  I think you are definitely right that IPv6 would mitigate, but I think that the real way to avoid this issue is for the kernel to make the tradeoff for you - it's so cheap to do an arp ping, you might as well on maybe every 100th packet or so. Or similar: timing out and clearing the arp entries more frequently. As for other cloud providers, I'd be really interested to hear from Digital Ocean, Rackspace, and Google!
  - pyvpx 12y ago
    cheap for one server...definitely not cheap at AWS scale.
  - jsolson 12y ago
    Answered parallel to you in this thread: https://news.ycombinator.com/item?id=8732356 https://news.ycombinator.com/item?id=8732356
- pquerna 12y ago
  It's common on these provider "isolated" networks to use a small /24 subnet by default. I'm more surprised that AWS isn't sending a gratuitous ARP when IPs are re-cycled to mitigate issues like this.
  - windexh8er 12y ago
    That would mean that one of two things would need to occur: 1) Amazon sends GARP on behalf of the guest OS, from outside the kernel 2) Amazon has hooks into the guest OS to instruct the GARP to occur Both are very bad things with regard to customer / Amazon separation of privileges and control. I think every network engineer was probably yelling ARP at the screen early on in this write up and is likely one of the first things those seasoned in L2/L3 would look at first. Also, as others have stated IPv6 removes this problem altogether. With the complete gutting of the concept of a broadcast and replaced by announcements within multicast groups which are much more efficient when you're talking about "small" prefixes that house 18,446,744,073,709,552,000 hosts (/64). ARP tables wouldn't be able to scale or work efficiently. Overall this makes me think that it would be interesting to build a best practices guide around IPv4 and IPv6 networking within cloud provider environments. I think the gratuitous ARP on boot is a relatively safe practice, especially with regard to environments that are in continual flux. $0.02.
- justincormack 12y ago
  ipv6 doesnt actually use arp, so it is not going to be exactly the same anyway.
- jsolson 12y ago
  This particular problem would not crop up on Google Compute Engine. See this cryptic snippet from /sbin/ifconfig on a production GCE VM: eth0 Link encap:Ethernet HWaddr 42:01:0a:f0:40:dd inet addr:10.240.64.221 Bcast:10.240.64.221 Mask:255.255.255.255 This isn't an intentional feature, it's just a property of how our Andromeda SDN[0] is wired up. In particular, we lift the business end of figuring out where the other hosts on your LAN are up into the SDN rather than relying on the guest to cache ARP entries (hence the /32 netmask). [0]: http://googlecloudplatform.blogspot.com/2014/04/enter-andromeda-zone-google-cloud-platforms-latest-networking-stack.html http://googlecloudplatform.blogspot.com/2014/04/enter-androm...
  - jbeda 12y ago
    Actually, this is intentional. Pre-andromeda, we hit ARP caching issues when testing scale for GCE. The model to let the SDN figure out the other end (and stub out things like ARP) was carried forward with Andromeda. (I'm the original TL for GCE)
    - jsolson 12y ago
      Oh, interesting. I didn't know it was actually in response to ARP issues. By the time I showed up it was just how bits were wired. Apparently I should've gone to the source :)
schimmy_changa 12y ago
I think the biggest thing I was surprised by with this investigation was the lack of documentation about data-layer tools. At one point I was looking through the source of the 'ip' command to try to find out exactly which conditions caused a 'STALE' entry in the ARP table...
tastynacho 12y ago
Welcome to 40 years ago :/ Pretty basic ARP issue you're suppose to learn in admin 101. It's also a fairly standard interview question for anyone interviewing for a devops/admin/operations position
- forgottenpass 12y ago
  This website is full of green kids either at startups or who want to be. They're generally not going to have juicy issues to write post-mortems about on an engineering blog so they can causally mention they're hiring. You either lower your standards or get downvoted.
- j-kidd 12y ago
  You shall read this first: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1331150 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1331150 There was a rather significant change to the kernel ARP caching behavior introduced in early 2013: https://github.com/torvalds/linux/commit/2724680bceee94eac391552863771af105a7355c https://github.com/torvalds/linux/commit/2724680bceee94eac39... It seems to work fine everywhere, except in EC2 VPC, where the arp cache can sometime becomes stale. We too reported this issue to AWS support, but have no idea if they are doing anything about it. The workaround is to apply a sysctl change to revert to the old behavior prior to the commit. Or to use a subnet larger than /24 to reduce the chance of getting the same IP.
ChuckMcM 12y ago
Interesting trace down to stale ARP entries. It gets worse when the switches are running mac address filtering and they get out of date. We had that issue with some Blade G8052 top of rack switches with their upstream 10G ports. They sometimes "forget" which upstream port has the MAC address that they are switching too, and those packets just spew out messily into the data center leaving a mess. The "fix" it to force the switch to ping up through a specific upstream port periodically to the center switch's management IP address. Sigh.
- p8952 12y ago
  I've seen the exact same issue in some Dell PowerConnect switches. We only saw the issue when using the "iSCSI Auto-Config" mode. Manually configuring the switches with the same config but entered by hand resolved the issue.
danesparza 12y ago
Am I weird because I actually muttered 'ARP caching issue' halfway through your article? :-) Love the technical write-up -- thanks!
- schimmy_changa 12y ago
  I don't think you're weird, I think you're probably a great candidate to work as a sysadmin / SRE, or at Clever, we're hiring: https://clever.com/about/jobs#engineer-distributed-systems https://clever.com/about/jobs#engineer-distributed-systems !
  - justincormack 12y ago
    Your job descriptions list a lot of technologies but dont mention anything to do with operating systems or networking. It seems odd to explicitly call out Node and redis but not anything down the stack.
    - meowface 12y ago
      I don't think there's anything wrong with that. Anyone in a sysadmin or devops is going to be expected to be proficient at Linux and bash, Windows and Powershell for Windows desktop/server admins, Cisco and Juniper for networking, etc. These are kind of default assumptions for positions like these. More modern and highly specific technologies like HAproxy, Salt, Node, Redis are not going to be known by every experienced, competent applicant though, and while you can often pick them up on the job if you have general experience in the area, there's always going to be a learning curve.
    - schimmy_changa 12y ago
      It's a good point, but right now we don't do anything that relies specifically on one technology at the very low levels (for instance, kernel modules or very specific networking optimizations we don't need and would be over-optimization). Instead we try to maintain a SOA which is agnostic to what underlying tools back our services, something that we've found Docker is useful for. That being said, we generally just use Ubuntu, and have some other AWS-specific setups around networking. If you like hacking on Cisco routers, I think it's perhaps not the right position :)
falcolas 12y ago
Try an arping from the new workers on first startup? Ran into this quite a bit when using VIPs for DB failover, and an arping fixed the caching issue in most cases.
wahnfrieden 12y ago
FYI, Clever: I click "Engineering Blog" at the top, and all links to blog posts on that page 404.
- rgarcia 12y ago
  sorry about that. It's fixed now!
spectre256 12y ago
This reminds me of a time at a previous company years ago, where we experienced an issue that felt similar, although the root cause was quite different. Basically, we had multiple teams all launching/terminating web servers. Unfortunately, they were all in the same EC2 deployment, and more often than not our load balancers from one team would send traffic to the web servers of another team. Furthermore, our setups were similar enough that this would sometimes cause bad results for users. We fixed it by making sure that our web servers on every team spoke on different ports. Not elegant, but effective (until two teams accidentally picked the same ports). These days we have good enough infrastructure tools that this problem should never happen. But in 2009, at a company that was overwhelmed with growth, those sort of things happen.
- vosper 12y ago
  Would a better fix have been to have a load balancer per team, and then have the EC2 instance bind to the correct load balancer?
  - spectre256 12y ago
    That's what we had, but because the IPs were often recycled, traffic sometimes went to the wrong place if the load balancers has stale data.
perlgeek 12y ago
Wouldn't it be a better solution to not reuse IP addresses quickly? If I understood it correctly, they are in a private network anyway, so they could afford it.
- daveloyall 12y ago
  I thought there was a mechanism that would allow some sort of failure notification packet to make it back to the origin, so that it would then perform a proper ARP lookup WHO HAS 1.2.3.4? TELL 4.3.2.1!. Perhaps AWS is blocking some class of packet?
  - X-Istence 12y ago
    The issue is that if it is on-network the only real "failure" notification is a time-out. No ICMP packets, nothing are sent back to indicate you are sending to a non-existent IP address.
jcollins 12y ago
We saw this earlier this year after upgrading to a new Linux kernel. The solution for us was to set this in sysctl.conf: net.ipv4.neigh.default.gc_thresh1=0 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1331150/ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1331150... https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1331150/comments/12 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1331150...
kiyoto 12y ago
Looking at the port number, it looks like Clever is a MongoDB user =)
- akx 12y ago
  You, uh, don't even have to just recognize the port number, reading the article is enough... > [...] These workers all connect to a MongoDB replicaset to update and read data [...] > 27017 = port mongod is listening on
girvo 12y ago
We had a fascinating bug on EC2 -- we could connect to the instance, but no network traffic made it out. It wasn't security group problems, it was literally a really weird bug in EC2's network that we somehow triggered, the engineer over at Amazon that looked at it was really excited when he came across our case as it was so weird, heh. They fixed it, I can't remember exactly what was done on their end, but it was one of the weirder problems I've attempted to debug. Nothing I tried worked!
zenocon 12y ago
I just experienced this early this week. Very frustrating. I also posted to AWS forums and got zero assistance; am currently not paying for AWS support plan. This article came at an opportune moment -- it makes sense and removes the shroud of mystery around why it "works sometimes" which leaves me with an uneasy feeling for a production setup.
- schimmy_changa 12y ago
  awesome, I was hoping it could help out someone else :)
legohead 12y ago
I was going to respond with my little story, but I see your article already linked it! ;)

185 points by schimmy_changa 12y ago

I wonder if this is a problem for any cloud provider, I also wonder if ipv6 could help mitigate? Then the IP collusions would be rarer.

schimmy_changa 12y ago
I think you are definitely right that IPv6 would mitigate, but I think that the real way to avoid this issue is for the kernel to make the tradeoff for you - it's so cheap to do an arp ping, you might as well on maybe every 100th packet or so. Or similar: timing out and clearing the arp entries more frequently. As for other cloud providers, I'd be really interested to hear from Digital Ocean, Rackspace, and Google!
- pyvpx 12y ago
  cheap for one server...definitely not cheap at AWS scale.
- jsolson 12y ago
  Answered parallel to you in this thread: https://news.ycombinator.com/item?id=8732356 https://news.ycombinator.com/item?id=8732356
pquerna 12y ago
It's common on these provider "isolated" networks to use a small /24 subnet by default. I'm more surprised that AWS isn't sending a gratuitous ARP when IPs are re-cycled to mitigate issues like this.
- windexh8er 12y ago
  That would mean that one of two things would need to occur: 1) Amazon sends GARP on behalf of the guest OS, from outside the kernel 2) Amazon has hooks into the guest OS to instruct the GARP to occur Both are very bad things with regard to customer / Amazon separation of privileges and control. I think every network engineer was probably yelling ARP at the screen early on in this write up and is likely one of the first things those seasoned in L2/L3 would look at first. Also, as others have stated IPv6 removes this problem altogether. With the complete gutting of the concept of a broadcast and replaced by announcements within multicast groups which are much more efficient when you're talking about "small" prefixes that house 18,446,744,073,709,552,000 hosts (/64). ARP tables wouldn't be able to scale or work efficiently. Overall this makes me think that it would be interesting to build a best practices guide around IPv4 and IPv6 networking within cloud provider environments. I think the gratuitous ARP on boot is a relatively safe practice, especially with regard to environments that are in continual flux. $0.02.
justincormack 12y ago
ipv6 doesnt actually use arp, so it is not going to be exactly the same anyway.
jsolson 12y ago
This particular problem would not crop up on Google Compute Engine. See this cryptic snippet from /sbin/ifconfig on a production GCE VM: eth0 Link encap:Ethernet HWaddr 42:01:0a:f0:40:dd inet addr:10.240.64.221 Bcast:10.240.64.221 Mask:255.255.255.255 This isn't an intentional feature, it's just a property of how our Andromeda SDN[0] is wired up. In particular, we lift the business end of figuring out where the other hosts on your LAN are up into the SDN rather than relying on the guest to cache ARP entries (hence the /32 netmask). [0]: http://googlecloudplatform.blogspot.com/2014/04/enter-andromeda-zone-google-cloud-platforms-latest-networking-stack.html http://googlecloudplatform.blogspot.com/2014/04/enter-androm...
- jbeda 12y ago
  Actually, this is intentional. Pre-andromeda, we hit ARP caching issues when testing scale for GCE. The model to let the SDN figure out the other end (and stub out things like ARP) was carried forward with Andromeda. (I'm the original TL for GCE)
  - jsolson 12y ago
    Oh, interesting. I didn't know it was actually in response to ARP issues. By the time I showed up it was just how bits were wired. Apparently I should've gone to the source :)

schimmy_changa 12y ago

I think the biggest thing I was surprised by with this investigation was the lack of documentation about data-layer tools. At one point I was looking through the source of the 'ip' command to try to find out exactly which conditions caused a 'STALE' entry in the ARP table...

tastynacho 12y ago

Welcome to 40 years ago :/ Pretty basic ARP issue you're suppose to learn in admin 101. It's also a fairly standard interview question for anyone interviewing for a devops/admin/operations position

forgottenpass 12y ago
This website is full of green kids either at startups or who want to be. They're generally not going to have juicy issues to write post-mortems about on an engineering blog so they can causally mention they're hiring. You either lower your standards or get downvoted.
j-kidd 12y ago
You shall read this first: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1331150 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1331150 There was a rather significant change to the kernel ARP caching behavior introduced in early 2013: https://github.com/torvalds/linux/commit/2724680bceee94eac391552863771af105a7355c https://github.com/torvalds/linux/commit/2724680bceee94eac39... It seems to work fine everywhere, except in EC2 VPC, where the arp cache can sometime becomes stale. We too reported this issue to AWS support, but have no idea if they are doing anything about it. The workaround is to apply a sysctl change to revert to the old behavior prior to the commit. Or to use a subnet larger than /24 to reduce the chance of getting the same IP.

ChuckMcM 12y ago

Interesting trace down to stale ARP entries. It gets worse when the switches are running mac address filtering and they get out of date. We had that issue with some Blade G8052 top of rack switches with their upstream 10G ports. They sometimes "forget" which upstream port has the MAC address that they are switching too, and those packets just spew out messily into the data center leaving a mess. The "fix" it to force the switch to ping up through a specific upstream port periodically to the center switch's management IP address. Sigh.

p8952 12y ago
I've seen the exact same issue in some Dell PowerConnect switches. We only saw the issue when using the "iSCSI Auto-Config" mode. Manually configuring the switches with the same config but entered by hand resolved the issue.

danesparza 12y ago

Am I weird because I actually muttered 'ARP caching issue' halfway through your article? :-) Love the technical write-up -- thanks!

schimmy_changa 12y ago
I don't think you're weird, I think you're probably a great candidate to work as a sysadmin / SRE, or at Clever, we're hiring: https://clever.com/about/jobs#engineer-distributed-systems https://clever.com/about/jobs#engineer-distributed-systems !
- justincormack 12y ago
  Your job descriptions list a lot of technologies but dont mention anything to do with operating systems or networking. It seems odd to explicitly call out Node and redis but not anything down the stack.
  - meowface 12y ago
    I don't think there's anything wrong with that. Anyone in a sysadmin or devops is going to be expected to be proficient at Linux and bash, Windows and Powershell for Windows desktop/server admins, Cisco and Juniper for networking, etc. These are kind of default assumptions for positions like these. More modern and highly specific technologies like HAproxy, Salt, Node, Redis are not going to be known by every experienced, competent applicant though, and while you can often pick them up on the job if you have general experience in the area, there's always going to be a learning curve.
  - schimmy_changa 12y ago
    It's a good point, but right now we don't do anything that relies specifically on one technology at the very low levels (for instance, kernel modules or very specific networking optimizations we don't need and would be over-optimization). Instead we try to maintain a SOA which is agnostic to what underlying tools back our services, something that we've found Docker is useful for. That being said, we generally just use Ubuntu, and have some other AWS-specific setups around networking. If you like hacking on Cisco routers, I think it's perhaps not the right position :)

falcolas 12y ago

Try an arping from the new workers on first startup? Ran into this quite a bit when using VIPs for DB failover, and an arping fixed the caching issue in most cases.

wahnfrieden 12y ago

FYI, Clever: I click "Engineering Blog" at the top, and all links to blog posts on that page 404.

rgarcia 12y ago
sorry about that. It's fixed now!

spectre256 12y ago

This reminds me of a time at a previous company years ago, where we experienced an issue that felt similar, although the root cause was quite different. Basically, we had multiple teams all launching/terminating web servers. Unfortunately, they were all in the same EC2 deployment, and more often than not our load balancers from one team would send traffic to the web servers of another team. Furthermore, our setups were similar enough that this would sometimes cause bad results for users. We fixed it by making sure that our web servers on every team spoke on different ports. Not elegant, but effective (until two teams accidentally picked the same ports). These days we have good enough infrastructure tools that this problem should never happen. But in 2009, at a company that was overwhelmed with growth, those sort of things happen.

vosper 12y ago
Would a better fix have been to have a load balancer per team, and then have the EC2 instance bind to the correct load balancer?
- spectre256 12y ago
  That's what we had, but because the IPs were often recycled, traffic sometimes went to the wrong place if the load balancers has stale data.

perlgeek 12y ago

Wouldn't it be a better solution to not reuse IP addresses quickly? If I understood it correctly, they are in a private network anyway, so they could afford it.

daveloyall 12y ago
I thought there was a mechanism that would allow some sort of failure notification packet to make it back to the origin, so that it would then perform a proper ARP lookup WHO HAS 1.2.3.4? TELL 4.3.2.1!. Perhaps AWS is blocking some class of packet?
- X-Istence 12y ago
  The issue is that if it is on-network the only real "failure" notification is a time-out. No ICMP packets, nothing are sent back to indicate you are sending to a non-existent IP address.

jcollins 12y ago

We saw this earlier this year after upgrading to a new Linux kernel. The solution for us was to set this in sysctl.conf: net.ipv4.neigh.default.gc_thresh1=0 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1331150/ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1331150... https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1331150/comments/12 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1331150...

kiyoto 12y ago

Looking at the port number, it looks like Clever is a MongoDB user =)

akx 12y ago
You, uh, don't even have to just recognize the port number, reading the article is enough... > [...] These workers all connect to a MongoDB replicaset to update and read data [...] > 27017 = port mongod is listening on

girvo 12y ago

We had a fascinating bug on EC2 -- we could connect to the instance, but no network traffic made it out. It wasn't security group problems, it was literally a really weird bug in EC2's network that we somehow triggered, the engineer over at Amazon that looked at it was really excited when he came across our case as it was so weird, heh. They fixed it, I can't remember exactly what was done on their end, but it was one of the weirder problems I've attempted to debug. Nothing I tried worked!

zenocon 12y ago

I just experienced this early this week. Very frustrating. I also posted to AWS forums and got zero assistance; am currently not paying for AWS support plan. This article came at an opportune moment -- it makes sense and removes the shroud of mystery around why it "works sometimes" which leaves me with an uneasy feeling for a production setup.

schimmy_changa 12y ago
awesome, I was hoping it could help out someone else :)

legohead 12y ago

I was going to respond with my little story, but I see your article already linked it! ;)