5 ms·

Am I weird because I actually muttered 'ARP caching issue' halfway through your article? :-) Love the technical write-up -- thanks!

schimmy_changa 12y ago
I don't think you're weird, I think you're probably a great candidate to work as a sysadmin / SRE, or at Clever, we're hiring: https://clever.com/about/jobs#engineer-distributed-systems https://clever.com/about/jobs#engineer-distributed-systems !
- justincormack 12y ago
  Your job descriptions list a lot of technologies but dont mention anything to do with operating systems or networking. It seems odd to explicitly call out Node and redis but not anything down the stack.
  - meowface 12y ago
    I don't think there's anything wrong with that. Anyone in a sysadmin or devops is going to be expected to be proficient at Linux and bash, Windows and Powershell for Windows desktop/server admins, Cisco and Juniper for networking, etc. These are kind of default assumptions for positions like these. More modern and highly specific technologies like HAproxy, Salt, Node, Redis are not going to be known by every experienced, competent applicant though, and while you can often pick them up on the job if you have general experience in the area, there's always going to be a learning curve.
  - schimmy_changa 12y ago
    It's a good point, but right now we don't do anything that relies specifically on one technology at the very low levels (for instance, kernel modules or very specific networking optimizations we don't need and would be over-optimization). Instead we try to maintain a SOA which is agnostic to what underlying tools back our services, something that we've found Docker is useful for. That being said, we generally just use Ubuntu, and have some other AWS-specific setups around networking. If you like hacking on Cisco routers, I think it's perhaps not the right position :)
  - hijinks 12y ago
    I interviewed for a DevOps role at Clever a few months ago and the interview was 100% python algorithm questions. I did it correctly and was told how do I make it faster.. So I did and was told.. how can I make it faster.. After 30 minutes of a skype interview I told the guy I wasn't interested and thanked him for his time. Seems they are more dev then ops for devops role.
josegonzalez 12y ago
A ops friend of mine also saw this exact issue last week - same conditions as well - so this was also fresh in my mind. He ended up disabling arp caching though, rather than clearing it.
- daveloyall 12y ago
  So his machines issue a WHO HAS ...? request and wait for a response before every outgoing packet? ...
  - ansible 12y ago
    So his machines issue a WHO HAS ...? request and wait for a response before every outgoing packet? Yeah, that seems like a really bad idea. That will drastically increase the latency and cut throughput.
invertedohm 12y ago
The "it works after traceroute" line was the moment I was sure it was ARP caching. Then I was just wondering if it had anything to do with AWS specifically, which could have pointed towards a serious VPC bug. That doesn't seem to be the case though. You're not weird. Just burnt by real experience. Nice write-up. Too bad I'm not in SF. :P
valarauca1 12y ago
I've reached the point that flushing the ARP cache is part of my network troubleshooting routine. Always gives you the weirdest hardest to trace errors.
- js2 12y ago
  Until that time when you flush the cache on Solaris[1] and remove the entries for its own interfaces. Then it stops responding to ARP requests for itself. Now you have an even more fun issue to troubleshoot! 1. something silly like: "arp -a | awk '{print $2}' | xargs -n 1 arp -d". It's been a while and I don't think I was the original culprit.
  - saganus 12y ago
    And how do you fix that? I'm no Solaris guru obviously, but this looks like a chicken-and-egg problem. How do you actually go about fixing this? It's sounds very... Kafkesque
    - valarauca1 12y ago
      re-add it manually # arp -s 192.168.0.1 00:18:4d:f8:a4:6e Wait your working over SSH? Oh ummmm, talk to the data center guy and have him break out his USB to serial dongle and a RS232 cable.
      - saganus 12y ago
        Oh...dayum.. that's not good..
      - js2 12y ago
        Actually if you think about it, you can reach the server even if it's not replying to ARPs as long as it's initiating connections to other devices on the same network. Say A is the server in question (not responding to ARPs) and it initiates a connection to B. In this case, B will learn A's MAC, and so B will be able to reach A until A's entry times out of B's MAC table. The router for that network also has A's MAC, but you may not have access to the network gear. The intermittent connectivity to server A is what makes this problem so fun to diagnose. :-)
daveloyall 12y ago
Let this be a lesson: teach your children about hubs before you teach them about switches! :) So then, that inevitable question will come up: "Well, how does it know WHICH port?"...
- justincormack 12y ago
  Kids now, never seen a hub!
  - schimmy_changa 12y ago
    It is true that I haven't seen a hub in about 5 years, which means (100% anecdotally) that if someone is just graduating college now and started learning exactly 4 years ago, they might never have seen a hub...
    - rkangel 12y ago
      I have a treasured and jealously guarded hub that I keep at work for network debugging stuff, when a port mirroring switch just won't do. Almost impossible to get hold of now.
X-Istence 12y ago
I guessed this as soon as he mentioned spinning up/down a lot of machines using scripting. I've seen the same thing happen within an OpenStack environment I used to maintain. As part of the cloud-init scripts on new machines we would install arping and have the newly provisioned VM automatically send gratuitous arps so that arp caches would get updated. I haven't seen the issue on my RHEL 7 VM's though, nor in my current environment, but we don't spin up/down as many VM's as we used to.
kelnos 12y ago
Don't feel bad; as soon as I read the bit about them using a small amount of IP space, and new instances getting assigned IPs that were recently used by terminated hosts, I thought the same thing.
mobiplayer 12y ago
Same here... I´ve been seing things like this everyday, for years (Network / NetSec Support Engineer here :)) Just today: Servers with a static IP address getting it changed to an APIPA just because some other 3rd party device sent a unicast ARP request to it that should've been a broadcast as per RFC4436. I personally find ARP to be one of the funniest protocols out there and I love the faces people make when they understand how ARP glues L2 and L3 and suddenly everything makes sense. It is funny to see articles like this, because the network seems to be the "last frontier" for IT companies/workers. Only a handful seem to be brave enough to work on it (and actually like it). It is supposed to work like the lights when we switch them on and any interaction with a network engineer is just to tell them about an issue :) In my opinion, the network guys are the geeks among the geeks. Don't get me wrong, it is easy to understand networking, it just takes dedication.

schimmy_changa 12y ago

I don't think you're weird, I think you're probably a great candidate to work as a sysadmin / SRE, or at Clever, we're hiring: https://clever.com/about/jobs#engineer-distributed-systems https://clever.com/about/jobs#engineer-distributed-systems !

justincormack 12y ago
Your job descriptions list a lot of technologies but dont mention anything to do with operating systems or networking. It seems odd to explicitly call out Node and redis but not anything down the stack.
- meowface 12y ago
  I don't think there's anything wrong with that. Anyone in a sysadmin or devops is going to be expected to be proficient at Linux and bash, Windows and Powershell for Windows desktop/server admins, Cisco and Juniper for networking, etc. These are kind of default assumptions for positions like these. More modern and highly specific technologies like HAproxy, Salt, Node, Redis are not going to be known by every experienced, competent applicant though, and while you can often pick them up on the job if you have general experience in the area, there's always going to be a learning curve.
- schimmy_changa 12y ago
  It's a good point, but right now we don't do anything that relies specifically on one technology at the very low levels (for instance, kernel modules or very specific networking optimizations we don't need and would be over-optimization). Instead we try to maintain a SOA which is agnostic to what underlying tools back our services, something that we've found Docker is useful for. That being said, we generally just use Ubuntu, and have some other AWS-specific setups around networking. If you like hacking on Cisco routers, I think it's perhaps not the right position :)
- hijinks 12y ago
  I interviewed for a DevOps role at Clever a few months ago and the interview was 100% python algorithm questions. I did it correctly and was told how do I make it faster.. So I did and was told.. how can I make it faster.. After 30 minutes of a skype interview I told the guy I wasn't interested and thanked him for his time. Seems they are more dev then ops for devops role.

josegonzalez 12y ago

A ops friend of mine also saw this exact issue last week - same conditions as well - so this was also fresh in my mind. He ended up disabling arp caching though, rather than clearing it.

daveloyall 12y ago
So his machines issue a WHO HAS ...? request and wait for a response before every outgoing packet? ...
- ansible 12y ago
  So his machines issue a WHO HAS ...? request and wait for a response before every outgoing packet? Yeah, that seems like a really bad idea. That will drastically increase the latency and cut throughput.

invertedohm 12y ago

The "it works after traceroute" line was the moment I was sure it was ARP caching. Then I was just wondering if it had anything to do with AWS specifically, which could have pointed towards a serious VPC bug. That doesn't seem to be the case though. You're not weird. Just burnt by real experience. Nice write-up. Too bad I'm not in SF. :P

valarauca1 12y ago

I've reached the point that flushing the ARP cache is part of my network troubleshooting routine. Always gives you the weirdest hardest to trace errors.

js2 12y ago
Until that time when you flush the cache on Solaris[1] and remove the entries for its own interfaces. Then it stops responding to ARP requests for itself. Now you have an even more fun issue to troubleshoot! 1. something silly like: "arp -a | awk '{print $2}' | xargs -n 1 arp -d". It's been a while and I don't think I was the original culprit.
- saganus 12y ago
  And how do you fix that? I'm no Solaris guru obviously, but this looks like a chicken-and-egg problem. How do you actually go about fixing this? It's sounds very... Kafkesque
  - valarauca1 12y ago
    re-add it manually # arp -s 192.168.0.1 00:18:4d:f8:a4:6e Wait your working over SSH? Oh ummmm, talk to the data center guy and have him break out his USB to serial dongle and a RS232 cable.
    - saganus 12y ago
      Oh...dayum.. that's not good..
    - js2 12y ago
      Actually if you think about it, you can reach the server even if it's not replying to ARPs as long as it's initiating connections to other devices on the same network. Say A is the server in question (not responding to ARPs) and it initiates a connection to B. In this case, B will learn A's MAC, and so B will be able to reach A until A's entry times out of B's MAC table. The router for that network also has A's MAC, but you may not have access to the network gear. The intermittent connectivity to server A is what makes this problem so fun to diagnose. :-)

daveloyall 12y ago

Let this be a lesson: teach your children about hubs before you teach them about switches! :) So then, that inevitable question will come up: "Well, how does it know WHICH port?"...

justincormack 12y ago
Kids now, never seen a hub!
- schimmy_changa 12y ago
  It is true that I haven't seen a hub in about 5 years, which means (100% anecdotally) that if someone is just graduating college now and started learning exactly 4 years ago, they might never have seen a hub...
  - rkangel 12y ago
    I have a treasured and jealously guarded hub that I keep at work for network debugging stuff, when a port mirroring switch just won't do. Almost impossible to get hold of now.

X-Istence 12y ago

I guessed this as soon as he mentioned spinning up/down a lot of machines using scripting. I've seen the same thing happen within an OpenStack environment I used to maintain. As part of the cloud-init scripts on new machines we would install arping and have the newly provisioned VM automatically send gratuitous arps so that arp caches would get updated. I haven't seen the issue on my RHEL 7 VM's though, nor in my current environment, but we don't spin up/down as many VM's as we used to.

kelnos 12y ago

Don't feel bad; as soon as I read the bit about them using a small amount of IP space, and new instances getting assigned IPs that were recently used by terminated hosts, I thought the same thing.

mobiplayer 12y ago

Same here... I´ve been seing things like this everyday, for years (Network / NetSec Support Engineer here :)) Just today: Servers with a static IP address getting it changed to an APIPA just because some other 3rd party device sent a unicast ARP request to it that should've been a broadcast as per RFC4436. I personally find ARP to be one of the funniest protocols out there and I love the faces people make when they understand how ARP glues L2 and L3 and suddenly everything makes sense. It is funny to see articles like this, because the network seems to be the "last frontier" for IT companies/workers. Only a handful seem to be brave enough to work on it (and actually like it). It is supposed to work like the lights when we switch them on and any interaction with a network engineer is just to tell them about an issue :) In my opinion, the network guys are the geeks among the geeks. Don't get me wrong, it is easy to understand networking, it just takes dedication.