3 ms·
Nice writeup, but it leaves me curious about the root cause: For some reason, our switches were unable to learn a significant percentage of our MAC addresses a
by xtacy 14y ago
Nice writeup, but it leaves me curious about the root cause:
For some reason, our switches were unable to learn a significant percentage of our MAC addresses and this aggregate traffic was enough to saturate all of the links between the access and aggregation switches, causing the poor performance we saw throughout the day.
Did you work with your vendor to understand what caused the above problem? Was it a lack of number of entries in the MAC table?
This problem aside, I am wondering why you still run layer 2 network in a tree-like configuration. These are known not to scale well, beyond a small LAN. An appropriate layer 3 network (with multipath routing) would ensure there is no such flooding, and ensure you use all the precious capacity in your switches!
- imbriaco 14y agoYeah, if you read a little further down we worked with the vendor to get them extensive diagnostics and get to the root problem. TL;DR: Lock contention on the CAM table. "We have worked with our network vendor to provide diagnostic information which led them to discover the root cause for the MAC learning issues. We expect a final fix for this issue within the next week or so and will be deploying a software update to our switches at that time. In the mean time we are closely monitoring our aggregation to access layer capacity and have a workaround process if the problem comes up again." In terms of network scalability, a properly designed layer 2 topology can scale quite a bit farther than our current needs. This is very much the least disruptive change we could introduce to solve the immediate network problems while we work on the future architecture.
- xtacy 14y agoOoh, lock contention on the hardware CAM table or the software in the switch that handles these updates? The first time I am hearing about this bug :-) Thanks a lot for sharing this bug story!
- imbriaco 14y agoIt was a lock that was being held by software but also prevented hardware updates to certain hash locations in the CAM. So MAC addresses that hashed to those locations couldn't be learned by any means.
- brooksbp 14y agoI wonder what hash was in use. Also, what workaround were you using to prevent flooding while this issue was happening?
- masklinn 14y ago> TL;DR: Lock contention on the CAM table. It's not really a tl;dr since (as far as I can see) it wasn't in the original post, though.