5 ms·
This reminds me of a time at a previous company years ago, where we experienced an issue that felt similar, although the root cause was quite different. Basic
by spectre256 12y ago
This reminds me of a time at a previous company years ago, where we experienced an issue that felt similar, although the root cause was quite different.
Basically, we had multiple teams all launching/terminating web servers. Unfortunately, they were all in the same EC2 deployment, and more often than not our load balancers from one team would send traffic to the web servers of another team. Furthermore, our setups were similar enough that this would sometimes cause bad results for users. We fixed it by making sure that our web servers on every team spoke on different ports. Not elegant, but effective (until two teams accidentally picked the same ports).
These days we have good enough infrastructure tools that this problem should never happen. But in 2009, at a company that was overwhelmed with growth, those sort of things happen.
- vosper 12y agoWould a better fix have been to have a load balancer per team, and then have the EC2 instance bind to the correct load balancer?
- spectre256 12y agoThat's what we had, but because the IPs were often recycled, traffic sometimes went to the wrong place if the load balancers has stale data.