5 ms·
Looks like a load balancer issue: http://status.37signals.com/status/campfire/2013/4/17 http://status.37signals.com/status/campfire/2013/4/17
by mikeevans 14y ago
Looks like a load balancer issue: http://status.37signals.com/status/campfire/2013/4/17 http://status.37signals.com/status/campfire/2013/4/17
- masonhensley 14y agoI'm not too familiar with load balancers. I'm slowly learning rails server stuff that's more complicated than heroku. Why would all of their applications be sitting behind a single load balancer? Wouldn't you want to silo them so that you do not have a single point of failure for all of your products?
- banachtarski 14y agoAll roads lead to a single URL (aka IP address) so somewhere along the way, you're going to have a single point of failure that does a one to many multiplexing, logically speaking.
- sokoloff 14y agoMason's question was about multiple URLs. IOW, "why does basecamp.37signals.com share a load balancer with campfire.37signals.com?" It's usually for cost reasons. Hardware load balancers are expensive, and the job they do isn't especially computationally expensive, so it's generally accepted to host multiple VIPs on a single load balancer. Looking at it the other way, being 100% down on one LB might be preferable to being 50% down when a single LB out of two fails. The alternative gives you twice as many chances for failure, even though each chance is half as "bad". Common-mode failure mechanisms would of course still bring you entirely down. (Bad config by a human being the most common cause.)
- ironchef 14y agoMost modern hardware LBs support active-active or active-passive connections with each other (deploying two A10s, etc.). It's been this way for 15 or so years (since local director, bigip, arrowpoint, etc.). The main reasons you would suffer from this would be misconfiguration or too cheap to buy 2 LBs.
- themcgruff 14y ago(I do ops at 37signals...) We have two top of the line ("high end") load balancers at each site. They malfunctioned. When they both malfunction at once, it doesn't matter whether it's active/active or active/passive or anything like that.
- masonhensley 14y ago> "why does basecamp.37signals.com share a load balancer with campfire.37signals.com?" That is what I was trying to understand. Thanks for the insight.
- regularfry 14y agoThat's not necessarily true. You can have more than one IP address serving the same hostname, and they'll get approximately round-robinned. The usual thing would be for these IP addresses to be on different machines. If you `dig www.google.com` you'll see an example.
- dilap 14y agoBut wouldn't you still get partial failure in that case?
- clebio 14y agoWouldn't that just increase latency, rather than induce failure, though?
- dilap 14y agoI was waiting for someone who really knows the answer to jump in, but I don't think so, because a client would have to know to jump back to the DNS again after reaching the IP address failed, which I don't think they typically do. But I might be wrong. Almost want to test it now... Probably 80% of hackernews knows this by heart.
- appplemac 14y agoWell, if you have a single point of failure, even if it's a load balancer, you've done something wrong.
- appplemac 14y agoI will always refresh the page before posting a reply I will always refresh the page before posting a reply I will always refresh the page before posting a reply
- Macsenour 14y agoThanks Bart :)
- wetherbeei 14y agoYou can have redundant load balancer setups that listen on the same IP [1]. Combined with round-robin DNS and there isn't a single point of failure unless your DNS servers go down. [1] http://siag.nu/pen/vrrpd-linux.shtml http://siag.nu/pen/vrrpd-linux.shtml
- SeoxyS 14y agoNot true: DNS servers can be configured with multiple A records for a single hostname, and the traffic will be spread round-robin. A DNS query will resolve to a different IP for different clients. At Chartboost, our systems are configured with every DNS query hitting one of 6 load balancers, which in turn spreads traffic to ~200 app servers.
- banachtarski 14y agoSure but then the DNS server is the one to many multiplexer. You can shove things around all you want, you'll still end up with either a single point of failure or weak consistency.
- jewel 14y agoSince DNS records change rarely, it's trivial to have multiple active DNS servers for a domain. In normal deployments, DNS doesn't have a single point of failure. It does have "weak consistency", like you say, but that doesn't matter if the records rarely change.
- wlll 14y agoI'm a sysadmin at 37signals (was working during this outage). The load balancers are meant to be redundant, and they are meant fail over automatically when something goes wrong. We're talking to the vendor about why this failed.
- tptacek 14y agoThere are something like 89345793 different documents on how best to test Rails application code. Someone should write the canonical reference on testing the web infrastructure that people stick in front of Rails applications (really, all web applications).