5 ms·
So they used a feature built into a load balancer to gracefully drain traffic from specific availability zones? Odd that a feature found in load balancers from
by skullone 3y ago
So they used a feature built into a load balancer to gracefully drain traffic from specific availability zones? Odd that a feature found in load balancers from the last 25 years is a blog post worthy thing.
- progbits 3y agoThe other bit is separating the service into isolated cells so issues in one don't affect dependent services everywhere like they had experienced before. But yeah any good SRE could point this out years ago.
- skullone 3y agoJust odd a company worth billions and billions of dollars is just now discovering HA models standard since the 90s. Can expand the Clos network architecture to these distributed service applications too. But judging by Slack's client quality, mature concepts such as those must be new to them.
- deleted 3y ago[deleted]
- antoniojtorres 3y agoThe linked AWS article specifically explains that it’s not just the typical single load balancer for cross AZ routing. I frankly don’t know where you’re getting that this means that HA is new to them.
- skullone 3y agoOf course this isn't a typical single load balancer for cross AZ - but the general gist of their "new" architecture is first principles level of design. But sure, we can celebrate their minor achievement I guess
- jameshart 3y agoThat seems like a shallow dismissal. In a distributed system, making sure that sub requests are handled across distributed nodes within the local AZ, and correctly draining traffic from AZs with partial component service outages, is not as trivial as 'using a feature built in to a load balancer'.
- skullone 3y agoIt may be shallow, but architecting for this is not really "advanced, FAANG-only accessible methodology". I'm surprised their services have been as "reliable" as they have been considering such trivial stuff is just now being employed in their architecture.
- robertlagrant 3y ago> architecting for this is not really "advanced, FAANG-only accessible methodology" Sorry - where are you quoting this claim from?
- colmmacc 3y agoClose but I don't think it's quite 25 years! I added graceful draining to Apache httpd's mod_proxy and mod_proxy_balancer either in 2003 or 2004, and at the time I'm nearly certain it was the first software load balancer to have the feature, and it wasn't available on the hardware load balancers of the time that I had access to ... though I later learned that at least BigIP load balancers had the feature. At the time, we had healthy debates about whether the feature was useful enough to justify additional complexity, and whether there could be cases where it would backfire. To this day, it's an underused feature. I still regularly run into customers and configurations that cause unnecessary blips to their end-users, so it's nice to see when people dig in and make sure that the next level of networking is working as well as it can.
- robertlagrant 3y agoWell played, HN.
- skullone 3y agoI migrated some old BigIP load balancers over to Apache in 2004ish, and extended some of mod_proxy to do some "unholy" things at the time. We also did a lot of direct server return stuff when no load balancer you could buy could handle the amount of traffic statefully. Man, how times have changed, and lesson forgotten.
- djbusby 3y agoMicrosoft bought Convoy in 1998[0]. Then incorporated it into NT4sp6a and Win2k as NLB/WLBS. One of its features was to gracefully remove a server from the cluster after all connections were closed - draining. But, cluster not the same as an LB. [0] https://news.microsoft.com/1998/08/24/microsoft-corp-acquires-valence-research-inc/ https://news.microsoft.com/1998/08/24/microsoft-corp-acquire...
- deleted 3y ago[deleted]