23 ms·

A terrible, horrible, no-good, very bad day at Slack

631 points by ceohockey60 6y ago

kd5bjo 6y ago
The title is an allusion to a popular children’s book[1]. I’m assuming that an automated algorithm pulled the “very;” hopefully the mods will consider restoring it. [1] https://en.wikipedia.org/wiki/Alexander_and_the_Terrible,_Horrible,_No_Good,_Very_Bad_Day https://en.wikipedia.org/wiki/Alexander_and_the_Terrible,_Ho...
- chrismorgan 6y ago
  HN automatically mangles titles in various ways I’m not fond of, dropping words that might or might not be significant, fiddling with capitalisation, &c., but the submitter can go back and edit the title back to what it was supposed to be, and the one time I’ve done that the system didn’t mangulate it again.
  - dang 6y ago
    That's by design, because the software is obviously imperfect. It does more good than harm, though, so we keep it.
- Stratoscope 6y ago
  That was the book that made me want to move to Australia! Or Timbuktu.
  - friseurtermin 6y ago
    Well, did you end up moving to Australia?
- SamuelAdams 6y ago
  There's also a film based on the book. https://en.wikipedia.org/wiki/Alexander_and_the_Terrible,_Horrible,_No_Good,_Very_Bad_Day_(film) https://en.wikipedia.org/wiki/Alexander_and_the_Terrible,_Ho...
  - cslawson 6y ago
    It was better than expected, IMO.
- dang 6y ago
  Ok, we've re-veried the title above.
fredsted 6y ago
Interesting bit: > The program which synced the host list generated by consul template with the HAProxy server state had a bug. It always attempted to find a slot for new webapp instances before it freed slots taken up by old webapp instances that were no longer running. This program began to fail and exit early because it was unable to find any empty slots, meaning that the running HAProxy instances weren’t getting their state updated
hyperman1 6y ago
I've just been bitten by this too: The broken monitoring hadn’t been noticed partly because this system ‘just worked’ for a long time, and didn’t require any change. Any experience on how to deal with it? Who watches the watchers?
- nickromano 6y ago
  If you use something like CloudWatch, you can set an alarm for “insufficient data” for when a monitor stops reporting.
  - EdwardDiego 6y ago
    Yep, and Grafana allows alerting on No Data states.
  - harpratap 6y ago
    I don't think it would have been enough for this exact situation. The solution you are providing is for cases where x==nil but in their case there was never an x, so you cannot do x==nil checks unless you deliberately set up x.
    - donavanm 6y ago
      Not quite. Cloudwatch provides for both behaviors. You can treat periods with no datapoints as “breaching” the threshold criteria. You can also independently alert for “insufficient data” where the source is unavailable, doesnt contain enough data to evaluate, etc. In the past I did both. Always emit a 0 datapoint every period + treat missing datapoints as breaching to discover if an application wasnt consistently emitting metrics. In addition a lower severity Insufficient Data alert was used to discover/validate when a meteic stream literally didnt exist (normally through misconfiguration of metric & alarm dimensions). Docs: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...
- jacquesm 6y ago
  This same sort of thing tends to creep into backup systems that work for a long time without if the backups are never restored. The more reliable the systems they backup the bigger the chance that a restore will fail when you need it most. So test your backups.
vecio 6y ago
What's the differences of using HAProxy or Envoy between using the cloud load balancers of AWS or Google Cloud?
- speedgoose 6y ago
  More control and pricing I guess.
- ublaze 6y ago
  Cloud load balancers can be sneakily expensive. Few months ago, we spent a few weeks replacing an ELB with naive client side load balancing via round robin, which saves us > 200k/year. ELBs charge per byte transmitted, which seems reasonable, but can end up really expensive.
  - user5994461 6y ago
    They charge per byte and per request if I remember well, which can be really expensive for serving both small API call and large files. Another limitation is that the ELB only works in AWS to AWS instances in the same location. Gotta use something else for geographic load balancing and for other datacenters.
  - cutemonster 6y ago
    > client side load balancing In the browser? Or a mobile app? They send 1 api req to server 1, then 1 to server 2 and so on? What about any session cookies maybe tied to a specific server?
    - Nextgrid 6y ago
      Presumably round-robin DNS. A DNS response would only return a handful of servers, of which the client will itself only pick one at random for the duration of the session. Now this approach has drawbacks (DNS responses are cached, and the DNS record picked initially by the client will typically be cached until the app/browser is restarted) but if they are acceptable to you then it's an easy, proven solution.
nik_0_0 6y ago
Super interesting post. Following blog links, the timeline in https://slack.engineering/all-hands-on-deck-91d6986c3ee https://slack.engineering/all-hands-on-deck-91d6986c3ee also offers a look at the play by play. However, as far as I can read it, they have somewhat different views on the root cause? "Soon, it became clear we had stale HAProxy configuration files, as a result of linting errors preventing re-rendering of the configuration." vs. "The program which synced the host list generated by consul template with the HAProxy server state had a bug. It always attempted to find a slot for new webapp instances before it freed slots taken up by old webapp instances that were no longer running. This program began to fail and exit early because it was unable to find any empty slots, meaning that the running HAProxy instances weren’t getting their state updated. As the day passed and the webapp autoscaling group scaled up and down, the list of backends in the HAProxy state became more and more stale." Maybe a combination of the two?
- ketzo 6y ago
  Honestly it’s a bit tough for me to parse, but the way I’m reading it, 1. Stale configs led to an overabundance of web apps, and then 2. Old instances of the web app couldn’t be removed because of the consul-template bug. so, yes, a combination (in sequence) of the two. Hard for me to be sure because I’m by no means knowledgeable on this stuff.
  - aeyes 6y ago
    Even easier way to understand what happened: - slots full - to update slots with a new host you need an empty slot - hosts went away but updating config was impossible -> errors because config referenced non-existing hosts
    - folkhack 6y ago
      Agree but one more: - monitoring was broke so we didn't learn about it until it was too late
  - a2tech 6y ago
    Real root cause was a poorly written home built tool to manage haproxy configs. The tool did not handle the slots being full and crashed. Respawn, rinse and repeat. Haproxy config got stale and when their automated tools removed servers it started with the machines haproxy actually knew about, and then the service died.
saagarjha 6y ago
> One of the incident’s effects was a significant scale-up of our main webapp tier. Sorry, I’m not very familiar with the terminology here; what is the “main webapp tier”?
- jsmeaton 6y ago
  Apps these days might have several groups of services. For a simple case you might have a web tier serving http requests from customers, and you might have a Worker/background task tier. They usually scale independently.
- waheoo 6y ago
  The things that serve the slack gui over the browser.
realtalk_sp 6y ago
Pretty fantastic case study in the perils of complex systems. Is there any place where these types of post-mortems are collected? Could be a very valuable resource for systems engineers.
- marksomnian 6y ago
  Here's a decent list: https://github.com/danluu/post-mortems https://github.com/danluu/post-mortems
- waheoo 6y ago
  I dunno, it sounds more like they just got caught with their pants down not finishing a migration to some new shiny. Also looks like they do blue green but dont confirm the replacements are live before considering the greens new blue.
m12k 6y ago
TL;DR First a performance bug was caught during rollout, and rolled back within a few minutes. However this triggered their auto-scaling of web apps to ramp up to more instances than a hard limit they had. This in turn triggered a bug in how they update the list of hosts in their load balancer, causing it to not get updated with new instances, and eventually go stale. After 8 hours the only real remaining instances in the list were the few oldest ones, so when they then scaled down the number of instances again, these old instances were the first to be shut down, causing the outage, because the instances that should have taken over were not listed in the stale load balancer host list.
- cuu508 6y ago
  That's an excellent TL;DR, thank you! Postmortems should start with a summary paragraph like the above, and then go into story and full details below.
  - jacksonpollock 6y ago
    agreed!
  - kergonath 6y ago
    This. The linked post is unreadable.
  - yawaramin 6y ago
    BLUF style–Bottom Line Up Front. Often used by the military for field action reports.
  - pradn 6y ago
    The internal Google postmortem format does require a 1-2 sentence summary. Absolutely vital for being able to learn from mistakes. No one is going to parse walls of text when browsing through post-mortems.
    - antoinealb 6y ago
      Here is an example of such a postmortem format if people wonder: https://landing.google.com/sre/sre-book/chapters/postmortem/ https://landing.google.com/sre/sre-book/chapters/postmortem/
jedberg 6y ago
Great writeup. It's cool that they were able to figure it out as quickly as they did, all things considered. If I were brought in as a consultant on this, my first question would be: why are you using a fleet of HAProxies instead of the ALB? I'm not saying that's a bad choice, but I'd want to know why that choice was made. The second question I would ask is what kind of Chaos Engineering they are doing. Are they doing major traffic failover tests? Rapid upscaling tests? Random terminations? Those are probably the first two things I'd want to solve.
- pcnix 6y ago
  In my experience, ALB scaling is opaque, and constantly lags behind herds of traffic. This is very noticeable if you're running services that experience sudden spikes, there's a noticeable lag of ALB scaling. That said though, it does do it's job for the most part.
- rhizome 6y ago
  I hope all of these questions would be asked only after everything was working again!
  - jedberg 6y ago
    Of course. :) These are things I would ask during a post-mortem.
sudhirj 6y ago
This is one of the biggest arguments I see for serverless (AWS Lambda + DynamoDB) or at least managed PaaS systems (Google App Engine, Heroku with RDS or CloudSQL). These systems may seem to cost more for some workload curves (or might even be cheaper for your curve), but the difference is worth it because you're paying for specialized 24/7 dev-ops teams whose only job is to keep these systems running smoothly, and by definition they're already familiar with running workloads orders of magnitude bigger than yours. Even then the platforms might be cheaper because you're only paying for a fraction of the dev-ops team's salaries, but you get their full benefit.
- gchamonlive 6y ago
  You can't assume that, these costs, specifically the server less stack at AWS, scale up quite opaquely. You can put billing alerts in place, but once you tie your infrastructure to one specific serverless vendor, even if you identify a harmful cost scale, you can't easily mitigate it. I am not saying serverless is expensive, all I am advocating is extensive planning and preparing before adopting any particular serverless solution. Once you give the green light to adopt a specific cloud solution, you tie yourself to that cloud, and that can turn out to be a bad idea in the long run. Lambda is a great tool when used right. It can take big workloads without costing too much. But in itself it does very little. It still needs to intergrate with something to be triggered, and if you go with AWS API Gateway that can cost you a pretty penny. Load balancer also incurs costs that are difficult to predict and is not so flexible as other load balancing tools, so sometimes you might still need to provision you own service discovery and load distribution layers, just like they had to do at slack. Serverless is nice, solves a lot of issues and give smaller teams a shot they otherwise wouldn't have, having to manage everything by themselves. But cloud costs can be opaque, cloud implementation can be very complex, cloud solutions can be too rigid sometimes and tying your product to one vendor can be detrimental in the long run.
  - jjoonathan 6y ago
    Not just costs! AWS services are opaque performance minefields, too. If you need reliable low latency -- which is probably core to the slack experience, or any GUI in a competitive space, really -- lambda is not a good option. They just don't share your priorities. AWS support will happily waste a lot of your time chasing Just One More Trick to mitigate the problem, though.
wayanon 6y ago
I bet if this story was told as an animation on YouTube it would be popular.
T3RMINATED 6y ago
Microsoft Teams is better.
xorcist 6y ago
Autoscaling is hard. Never ever use one that you don't thoroughly understand. An autoscaler that keeps chugging when deploys aren't green is outright dangerous.
- harpratap 6y ago
  I think this one is more of a service discovery bug than auto-scaling.
  - xorcist 6y ago
    Perhaps you know more than is in this blog post, but it sounds more like a rather standard load balancer. Deployment broke. Yeah, that probably should have been caught. Even if it didn't, monitoring should have caught the stale load balancer config. It didn't, for some reason unknown to us. These things happen. Things break. The autoscaler then proceeded to kill customer traffic. That was the part that worked as designed, so another design would have avoided escalating the situation (if you forgive some armchair engineering here).
    - cutemonster 6y ago
      Maybe what's missing was to test high load and autoscaling to 5x the traffic on, say, a holiday (when Slacks customers don't work)
snvzz 6y ago
>terrible, horrible, no-good, bad day Traditionally, a "black day". I guess that's not OK anymore, but I find the effort to work around established language cute. Edit: Apparently a meme.
- dralley 6y ago
  https://en.wikipedia.org/wiki/Alexander_and_the_Terrible,_Horrible,_No_Good,_Very_Bad_Day https://en.wikipedia.org/wiki/Alexander_and_the_Terrible,_Ho...
- flamble 6y ago
  It's a reference to Alexander and the Terrible, Horrible, No Good, Very Bad Day. I find the effort to make something innocuous into a confirmation of personal prejudices cute.
- bitesociety 6y ago
  It's a phrase from pop culture
- deleted 6y ago
  [deleted]
k__ 6y ago
After I used Discord in different contexts for months now (and Slack for years), I can't understand why someone willingly chooses Slack. It's the Atlassian of chat tools. Horrible performance and bad usabillity.
- jrockway 6y ago
  As someone who uses Discord, I can tell you it goes down too. My favorite part is looking at their status page, seeing that API response time is exceedingly high, and getting no updates from the team about whether or not they're fixing it.
  - k__ 6y ago
    I'm not saying Discord is perfect or "always up", it's just that their client UI is better structured and more responsive in most cases. I am on multiple OSS Discord servers with thousands of users, and it works just fine most of the time. I am on multiple Slack servers with just 10-20 users and it is unbearably slow.
    - rpdillon 6y ago
      I admined a multi-hundred user Slack org and now am part of a multi-thousand user slack org and I've never experienced (or heard about) the slowness that you're describing. Do you have a feel for whether it was client-side or server-side?
      - k__ 6y ago
        More client-side. Server-side scaling was only an issue with OSS communities and for hackathons, where >50k people were online.
    - pier25 6y ago
      Yeah the performance of Discord is much better but it looks and feels like a product for gamers. Slack has a more mainstream look and feel which is probably one of the reasons it's preferred by companies.
jrockway 6y ago
I had trouble getting through this article because my internal monologue was screaming "Envoy and xDS wouldn't have this problem". But that's exactly what they decided ;) HAProxy is a little behind the state of the art on "hey I could just ask some server where the backends are", and it shows in this case. (The "slots" are particularly alarming, as is having to restart when backends come and go.) xDS lets you give your frontend proxy a complete view of your whole system -- where the other proxies are (region/AZ/machine) and where the backends are, and how many of those. It can then make very good load balancing decisions -- preferring backends in the AZ that the frontend proxy is in, but intelligently spilling over if some other AZ is missing a frontend proxy or has fewer backends. And it emits metrics for each load balancing decision, so you can detect problems or unexpected balancing decisions before it results in an outage. https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/load_balancing/zone_aware https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overv... I also like the other features that Envoy has -- it can start distributed traces, it gives every request a unique ID so you can correlate applications and frontend proxy logs, it has a ton of counters/metrics for everything it does, and it can pick apart HTTP to balance requests (rather than TCP connections) between backends. It can also retry failed requests, so that users don't see transient errors (especially during rollouts). And it's retry logic is smart, so that if your requests are failing because a shared backend is down (i.e. your database blew up), it breaks the circuit for a period of time and lets your app potentially recover. The result is a good experience for end users sending you traffic, and extreme visibility into every incoming request. Mysterious problems become easy to debug just by looking at a dashboard, or perhaps by clicking into your tracing UI in the worst case. The disadvantage is that it doesn't really support any service discovery other than DNS out of the box. I had to write github.com/jrockway/ekglue to use Kubernetes service discovery to map services to Envoy's "clusters" (upstreams/backends), but I'm glad I did because it works beautifully. Envoy can take advantage of everything that Kubernetes knows about the service, which results in less config to write and a more robust application. (For example, it knows about backends that Kubernetes considers unready -- if all your backends are unready, Envoy will "panic" and try sending them traffic anyway. This can result in less downtime if your readiness check is broken or there's a lot of churn during a big rollout.)
- harpratap 6y ago
anderspitman 6y ago
Seems to me that most of the problems came from the sheer scale Slack operates at. A single instance of a self-hosted chat application wouldn't require any of the load balancing infrastructure. But SaaS is really convenient. I wonder if there's a market for "local cloud" companies that operate out of your city and offer hosting of popular open source projects. The complexity would be much lower, and hopefully the reliability higher. Plus you get the benefit of lower latency. 90% of the people in my team's slack channels live with 5 miles of each other.
- dannyw 6y ago
  A single instance of a self-hosted chat application would also result in unbounded hours investigating some obscure bug that affects 1 in 100,000 deployments, etc; while the developers don't know because they don't have enough ability to remotely dig in and investigate. Self hosting is not a pancea and I would not think it would be more reliable. There's also limited reason why you really want your servers in your city, 5ms of latency savings isn't it,instead of economies of scale in large datacentres with good network uplinks and centralized reliability teams.
MapleWalnut 6y ago
> It’s worth noting that HAProxy can integrate with Consul’s DNS interface, but this adds lag due to the DNS TTL, it limits the ability to use Consul tags, and managing very large DNS responses often seems to lead to hitting painful edge-cases and bugs. I was surprised how they dismissed HAProxy integration with Consul using SRV DNS records. Can anyone confirm the problems they highlight? It seems like their service that broke would not be needed if they went the DNS route.
- blyry 6y ago
  Pre 2.0 there were a few bugs with SRV discovery, maybe they adopted early and got bit? Just an anecdote but we've been using it since 1.9 without issue. Massively different scales though. Pre k8s and before srv support we used consul template in prod as well but it always scared me, seemed like too many moving pieces for what should've been a simple system.
  - blyry 6y ago
    I asked internally and figured out the gotcha that bit us: default dns payload size is 512b, which is enough for a few backend hosts but for sure not 12 or 30. Limit is 8kb, which probably wouldn't work for whatever slack is doing. https://cbonte.github.io/haproxy-dconv/2.1/configuration.html#5.3.2-accepted_payload_size https://cbonte.github.io/haproxy-dconv/2.1/configuration.htm.... Because DNS records come back in random order for each response, those truncated dns responses caused the backend slots to constantly rotate between different pod instances. Haproxy was graceful about the rotations, but it showed up as suddenly very strange latency / perf numbers when a backend was scaled up to say 10 instances from the normal 3
- user5994461 6y ago
  Dynamic DNS support is only since haproxy 1.8, maybe 2 years old give or take. Slack infrastructure must be older than that so they improvised. There were few bugs in the first implementations, should be good now. Slack don't mention their scale but I can imagine some UDP/DNS edge cases if there are hundreds of instances behind one domain.
sfpoet 6y ago
Scalability is always such a spur of the moment implementation at a startup. This seems to be cruft left over from that startup phase. Would a scalability audit have caught it? Tough to say as Slack came from that build fast and break things era.
- cutemonster 6y ago
  Scale fast and break things
Pxtl 6y ago
Not being involved in this kind of scaled-up devops kind of stuff, my read of this article was "HAProxy sounds awful"... but they noted in the article 1) Migrating to better software, and 2) Newer versions of HAProxy are also better software. But yeah, the first half where they're discussing "we don't do X, Y, or Z clean, idempotent way of doing config because the HAProxy doesn't perform right if you do it that way." sounded painfully familiar for a bunch of awful tools that I've used.
- kevmo314 6y ago
  I don't think that's fair. HAProxy is good and reliable. It's just old for the use cases we need today. Even the article has this opinion: > While HAProxy has served us well and reliably for many years, it also has some operational sharp edges Saying it sounds awful is like saying Vanilla JS sounds awful because it doesn't scale to the demanding webapp use cases of today. And, as you mentioned, it's really compounded by the fact that they're using an outdated version of HAProxy, but that doesn't make it awful software.
  - Pxtl 6y ago
    > Vanilla JS sounds awful ... it kinda is?
    - kevmo314 6y ago
      I don't think so. Just because something isn't a swiss army knife doesn't make it awful in my book.
ulisesrmzroche 6y ago
Why was the “very” removed? It sounds so much better like that.
- dang 6y ago
  HN's debaiting software took it out. We put it back. https://news.ycombinator.com/item?id=23756384 https://news.ycombinator.com/item?id=23756384 https://quoteinvestigator.com/2012/08/29/substitute-damn/ https://quoteinvestigator.com/2012/08/29/substitute-damn/
dpix 6y ago
Discussion during the outage: https://news.ycombinator.com/item?id=23161623 https://news.ycombinator.com/item?id=23161623
SilasX 6y ago
Stupid question, and admittedly off topic: What's with the "terrible, horrible, no-good, very bad" expression I see a lot? It's a reference to something? From googling, it seems to be this [1], but ... why? Why do people reference it? Usually you reference some work like this because a) the phrase is unusually creative, or b) the work is unusually memorable. Neither is true here. [1] https://en.wikipedia.org/wiki/Alexander_and_the_Terrible,_Horrible,_No_Good,_Very_Bad_Day https://en.wikipedia.org/wiki/Alexander_and_the_Terrible,_Ho...
- nikanj 6y ago
  This book sold well enough to score a tv series, a disney movie, a musical, and a theater play. How much more successful do you need it to be? Odds seem very high that a kid growing up in the past 50 years was exposed to this story and phrase
  - SilasX 6y ago
    I wasn’t. Or if I was, it obviously didn’t make an impression, and properly so: combining random negative words isn’t creative.
    - narag 6y ago
      It seems there's some kind of "all your base are belong to us" weird resonance for it. I had my own explanation why it works, but the only proof that's really needed is that it became a meme.
- solumos 6y ago
  It's memorable in that it's a lighthearted way of describing a pretty disastrous event without using expletives, with at least a dash of cultural significance. I agree it's a lame title.
- mattyb678 6y ago
  A couple points. Just because you personally don't remember or know the book does not mean that it is not memorable. Also, it could be that those who weren't aware of the book used it after seeing it used elsewhere.
nikanj 6y ago
> ” The reason that we haven’t been doing any significant work on this HAProxy stack is that we’re moving towards Envoy Proxy ” I truly don’t understand the cycle of churn. Once most edge cases and bugs of Haproxy have been found, the right decision is not migrating to completely unknown territory again. No project is a silver bullet, and changing stacks after you find the bugs makes for a terrible return on your bug-hunting investments.
- bastawhiz 6y ago
  And if there is no path forward to resolve the bugs? How many rough edges and bugs can you tolerate before a system is no longer worth investing in, especially when another newer system does not suffer from any of those bugs? To say there is no tipping point where the cost of continuing to use a system which no longer fits your needs exceeds the cost of migrating to a system which does (and will continue to) fit your needs ignores some amount of common sense.
- brown9-2 6y ago
  I think if you read this as they’re moving to Envoy because of this incident then you’ve misread. But it also sounds like Envoy and HAProxy have fundamentally different approaches to service discovery: https://twitter.com/mattklein123/status/1277729102271676416?s=21 https://twitter.com/mattklein123/status/1277729102271676416?... https://twitter.com/mattklein123/status/1278114953497436161?s=21 https://twitter.com/mattklein123/status/1278114953497436161?...
deleted 6y ago
[deleted]
matiasfernandez 6y ago
"I'm still not understanding why it's so hard to display the birthday date on the settings page. Why can't we get this done this quarter?" Look, I'm sorry, we've been over this. It's the design of our back-end. First there's this thing called the Bingo service. See, Bingo knows everyone's name-o, so we get the user's ID out of there. And from Bingo, we can call Papaya and MBS (Magic Baby Service) to get that user ID and turn it into a user session token. We can validate those with LNMOP. And then once we have that we can finally pull the users info down from Raccoon.
- fireflux_ 6y ago
  Reference: https://www.youtube.com/watch?v=y8OnoxKotPQ https://www.youtube.com/watch?v=y8OnoxKotPQ I revisit this video every now and then.
  - matiasfernandez 6y ago
    It's all I could think of while reading the article. The spiraling-down into complexity and service names had me smiling all the way. The "we should have predicted this" in last paragraph sealed the deal. And the HN comments saying they should have known better and used ZingBing from the start had me rolling. "You think you know what it takes to tell the user it's their birthday?!"
  - celim307 6y ago
    This one always cuts too close to home https://youtu.be/_o7qjN3KF8U https://youtu.be/_o7qjN3KF8U
    - matiasfernandez 6y ago
      Can't wait for BallmerCon this year. The XLOOKUP meta is going to spice things up. https://www.youtube.com/watch?v=xubbVvKbUfY https://www.youtube.com/watch?v=xubbVvKbUfY
- RonanTheGrey 6y ago
  Complete with indefatigable acronyms and insane naming conventions. This guy engineerings.
deleted 6y ago
[deleted]
astral303 6y ago
Another insight for me was coding the original system to have N slots. This should’ve been a red flag—why is there an arbitrary constraint there? Why allow the system to have a fixed limit there? If you choose to go down the slots road, then you need to put in alerts and discovery for reaching slot limits—-which means monitoring and tracking them, then setting up alerts.

631 points by ceohockey60 6y ago

The title is an allusion to a popular children’s book[1]. I’m assuming that an automated algorithm pulled the “very;” hopefully the mods will consider restoring it. [1] https://en.wikipedia.org/wiki/Alexander_and_the_Terrible,_Horrible,_No_Good,_Very_Bad_Day https://en.wikipedia.org/wiki/Alexander_and_the_Terrible,_Ho...

chrismorgan 6y ago
HN automatically mangles titles in various ways I’m not fond of, dropping words that might or might not be significant, fiddling with capitalisation, &c., but the submitter can go back and edit the title back to what it was supposed to be, and the one time I’ve done that the system didn’t mangulate it again.
- dang 6y ago
  That's by design, because the software is obviously imperfect. It does more good than harm, though, so we keep it.
Stratoscope 6y ago
That was the book that made me want to move to Australia! Or Timbuktu.
- friseurtermin 6y ago
  Well, did you end up moving to Australia?
SamuelAdams 6y ago
There's also a film based on the book. https://en.wikipedia.org/wiki/Alexander_and_the_Terrible,_Horrible,_No_Good,_Very_Bad_Day_(film) https://en.wikipedia.org/wiki/Alexander_and_the_Terrible,_Ho...
- cslawson 6y ago
  It was better than expected, IMO.
dang 6y ago
Ok, we've re-veried the title above.

fredsted 6y ago

Interesting bit: > The program which synced the host list generated by consul template with the HAProxy server state had a bug. It always attempted to find a slot for new webapp instances before it freed slots taken up by old webapp instances that were no longer running. This program began to fail and exit early because it was unable to find any empty slots, meaning that the running HAProxy instances weren’t getting their state updated

hyperman1 6y ago

I've just been bitten by this too: The broken monitoring hadn’t been noticed partly because this system ‘just worked’ for a long time, and didn’t require any change. Any experience on how to deal with it? Who watches the watchers?

nickromano 6y ago
If you use something like CloudWatch, you can set an alarm for “insufficient data” for when a monitor stops reporting.
- EdwardDiego 6y ago
  Yep, and Grafana allows alerting on No Data states.
- harpratap 6y ago
  I don't think it would have been enough for this exact situation. The solution you are providing is for cases where x==nil but in their case there was never an x, so you cannot do x==nil checks unless you deliberately set up x.
  - donavanm 6y ago
    Not quite. Cloudwatch provides for both behaviors. You can treat periods with no datapoints as “breaching” the threshold criteria. You can also independently alert for “insufficient data” where the source is unavailable, doesnt contain enough data to evaluate, etc. In the past I did both. Always emit a 0 datapoint every period + treat missing datapoints as breaching to discover if an application wasnt consistently emitting metrics. In addition a lower severity Insufficient Data alert was used to discover/validate when a meteic stream literally didnt exist (normally through misconfiguration of metric & alarm dimensions). Docs: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...
jacquesm 6y ago
This same sort of thing tends to creep into backup systems that work for a long time without if the backups are never restored. The more reliable the systems they backup the bigger the chance that a restore will fail when you need it most. So test your backups.

vecio 6y ago

What's the differences of using HAProxy or Envoy between using the cloud load balancers of AWS or Google Cloud?

speedgoose 6y ago
More control and pricing I guess.
ublaze 6y ago
Cloud load balancers can be sneakily expensive. Few months ago, we spent a few weeks replacing an ELB with naive client side load balancing via round robin, which saves us > 200k/year. ELBs charge per byte transmitted, which seems reasonable, but can end up really expensive.
- user5994461 6y ago
  They charge per byte and per request if I remember well, which can be really expensive for serving both small API call and large files. Another limitation is that the ELB only works in AWS to AWS instances in the same location. Gotta use something else for geographic load balancing and for other datacenters.
- cutemonster 6y ago
  > client side load balancing In the browser? Or a mobile app? They send 1 api req to server 1, then 1 to server 2 and so on? What about any session cookies maybe tied to a specific server?
  - Nextgrid 6y ago
    Presumably round-robin DNS. A DNS response would only return a handful of servers, of which the client will itself only pick one at random for the duration of the session. Now this approach has drawbacks (DNS responses are cached, and the DNS record picked initially by the client will typically be cached until the app/browser is restarted) but if they are acceptable to you then it's an easy, proven solution.

nik_0_0 6y ago

Super interesting post. Following blog links, the timeline in https://slack.engineering/all-hands-on-deck-91d6986c3ee https://slack.engineering/all-hands-on-deck-91d6986c3ee also offers a look at the play by play. However, as far as I can read it, they have somewhat different views on the root cause? "Soon, it became clear we had stale HAProxy configuration files, as a result of linting errors preventing re-rendering of the configuration." vs. "The program which synced the host list generated by consul template with the HAProxy server state had a bug. It always attempted to find a slot for new webapp instances before it freed slots taken up by old webapp instances that were no longer running. This program began to fail and exit early because it was unable to find any empty slots, meaning that the running HAProxy instances weren’t getting their state updated. As the day passed and the webapp autoscaling group scaled up and down, the list of backends in the HAProxy state became more and more stale." Maybe a combination of the two?

ketzo 6y ago
Honestly it’s a bit tough for me to parse, but the way I’m reading it, 1. Stale configs led to an overabundance of web apps, and then 2. Old instances of the web app couldn’t be removed because of the consul-template bug. so, yes, a combination (in sequence) of the two. Hard for me to be sure because I’m by no means knowledgeable on this stuff.
- aeyes 6y ago
  Even easier way to understand what happened: - slots full - to update slots with a new host you need an empty slot - hosts went away but updating config was impossible -> errors because config referenced non-existing hosts
  - folkhack 6y ago
    Agree but one more: - monitoring was broke so we didn't learn about it until it was too late
- a2tech 6y ago
  Real root cause was a poorly written home built tool to manage haproxy configs. The tool did not handle the slots being full and crashed. Respawn, rinse and repeat. Haproxy config got stale and when their automated tools removed servers it started with the machines haproxy actually knew about, and then the service died.

saagarjha 6y ago

> One of the incident’s effects was a significant scale-up of our main webapp tier. Sorry, I’m not very familiar with the terminology here; what is the “main webapp tier”?

jsmeaton 6y ago
Apps these days might have several groups of services. For a simple case you might have a web tier serving http requests from customers, and you might have a Worker/background task tier. They usually scale independently.
waheoo 6y ago
The things that serve the slack gui over the browser.

realtalk_sp 6y ago

Pretty fantastic case study in the perils of complex systems. Is there any place where these types of post-mortems are collected? Could be a very valuable resource for systems engineers.

marksomnian 6y ago
Here's a decent list: https://github.com/danluu/post-mortems https://github.com/danluu/post-mortems
waheoo 6y ago
I dunno, it sounds more like they just got caught with their pants down not finishing a migration to some new shiny. Also looks like they do blue green but dont confirm the replacements are live before considering the greens new blue.

m12k 6y ago

TL;DR First a performance bug was caught during rollout, and rolled back within a few minutes. However this triggered their auto-scaling of web apps to ramp up to more instances than a hard limit they had. This in turn triggered a bug in how they update the list of hosts in their load balancer, causing it to not get updated with new instances, and eventually go stale. After 8 hours the only real remaining instances in the list were the few oldest ones, so when they then scaled down the number of instances again, these old instances were the first to be shut down, causing the outage, because the instances that should have taken over were not listed in the stale load balancer host list.

cuu508 6y ago
That's an excellent TL;DR, thank you! Postmortems should start with a summary paragraph like the above, and then go into story and full details below.
- jacksonpollock 6y ago
  agreed!
- kergonath 6y ago
  This. The linked post is unreadable.
- yawaramin 6y ago
  BLUF style–Bottom Line Up Front. Often used by the military for field action reports.
- pradn 6y ago
  The internal Google postmortem format does require a 1-2 sentence summary. Absolutely vital for being able to learn from mistakes. No one is going to parse walls of text when browsing through post-mortems.
  - antoinealb 6y ago
    Here is an example of such a postmortem format if people wonder: https://landing.google.com/sre/sre-book/chapters/postmortem/ https://landing.google.com/sre/sre-book/chapters/postmortem/

jedberg 6y ago

Great writeup. It's cool that they were able to figure it out as quickly as they did, all things considered. If I were brought in as a consultant on this, my first question would be: why are you using a fleet of HAProxies instead of the ALB? I'm not saying that's a bad choice, but I'd want to know why that choice was made. The second question I would ask is what kind of Chaos Engineering they are doing. Are they doing major traffic failover tests? Rapid upscaling tests? Random terminations? Those are probably the first two things I'd want to solve.

pcnix 6y ago
In my experience, ALB scaling is opaque, and constantly lags behind herds of traffic. This is very noticeable if you're running services that experience sudden spikes, there's a noticeable lag of ALB scaling. That said though, it does do it's job for the most part.
rhizome 6y ago
I hope all of these questions would be asked only after everything was working again!
- jedberg 6y ago
  Of course. :) These are things I would ask during a post-mortem.

sudhirj 6y ago

This is one of the biggest arguments I see for serverless (AWS Lambda + DynamoDB) or at least managed PaaS systems (Google App Engine, Heroku with RDS or CloudSQL). These systems may seem to cost more for some workload curves (or might even be cheaper for your curve), but the difference is worth it because you're paying for specialized 24/7 dev-ops teams whose only job is to keep these systems running smoothly, and by definition they're already familiar with running workloads orders of magnitude bigger than yours. Even then the platforms might be cheaper because you're only paying for a fraction of the dev-ops team's salaries, but you get their full benefit.

gchamonlive 6y ago
You can't assume that, these costs, specifically the server less stack at AWS, scale up quite opaquely. You can put billing alerts in place, but once you tie your infrastructure to one specific serverless vendor, even if you identify a harmful cost scale, you can't easily mitigate it. I am not saying serverless is expensive, all I am advocating is extensive planning and preparing before adopting any particular serverless solution. Once you give the green light to adopt a specific cloud solution, you tie yourself to that cloud, and that can turn out to be a bad idea in the long run. Lambda is a great tool when used right. It can take big workloads without costing too much. But in itself it does very little. It still needs to intergrate with something to be triggered, and if you go with AWS API Gateway that can cost you a pretty penny. Load balancer also incurs costs that are difficult to predict and is not so flexible as other load balancing tools, so sometimes you might still need to provision you own service discovery and load distribution layers, just like they had to do at slack. Serverless is nice, solves a lot of issues and give smaller teams a shot they otherwise wouldn't have, having to manage everything by themselves. But cloud costs can be opaque, cloud implementation can be very complex, cloud solutions can be too rigid sometimes and tying your product to one vendor can be detrimental in the long run.
- jjoonathan 6y ago
  Not just costs! AWS services are opaque performance minefields, too. If you need reliable low latency -- which is probably core to the slack experience, or any GUI in a competitive space, really -- lambda is not a good option. They just don't share your priorities. AWS support will happily waste a lot of your time chasing Just One More Trick to mitigate the problem, though.

wayanon 6y ago

I bet if this story was told as an animation on YouTube it would be popular.

T3RMINATED 6y ago

Microsoft Teams is better.

xorcist 6y ago

Autoscaling is hard. Never ever use one that you don't thoroughly understand. An autoscaler that keeps chugging when deploys aren't green is outright dangerous.

harpratap 6y ago
I think this one is more of a service discovery bug than auto-scaling.
- xorcist 6y ago
  Perhaps you know more than is in this blog post, but it sounds more like a rather standard load balancer. Deployment broke. Yeah, that probably should have been caught. Even if it didn't, monitoring should have caught the stale load balancer config. It didn't, for some reason unknown to us. These things happen. Things break. The autoscaler then proceeded to kill customer traffic. That was the part that worked as designed, so another design would have avoided escalating the situation (if you forgive some armchair engineering here).
  - cutemonster 6y ago
    Maybe what's missing was to test high load and autoscaling to 5x the traffic on, say, a holiday (when Slacks customers don't work)

snvzz 6y ago

>terrible, horrible, no-good, bad day Traditionally, a "black day". I guess that's not OK anymore, but I find the effort to work around established language cute. Edit: Apparently a meme.

dralley 6y ago
https://en.wikipedia.org/wiki/Alexander_and_the_Terrible,_Horrible,_No_Good,_Very_Bad_Day https://en.wikipedia.org/wiki/Alexander_and_the_Terrible,_Ho...
flamble 6y ago
It's a reference to Alexander and the Terrible, Horrible, No Good, Very Bad Day. I find the effort to make something innocuous into a confirmation of personal prejudices cute.
bitesociety 6y ago
It's a phrase from pop culture
deleted 6y ago
[deleted]

k__ 6y ago

After I used Discord in different contexts for months now (and Slack for years), I can't understand why someone willingly chooses Slack. It's the Atlassian of chat tools. Horrible performance and bad usabillity.

jrockway 6y ago
As someone who uses Discord, I can tell you it goes down too. My favorite part is looking at their status page, seeing that API response time is exceedingly high, and getting no updates from the team about whether or not they're fixing it.
- k__ 6y ago
  I'm not saying Discord is perfect or "always up", it's just that their client UI is better structured and more responsive in most cases. I am on multiple OSS Discord servers with thousands of users, and it works just fine most of the time. I am on multiple Slack servers with just 10-20 users and it is unbearably slow.
  - rpdillon 6y ago
    I admined a multi-hundred user Slack org and now am part of a multi-thousand user slack org and I've never experienced (or heard about) the slowness that you're describing. Do you have a feel for whether it was client-side or server-side?
    - k__ 6y ago
      More client-side. Server-side scaling was only an issue with OSS communities and for hackathons, where >50k people were online.
  - pier25 6y ago
    Yeah the performance of Discord is much better but it looks and feels like a product for gamers. Slack has a more mainstream look and feel which is probably one of the reasons it's preferred by companies.

jrockway 6y ago

I had trouble getting through this article because my internal monologue was screaming "Envoy and xDS wouldn't have this problem". But that's exactly what they decided ;) HAProxy is a little behind the state of the art on "hey I could just ask some server where the backends are", and it shows in this case. (The "slots" are particularly alarming, as is having to restart when backends come and go.) xDS lets you give your frontend proxy a complete view of your whole system -- where the other proxies are (region/AZ/machine) and where the backends are, and how many of those. It can then make very good load balancing decisions -- preferring backends in the AZ that the frontend proxy is in, but intelligently spilling over if some other AZ is missing a frontend proxy or has fewer backends. And it emits metrics for each load balancing decision, so you can detect problems or unexpected balancing decisions before it results in an outage. https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/load_balancing/zone_aware https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overv... I also like the other features that Envoy has -- it can start distributed traces, it gives every request a unique ID so you can correlate applications and frontend proxy logs, it has a ton of counters/metrics for everything it does, and it can pick apart HTTP to balance requests (rather than TCP connections) between backends. It can also retry failed requests, so that users don't see transient errors (especially during rollouts). And it's retry logic is smart, so that if your requests are failing because a shared backend is down (i.e. your database blew up), it breaks the circuit for a period of time and lets your app potentially recover. The result is a good experience for end users sending you traffic, and extreme visibility into every incoming request. Mysterious problems become easy to debug just by looking at a dashboard, or perhaps by clicking into your tracing UI in the worst case. The disadvantage is that it doesn't really support any service discovery other than DNS out of the box. I had to write github.com/jrockway/ekglue to use Kubernetes service discovery to map services to Envoy's "clusters" (upstreams/backends), but I'm glad I did because it works beautifully. Envoy can take advantage of everything that Kubernetes knows about the service, which results in less config to write and a more robust application. (For example, it knows about backends that Kubernetes considers unready -- if all your backends are unready, Envoy will "panic" and try sending them traffic anyway. This can result in less downtime if your readiness check is broken or there's a lot of churn during a big rollout.)

harpratap 6y ago

anderspitman 6y ago

Seems to me that most of the problems came from the sheer scale Slack operates at. A single instance of a self-hosted chat application wouldn't require any of the load balancing infrastructure. But SaaS is really convenient. I wonder if there's a market for "local cloud" companies that operate out of your city and offer hosting of popular open source projects. The complexity would be much lower, and hopefully the reliability higher. Plus you get the benefit of lower latency. 90% of the people in my team's slack channels live with 5 miles of each other.

dannyw 6y ago
A single instance of a self-hosted chat application would also result in unbounded hours investigating some obscure bug that affects 1 in 100,000 deployments, etc; while the developers don't know because they don't have enough ability to remotely dig in and investigate. Self hosting is not a pancea and I would not think it would be more reliable. There's also limited reason why you really want your servers in your city, 5ms of latency savings isn't it,instead of economies of scale in large datacentres with good network uplinks and centralized reliability teams.

MapleWalnut 6y ago

> It’s worth noting that HAProxy can integrate with Consul’s DNS interface, but this adds lag due to the DNS TTL, it limits the ability to use Consul tags, and managing very large DNS responses often seems to lead to hitting painful edge-cases and bugs. I was surprised how they dismissed HAProxy integration with Consul using SRV DNS records. Can anyone confirm the problems they highlight? It seems like their service that broke would not be needed if they went the DNS route.

blyry 6y ago
Pre 2.0 there were a few bugs with SRV discovery, maybe they adopted early and got bit? Just an anecdote but we've been using it since 1.9 without issue. Massively different scales though. Pre k8s and before srv support we used consul template in prod as well but it always scared me, seemed like too many moving pieces for what should've been a simple system.
- blyry 6y ago
  I asked internally and figured out the gotcha that bit us: default dns payload size is 512b, which is enough for a few backend hosts but for sure not 12 or 30. Limit is 8kb, which probably wouldn't work for whatever slack is doing. https://cbonte.github.io/haproxy-dconv/2.1/configuration.html#5.3.2-accepted_payload_size https://cbonte.github.io/haproxy-dconv/2.1/configuration.htm.... Because DNS records come back in random order for each response, those truncated dns responses caused the backend slots to constantly rotate between different pod instances. Haproxy was graceful about the rotations, but it showed up as suddenly very strange latency / perf numbers when a backend was scaled up to say 10 instances from the normal 3
user5994461 6y ago
Dynamic DNS support is only since haproxy 1.8, maybe 2 years old give or take. Slack infrastructure must be older than that so they improvised. There were few bugs in the first implementations, should be good now. Slack don't mention their scale but I can imagine some UDP/DNS edge cases if there are hundreds of instances behind one domain.

sfpoet 6y ago

Scalability is always such a spur of the moment implementation at a startup. This seems to be cruft left over from that startup phase. Would a scalability audit have caught it? Tough to say as Slack came from that build fast and break things era.

cutemonster 6y ago
Scale fast and break things

Pxtl 6y ago

Not being involved in this kind of scaled-up devops kind of stuff, my read of this article was "HAProxy sounds awful"... but they noted in the article 1) Migrating to better software, and 2) Newer versions of HAProxy are also better software. But yeah, the first half where they're discussing "we don't do X, Y, or Z clean, idempotent way of doing config because the HAProxy doesn't perform right if you do it that way." sounded painfully familiar for a bunch of awful tools that I've used.

kevmo314 6y ago
I don't think that's fair. HAProxy is good and reliable. It's just old for the use cases we need today. Even the article has this opinion: > While HAProxy has served us well and reliably for many years, it also has some operational sharp edges Saying it sounds awful is like saying Vanilla JS sounds awful because it doesn't scale to the demanding webapp use cases of today. And, as you mentioned, it's really compounded by the fact that they're using an outdated version of HAProxy, but that doesn't make it awful software.
- Pxtl 6y ago
  > Vanilla JS sounds awful ... it kinda is?
  - kevmo314 6y ago
    I don't think so. Just because something isn't a swiss army knife doesn't make it awful in my book.

ulisesrmzroche 6y ago

Why was the “very” removed? It sounds so much better like that.

dang 6y ago
HN's debaiting software took it out. We put it back. https://news.ycombinator.com/item?id=23756384 https://news.ycombinator.com/item?id=23756384 https://quoteinvestigator.com/2012/08/29/substitute-damn/ https://quoteinvestigator.com/2012/08/29/substitute-damn/

dpix 6y ago

Discussion during the outage: https://news.ycombinator.com/item?id=23161623 https://news.ycombinator.com/item?id=23161623

SilasX 6y ago

Stupid question, and admittedly off topic: What's with the "terrible, horrible, no-good, very bad" expression I see a lot? It's a reference to something? From googling, it seems to be this [1], but ... why? Why do people reference it? Usually you reference some work like this because a) the phrase is unusually creative, or b) the work is unusually memorable. Neither is true here. [1] https://en.wikipedia.org/wiki/Alexander_and_the_Terrible,_Horrible,_No_Good,_Very_Bad_Day https://en.wikipedia.org/wiki/Alexander_and_the_Terrible,_Ho...

nikanj 6y ago
This book sold well enough to score a tv series, a disney movie, a musical, and a theater play. How much more successful do you need it to be? Odds seem very high that a kid growing up in the past 50 years was exposed to this story and phrase
- SilasX 6y ago
  I wasn’t. Or if I was, it obviously didn’t make an impression, and properly so: combining random negative words isn’t creative.
  - narag 6y ago
    It seems there's some kind of "all your base are belong to us" weird resonance for it. I had my own explanation why it works, but the only proof that's really needed is that it became a meme.
solumos 6y ago
It's memorable in that it's a lighthearted way of describing a pretty disastrous event without using expletives, with at least a dash of cultural significance. I agree it's a lame title.
mattyb678 6y ago
A couple points. Just because you personally don't remember or know the book does not mean that it is not memorable. Also, it could be that those who weren't aware of the book used it after seeing it used elsewhere.

nikanj 6y ago

> ” The reason that we haven’t been doing any significant work on this HAProxy stack is that we’re moving towards Envoy Proxy ” I truly don’t understand the cycle of churn. Once most edge cases and bugs of Haproxy have been found, the right decision is not migrating to completely unknown territory again. No project is a silver bullet, and changing stacks after you find the bugs makes for a terrible return on your bug-hunting investments.

bastawhiz 6y ago
And if there is no path forward to resolve the bugs? How many rough edges and bugs can you tolerate before a system is no longer worth investing in, especially when another newer system does not suffer from any of those bugs? To say there is no tipping point where the cost of continuing to use a system which no longer fits your needs exceeds the cost of migrating to a system which does (and will continue to) fit your needs ignores some amount of common sense.
brown9-2 6y ago
I think if you read this as they’re moving to Envoy because of this incident then you’ve misread. But it also sounds like Envoy and HAProxy have fundamentally different approaches to service discovery: https://twitter.com/mattklein123/status/1277729102271676416?s=21 https://twitter.com/mattklein123/status/1277729102271676416?... https://twitter.com/mattklein123/status/1278114953497436161?s=21 https://twitter.com/mattklein123/status/1278114953497436161?...

deleted 6y ago

[deleted]

matiasfernandez 6y ago

"I'm still not understanding why it's so hard to display the birthday date on the settings page. Why can't we get this done this quarter?" Look, I'm sorry, we've been over this. It's the design of our back-end. First there's this thing called the Bingo service. See, Bingo knows everyone's name-o, so we get the user's ID out of there. And from Bingo, we can call Papaya and MBS (Magic Baby Service) to get that user ID and turn it into a user session token. We can validate those with LNMOP. And then once we have that we can finally pull the users info down from Raccoon.

fireflux_ 6y ago
Reference: https://www.youtube.com/watch?v=y8OnoxKotPQ https://www.youtube.com/watch?v=y8OnoxKotPQ I revisit this video every now and then.
- matiasfernandez 6y ago
  It's all I could think of while reading the article. The spiraling-down into complexity and service names had me smiling all the way. The "we should have predicted this" in last paragraph sealed the deal. And the HN comments saying they should have known better and used ZingBing from the start had me rolling. "You think you know what it takes to tell the user it's their birthday?!"
- celim307 6y ago
  This one always cuts too close to home https://youtu.be/_o7qjN3KF8U https://youtu.be/_o7qjN3KF8U
  - matiasfernandez 6y ago
    Can't wait for BallmerCon this year. The XLOOKUP meta is going to spice things up. https://www.youtube.com/watch?v=xubbVvKbUfY https://www.youtube.com/watch?v=xubbVvKbUfY
RonanTheGrey 6y ago
Complete with indefatigable acronyms and insane naming conventions. This guy engineerings.

deleted 6y ago

[deleted]

astral303 6y ago

Another insight for me was coding the original system to have N slots. This should’ve been a red flag—why is there an arbitrary constraint there? Why allow the system to have a fixed limit there? If you choose to go down the slots road, then you need to put in alerts and discovery for reaching slot limits—-which means monitoring and tracking them, then setting up alerts.