15 ms·
We built a self-healing system to survive a concurrency bug at Netflix
- deleted 2y ago[deleted]
- kristiandupont 2y agoThat was a bit underwhelming compared to what the headline set my expectations up for, but definitely a good idea and neat solution.
- vanjajaja1 2y agofrom the headline alone I got linkedin ceo vibe. "Built a Self-Healing System to Survive a Concurrency Bug" is how I could describe wrapping a failing method in a retry loop
- fragmede 2y agoPut in a couple more if statements checking the output of rand(), call it AI, and you'll be CEO in no time!
- coolgoose 2y agoOne of the things I am greatful for kubernetes and the killing of pods. Had a similar problem but memory wise with a pesky memory leak, and the short term solution was to do nothing as instances would to do nothing.
- maximinus_thrax 2y agoDuring one of my past gigs, this exact feature hid a huge memory leak, in old code, always running on k8s which we found out only when we moved some instances to bare metal.
- esprehn 2y agoWe hit this in a past gig too. One of the big services had a leak, but deployed every 24 hours which was hiding it. When the holiday deploy freeze hit the pods lived much longer than normal and caused an OOM storm. At first I thought maybe we should add a "hack" to cycle all the pods over 24 hours old, but then I wondered if making holiday freezes behave like normal weeks was really a hack at all or just reasonable predictability. In the end folks managed to fix the leak and we didn't resolve the philosophical question though.
- deleted 2y ago[deleted]
- ksd482 2y agoThis was a nice short read. A simple (temporary) solution, yet a clever one. How was he managing the instances? Was he using kubernetes, or did he write some script to manage the auto terminating of the instances? It would also be nice to know why: 1. Killing was quicker than restarting. Perhaps because of the business logic built into the java application? 2. Killing was safe. How was the system architectured so that the requests weren't dropped altogether. EDIT: formatting
- deleted 2y ago[deleted]
- jumploops 2y agoThe author mentions 2011 as the time they switched from REST to RPC-ish APIs, and this issue was related to that migration. Kubernetes launched in 2014, if memory serves, and it took a bit before widespread adoption, so I’m guessing this was some internal solution. This was a great read, and harkens back to the days of managing 1000s of cores on bare metal!
- braggerxyz 2y ago> It would also be nice to know why: 1. Killing was quicker than restarting. If you happen to restart one of the instances that was hanging in the infinite thread, you can wait a very long time until the Java container actually decides to kill itself because it did not finish its graceful shutdown within the alotted timeout period. Some Java containers have a default of 300s for this. In this circumstance kill -9 is faster by a lot ;) Also we had circumstances where the affected Java container did not stop even if the timeout was reached because the misbehaving thread did consume the whole cpu and none was left for the supervisor thread. Then you can only kill the host process of the JVM.
- mhawthorne 2y ago[dead]
- dhruvrrp 2y agoInteresting read, the fix seems to be straightforward, but I'd have a few more questions if I was trying to do something similar. Is software deployed regularly on this cluster? Does that deployment happen faster than the rate at which they were losing CPUs? Why not just periodically force a deployment, given it's a repeated process that probably already happens frequently. What happens to the clients trying to connect to the stuck instances? Did they just get stuck/timeout? Would it have been better to have more targeted terminations/full terminations instead?
- nikita2206 2y agoAn answer to basically all your questions is: doesn’t matter, they did their best to stabilize in a short amount of time, and it worked - that’s what mattered.
- mhawthorne 2y ago[dead]
- rukugu 2y agoI like the practicality of this
- est 2y agoReminds me of the famous quote by Rasmus Lerdorf, creator of PHP > I’m not a real programmer. I throw together things until it works then I move on. The real programmers will say “Yeah it works but you’re leaking memory everywhere. Perhaps we should fix that.” I’ll just restart Apache every 10 requests.
- nicman23 2y agoi ll argue that doing the restart is more important until someone else finds the leak
- fragmede 2y agoOr future me. It hurts on the inside to just kick EC2 every hour because every 61 minutes something goes awry in the process. But the show must go on, so you put in the temporary fix knowing that it's not going to be temporary. Still, weeks/months/years down the line you could get lucky and the problem will go away and you can remove the kludge. But if you're ridiculously lucky, not only will the problem get fixed, but you'll get to understand exactly why the mysterious problem was happening in the first place. Like the gunicorn 500 upgrade bug, or the Postgres TOAST json thing. That sort of satisfaction isn't something money can buy. (Though it will help pay for servers in the interim until you find the bug.)
- nicman23 2y agoor at least after the weekend :P
- morning-coffee 2y agoAlso uttered by others who thought borrowing money was more important until they could figure out a way to control spending.
- raverbashing 2y ago> and to my memory, some calls to ConcurrentHashMap.get() seemed to be running infinitely. Of course they did. And whoever though "Concurrent" meant it would work fine gets burned by it. Of course. And of course it doesn't work properly or intuitively for some very stupid reason. Sigh
- xxs 2y agoIt has to be an error - it could happen to HashMap, it has never been an issue w/ CHM.
- keeganpoppen 2y agothis sounds more like citing chapter and verse in an exegesis than anything of direct relevance to the Mortal Plane...
- mhawthorne 2y ago[dead]
- kenhwang 2y agoMy workplace currently has a similar problem where a resource leak can be greatly increased with certain unpredictable/unknown traffic conditions. Our half-day workaround implementation was the same thing, just cycle the cluster regularly automatically. Since we're running on AWS, we just double the size of the cluster, wait for the instances to initialize, then rapidly decommission the old instances. Every 2 hours. It's shockingly stable. So much so that resolving the root cause isn't considered a priority and so we've had this running for months.
- gorkempacaci 2y agoHow about the costs? Isn’t this a very expensive bandaid? How is it not a priority? :)
- bratbag 2y agoDepends what else it's solving for. I've seen multiple issues solved like this after engineering teams have been cut to the bone. If the cost of maintaining enough engineers to keep systems stable for more than 24 hours, is more than the cost of doubling the container count, then this is what happens
- JSDevOps 2y agoThis. All the domain knowledge has left. This sounds like a Hacky work around at best which AWS will welcome you with open arms come invoice day.
- kenhwang 2y agoDepends on how long it takes for the incoming instances to initialize and outgoing instances to fully decommission. x = time it takes to switchover y = length of the cycles x/y = % increase in cost For us, it's 15 minutes / 120 minutes = 12.5% increase, which was deemed acceptable enough for a small service.
- toast0 2y agoShouldn't be too high cost if you only run 2x the instances for a short amount of time. A reasonable use of Cloud, IMHO, if you can't figure out a less disruptive bandaid.
- xxs 2y agoThey were just lucky not to have data corruption due to concurrency issue, and the manifestation was infinite get. Overall if you can randomly "kill -9", the case is rather trivial. Likely replacing HashMap with CHM would not solve the concurrency issue either, but it'd prevent an infinite loop. (Edit) It appear that part is just wrong: "some calls to ConcurrentHashMap.get() seemed to be running infinitely." <-- it's possible to happen no hashmap during concurrent put(s), but not to ConcurrentHashMap
- ay 2y agoit wasn’t luck, it was very deliberately engineered for. The article does lack a good bit of context about the Netflix infra: https://netflixtechblog.com/the-netflix-simian-army-16e57fbab116 https://netflixtechblog.com/the-netflix-simian-army-16e57fba... https://github.com/Netflix/chaosmonkey https://github.com/Netflix/chaosmonkey
- vladak 2y agoyep, the link in the "some calls to ConcurrentHashMap.get() seemed to be running infinitely." sentence points to HashMap.html#get(java.lang.Object)
- xxs 2y agoI have seen that part myself (infinite loops), also I have quite extensive experience with CHM (and HashMap). Overall such a mistake alone undermines the effort/article.
- mhawthorne 2y ago[dead]
- conradfr 2y agoI have a project where one function (reading metadata from an Icecast stream [0]) was causing a memory leak and ultimately consuming all of it. I don't remember all the details but I've still not be able to find the bug. But this being in Elixir I "fixed it" with Task, TaskSupervisor and try/catch/rescue. Not really a win but it is still running fine to this day. [0] https://github.com/conradfr/ProgRadio/blob/1fa12ca73a40aedb949635dcaa3a486f432d0cb9/progradio_api/lib/progradio_api/song_provider/icecast.ex#L48 https://github.com/conradfr/ProgRadio/blob/1fa12ca73a40aedb9...
- cpursley 2y agoHalf of hn posts are people showing off things where they spent a herculean amount of effort reinventing something that elixir/erlang has had solved 30+ years already.
- nesarkvechnep 2y agoSome are even proud of their ignorance and belittle Erlang and Elixir.
- rikthevik 2y agoI'm fine with it. If people want to belittle something, either we aren't trying to solve the same problem (sure) or they're actively turning people away from what could be a serious advantage (more for me!) If the cost of switching wasn't so high, I'd love to write Elixir all day. It's a joy.
- girishso 2y agoNot much familiar with Elixir OTP, but isn’t the approach OP took similar to Let It Crash philosophy of OTP?
- ramchip 2y agoNot really, you wouldn't normally kill or restart processes randomly in an OTP system. "Let it crash" is more about separating error handling from business logic.
- btbuilder 2y agoI mean, it worked for Boeing[1] too. 1 - https://www.theregister.com/2020/04/02/boeing_787_power_cycle_51_days_stale_data/ https://www.theregister.com/2020/04/02/boeing_787_power_cycl...
- keeganpoppen 2y ago"worked" (not that i don't get the sarcasm)
- shiroiushi 2y agoTrue, but for somewhat different reasons. For the OP, they take this approach because they simply don't know yet what the problem is, and it would take some time to track it down and fix it and they don't want to bother. For Boeing, it's probably something fairly simple actually, but they don't want to fix it because their software has to go through a strict development process based on requirements and needing certification and testing, so fixing even a trivial bug is extremely time-consuming and expensive, so it's easier to just put a directive in the manual saying the equipment needs to be power-cycled every so often and let the users deal with it. The OP isn't dealing with this kind of situation.
- iLoveOncall 2y agoYeah I had the same issue of my EC2 that I used to host my personal websites randomly getting to 100% CPU and being unreachable. I put a CloudWatch alarm at 90% CPU usage which would trigger a reboot (which completed way before anyone would notice a downtime). Never had issues again.
- iluvcommunism 2y agoKill and restart the service. This seems to be the coder solution to everything. We do it for our service as well. The programmer could fix their stuff but alas, that’s too much to ask.
- edf13 2y agoYes - lots of writing for a common solution to a bug... Memory leaks are often "resolved" this way... until time allows for a proper fix.
- Cthulhu_ 2y ago"Did you try turning it off and on again?"
- perseysinowatz 2y ago[dead]
- jumploops 2y agoThis reminds me of a couple startups I knew running Node.js circa ~2014, where they would just restart their servers every night due to memory issues. iirc it was mostly folks with websocket issues, but fixing the upstream was harder 10 years later and specific software has gotten better, but this type of problem is certainly still prevalent!
- JonChesterfield 2y agoTitle is grossly misleading. That Netflix had already built a self-healing system means they were able to handle a memory leak by killing random servers faster than memory was leaking. This post isn't about how they've managed that, it's just showing off that their existing system is robust enough that you can do hacks like this to it.
- 4star3star 2y agoYour take is much different than mine. The issue was a practical one of sparing people from working too much over one weekend since the bug would have to wait until Monday, and the author willingly described the solution as the worst.
- rullelito 2y ago> Why not just reboot them? Terminating was faster. If you don't know why you should reboot servers/services properly instead of terminating them..
- cnity 2y agoWell, why? This comment seems counter to the now-popular "cattle not pets" approach.
- merizian 2y agoThis reminds me of LLM pretraining and how there are so many points at which the program could fail and so you need clever solutions to keep uptime high. And it's not possible to just fix the bugs--GPUs will often just crash (e.g. in graphics, if a pixel flips the wrong color for a frame, it's fine, whereas such things can cause numerical instability in deep learning so ECC catches them). You also often have a fixed sized cluster which you want to maximize utilization of. So improving uptime involves holding out a set of GPUs to swap out failed ones while they reboot. But also the whole run can just randomly deadlock, so you might solve that by listening to the logs and restarting after a certain amount of inactivity. And you have to be clever with how to save/load checkpoints, since that can start to become a huge bottleneck. After many layers of self healing, we managed to take a vacation for a few days without any calls :)
- rvnx 2y agoMeta has a similar strategy, and this is why memory leak bugs in HHVM are not fixed (they consider that instances are going to be regularly killed anyway)
- fidotron 2y agoThis is a bit odd coming from the company of chaos engineering - has the chaos monkey been abandoned at Netflix? I have long advocated randomly restarting things with different thresholds partly for reasons like this* and to ensure people are not complacent wrt architecture choices. The resistance, which you can see elsewhere here, is huge, but at scale it will happen regardless of how clever you try to be. (A lesson from the erlang people that is often overlooked). * Many moons ago I worked on a video player which had a low level resource leak in some decoder dependency. Luckily the leak was attached to the process, so it was a simple matter of cycling the process every 5 minutes and seamlessly attaching a new one. That just kept going for months on end, and eventually the dependency vendor fixed the leak, but many years later.
- ricardobeat 2y agoIn cases like this won't Chaos Monkey actually hide the problem, since it's basically doing exactly the same as their mitigation strategy - randomly restarting services?
- fidotron 2y agoRight. The point of the question is why not ramp up the monkey? They seem to imply it isn’t there now, which wouldn’t surprise me with the cultural shifts that have occurred in the tech world.
- mhawthorne 2y ago[dead]
- louison11 2y agoYou gotta pick your battles. Part of being in a startup is to be comfortable with quick and dirty when necessary. It’s when things get bigger, too corporate and slow that companies stop moving fast.
- Sl1mb0 2y agoWe are talking about Netflix. You know, the 'N' in FAANG/MAANG or whatever.
- afavour 2y agoAs a non-FAANGer Netflix has always intrigued me because of this. While Google, Facebook and others seem to have bogged themselves down in administrative mess, Netflix still seems agile. From the outside at least. (also worth noting this post seems to be discussing an event that occurred many years ago, circa 2011, so might not be a reflection of where they are today)
- wil421 2y agoNetflix isn’t trying to be a search engine, hardware manufacturer, consumer cloud provider (email, OneDrive, etc), cloud infrastructure provider, and an ad company at the same time. Or an Online Walmart who does all the rest and more.
- jldugger 2y agoNetflix is a much smaller enterprise. It got included because it was high growth at the time, not because it was destined to become a trillion dollar company.
- alecco 2y agoSelf-healing system: increase cluster size and replace servers randomly. It works because it was a problem of threads occasionally entering an infinite loop but not corrupting data. And the whole system can tolerate these kind of whole server crashes. IMHO an unusual combination of preconditions. It's not explained why they couldn't write a monitor script instead to find servers having the issue and only killing those.
- 4star3star 2y agoI think they just needed a quick and dirty solution that was good enough for a few days. They figured that for 1% failure per hour, they needed to kill x processes every y minutes to keep ahead of the failures. I'm sure it would be much more efficient but also more complicated to try to target the specific failures, and the "good enough" solution was acceptable.
- starlite-5008 2y ago[flagged]
- btbuildem 2y agoOn a long enough timescale, everything eventually converges to Erlang
- pmarreck 2y agoHah, hinted at that in my comment: https://news.ycombinator.com/item?id=42126301 https://news.ycombinator.com/item?id=42126301 It really is a fundamental advantage against the worst kinds of this category of bug
- pmarreck 2y agoI had to deal with a concurrency bug in Ruby once and it was so bad* that it pushed me into Elixir, which makes the vast majority of concurrency bugs impossible at the language-design level, thus enabling more sanity. Ingeniously simple solution for this particular bug though. *as I recall, it had to do with merging a regular Hash in the ENV with a HashWithIndifferentAccess, which as it turns out was ill-conceived at the time and had undefined corner cases (example: what should happen when you merge a regular Hash containing either a string or symbol key (or both) into a HashWithIndifferentAccess containing the same key but internally only represented as a string? Which takes precedence was undefined at the time.)
- ken47 2y agoIf the principles of languages like Erlang were taught in American school, things like this would be much likely to occur. Silly that Computer Science is regarded more highly by many than Software Engineering for Software Engineering jobs.
- nextos 2y agoIdeas stemming from Erlang and Mozart/Oz are indeed a big blind spot in most undergrad programs. Sadly, even in EU all this is becoming a niche topic, which is weird as today's applications are more concurrent and data-intensive than ever.
- pronoiac 2y agoI've dealt with something similar. We were able to spin up zombie reapers, looking for the cores / CPUs that were pegged at 100%, and prioritize the instances that were worst hit.
- TZubiri 2y agoNetflix is supposed to be the bastion of microservices and the trailblazer of all-aws infrastructure. But as time goes by I just ask, all this work and costs and complexity, to serve files? Yeah don't get me wrong, the size of the files are really big, AND they are streamed, noted. But it's not the programming complexity challenge that one would expect, almost all of the complexity seems to stem from metadata like when users stop watching, and how to recommend them titles to keep them hooked, and when to cut the titles and autoplay the next video to make them addicted to binge watching. Case in point, the blogpost speaks of a CPU concurrency bug and clients being servers? But never once refers to an actual business domain purpose. Like are these servers even loading video content? My bet is they are more on the optimizing engagement side of things. And I make this bet knowing that these are servers with high video-like load, but I'm confident that these guys are juggling 10TB/s of mouse metadata into some ML system more than I'm confident that they have some problem with the core of their technology which has worked since launch. As I say this, I know I'm probably wrong, surely the production issues are cause by high peak loads like a new chapter of the latest series or whatever. I'm all over the place, I just don't like netflix is what I'm saying
- romanobro56 2y ago[flagged]
- TZubiri 2y agoRizz skibidi bro
- PittleyDunkin 2y ago> But as time goes by I just ask, all this work and costs and complexity, to serve files? You could say the same thing about the entire web.
- nicce 2y agoNot really. People are not posting data into Netlix. Netflix is mostly read-only. That is huge complexity reducer.
- posix_compliant 2y agoWhat's neat is that this is a differential equation. If you kill 5% of instances each hour, the reduction in bad instances is proportional to the current number of instances. i.e. if bad(t) = fraction of bad instances at time t and bad(0) = 0 then d(bad(t))/dt = -0.05 * bad(t) + 0.01 * (1 - bad(t)) so bad(t) = 0.166667 - 0.166667 e^(-0.06 t) Which looks a mighty lot like the graph of bad instances in the blog post.
- uvdn7 2y agoLove it! I wonder if the team knew this explicitly or intuitively when they deployed the strategy. > We created a rule in our central monitoring and alerting system to randomly kill a few instances every 15 minutes. Every killed instance would be replaced with a healthy, fresh one. It doesn't look like they worked out the numbers ahead of the time.
- eigenvalue 2y agoI understand how their approach worked well enough, but I don’t get why they couldn’t selectively target the VMs that were currently experiencing problems rather than randomly select any VM to terminate. If they were exhausting all their CPU resources, wouldn’t that be easy enough to search for using something like ansible?
- rdoherty 2y agoI agree, I've been at places that can tie alerts at a host level to an automated task runner. Basically a workflow system that gets kicked off on an alert. Alert fires, host is rebooted or terminated. Helpful for things like this.
- pjdesno 2y agoVaguely related anecdote: 30 years ago or so I worked at a tiny networking company where several coworkers came from a small company (call it C) that made AppleTalk routers. They recounted being puzzled that their competitor (company S) had a reputation for having a rock-solid product, but when they got it into the lab they found their competitor's product crashed maybe 10 times more often than their own. It turned out that the competing device could reboot faster than the end-to-end connection timeout in the higher-level protocol, so in practice failures were invisible. Their router, on the other hand, took long enough to reboot that your print job or file server copy would fail. It was as simple as that, and in practice the other product was rock-solid and theirs wasn't. (This is a fairly accurate summary of what I was told, but there's a chance my coworkers were totally wrong. The conclusion still stands, I think - fast restarts can save your ass.)
- cruffle_duffle 2y agoSeems like the next priority would be to make your product reboot just as fast if not faster then theirs.
- rtkwe 2y agoClearly but maybe the thing that makes your product crash less makes it take longer to reboot. Also the story isn't that they couldn't just that they were measuring the actual failure rate not the effective failure rate because the device could recover faster than the failure caused actual issues.
- kevin_nisbet 2y agoThis is along the lines of how one of the wireless telecom products I really liked worked. Each running process had a backup on another blade in the chassis. All internal state was replicated. And the process was written in a crash only fashion, anything unexpected happened and the process would just minicore and exit. One day I think I noticed that we had over a hundred thousand crashes in the previous 24 hours, but no one complained and we just sent over the minicores to the devs and got them fixed. In theory some users would be impacted that were triggering the crashes, their devices might have a glitch and need to re-associate with the network, but the crashes caused no widespread impacts in that case. To this day I'm a fan of crash only software as a philosophy, even though I haven't had the opportunity to implement it in the software I work on.
- otterley 2y ago> It was Friday afternoon > Rolling back was cumbersome It's a fundamental principle of modern DevOps practice that rollbacks should be quick and easy, done immediately when you notice a production regression, and ideally automated. And at Netflix's scale, one would have wanted this rollout to be done in waves to minimize risk. Apparently this happened back in 2021. Did the team investigate later why you couldn't do this, and address it?
- __turbobrew__ 2y ago> Could we roll back? Not easily. I can’t recall why I can appreciate the hack to deal with this (I actually came up with the same solution in my head as reading) but if you cannot rollback and you cannot roll forward you are stuck in a special purgatory of CD hell that you should be spending every moment of time getting out of before doing anything else.
- Scubabear68 2y agoThe real key here is to understand Netflix’s business, and also many social media companies too. These companies have achieved vast scale because correctness doesn’t matter that much so long as it is “good enough” for a large enough statistical population, and their Devops practices and coding practices have evolved with this as a key factor. It is not uncommon at all for Netflix or Hulu or Facebook or Instagram to throw an error or do something bone headed. When it happens you shrug and try again. Now imagine if this was applied to credit card payments systems, or your ATM network, or similar. The reality of course is that some financial systems do operate this way, but it’s recognized as a problem and usually gets on people’s radar to fix as failed transaction rates creep up and it starts costing money directly or clients. “Just randomly kill shit” is perfectly fine in the Netflix world. In other domains, not so much (but again it can and will be used as an emergency measure!).
- asdga 2y ago[dead]
- nitwit005 2y agoThat presumably fixed things from a deployment point of view, but if there was a concurrency bug involving a hashmap, the service may have been emitting incorrect results. For example: calculate hash code of string, determine index, find apparent match, hashmap is modified by another thread, return value at that index which no longer matches. I don't think that particular issue can happen with Java's HashMap, but there's probably some sort of similar goofiness.
- ec109685 2y agoApache where I used to work was configured to restart each child after 1000 requests. So many outages prevented at very little cost.
- SnoozingBoa 2y agoIn technical context leaving unknowns with unknown boundaries behind is an approach that makes it hard to assess next actions in error cases, but even more importantly it makes the future system design work rooted on uncertainty. To me that is blocker to my thinking. I really need to understand the impact of leaving something behind before continuing. I’d likely do everything in my power to remove that unknown from current state for the sake of sanity.
- vishaldalwadi26 2y ago> Practical engineering can mean many things, but a definition I often return to is: having clear goals and making choices that are aligned with them. The main takeaway from this post. You will not encounter exactly this or similar issues at your workplace, but this single piece of advice can help you fix any issues that you do encounter.