11 ms·

Makes me remember how Facebook operates memory leaks at scale; "won't fix, we just reboot servers when they crash".

by rvnx 5y ago

silisili 5y ago
As a former linux expert and sysadmin turned developer, I think this attitude is what pissed me off most about current developers. I thought I'd have a huge upper hand, understanding dark magic. Come to find out most developers don't know and don't care about any of it, and have infected the whole community with the idea that none of it matters and servers and processes are just cattle. /rant
- anonymoushn 5y ago
  Devs don't generally have the time to take good care of these things while meeting the goals of the business, especially because they probably work at a company with 0 sysadmins.
- cuddlecake 5y ago
  As a current developer, what pisses me off about anyone is sweeping statements like this :(
  - silisili 5y ago
    I'm also a current developer. This isn't a slight against developers, but the prevailing mindset. And that mindset is little more than 'move fast and keep rebooting things'
    - zerocount 5y ago
      Yep, the quickest solution with the least amount of effort, unfortunately.
      - jeltz 5y ago
        But the amount of technical debt created by always using the leadt effort solution can be huge.
  - jiveturkey 5y ago
    Please, where do you work such that you are afforded adequate time to debug deep OS level issues? That it's acceptable to code to handle the 1-10^12 type error? I want to work there too!
- lillecarl 5y ago
  I mean, while we shouldn't leak but the systems should be able to deal with it right?
  - silisili 5y ago
    Different mindsets. I was cutting my teeth on a set of 12 static servers/processes, with no autoscaling or auto reboots. Memory leaks were rare but reported. Usually FD leaks were culprits. So learned a ton about how to write really correct applications, and today it just feels like none of that matters. And in complete fairness, with today's ops tools, it probably doesn't.
    - Arainach 5y ago
      At scale, server failures are going to happen. Maybe it's a power outage. Maybe it's a failed drive. Maybe it's a kernel crash. Only one of those can be remedied (not even prevented) with careful debugger investigation, so you're going to have to do the work to be able to handle such failures gracefully. At that point, unless the leaks are causing outages so rapidly that the machines can't handle, then rebooting becomes not that big of a deal, while wasting senior eng days trying to track them down is a real cost.
      - hvgk 5y ago
        There are some nasty escalation scenarios I’ve seen in that space. Cost increases as does workload backlogs and latency. Eventually something breaks and causes an outage. All leaks should be resolved lest you end up with floods.
  - numpad0 5y ago
    So long as it isn’t running anything realtime and process restart cost isn’t a problem (realtime == consistent run time, predictable behavior, != “fast”)
- dmitriid 5y ago
  > As a former linux expert and sysadmin turned developer, I think this attitude is what pissed me off most about current developers Where by "current" you mean "always has been" (I remember this being a solution to early Rails problems, for example)
  - samhw 5y ago
    He's getting it from the line of code in everyone's head that goes `let imageOfWorld = exaggeratedAntithesis(self.ideals)`
- onion2k 5y ago
  "My app has a memory leak. Please can I have the engineering time to debug an operating system." wouldn't fly at any company. It's not that developers don't care; it's that developers don't have any choice.
  - ajuc 5y ago
    I'm pretty sure I did this several times and I worked in a company of 15 people where our whole IT department was me and the other guy. Depends on the domain. For example we had our devices for impaired vision people stop responding to touch events on our cheap OEM touchscreens when German locale was enabled. Turns out linux driver used locale-specific code to read calibration configuration so each event position was scaled by orders of magnitude :) The funny thing is that then they fixed that in the driver and if you had commas in the config file as a workaround - you get a kernel panic :)
    - onion2k 5y ago
      That's a problem where you had no real choice to do anything else by the sounds of it though. You couldn't just "turn it off and on again" to fix it. In the case of a server where you can just have health check that reboots the server if memory gets a bit too low, it's far simpler just to automate a power cycle. It's not laziness or incompetence; it's understanding where to spend your effort best.
      - ajuc 5y ago
        If we had memory leaks caused by drivers we would have to debug them all the way as well, restarting the device every few hours wasn't an option.
        onion2k 5y ago
        Exactly. We all choose the most cost effective solution - it's a balance between engineering effort and how effective the fix is. In the case of a touch screen, rebootingit every few hours isn't as effective as debugging the underlying cause. In the case of a web server having a memory leak, killing a pod and bringing up a new one automatically when it gets unhealthy definitely is. It's not surprising that each dev team settles on different solutions to each problem.
- deleted 5y ago
  [deleted]
- hvgk 5y ago
  That’s about right. I’ve got a dev who’s container just exits 0 ten times a day. He told me kubernetes takes care of it so it’s not a priority.
- _def 5y ago
  Of course I can't speak about your experiences, but I think the reason for this are mostly business goals. At least I have the impression that lots of developers care for technical excellence, but often that is not desirable from a business perspective. That said, I really wish this wasn't the case. I miss the days when efficient software really mattered (just look at modern tv firmwares, or even mobile stuff).
  - silisili 5y ago
    You're probably right. I still remember coming in to 'take over' a new product one (crazy) guy had written. It had no safe guards at all - no queueing, kept everything in memory, no retries or really error handling other than printlines. I remember pointing out 'what if x happens', where x is a really realistic scenario, and the author(and VP) kept saying 'oh thats a caviar problem.' When I asked what that meant, they said "by the time we have to worry about that scenario, we'll be so rich we'll all be eating caviar." Apparently retrying network requests or getting say, 50 requests at a time would make us all millionaires or something. Predictably, the thing fell over multiple times per day until we trashed it and started from scratch...and I still have no idea what caviar tastes like.
- marcan_42 5y ago
  They don't care, until a kernel panic causes disk corruption in just the right way to cause a cross-security-boundary information leak that also ends up breaking an entire global pipeline, or a rare buffer overread in a ubiquitous piece of code causes once in a petabyte mismatches in cross-checked processes with compliance implications. Source: xoogler, developed a reputation for black magic debugging, debugged both of those things. Correctness still matters, no matter how big you are or how much you think auto-restarts help.
  - jasmas 5y ago
    Nah they still don't care
    - nisegami 5y ago
      Never stay at one place long enough for these things to come up /s
    - bluGill 5y ago
      That depends on how noticed it gets. My company makes embedded systems that in some failure cases can kill - we have a history of losing large lawsuits over the base 70 years (before computers of course). As such we have a powerful safety review system in place because million dollar lawsuits make CEOs care - it isn't perfect but there are a lot of things that we don't do that our customers would love.
  - discardable_dan 5y ago
    Yes, they really don't care unless it results in a site going down. An industrial giant is happy to throw infinite machines at a problem until it is resolved. The issue isn't in their behavior, it's in Linux's implementation being so poor that they have to. But what are the alternatives? Should Google/Facebook/Amazon throw infinite developer time resolving kernel issues? Should they walk away from Linux?
    - mhoad 5y ago
      I mean one of them at least went ahead and wrote (technically is in the process of writing) an entire operating system from scratch in response. I say this as someone who uses and has used linux as a daily driver for years now but honestly I don’t feel good about the future of the platform in the medium term at all. If anyone has some time to throw at understanding why this is probably the single most sober and rational set of arguments I’ve found of the topic yet https://m.youtube.com/watch?v=WtJ9T_IJOPE https://m.youtube.com/watch?v=WtJ9T_IJOPE
- einpoklum 5y ago
  > Come to find out most developers don't know and don't care about any of it Can you better define "it"? > and have infected the whole community with the idea that none of it matters and servers and processes are just cattle. Perhaps if by "community" you mean the "run everything on VMs on the cloud" people...
- dexen 5y ago
  >"won't fix, we just reboot servers when they crash" >understanding dark magic That is the original Unix black magic: KISS and do the simplest possible thing that works and when in doubt use brute force and so forth. That is the "C" language of infrastructure. In the particular in the scenario we're discussing, there's always a pool of several servers ready to handle a HTTP request (or DB query, or multimedia transformation, etc.). Rebooting a small % of this pool, rather than debugging every single memory leak is a pragmatic, cost-effective solution. "Zero bugs" is a very abstract and detached goal; "cost effectiveness" is a very realistic and wholesome goal.
- solididiot 5y ago
  You have only this much bandwidth between business logic, the last trendy framework/language thrust upon you, the billion overengineering "good practices" adding clutter to your code base and the zillion intricacies of the latest kubernetes platform that "makes your life easier". Most devs have way more than they can work with a single human mind on their plates. Accusing them of missing out on the low level kernel stuff that is about 5 or 6 levels of indirection deeper than the stuff they work day in and day out is like accusing pilots of insufficient knowledge of organic chemistry to understand the jet fuel composition.
- elesbao 5y ago
  They have not infected, that's not fair. they just don't care and shouldn't be blamed just because you care too much. Good that you and others care and contribute to improve tools and libraries.
- ninkendo 5y ago
  I feel like the overall direction of the software world has been aiming to maximize developer productivity -- at the expense of rock solid reliability -- for quite some time now. You can think of garbage collection as being an early admittance that it's far simpler to just clean things up in bulk on a periodic basis, than it is to keep track of every resource you're allocating, and making sure that it's freed at the earliest possible time. Restarting a service after it's been running for hours, because of memory leaks, seems like a broader implementation of a similar notion... despite best efforts, sometimes services just accumulate cruft, and so long as you can properly drain clients to another server before restarting, is it really that bad to have to restart it periodically? I think of it as an extra long GC pause that frees everything up. There's a sweet spot where "good enough" really is good enough, and I don't lament it all that much.
- skyde 5y ago
  it’s a good idea to assume server will fail (power supply, hard drive, network card) because we you use 3k of them they do. But it’s a not a good excuse to be wasteful! which is why I like that Rust and Go are getting more popular
namdnay 5y ago
Every big org I’ve worked for had some form of this. To be honest, if the estimated cost of fixing the leak is an order of magnitude bigger than the cost of recycling the nodes every week for the next 5 years, then it’s the best thing to do from a business perspective, however sad that may be as engineers :)
spullara 5y ago
This is a long time thing though. Tuxedo (a transaction processing system from the early 90s) would periodically restart processes to ensure that memory leaks didn't compromise the system. The Apache webserver has a knob for how many times you should run code in a process before restarting it. At the end of the day it is an extremely practical way to ensure that memory leaks don't bring your service down.
hahnchen 5y ago
is there a src on this? Or are you just kidding around
- lclarkmichalek 5y ago
  I don't know any major service at FB that works this way. There could be risks from increased uptime, but I don't know any service that has OOMing as a regular, expected behaviour.
- pdpi 5y ago
  Memory is big enough an issue that they have a whole team working on jemalloc internally. It’s a big enough issue that, when a release of jemalloc messed up the memory usage profile on my team’s service, one of my team mates spent a couple of months figuring out why. That sort of cavalier attitude to memory leaks doesn’t really fly when we’re talking potentially millions of dollars in server costs going down the drain.
dig1 5y ago
Not only Facebook, but the whole industry, if I may say. For example, I had a client who could not get rid of weird JVM GC pauses that would come from nowhere, after which it would render the application unusable for some time. The cheapest solution (as they could not scale it to more machines because it would just replicate the matter) was to add a custom code that would ping systemd watchdog every X seconds. If that stopped, systemd would restart the app. I think there are two problems with "hard core" debugging: 1) these things are costly and will usually pull in the best devs (so no one will be working on that "important" UI-related ticket that should have been finished a week ago /s). 2) for the management team, spending 10-15 hours fixing one-line or/xor/bit-flip magic stuff doesn't make much sense, and I'd usually get responses like: "oh, it was THAT simple?". Yeah, right, none of you'd seen I've read that Intel manual twice and re-evaluated my math knowledge multiple times over the last two nights :D
- oblio 5y ago
  IIS literally has functionality built-in to restart its subprocesses or whatever they're called, if the memory usage goes of X or if the subprocess crashes. And yes, it is used in production in very big companies. The main thing that was monitored was actually the restart rate/memory increase rate, i.e. a restart every 6 hours: ok, a restart every 30 minutes, not ok.
  - nvr219 5y ago
    If it's stupid and it works it ain't stupid.
PragmaticPulp 5y ago
> "won't fix, we just reboot servers when they crash". Where are you getting this? The article is about a tool Facebook made specifically because they have teams doing deep-dive analysis of even rare bugs that occur in their systems. This suggests the opposite of Facebook being the type of company that looks the other way when crashes occur.
wodenokoto 5y ago
That’s almost also how airplanes work. They just reboot before the crash. https://www.theregister.com/AMP/2020/04/02/boeing_787_power_cycle_51_days_stale_data/ https://www.theregister.com/AMP/2020/04/02/boeing_787_power_...
- david422 5y ago
  IIRC - it's an integer overflow problem. And power cycles generally come much sooner than than the 51 days. From a software engineering perspective - seems like something that should be fixed. But apparently someone else decided not to. Maybe complexity? Cost?
  - trinovantes 5y ago
    Definitely cost Why go through the entire recertification process when the alternative solution is simply reboot every couple of weeks?
1024core 5y ago
A long time ago, I used to work at one of the search engines. We had a problem where the server would OOM after about ~10K search requests. Solution: put in a counter and exit(0) after 9K requests (a config option). Time goes on. Server starts OOM-ing again. Solution: change the config to exit after 8K requests. Time goes on. Server starts OOM-ing again. Solution: change the config to exit after 6K requests (daring move, one would say). Time goes on. Server starts OOM-ing again. Solution: change the config to exit after 4K requests. That's around when I left the company
forgotmyoldacc 5y ago
I'm disappointed that this irrelevant trivia is the top voted comment on a post about an interesting kernel debugger.
jodrellblank 5y ago
See talk "The Mess We're In" by Joe Armstrong (Erlang co-creator) https://www.youtube.com/watch?v=lKXe3HUG2l4 https://www.youtube.com/watch?v=lKXe3HUG2l4 The tl;dr version is that there are 10^80 atoms in the known universe, a computer with 1MB of memory can be in 2^1,000,000 or 10^300,000 possible states. Worse with a thousand times more memory. The easiest, quickest, perhaps the only, way to get it into a known state is to reboot it. (Amusing introduction: "I took Friday off to be at home, because I can't get any work done at work, too many interruptions") (I like Joe Armstrong's sense of humour here, dry, understated: "A prompt came up saying 'there's a new version of Open Office' and I thought 'That's good, it'll be better'.".)

Makes me remember how Facebook operates memory leaks at scale; "won't fix, we just reboot servers when they crash".

by rvnx 5y ago

Makes me remember how Facebook operates memory leaks at scale; "won't fix, we just reboot servers when they crash".

As a former linux expert and sysadmin turned developer, I think this attitude is what pissed me off most about current developers. I thought I'd have a huge upper hand, understanding dark magic. Come to find out most developers don't know and don't care about any of it, and have infected the whole community with the idea that none of it matters and servers and processes are just cattle. /rant

anonymoushn 5y ago
Devs don't generally have the time to take good care of these things while meeting the goals of the business, especially because they probably work at a company with 0 sysadmins.
cuddlecake 5y ago
As a current developer, what pisses me off about anyone is sweeping statements like this :(
- silisili 5y ago
  I'm also a current developer. This isn't a slight against developers, but the prevailing mindset. And that mindset is little more than 'move fast and keep rebooting things'
  - zerocount 5y ago
    Yep, the quickest solution with the least amount of effort, unfortunately.
    - jeltz 5y ago
      But the amount of technical debt created by always using the leadt effort solution can be huge.
- jiveturkey 5y ago
  Please, where do you work such that you are afforded adequate time to debug deep OS level issues? That it's acceptable to code to handle the 1-10^12 type error? I want to work there too!
lillecarl 5y ago
I mean, while we shouldn't leak but the systems should be able to deal with it right?
- silisili 5y ago
  Different mindsets. I was cutting my teeth on a set of 12 static servers/processes, with no autoscaling or auto reboots. Memory leaks were rare but reported. Usually FD leaks were culprits. So learned a ton about how to write really correct applications, and today it just feels like none of that matters. And in complete fairness, with today's ops tools, it probably doesn't.
  - Arainach 5y ago
    At scale, server failures are going to happen. Maybe it's a power outage. Maybe it's a failed drive. Maybe it's a kernel crash. Only one of those can be remedied (not even prevented) with careful debugger investigation, so you're going to have to do the work to be able to handle such failures gracefully. At that point, unless the leaks are causing outages so rapidly that the machines can't handle, then rebooting becomes not that big of a deal, while wasting senior eng days trying to track them down is a real cost.
    - hvgk 5y ago
      There are some nasty escalation scenarios I’ve seen in that space. Cost increases as does workload backlogs and latency. Eventually something breaks and causes an outage. All leaks should be resolved lest you end up with floods.
- numpad0 5y ago
  So long as it isn’t running anything realtime and process restart cost isn’t a problem (realtime == consistent run time, predictable behavior, != “fast”)
dmitriid 5y ago
> As a former linux expert and sysadmin turned developer, I think this attitude is what pissed me off most about current developers Where by "current" you mean "always has been" (I remember this being a solution to early Rails problems, for example)
- samhw 5y ago
  He's getting it from the line of code in everyone's head that goes `let imageOfWorld = exaggeratedAntithesis(self.ideals)`
onion2k 5y ago
"My app has a memory leak. Please can I have the engineering time to debug an operating system." wouldn't fly at any company. It's not that developers don't care; it's that developers don't have any choice.
- ajuc 5y ago
  I'm pretty sure I did this several times and I worked in a company of 15 people where our whole IT department was me and the other guy. Depends on the domain. For example we had our devices for impaired vision people stop responding to touch events on our cheap OEM touchscreens when German locale was enabled. Turns out linux driver used locale-specific code to read calibration configuration so each event position was scaled by orders of magnitude :) The funny thing is that then they fixed that in the driver and if you had commas in the config file as a workaround - you get a kernel panic :)
  - onion2k 5y ago
    That's a problem where you had no real choice to do anything else by the sounds of it though. You couldn't just "turn it off and on again" to fix it. In the case of a server where you can just have health check that reboots the server if memory gets a bit too low, it's far simpler just to automate a power cycle. It's not laziness or incompetence; it's understanding where to spend your effort best.
    - ajuc 5y ago
      If we had memory leaks caused by drivers we would have to debug them all the way as well, restarting the device every few hours wasn't an option.
      - onion2k 5y ago
        Exactly. We all choose the most cost effective solution - it's a balance between engineering effort and how effective the fix is. In the case of a touch screen, rebootingit every few hours isn't as effective as debugging the underlying cause. In the case of a web server having a memory leak, killing a pod and bringing up a new one automatically when it gets unhealthy definitely is. It's not surprising that each dev team settles on different solutions to each problem.
deleted 5y ago
[deleted]
hvgk 5y ago
That’s about right. I’ve got a dev who’s container just exits 0 ten times a day. He told me kubernetes takes care of it so it’s not a priority.
_def 5y ago
Of course I can't speak about your experiences, but I think the reason for this are mostly business goals. At least I have the impression that lots of developers care for technical excellence, but often that is not desirable from a business perspective. That said, I really wish this wasn't the case. I miss the days when efficient software really mattered (just look at modern tv firmwares, or even mobile stuff).
- silisili 5y ago
  You're probably right. I still remember coming in to 'take over' a new product one (crazy) guy had written. It had no safe guards at all - no queueing, kept everything in memory, no retries or really error handling other than printlines. I remember pointing out 'what if x happens', where x is a really realistic scenario, and the author(and VP) kept saying 'oh thats a caviar problem.' When I asked what that meant, they said "by the time we have to worry about that scenario, we'll be so rich we'll all be eating caviar." Apparently retrying network requests or getting say, 50 requests at a time would make us all millionaires or something. Predictably, the thing fell over multiple times per day until we trashed it and started from scratch...and I still have no idea what caviar tastes like.
marcan_42 5y ago
They don't care, until a kernel panic causes disk corruption in just the right way to cause a cross-security-boundary information leak that also ends up breaking an entire global pipeline, or a rare buffer overread in a ubiquitous piece of code causes once in a petabyte mismatches in cross-checked processes with compliance implications. Source: xoogler, developed a reputation for black magic debugging, debugged both of those things. Correctness still matters, no matter how big you are or how much you think auto-restarts help.
- jasmas 5y ago
  Nah they still don't care
  - nisegami 5y ago
    Never stay at one place long enough for these things to come up /s
  - bluGill 5y ago
    That depends on how noticed it gets. My company makes embedded systems that in some failure cases can kill - we have a history of losing large lawsuits over the base 70 years (before computers of course). As such we have a powerful safety review system in place because million dollar lawsuits make CEOs care - it isn't perfect but there are a lot of things that we don't do that our customers would love.
- discardable_dan 5y ago
  Yes, they really don't care unless it results in a site going down. An industrial giant is happy to throw infinite machines at a problem until it is resolved. The issue isn't in their behavior, it's in Linux's implementation being so poor that they have to. But what are the alternatives? Should Google/Facebook/Amazon throw infinite developer time resolving kernel issues? Should they walk away from Linux?
  - mhoad 5y ago
    I mean one of them at least went ahead and wrote (technically is in the process of writing) an entire operating system from scratch in response. I say this as someone who uses and has used linux as a daily driver for years now but honestly I don’t feel good about the future of the platform in the medium term at all. If anyone has some time to throw at understanding why this is probably the single most sober and rational set of arguments I’ve found of the topic yet https://m.youtube.com/watch?v=WtJ9T_IJOPE https://m.youtube.com/watch?v=WtJ9T_IJOPE
einpoklum 5y ago
> Come to find out most developers don't know and don't care about any of it Can you better define "it"? > and have infected the whole community with the idea that none of it matters and servers and processes are just cattle. Perhaps if by "community" you mean the "run everything on VMs on the cloud" people...
dexen 5y ago
>"won't fix, we just reboot servers when they crash" >understanding dark magic That is the original Unix black magic: KISS and do the simplest possible thing that works and when in doubt use brute force and so forth. That is the "C" language of infrastructure. In the particular in the scenario we're discussing, there's always a pool of several servers ready to handle a HTTP request (or DB query, or multimedia transformation, etc.). Rebooting a small % of this pool, rather than debugging every single memory leak is a pragmatic, cost-effective solution. "Zero bugs" is a very abstract and detached goal; "cost effectiveness" is a very realistic and wholesome goal.
solididiot 5y ago
You have only this much bandwidth between business logic, the last trendy framework/language thrust upon you, the billion overengineering "good practices" adding clutter to your code base and the zillion intricacies of the latest kubernetes platform that "makes your life easier". Most devs have way more than they can work with a single human mind on their plates. Accusing them of missing out on the low level kernel stuff that is about 5 or 6 levels of indirection deeper than the stuff they work day in and day out is like accusing pilots of insufficient knowledge of organic chemistry to understand the jet fuel composition.
elesbao 5y ago
They have not infected, that's not fair. they just don't care and shouldn't be blamed just because you care too much. Good that you and others care and contribute to improve tools and libraries.
ninkendo 5y ago
I feel like the overall direction of the software world has been aiming to maximize developer productivity -- at the expense of rock solid reliability -- for quite some time now. You can think of garbage collection as being an early admittance that it's far simpler to just clean things up in bulk on a periodic basis, than it is to keep track of every resource you're allocating, and making sure that it's freed at the earliest possible time. Restarting a service after it's been running for hours, because of memory leaks, seems like a broader implementation of a similar notion... despite best efforts, sometimes services just accumulate cruft, and so long as you can properly drain clients to another server before restarting, is it really that bad to have to restart it periodically? I think of it as an extra long GC pause that frees everything up. There's a sweet spot where "good enough" really is good enough, and I don't lament it all that much.
skyde 5y ago
it’s a good idea to assume server will fail (power supply, hard drive, network card) because we you use 3k of them they do. But it’s a not a good excuse to be wasteful! which is why I like that Rust and Go are getting more popular

namdnay 5y ago

Every big org I’ve worked for had some form of this. To be honest, if the estimated cost of fixing the leak is an order of magnitude bigger than the cost of recycling the nodes every week for the next 5 years, then it’s the best thing to do from a business perspective, however sad that may be as engineers :)

spullara 5y ago

This is a long time thing though. Tuxedo (a transaction processing system from the early 90s) would periodically restart processes to ensure that memory leaks didn't compromise the system. The Apache webserver has a knob for how many times you should run code in a process before restarting it. At the end of the day it is an extremely practical way to ensure that memory leaks don't bring your service down.

hahnchen 5y ago

is there a src on this? Or are you just kidding around

lclarkmichalek 5y ago
I don't know any major service at FB that works this way. There could be risks from increased uptime, but I don't know any service that has OOMing as a regular, expected behaviour.
pdpi 5y ago
Memory is big enough an issue that they have a whole team working on jemalloc internally. It’s a big enough issue that, when a release of jemalloc messed up the memory usage profile on my team’s service, one of my team mates spent a couple of months figuring out why. That sort of cavalier attitude to memory leaks doesn’t really fly when we’re talking potentially millions of dollars in server costs going down the drain.

dig1 5y ago

Not only Facebook, but the whole industry, if I may say. For example, I had a client who could not get rid of weird JVM GC pauses that would come from nowhere, after which it would render the application unusable for some time. The cheapest solution (as they could not scale it to more machines because it would just replicate the matter) was to add a custom code that would ping systemd watchdog every X seconds. If that stopped, systemd would restart the app. I think there are two problems with "hard core" debugging: 1) these things are costly and will usually pull in the best devs (so no one will be working on that "important" UI-related ticket that should have been finished a week ago /s). 2) for the management team, spending 10-15 hours fixing one-line or/xor/bit-flip magic stuff doesn't make much sense, and I'd usually get responses like: "oh, it was THAT simple?". Yeah, right, none of you'd seen I've read that Intel manual twice and re-evaluated my math knowledge multiple times over the last two nights :D

oblio 5y ago
IIS literally has functionality built-in to restart its subprocesses or whatever they're called, if the memory usage goes of X or if the subprocess crashes. And yes, it is used in production in very big companies. The main thing that was monitored was actually the restart rate/memory increase rate, i.e. a restart every 6 hours: ok, a restart every 30 minutes, not ok.
- nvr219 5y ago
  If it's stupid and it works it ain't stupid.

PragmaticPulp 5y ago

> "won't fix, we just reboot servers when they crash". Where are you getting this? The article is about a tool Facebook made specifically because they have teams doing deep-dive analysis of even rare bugs that occur in their systems. This suggests the opposite of Facebook being the type of company that looks the other way when crashes occur.

wodenokoto 5y ago

That’s almost also how airplanes work. They just reboot before the crash. https://www.theregister.com/AMP/2020/04/02/boeing_787_power_cycle_51_days_stale_data/ https://www.theregister.com/AMP/2020/04/02/boeing_787_power_...

david422 5y ago
IIRC - it's an integer overflow problem. And power cycles generally come much sooner than than the 51 days. From a software engineering perspective - seems like something that should be fixed. But apparently someone else decided not to. Maybe complexity? Cost?
- trinovantes 5y ago
  Definitely cost Why go through the entire recertification process when the alternative solution is simply reboot every couple of weeks?

1024core 5y ago

A long time ago, I used to work at one of the search engines. We had a problem where the server would OOM after about ~10K search requests. Solution: put in a counter and exit(0) after 9K requests (a config option). Time goes on. Server starts OOM-ing again. Solution: change the config to exit after 8K requests. Time goes on. Server starts OOM-ing again. Solution: change the config to exit after 6K requests (daring move, one would say). Time goes on. Server starts OOM-ing again. Solution: change the config to exit after 4K requests. That's around when I left the company

forgotmyoldacc 5y ago

I'm disappointed that this irrelevant trivia is the top voted comment on a post about an interesting kernel debugger.

jodrellblank 5y ago

See talk "The Mess We're In" by Joe Armstrong (Erlang co-creator) https://www.youtube.com/watch?v=lKXe3HUG2l4 https://www.youtube.com/watch?v=lKXe3HUG2l4 The tl;dr version is that there are 10^80 atoms in the known universe, a computer with 1MB of memory can be in 2^1,000,000 or 10^300,000 possible states. Worse with a thousand times more memory. The easiest, quickest, perhaps the only, way to get it into a known state is to reboot it. (Amusing introduction: "I took Friday off to be at home, because I can't get any work done at work, too many interruptions") (I like Joe Armstrong's sense of humour here, dry, understated: "A prompt came up saying 'there's a new version of Open Office' and I thought 'That's good, it'll be better'.".)