3 ms·
Not only Facebook, but the whole industry, if I may say. For example, I had a client who could not get rid of weird JVM GC pauses that would come from nowhere,
by dig1 5y ago
Not only Facebook, but the whole industry, if I may say. For example, I had a client who could not get rid of weird JVM GC pauses that would come from nowhere, after which it would render the application unusable for some time. The cheapest solution (as they could not scale it to more machines because it would just replicate the matter) was to add a custom code that would ping systemd watchdog every X seconds. If that stopped, systemd would restart the app.
I think there are two problems with "hard core" debugging: 1) these things are costly and will usually pull in the best devs (so no one will be working on that "important" UI-related ticket that should have been finished a week ago /s). 2) for the management team, spending 10-15 hours fixing one-line or/xor/bit-flip magic stuff doesn't make much sense, and I'd usually get responses like: "oh, it was THAT simple?". Yeah, right, none of you'd seen I've read that Intel manual twice and re-evaluated my math knowledge multiple times over the last two nights :D
- oblio 5y agoIIS literally has functionality built-in to restart its subprocesses or whatever they're called, if the memory usage goes of X or if the subprocess crashes. And yes, it is used in production in very big companies. The main thing that was monitored was actually the restart rate/memory increase rate, i.e. a restart every 6 hours: ok, a restart every 30 minutes, not ok.
- nvr219 5y agoIf it's stupid and it works it ain't stupid.