4 ms·

I quite like the article and the advice it presents, building what I'd call modular monoliths (that can have modules be enabled or disabled based on feature fla

I quite like the article and the advice it presents, building what I'd call modular monoliths (that can have modules be enabled or disabled based on feature flags) is indeed a good approach for both increasing resiliency and decreasing the blast radius of various issues. However, this bit stuck out to me: > When a bad Pub/Sub message was pulled into the binary, an unhandled panic would crash the entire app, meaning web, workers and crons all died. I've never seen a production ready web framework or library that would let your entire process crash because of a bad request. There's always some error handling logic wrapping around the request handling, so that even if you let something slip by, the fallout will still be limited to the threads handling the requests and its context. Now, of course there can still be greater failures, such as issues with DB access stalling for whatever reason, leading to the HTTP requests not being processed, leading to the corresponding thread pool for those filling up, as well as any request queues maxing out and thus all new requests getting rejected. But for the most part, requests with bad data and such, short of CVEs, always have limited impact. I can imagine very few situations where things working differently wouldn't be jarring.

lawrjone 4y ago
Have you worked with Go codebases before? Standard practice is to wrap all your entrypoints - the start of a web request, the moment you begin to run a job - in a defer recover() which will catch panics. Sadly, recover won’t apply to any subsequently goroutine’d work. That means even if your entrypoints recover, if anything go func()s down the call stack, then if that function then panics it will bring down the entire process. We were aware of this but ended up including one by accident anyway. It’s very sad to me that you can’t apply a global process handler to ensure this type of thing doesn’t happen, but to my knowledge that isn’t possible. Worth mentioning Go doesn’t really encourage ‘frameworks’, and most Go apps compose together various libraries to make their app, rather than using something that packages everything together. Failures like this are an obvious downside to not having the well reviewed nuts-and-bolts-included framework to handle this for you.
- Gys 4y ago
  I probably misunderstand what you wrote. Because I think a (wrapped) panic will only result in crashing the one request that caused it? For example Gin provides a middleware wrapper for handling panics: CustomRecoveryWithWriter returns a middleware for a given writer that recovers from any panics and calls the provided handle func to handle it. (https://github.com/gin-gonic/gin/blob/master/recovery.go https://github.com/gin-gonic/gin/blob/master/recovery.go)
  - deleted 4y ago
    [deleted]
  - lawrjone 4y ago
    The post-mortem we published about this outage explains the details of how this crashed the binary. It gives some code examples and explanations that should clear this up: https://incident.io/blog/intermittent-downtime#mitigation-1-ensuring-we-handle-message-panics https://incident.io/blog/intermittent-downtime#mitigation-1-... ^ link should go to the relevant section
    - Gys 4y ago
      Will have a look, thanks!
- cle 4y ago
  To me, it's unclear what the best solution is here. Other languages solve this differently with tradeoffs, e.g. Java defaults to threads silently dying when an exception isn't caught. Your program will continue to run, but it's probably in some undefined state at that point. There are mechanisms for propagating exceptions elsewhere, but they have to be explicitly set up (like in Go). You can set a default uncaught exception handler, but that's effectively a global variable with all the subsequent "fun", and the uncaught exception handler has to know how to clean up and restore state if the exception was thrown from anywhere, which seems generally difficult to do correctly.
  - 4y ago
pmontra 4y ago
We had a single supervisor for both a Elixir/Phoenix web app and all the tasks it spawned to do long running jobs and scheduled tasks. That was very naive. A job too quickly too many times would bring down the supervisor and the web app. We moved the web app to its own supervisor tree and isolated tasks to their own supervisor too. I don't remember the details but at the end everything could crash and everything else would keep running.
- nesarkvechnep 4y ago
  That's why you use a separate `Task.Supervisor`.
Aeolun 4y ago
> I've never seen a production ready web framework or library that would let your entire process crash because of a bad request. Node does this (regardless of framework/server) if you have some unhandled promise rejection/error in an asynchronous function.
- gbuk2013 4y ago
  You can catch these with Node but I wouldn’t do it - if it’s not explicitly handled then it’s either a bug or something really bad happened and the app needs to get alarm bells ringing.
  - wruza 4y ago
    Error recovery in long-running dynamic-y processes is nontrivial, especially for “programming errors”. While it can be done, much easier to stick to crashonly practices. Also helps in production where you can safely crash it and restart without worrying about stale something.
- fendy3002 4y ago
  which can be handled easily with unhandledrejection. Though ideally, web frameworks such as NestJs is already handled the rejection on request level and can be override easily with Filters. Yeah I know express isn't handled, you'll need to handle it yourself.
EGreg 4y ago
Erlang and any Actors models I think
- maeln 4y ago
  Yes and no. Erlang does apply the "fail fast" idea and recommend to let actor crash instead of trying to handle the error (with some pragmatism ofc). But we are not talking about crashing the process of the whole app. In most case in Erlang (but you find the same idea with over massively parallel system or with micro-services), your actor will be handled by a supervisor, which job it is to keep it alive, and so restart it when it crash. Also, most of the time in Erlang, having a actor crash and spawning a new one is really fast, so you can afford to crash even for something as basic as a malformed json request or similar. Not something you can really afford when spawning a new system is more in the order of several hundred ms.
fabian2k 4y ago
The most dangerous kinds of mistakes in that area are bugs or oversights that cause massive resource usage or endless loops. Those can easily bring down parts of the application that are otherwise robustly separated in the way you mentioned. Consuming too much memory will kill the app at some point, just using lots of CPU or IO will make everything else grind to a halt. Holding onto resources that are pooled like DB connections is a good way to break everything as well. There are some languages and frameworks that are designed around even handling and isolating issues like these to some extent, Erlang/Elixir for example. But in general I would expect a web application backend to isolate services the way you mentioned. But I don't think this is really a preconfigured default in every case, and it's not safe do catch everything and keep running. Isolating web requests and only failing the individual request is of course reasonable and should be the default. But for background services your framework can't know which ones are required and which ones are optional and when to crash the process. Especially as most languages allow shared state there, so the framework can't know how independent they are.
- hyperpape 4y ago
  Indeed, I remember a significant degradation of service that happened when some data files were corrupted. Each request would need to work with one or more of them. There was a cache, and the code was written to avoid a thundering herd. But in the case where the data was corrupt, an exception was thrown, and nothing was put in the cache. So the application sat dedicating multiple cores to a cycle of loading and parsing these data files then errorring out.
thraxil 4y ago
> I've never seen a production ready web framework or library that would let your entire process crash because of a bad request. It's rare, but I've seen linked C libraries trigger a segfault that takes the whole thing down; no global try/catch strategy can help you when that happens. There should generally be something that supervises and restarts the whole server process, but it's definitely painful and can affect threads other than the one handling the bad request.
marcosdumay 4y ago
> There's always some error handling logic wrapping around the request handling Yeah, some languages make this easier than others. The way you described it made me think it was Rust, a language where those handlers are not trivial, but not incredibly hard either. But it seems that Go developers have it worse (no big surprise here). Of course, the champions on unhandled failures are always C and C++, where almost any issue is impossible to recover from. It's no coincidence that those are low level languages (and Go). When you move from the more controlled languages into those, you tend to also move from in-language error recovery to system-wide error recovery.
- curioussavage 4y ago
  I actually found that in practice my go applications were much more resilient. First one in production was handling well over 100k req/s and it was extremely reliable. As a beginner it was very satisfying.
tkahnoski 4y ago
Depends on the underlying framework on how it isolates processes... I could see this happening in a monolithic JVM application where it pretends there are separate containers, but a fatal error in the JVM will crash the world on that server. A related example I lived through with a Perl application, someone decided to use this library 'Storable' that would serialize the memory in a binary format. We upgraded the library and started seeing "slow" performance across the server farm. We recognized the processes were intermittently crashing and after decoding core dumps... figured out it was this upgrade to the Storable library. Apache httpd server chugged along just fine restarting processes. So different run-time, different type of crash resiliency. Long-term lesson... be extra cautious with memory-serialized objects. Newer libraries have better protection on this to parse a header to detect compatible issues before loading the raw object into memory, but the potential is there especially with distributed systems today.
Arbortheus 4y ago
My team has experienced _exactly_ the same cyclical panicking in production before because of the same library. The Google Pub/Sub Go library does not handle panics by default, so if any message payload plays badly with your code and panics, you cyclically panic the service. That's because the message keeps getting retried because you don't `Ack` it. Non acknowledgements get retried automatically, you get the picture.

lawrjone 4y ago

Have you worked with Go codebases before? Standard practice is to wrap all your entrypoints - the start of a web request, the moment you begin to run a job - in a defer recover() which will catch panics. Sadly, recover won’t apply to any subsequently goroutine’d work. That means even if your entrypoints recover, if anything go func()s down the call stack, then if that function then panics it will bring down the entire process. We were aware of this but ended up including one by accident anyway. It’s very sad to me that you can’t apply a global process handler to ensure this type of thing doesn’t happen, but to my knowledge that isn’t possible. Worth mentioning Go doesn’t really encourage ‘frameworks’, and most Go apps compose together various libraries to make their app, rather than using something that packages everything together. Failures like this are an obvious downside to not having the well reviewed nuts-and-bolts-included framework to handle this for you.

Gys 4y ago
I probably misunderstand what you wrote. Because I think a (wrapped) panic will only result in crashing the one request that caused it? For example Gin provides a middleware wrapper for handling panics: CustomRecoveryWithWriter returns a middleware for a given writer that recovers from any panics and calls the provided handle func to handle it. (https://github.com/gin-gonic/gin/blob/master/recovery.go https://github.com/gin-gonic/gin/blob/master/recovery.go)
- deleted 4y ago
  [deleted]
- lawrjone 4y ago
  The post-mortem we published about this outage explains the details of how this crashed the binary. It gives some code examples and explanations that should clear this up: https://incident.io/blog/intermittent-downtime#mitigation-1-ensuring-we-handle-message-panics https://incident.io/blog/intermittent-downtime#mitigation-1-... ^ link should go to the relevant section
  - Gys 4y ago
    Will have a look, thanks!
cle 4y ago
To me, it's unclear what the best solution is here. Other languages solve this differently with tradeoffs, e.g. Java defaults to threads silently dying when an exception isn't caught. Your program will continue to run, but it's probably in some undefined state at that point. There are mechanisms for propagating exceptions elsewhere, but they have to be explicitly set up (like in Go). You can set a default uncaught exception handler, but that's effectively a global variable with all the subsequent "fun", and the uncaught exception handler has to know how to clean up and restore state if the exception was thrown from anywhere, which seems generally difficult to do correctly.
- 4y ago

pmontra 4y ago

We had a single supervisor for both a Elixir/Phoenix web app and all the tasks it spawned to do long running jobs and scheduled tasks. That was very naive. A job too quickly too many times would bring down the supervisor and the web app. We moved the web app to its own supervisor tree and isolated tasks to their own supervisor too. I don't remember the details but at the end everything could crash and everything else would keep running.

nesarkvechnep 4y ago
That's why you use a separate `Task.Supervisor`.

Aeolun 4y ago

> I've never seen a production ready web framework or library that would let your entire process crash because of a bad request. Node does this (regardless of framework/server) if you have some unhandled promise rejection/error in an asynchronous function.

gbuk2013 4y ago
You can catch these with Node but I wouldn’t do it - if it’s not explicitly handled then it’s either a bug or something really bad happened and the app needs to get alarm bells ringing.
- wruza 4y ago
  Error recovery in long-running dynamic-y processes is nontrivial, especially for “programming errors”. While it can be done, much easier to stick to crashonly practices. Also helps in production where you can safely crash it and restart without worrying about stale something.
fendy3002 4y ago
which can be handled easily with unhandledrejection. Though ideally, web frameworks such as NestJs is already handled the rejection on request level and can be override easily with Filters. Yeah I know express isn't handled, you'll need to handle it yourself.

EGreg 4y ago

Erlang and any Actors models I think

maeln 4y ago
Yes and no. Erlang does apply the "fail fast" idea and recommend to let actor crash instead of trying to handle the error (with some pragmatism ofc). But we are not talking about crashing the process of the whole app. In most case in Erlang (but you find the same idea with over massively parallel system or with micro-services), your actor will be handled by a supervisor, which job it is to keep it alive, and so restart it when it crash. Also, most of the time in Erlang, having a actor crash and spawning a new one is really fast, so you can afford to crash even for something as basic as a malformed json request or similar. Not something you can really afford when spawning a new system is more in the order of several hundred ms.

fabian2k 4y ago

The most dangerous kinds of mistakes in that area are bugs or oversights that cause massive resource usage or endless loops. Those can easily bring down parts of the application that are otherwise robustly separated in the way you mentioned. Consuming too much memory will kill the app at some point, just using lots of CPU or IO will make everything else grind to a halt. Holding onto resources that are pooled like DB connections is a good way to break everything as well. There are some languages and frameworks that are designed around even handling and isolating issues like these to some extent, Erlang/Elixir for example. But in general I would expect a web application backend to isolate services the way you mentioned. But I don't think this is really a preconfigured default in every case, and it's not safe do catch everything and keep running. Isolating web requests and only failing the individual request is of course reasonable and should be the default. But for background services your framework can't know which ones are required and which ones are optional and when to crash the process. Especially as most languages allow shared state there, so the framework can't know how independent they are.

hyperpape 4y ago
Indeed, I remember a significant degradation of service that happened when some data files were corrupted. Each request would need to work with one or more of them. There was a cache, and the code was written to avoid a thundering herd. But in the case where the data was corrupt, an exception was thrown, and nothing was put in the cache. So the application sat dedicating multiple cores to a cycle of loading and parsing these data files then errorring out.

thraxil 4y ago

> I've never seen a production ready web framework or library that would let your entire process crash because of a bad request. It's rare, but I've seen linked C libraries trigger a segfault that takes the whole thing down; no global try/catch strategy can help you when that happens. There should generally be something that supervises and restarts the whole server process, but it's definitely painful and can affect threads other than the one handling the bad request.

marcosdumay 4y ago

> There's always some error handling logic wrapping around the request handling Yeah, some languages make this easier than others. The way you described it made me think it was Rust, a language where those handlers are not trivial, but not incredibly hard either. But it seems that Go developers have it worse (no big surprise here). Of course, the champions on unhandled failures are always C and C++, where almost any issue is impossible to recover from. It's no coincidence that those are low level languages (and Go). When you move from the more controlled languages into those, you tend to also move from in-language error recovery to system-wide error recovery.

curioussavage 4y ago
I actually found that in practice my go applications were much more resilient. First one in production was handling well over 100k req/s and it was extremely reliable. As a beginner it was very satisfying.

tkahnoski 4y ago

Depends on the underlying framework on how it isolates processes... I could see this happening in a monolithic JVM application where it pretends there are separate containers, but a fatal error in the JVM will crash the world on that server. A related example I lived through with a Perl application, someone decided to use this library 'Storable' that would serialize the memory in a binary format. We upgraded the library and started seeing "slow" performance across the server farm. We recognized the processes were intermittently crashing and after decoding core dumps... figured out it was this upgrade to the Storable library. Apache httpd server chugged along just fine restarting processes. So different run-time, different type of crash resiliency. Long-term lesson... be extra cautious with memory-serialized objects. Newer libraries have better protection on this to parse a header to detect compatible issues before loading the raw object into memory, but the potential is there especially with distributed systems today.

Arbortheus 4y ago

My team has experienced _exactly_ the same cyclical panicking in production before because of the same library. The Google Pub/Sub Go library does not handle panics by default, so if any message payload plays badly with your code and panics, you cyclically panic the service. That's because the message keeps getting retried because you don't `Ack` it. Non acknowledgements get retried automatically, you get the picture.