4 ms·

I don't agree with this approach. Say you have a network service that relies on other network services. It is not difficult to write those such that they know

I don't agree with this approach. Say you have a network service that relies on other network services. It is not difficult to write those such that they know to back off / retry when something disappears. It's extremely useful in a lot of situations: if you do work on a laptop that gets regularly unplugged, having running test services that know to reconnect makes your life easier. In production, having things automatically reconnect means a lot less restarting of services once whatever root cause problem is corrected. Just shrugging and giving up ends up being a lot more work in the end. I like to tell junior developers to catch everything they can, and handle it or die as nicely as possible. Of course you can't plan for everything, but you can write around network and disk issues and issue warnings in a way that makes the root cause more obvious. That involves catching errors.

verdagon 4y ago
I'd imagine retrying/reconnecting is compatible with the general "let it fail" approach. If you just sent a message/request to an actor/server and it still hasn't responded after 5 seconds, you can send another. It wouldn't matter whether that actor/server died from a regular error or a "let it fail" error, the retrying would still work the same.
- _gabe_ 4y ago
  > If you just sent a message/request to an actor/server and it still hasn't responded after 5 seconds, you can send another. This depends on the message. I hope amazon doesn't just send another message if my transaction didn't complete in 5 seconds. I think like all pieces of wisdom, sometimes it's OK to let it fail, and sometimes its OK to handle the errors. If anyone ever tells me to always do X or never do X, it's typically not sound advice. The one thing we can always count on is generic advice failing sometimes :) (And even in this article, you shouldn't always try to handle known errors and never try to handle unknown errors, there will be exceptions)
  - ShroudedNight 4y ago
    > I hope amazon doesn't just send another message if my transaction didn't complete in 5 seconds. I do, but only while also associating a stable nonce to the transaction.
akdor1154 4y ago
While the article focuses on the programming side, the other side is BEAM's Links and Supervisors which is what really allow this. Letting BEAM handle that stuff like it is designed to could probably do a better job than your junior devs, and of course then free them up to be writing useful stuff instead.
TameAntelope 4y ago
What you're describing are "known" states; the idea behind "let it fail" is that you shouldn't write code that exhaustively handles every single potential outcome, just the ones that are part of your code's path in general use. Definitely write code to handle network issues. Don't write code to handle random bitflips, ways to handle garbage coming back from the service you're connecting to, or try to handle OOM errors. Just let those fail. Do not catch everything you can. That's the whole point of "let it fail". An app crashing is totally fine and expected behavior, in a lot of cases (of course if it's not fine, e.g. someone dies, don't do that but if you're working on that kind of software and taking advice from me, you're super duper screwed).
- hansvm 4y ago
  Adding to that, even stuff like OOM errors _can_ be known states. It's not unreasonable for stuff like "one database per machine" to be able to adapt to the available memory. The point of "let it fail" is _just_ to drop the outcomes of your code's path in general use.
- calvinmorrison 4y ago
  One of the best protocol family is email protocols! It says - call back later, the line is down, give me an hour. Eventual consistency is better than failed calls. When my email went down last week, I wasn't worried. Eventually all messages would arrive. I like that system.
- IshKebab 4y ago
  > Don't write code to handle random bitflips It depends what you're doing. There's no fixed threshold for "errors that you should handle" so smackeyacky is right - handle the errors you can (but don't spend an inordinate amount of time handling very unlikely errors). Bitflips are not very unlikely on huge systems so you need to handle them. In my experience trying to distinguish between "expected" or "normal" errors and "unexpected" or "exceptional" errors is pointless and impossible. It's better to think about the likelihood of errors.
  - TameAntelope 4y ago
joshuamorton 4y ago
I've said this often specifically in the context of golang, but while you're right that retries and similar are a common case, they are fairly similar to the 'expected' error case in the article, and can almost always be handled at precisely the place where you raise the error. In python this is @retry.retry(exceptions=[RpcException], tries=5, backoff=2, jitter=1) def my_external_rpc_call(...): .... And RpcException will only be raised beyond this if the backend is unreachable for ~30 seconds. Similarly, rpc services can abstract over this entirely, grpc (and presumably others) allow you to configure the retry policy per rpc service or method, and have it reflected everywhere that is used, without writing wrappers[0]. Which really all is to say, once you have solid libraries that handle retries of operations that are known to be error prone (file IO, network IO, things that could lock/block, etc.) you pretty quickly get into "any error implies we're totally boned". [0]: https://github.com/grpc/grpc-go/blob/f601dfac73c9/examples/features/retry/client/main.go https://github.com/grpc/grpc-go/blob/f601dfac73c9/examples/f...
- guitarbill 4y ago
  A warning about this; retries should only be done at boundaries. And it's important to know if e.g. the http or API library already implements retires, not to mention which errors should be retried. I have seen at least one codebase where the retries where completely out of hand. In short, I've found retrying well is harder than it looks.
  - sarchertech 4y ago
    This is important. I’ve seen a case where retries were happening in the service mesh, the http client library, and the application code.
toast0 4y ago
Retrying is ideally handled at a single place though. If the original client is going to retry for failure, including timeout, any intermediate retries are likely to result in signficant multiplication during outages, and that makes for a more difficult recovery. It's also easy to miss reporting on intermediate retries and your system is running poorly and you didn't know. Having things automatically reconnect is separate from automatic retries of individual requests.

verdagon 4y ago

I'd imagine retrying/reconnecting is compatible with the general "let it fail" approach. If you just sent a message/request to an actor/server and it still hasn't responded after 5 seconds, you can send another. It wouldn't matter whether that actor/server died from a regular error or a "let it fail" error, the retrying would still work the same.

_gabe_ 4y ago
> If you just sent a message/request to an actor/server and it still hasn't responded after 5 seconds, you can send another. This depends on the message. I hope amazon doesn't just send another message if my transaction didn't complete in 5 seconds. I think like all pieces of wisdom, sometimes it's OK to let it fail, and sometimes its OK to handle the errors. If anyone ever tells me to always do X or never do X, it's typically not sound advice. The one thing we can always count on is generic advice failing sometimes :) (And even in this article, you shouldn't always try to handle known errors and never try to handle unknown errors, there will be exceptions)
- ShroudedNight 4y ago
  > I hope amazon doesn't just send another message if my transaction didn't complete in 5 seconds. I do, but only while also associating a stable nonce to the transaction.

akdor1154 4y ago

While the article focuses on the programming side, the other side is BEAM's Links and Supervisors which is what really allow this. Letting BEAM handle that stuff like it is designed to could probably do a better job than your junior devs, and of course then free them up to be writing useful stuff instead.

TameAntelope 4y ago

What you're describing are "known" states; the idea behind "let it fail" is that you shouldn't write code that exhaustively handles every single potential outcome, just the ones that are part of your code's path in general use. Definitely write code to handle network issues. Don't write code to handle random bitflips, ways to handle garbage coming back from the service you're connecting to, or try to handle OOM errors. Just let those fail. Do not catch everything you can. That's the whole point of "let it fail". An app crashing is totally fine and expected behavior, in a lot of cases (of course if it's not fine, e.g. someone dies, don't do that but if you're working on that kind of software and taking advice from me, you're super duper screwed).

hansvm 4y ago
Adding to that, even stuff like OOM errors _can_ be known states. It's not unreasonable for stuff like "one database per machine" to be able to adapt to the available memory. The point of "let it fail" is _just_ to drop the outcomes of your code's path in general use.
calvinmorrison 4y ago
One of the best protocol family is email protocols! It says - call back later, the line is down, give me an hour. Eventual consistency is better than failed calls. When my email went down last week, I wasn't worried. Eventually all messages would arrive. I like that system.
IshKebab 4y ago
> Don't write code to handle random bitflips It depends what you're doing. There's no fixed threshold for "errors that you should handle" so smackeyacky is right - handle the errors you can (but don't spend an inordinate amount of time handling very unlikely errors). Bitflips are not very unlikely on huge systems so you need to handle them. In my experience trying to distinguish between "expected" or "normal" errors and "unexpected" or "exceptional" errors is pointless and impossible. It's better to think about the likelihood of errors.
- TameAntelope 4y ago

joshuamorton 4y ago

I've said this often specifically in the context of golang, but while you're right that retries and similar are a common case, they are fairly similar to the 'expected' error case in the article, and can almost always be handled at precisely the place where you raise the error. In python this is @retry.retry(exceptions=[RpcException], tries=5, backoff=2, jitter=1) def my_external_rpc_call(...): .... And RpcException will only be raised beyond this if the backend is unreachable for ~30 seconds. Similarly, rpc services can abstract over this entirely, grpc (and presumably others) allow you to configure the retry policy per rpc service or method, and have it reflected everywhere that is used, without writing wrappers[0]. Which really all is to say, once you have solid libraries that handle retries of operations that are known to be error prone (file IO, network IO, things that could lock/block, etc.) you pretty quickly get into "any error implies we're totally boned". [0]: https://github.com/grpc/grpc-go/blob/f601dfac73c9/examples/features/retry/client/main.go https://github.com/grpc/grpc-go/blob/f601dfac73c9/examples/f...

guitarbill 4y ago
A warning about this; retries should only be done at boundaries. And it's important to know if e.g. the http or API library already implements retires, not to mention which errors should be retried. I have seen at least one codebase where the retries where completely out of hand. In short, I've found retrying well is harder than it looks.
- sarchertech 4y ago
  This is important. I’ve seen a case where retries were happening in the service mesh, the http client library, and the application code.

toast0 4y ago

Retrying is ideally handled at a single place though. If the original client is going to retry for failure, including timeout, any intermediate retries are likely to result in signficant multiplication during outages, and that makes for a more difficult recovery. It's also easy to miss reporting on intermediate retries and your system is running poorly and you didn't know. Having things automatically reconnect is separate from automatic retries of individual requests.