6 ms·

I've always found myself in fundamental disagreement with the Erlang/Elixir philosophy. Resilience, within the context of this presentation, and the philos

I've always found myself in fundamental disagreement with the Erlang/Elixir philosophy. Resilience, within the context of this presentation, and the philosophy of Erlang, would be the capability to recover from errors while unattended. I prefer systems that don't fail quite so smoothly. Take this example: The application runs out of memory due to too many transactions being processed at once. The problem happens once every day. However the system gracefully recovers, and the team is able to resolve the issue peacefully without the problem escalating to an active incident. If I was funding development of such a system, I would find this situation unacceptable. I'm paying for an application that doesn't crap out in production every single day, not one that "gracefully recovers" after each such event so the dev team can "peacefully" manage it. The users whose data is lost during these incidents probably wouldn't feel quite so peaceful about it, either.

draw_down 3y ago
[dead]
lobstrosity420 3y ago
Is your argument that the application gracefully recovering from the scenario will somehow make the dev team accommodated? Hard crashes are not an acceptable substitute for observability, or continuous improvement.
- munchler 3y ago
  I'm not sure what you mean by "accommodated". This example doesn't even rise to the level of an "active incident" in the Erlang philosophy. In other words, it's not a bug, so there's no urgency to improve it.
  - lobstrosity420 3y ago
    If there is data loss it’s an incident, full stop. Your observability layer should be letting you know.
    - munchler 3y ago
      I agree. However, the linked article that I was quoting from seems to see things differently. It describes a situation in which transactions are failing (i.e. data is being lost), but it's not an incident.
      - throwawaymaths 3y ago
        It's still in your logs and you're still tracking it with whatever o11y suite you're using.
      - di4na 3y ago
        Transaction failing does not mean dataloss. If you think it is, you do not understand what graceful recovery means. Graceful recovery means that something handle that failure after these transactions failed. There is no data loss. They may have been slower, but i think we can agree that a slight temporary latency for no dataloss and graceful handling of unexpected stuff like your database machine being on fire is not so bad?
  - lawn 3y ago
    You're just misunderstanding the philosophy. It's still a bug that should be fixed, it's just that the effects are better contained thanks to the ability to self-heal.
    - munchler 3y ago
      I'm just quoting the article. It's not an "active incident", whatever that means.
      - bcrosby95 3y ago
        It means resolving it can wait until work hours instead of waking someone up in the middle of the night on Saturday.
      - pessimizer 3y ago
        It means that it's not causing a service outage.
sph 3y ago
Hard to respond to this, it's too high level and generic to disagree with. But in practice, if you're on the BEAM and use OTP as intended, your code is fully parallel, split into fault-tolerant pieces, and one crash does not affect the entire system. I have been responsible of two Elixir monoliths in the last 6 years, and I have not seen them crash even once. Not even during prototyping, not even during load testing, never once. Without having to adopt defensive coding strategies. I write 1/4 the error checking code than if it were Rust, and 1/10 than if it were Python or Go. How many platforms can realistically promise this? Note, I haven't said my code is bug free. It's buggy because I'm only human and make plenty of mistakes. But the BEAM saves me from myself, saves me from third-party JSON endpoints that for a split second return corrupted data, saves me because the DB connection dropped for a second, and saves me from the myriad of transient heisenbugs that happen in production. The bugs are still there. But the system is fault-tolerant because it has been designed to be. -- If Saša Jurić can't prove it to you, then no one can: https://youtu.be/JvBT4XBdoUE https://youtu.be/JvBT4XBdoUE (I'm pretty sure every time someone posts this link, the number of Elixir converts increases a notch)
- munchler 3y ago
  I'm sure your software is great, and I don't mean this personally, but I just find this philosophy really strange. You don't write defensive code, and you don't often check for errors. You just let pieces of your application fail, and feel that this is OK because the overall application continues on. IMHO, Elixir isn't "saving" you from "transient heisenbugs", it's encouraging you not worry about them, which means that they will proliferate.
  - sph 3y ago
    How can I ever defend my position of lead engineer responsible for these systems, when someone on the Internet, that has never seen my code running, is adamant my systems are unstable and lousy with bugs. Trying to continue this conversation is a waste of our time. Feel free not to use Elixir.
    - munchler 3y ago
Jtsummers 3y ago
> The users whose data is lost during these incidents probably wouldn't feel quite so peaceful about it, either. Who says data has been lost? Only if your protocol is fire-and-forget with no checking. But then you'd have data loss no matter the language(s) you implemented it in if your system became inundated with requests and crashed.
- evadne 3y ago
  I think robustness is best managed in layers Databases have WAL -> Systems have task queues The executor itself can fail, if the job is not lost :)

draw_down 3y ago

[dead]

lobstrosity420 3y ago

Is your argument that the application gracefully recovering from the scenario will somehow make the dev team accommodated? Hard crashes are not an acceptable substitute for observability, or continuous improvement.

munchler 3y ago
I'm not sure what you mean by "accommodated". This example doesn't even rise to the level of an "active incident" in the Erlang philosophy. In other words, it's not a bug, so there's no urgency to improve it.
- lobstrosity420 3y ago
  If there is data loss it’s an incident, full stop. Your observability layer should be letting you know.
  - munchler 3y ago
    I agree. However, the linked article that I was quoting from seems to see things differently. It describes a situation in which transactions are failing (i.e. data is being lost), but it's not an incident.
    - throwawaymaths 3y ago
      It's still in your logs and you're still tracking it with whatever o11y suite you're using.
    - di4na 3y ago
      Transaction failing does not mean dataloss. If you think it is, you do not understand what graceful recovery means. Graceful recovery means that something handle that failure after these transactions failed. There is no data loss. They may have been slower, but i think we can agree that a slight temporary latency for no dataloss and graceful handling of unexpected stuff like your database machine being on fire is not so bad?
- lawn 3y ago
  You're just misunderstanding the philosophy. It's still a bug that should be fixed, it's just that the effects are better contained thanks to the ability to self-heal.
  - munchler 3y ago
    I'm just quoting the article. It's not an "active incident", whatever that means.
    - bcrosby95 3y ago
      It means resolving it can wait until work hours instead of waking someone up in the middle of the night on Saturday.
    - pessimizer 3y ago
      It means that it's not causing a service outage.

sph 3y ago

Hard to respond to this, it's too high level and generic to disagree with. But in practice, if you're on the BEAM and use OTP as intended, your code is fully parallel, split into fault-tolerant pieces, and one crash does not affect the entire system. I have been responsible of two Elixir monoliths in the last 6 years, and I have not seen them crash even once. Not even during prototyping, not even during load testing, never once. Without having to adopt defensive coding strategies. I write 1/4 the error checking code than if it were Rust, and 1/10 than if it were Python or Go. How many platforms can realistically promise this? Note, I haven't said my code is bug free. It's buggy because I'm only human and make plenty of mistakes. But the BEAM saves me from myself, saves me from third-party JSON endpoints that for a split second return corrupted data, saves me because the DB connection dropped for a second, and saves me from the myriad of transient heisenbugs that happen in production. The bugs are still there. But the system is fault-tolerant because it has been designed to be. -- If Saša Jurić can't prove it to you, then no one can: https://youtu.be/JvBT4XBdoUE https://youtu.be/JvBT4XBdoUE (I'm pretty sure every time someone posts this link, the number of Elixir converts increases a notch)

munchler 3y ago
I'm sure your software is great, and I don't mean this personally, but I just find this philosophy really strange. You don't write defensive code, and you don't often check for errors. You just let pieces of your application fail, and feel that this is OK because the overall application continues on. IMHO, Elixir isn't "saving" you from "transient heisenbugs", it's encouraging you not worry about them, which means that they will proliferate.
- sph 3y ago
  How can I ever defend my position of lead engineer responsible for these systems, when someone on the Internet, that has never seen my code running, is adamant my systems are unstable and lousy with bugs. Trying to continue this conversation is a waste of our time. Feel free not to use Elixir.
  - munchler 3y ago

Jtsummers 3y ago

> The users whose data is lost during these incidents probably wouldn't feel quite so peaceful about it, either. Who says data has been lost? Only if your protocol is fire-and-forget with no checking. But then you'd have data loss no matter the language(s) you implemented it in if your system became inundated with requests and crashed.

evadne 3y ago
I think robustness is best managed in layers Databases have WAL -> Systems have task queues The executor itself can fail, if the job is not lost :)