6 ms·

Since these insert events are being buffered, what happens if the process dies? Are all of those inserts lost? I feel like Erlang/Elixir is designed to handle

Since these insert events are being buffered, what happens if the process dies? Are all of those inserts lost? I feel like Erlang/Elixir is designed to handle cases like this robustly, but it's not clear to me how this code avoids losing data when a process crashes, potentially up to 5s worth of updates!

nrmitchi 6y ago
I'd like a solid answer to this too; it's my biggest concern with any sort of "batch online requests" functionality in any language; unsafe shutdowns can/will always happen at some point, and it feels like this is always a data loss risk.
- dnautics 6y ago
  To some degree there's always data loss risk... A backhoe could cut a line, you could lose power to a PSU, rack, data center, a meteor could destroy the willamette valley, etc. What you want your system to provide you with is the framework to understand the risks and what the recovery looks like and when recovery is a lost cause.
tomjakubowski 6y ago
I think you might solve this by writing a "reliable" supervisor process which actually receives the messages, and preserves message or state history to replay for its supervised process in case it crashes. Of course this really just shoves the problem up the stack, but at least you can stick to "let it crash" when writing the supervised process, and take more care when writing the supervisor which has a smaller scope of responsibility.
paultannenbaum 6y ago
You can add retry logic in your genServer, but for anything but the most simple use cases, you would want to add a Supervisor and define a retry strategy. https://elixir-lang.org/getting-started/mix-otp/supervisor-and-application.html https://elixir-lang.org/getting-started/mix-otp/supervisor-a...
therein 6y ago
Furthermore I have heard it is actually possible for casts to be dropped under load. I am having a hard time confirming this, and had a hard time confirming it back then too. And even worse, I realize most of the time we code as if the casts will be reliably delivered. Can someone chime in on this?
- dnautics 6y ago
  I think that casts are guaranteed delivery if the process exists. Problem is, the caster can't know that with certainty (there could be a race between checking, process death and sending). Or the recipient could just ignore the cast altogether. Out if the box it's impossible to know which happened. Calls by contrast check out a monitor on the counterparty and crash with a timeout so you have guaranteed delivery and acknowledgement, or crash the calling process. A lot of times people from other plarforms rush to use casts when they "don't need a response" but the actual meanings have more to do with failure domains and rate limiting back pressure; my personal feeling is you should default to call and only use cast when you need failure isolation.
- toast0 6y ago
  I don't think that's quite true, at least not of a gen_X cast. If you use erlang:send/3 with options or erlang:send_nosuspend/2, you can have some cases where messages would be dropped without trying. I think that may be the source of the drop under load you're thinking of? Without nosuspend, if you send to a node that's connected, dist will queue it to be sent, but there's no guarantee it's received because networking, and it may not even be sent if the other side has gone away and the send queue is already too large; There is a hard to express constraint that if your message doesn't get received in this case, dist will disconnect eventually, but it's important to note that the tick time-out doesn't guarantee timely delivery either; as long as some data is flowing, you can get quite a backlog; I think I've gotten net_adm:ping times above 30 minutes in some cases. Also, if the dist connection is dropped, but both nodes are online, it will likely reconnect shortly, and some messages will have been lost. If you send to a node that's not connected, and didn't specify no_connect, dist will queue the message while attempting to connect, but if that attempt fails, the messages will be dropped. It's also possible for code running as a gen_server (or GenServer, I suppose) to check how many messages are queued for it, and run different logic. I've written gen_servers that would drop optional requests if the queue was large. Also, if client timeouts are known, there are ways to approximate the time spent waiting in queue, and drop requests if they are received after the client already timed out; it's a little tricky to do this though.
karmajunkie 6y ago
In this case, that's correct—you could lose up to 5s worth of state, plus any buffered messages. That's by design here. The business impact of that is going to depend on use-case, but I personally would find it difficult to believe that there would be any real impact in that event, in this case, under high load. Obviously, different tolerances to that are going to necessitate different designs. You can configure the size of the message buffer when a GenServer starts, and if you have very low tolerance for lost data, you'd want to use a synchronous message (using call instead of cast, which blocks the sender until it returns) and appropriate error handling. BEAM has a plethora of features for reliable applications, you just have to apply them appropriately.
rozap 6y ago
Yea, they're just in memory, so they're lost. Erlang/Elixir doesn't solve this. There's no getting around the fact that you need an ack that your event was processed and saved to some durable store. A supervisor won't save you from lack of acknowledgment at the app level. Any restart or retry logic within beam itself is useless if a beaver chews through a power cord and the machine halts. And what is an ack, anyway? Is an ack from mongo as good as an ack from postgres? This is all stuff the language can't know. These are all business questions, all the language can do is give you tools handle these cases, which Erlang/Elixir does.
pmontra 6y ago
It doesn't. We read data from the db in the init function when the GenServer starts and make callers persist it before sending it to the GenServer. Our GenServers keep data in memory as the one in the post but we could read it from the db each time it ticks to process a record.
deleted 6y ago
[deleted]
lostcolony 6y ago
Short Answer - yes, data will be lost. THAT SAID - this is true of any buffering solution, it's why even the 'sync' style of request can still time out. Long Answer - What happens in any language where you batch stuff in memory? If the system breaks down, you lose that data. This isn't unique to Erlang/Elixir. There's a tradeoff that persistent storage is slower than memory, but it's persistent. So do you want to be fast, or do you want to be durable? However, there is some flexibility to address this at a system level. Namely, you wait for confirmation. A client, be that an actual user, another process, etc, wants to know if the write has been persisted. Whereas many other languages make this sort of caching layer completely transparent to the client (i.e., they return success immediately, and so failures mean invisible data loss as the cache is dropped), Erlang/Elixir's model makes it so you HAVE to think about this. I sent a message to the downstream process; do I care about a response? If I don't get a response for any given interval, I don't know what happened to that message. I can still do useful work in the meantime, but I don't know that my message was fully handled (persisted). I can retry or I can report failure or whatever. This is true in any distributed system, and in Erlang/Elixir it's expressed very evidently in the language constructs (rather than being hidden from you). You can build local disk caches if you want (in fact, there are included tools to make this super easy for you; ETS, DETS, and Mnesia allow you to shove Erlang terms into memory storage, disk based storage, and a hybrid of the two with some nice DB-like behaviors, respectively), but you need to choose to slow your message ingestion to the speed of local disk writes in that case (as well as handle synchronization of deletes in the event of multiple writers to your downstream). An depending what your upstream is, that doesn't provide a guarantee (i.e., a write to local disk != a persisted write from a user perspective, because the disk could crash before it ever makes it off the local one).
- rjknight 6y ago
  > What happens in any language where you batch stuff in memory? If the system breaks down, you lose that data. This isn't unique to Erlang/Elixir. A little over a decade ago I did a project using Erlang. My next project was using Ruby, and I was suddenly _horrified_ by the fact that I had no idea what would happen if the application crashed (and Ruby apps, at least in those days, crashed fairly often). Erlang/Elixir both forces you to think about these things, and gives you tools to address them, where many other languages (or their libraries) simply assume that we will stay on the happy path. Of course, you can be very productive using languages that just ignore the possibility of very rare failures. Many successful systems work on the basis that sometimes shit just happens and maybe some data does get lost. But, after programming with Erlang or Elixir for a while, that situation starts to feel less acceptable!
dudul 6y ago
Isn't it true of any, in-memory buffer implementation? It is very easy to add some form of recovery/persistence to a GenServer. You can have it write to a DB, ETS or disk. You lose some throughput obviously, it's all about balance.
edisonywh 6y ago
The article touched on it briefly, but perhaps it wasn't clear. If the process dies, the Supervisor invokes the `terminate/2` callback so you're still able to process events. Here are the relevant lines: https://github.com/plausible/analytics/blob/b724def948d51a0f07a46f5bd2568e84f708d5a2/lib/plausible/event/write_buffer.ex#L54-L57 https://github.com/plausible/analytics/blob/b724def948d51a0f... Keep in mind you need to trap exit signal to tell the supervisor to invoke the callback, as done so here: https://github.com/plausible/analytics/blob/b724def948d51a0f07a46f5bd2568e84f708d5a2/lib/plausible/event/write_buffer.ex#L12 https://github.com/plausible/analytics/blob/b724def948d51a0f... The erlang docs also mentions this: https://erlang.org/doc/design_principles/gen_server_concepts.html#stopping https://erlang.org/doc/design_principles/gen_server_concepts...
- brightball 6y ago
  I usually utilize this with an ETS cache to save and recover GenServer state in the event of a crash.
- silly-silly 6y ago
  I use this callback to capture the 'data' that caused the crash, so that it can be used to recover later or better understand why the crash happened. If I have as significantly long transaction, I can restart the transaction from transaction_id + 1 forward and then handle the actual transaction out of band by hand if its something very odd. I usually insert a logger in these code paths too, but the crash handler is usually after the logger, which makes it easier to reason what has happened if you are dealing with mutable state.
hifly 6y ago
Actually in Erlang/elixir, your own crash is something you can plan for and handle. if you check the code you will see in init() that the author captures exits. then there is a function terminate () which is called when a process exits for some reason (including crashes) So for example one possible implementation of state is that you catch your own crash, save some state and then restore it when you restart You aren't supposed to ever crash, however the whole language it's based around designing a framework to handle what happens if you do crash So you design a whole framework of supervisors, watching processes, that all have a defined startup and shutdown order. You build structures with rules such as: if my cache module fails, them restart this whole chunk of application over here. it's very cool

nrmitchi 6y ago

I'd like a solid answer to this too; it's my biggest concern with any sort of "batch online requests" functionality in any language; unsafe shutdowns can/will always happen at some point, and it feels like this is always a data loss risk.

dnautics 6y ago
To some degree there's always data loss risk... A backhoe could cut a line, you could lose power to a PSU, rack, data center, a meteor could destroy the willamette valley, etc. What you want your system to provide you with is the framework to understand the risks and what the recovery looks like and when recovery is a lost cause.

tomjakubowski 6y ago

I think you might solve this by writing a "reliable" supervisor process which actually receives the messages, and preserves message or state history to replay for its supervised process in case it crashes. Of course this really just shoves the problem up the stack, but at least you can stick to "let it crash" when writing the supervised process, and take more care when writing the supervisor which has a smaller scope of responsibility.

paultannenbaum 6y ago

You can add retry logic in your genServer, but for anything but the most simple use cases, you would want to add a Supervisor and define a retry strategy. https://elixir-lang.org/getting-started/mix-otp/supervisor-and-application.html https://elixir-lang.org/getting-started/mix-otp/supervisor-a...

therein 6y ago

Furthermore I have heard it is actually possible for casts to be dropped under load. I am having a hard time confirming this, and had a hard time confirming it back then too. And even worse, I realize most of the time we code as if the casts will be reliably delivered. Can someone chime in on this?

dnautics 6y ago
I think that casts are guaranteed delivery if the process exists. Problem is, the caster can't know that with certainty (there could be a race between checking, process death and sending). Or the recipient could just ignore the cast altogether. Out if the box it's impossible to know which happened. Calls by contrast check out a monitor on the counterparty and crash with a timeout so you have guaranteed delivery and acknowledgement, or crash the calling process. A lot of times people from other plarforms rush to use casts when they "don't need a response" but the actual meanings have more to do with failure domains and rate limiting back pressure; my personal feeling is you should default to call and only use cast when you need failure isolation.
toast0 6y ago
I don't think that's quite true, at least not of a gen_X cast. If you use erlang:send/3 with options or erlang:send_nosuspend/2, you can have some cases where messages would be dropped without trying. I think that may be the source of the drop under load you're thinking of? Without nosuspend, if you send to a node that's connected, dist will queue it to be sent, but there's no guarantee it's received because networking, and it may not even be sent if the other side has gone away and the send queue is already too large; There is a hard to express constraint that if your message doesn't get received in this case, dist will disconnect eventually, but it's important to note that the tick time-out doesn't guarantee timely delivery either; as long as some data is flowing, you can get quite a backlog; I think I've gotten net_adm:ping times above 30 minutes in some cases. Also, if the dist connection is dropped, but both nodes are online, it will likely reconnect shortly, and some messages will have been lost. If you send to a node that's not connected, and didn't specify no_connect, dist will queue the message while attempting to connect, but if that attempt fails, the messages will be dropped. It's also possible for code running as a gen_server (or GenServer, I suppose) to check how many messages are queued for it, and run different logic. I've written gen_servers that would drop optional requests if the queue was large. Also, if client timeouts are known, there are ways to approximate the time spent waiting in queue, and drop requests if they are received after the client already timed out; it's a little tricky to do this though.

karmajunkie 6y ago

In this case, that's correct—you could lose up to 5s worth of state, plus any buffered messages. That's by design here. The business impact of that is going to depend on use-case, but I personally would find it difficult to believe that there would be any real impact in that event, in this case, under high load. Obviously, different tolerances to that are going to necessitate different designs. You can configure the size of the message buffer when a GenServer starts, and if you have very low tolerance for lost data, you'd want to use a synchronous message (using call instead of cast, which blocks the sender until it returns) and appropriate error handling. BEAM has a plethora of features for reliable applications, you just have to apply them appropriately.

rozap 6y ago

Yea, they're just in memory, so they're lost. Erlang/Elixir doesn't solve this. There's no getting around the fact that you need an ack that your event was processed and saved to some durable store. A supervisor won't save you from lack of acknowledgment at the app level. Any restart or retry logic within beam itself is useless if a beaver chews through a power cord and the machine halts. And what is an ack, anyway? Is an ack from mongo as good as an ack from postgres? This is all stuff the language can't know. These are all business questions, all the language can do is give you tools handle these cases, which Erlang/Elixir does.

pmontra 6y ago

It doesn't. We read data from the db in the init function when the GenServer starts and make callers persist it before sending it to the GenServer. Our GenServers keep data in memory as the one in the post but we could read it from the db each time it ticks to process a record.

deleted 6y ago

[deleted]

lostcolony 6y ago

Short Answer - yes, data will be lost. THAT SAID - this is true of any buffering solution, it's why even the 'sync' style of request can still time out. Long Answer - What happens in any language where you batch stuff in memory? If the system breaks down, you lose that data. This isn't unique to Erlang/Elixir. There's a tradeoff that persistent storage is slower than memory, but it's persistent. So do you want to be fast, or do you want to be durable? However, there is some flexibility to address this at a system level. Namely, you wait for confirmation. A client, be that an actual user, another process, etc, wants to know if the write has been persisted. Whereas many other languages make this sort of caching layer completely transparent to the client (i.e., they return success immediately, and so failures mean invisible data loss as the cache is dropped), Erlang/Elixir's model makes it so you HAVE to think about this. I sent a message to the downstream process; do I care about a response? If I don't get a response for any given interval, I don't know what happened to that message. I can still do useful work in the meantime, but I don't know that my message was fully handled (persisted). I can retry or I can report failure or whatever. This is true in any distributed system, and in Erlang/Elixir it's expressed very evidently in the language constructs (rather than being hidden from you). You can build local disk caches if you want (in fact, there are included tools to make this super easy for you; ETS, DETS, and Mnesia allow you to shove Erlang terms into memory storage, disk based storage, and a hybrid of the two with some nice DB-like behaviors, respectively), but you need to choose to slow your message ingestion to the speed of local disk writes in that case (as well as handle synchronization of deletes in the event of multiple writers to your downstream). An depending what your upstream is, that doesn't provide a guarantee (i.e., a write to local disk != a persisted write from a user perspective, because the disk could crash before it ever makes it off the local one).

rjknight 6y ago
> What happens in any language where you batch stuff in memory? If the system breaks down, you lose that data. This isn't unique to Erlang/Elixir. A little over a decade ago I did a project using Erlang. My next project was using Ruby, and I was suddenly _horrified_ by the fact that I had no idea what would happen if the application crashed (and Ruby apps, at least in those days, crashed fairly often). Erlang/Elixir both forces you to think about these things, and gives you tools to address them, where many other languages (or their libraries) simply assume that we will stay on the happy path. Of course, you can be very productive using languages that just ignore the possibility of very rare failures. Many successful systems work on the basis that sometimes shit just happens and maybe some data does get lost. But, after programming with Erlang or Elixir for a while, that situation starts to feel less acceptable!

dudul 6y ago

Isn't it true of any, in-memory buffer implementation? It is very easy to add some form of recovery/persistence to a GenServer. You can have it write to a DB, ETS or disk. You lose some throughput obviously, it's all about balance.

edisonywh 6y ago

The article touched on it briefly, but perhaps it wasn't clear. If the process dies, the Supervisor invokes the `terminate/2` callback so you're still able to process events. Here are the relevant lines: https://github.com/plausible/analytics/blob/b724def948d51a0f07a46f5bd2568e84f708d5a2/lib/plausible/event/write_buffer.ex#L54-L57 https://github.com/plausible/analytics/blob/b724def948d51a0f... Keep in mind you need to trap exit signal to tell the supervisor to invoke the callback, as done so here: https://github.com/plausible/analytics/blob/b724def948d51a0f07a46f5bd2568e84f708d5a2/lib/plausible/event/write_buffer.ex#L12 https://github.com/plausible/analytics/blob/b724def948d51a0f... The erlang docs also mentions this: https://erlang.org/doc/design_principles/gen_server_concepts.html#stopping https://erlang.org/doc/design_principles/gen_server_concepts...

brightball 6y ago
I usually utilize this with an ETS cache to save and recover GenServer state in the event of a crash.
silly-silly 6y ago
I use this callback to capture the 'data' that caused the crash, so that it can be used to recover later or better understand why the crash happened. If I have as significantly long transaction, I can restart the transaction from transaction_id + 1 forward and then handle the actual transaction out of band by hand if its something very odd. I usually insert a logger in these code paths too, but the crash handler is usually after the logger, which makes it easier to reason what has happened if you are dealing with mutable state.

hifly 6y ago

Actually in Erlang/elixir, your own crash is something you can plan for and handle. if you check the code you will see in init() that the author captures exits. then there is a function terminate () which is called when a process exits for some reason (including crashes) So for example one possible implementation of state is that you catch your own crash, save some state and then restore it when you restart You aren't supposed to ever crash, however the whole language it's based around designing a framework to handle what happens if you do crash So you design a whole framework of supervisors, watching processes, that all have a defined startup and shutdown order. You build structures with rules such as: if my cache module fails, them restart this whole chunk of application over here. it's very cool