3 ms·
> Implement backpressure throughout your system. This is a subtle one and gets lost among CAP theorem talk, but it is important. Just recently had to struggle
by rdtsc 10y ago
> Implement backpressure throughout your system.
This is a subtle one and gets lost among CAP theorem talk, but it is important.
Just recently had to struggle with adding backpressure. Did some things as just sending a message (a "gen_server:cast" in Erlang). The sender shoved all 300K+ messages in the mailbox of receiver which could not process them in time because of a bottleneck and bug. Receiver's memory usage went to 120GB when it eventually restarted.
The simple solution was just to switch a gen_server:cast with a gen_server:call i.e. make the receiver acknowledge, and the sender wait for acknowledge before sending another message. (As an aside that's the nice benefit of Erlang/Elixir there, it was really a 2 line change. If it was some kind of serialization library + rpc socket layer, it would have been a lot more code to write).
An obvious corollary of this is "always test on production workload and conditions, not just your laptop". What had happened locally is with a smaller test data it would fit in memory and for a 10Gb or so it wasn't noticeable. So it seemed like everything was working fine.
But speaking of CAP theorem this is always a great fun:
http://ferd.ca/beating-the-cap-theorem-checklist.html http://ferd.ca/beating-the-cap-theorem-checklist.html
---
Beating the CAP Theorem Checklist
Your ( ) tweet ( ) blog post ( ) marketing material ( ) online comment
advocates a way to beat the CAP theorem. Your idea will not work. Here is why
it won't work:
...
---
- alanning 10y agoThanks for sharing this. A few questions from someone interested in learning how to use BEAM-based systems in production: * How did you debug the issue? I know Erlang in Anger [1] is kind of written just for this but it feels pretty intense to me as a newbie. I don't know enough to tell whether the issues there are only things I'd need to worry about at larger scale but I found it pretty intimidating, to the point where I have delayed actually designing/deploying a Erlang solution. Designing for Scalability with Erlang/OTP [2] has a chapter on monitoring that I'm looking forward to reading. Wondering if there's some resources you could recommend to get started running a prod Erlang system re: debugging, monitoring, etc. * How do you decide when to use a message queue (like sqs, RabbitMQ) vs regular processes? Do you have any guidelines you could share or is it more just, "use processes when you can; more formal MQ if interfacing with non-beam systems"? I struggle since conceptually each sender/receiver has its own queue via its mailbox. 1. https://www.erlang-in-anger.com/ https://www.erlang-in-anger.com/ 2. http://shop.oreilly.com/product/0636920024149.do http://shop.oreilly.com/product/0636920024149.do
- rdtsc 10y agoGood question. So first I noticed in metrics dashboard (so it is important to have metrics) the receiver never seemed to have gotten the expected number of messages. Then noticed the count of messages was being reset. Suspected something restarted the node. Browsed through metrics, noticed both memory usage was spiking too high and node was indeed restarting (uptime kept going up and down). Focused on memory usage. We have a small function which returns top N memory hungry processes. Notice a particular one. Noticed its mailbox was tens of gigabytes. Then used recon_trace to trace that process to see what it was doing. recon_trace is written by author of Erlang In Anger. So did something like: recon_trace:calls[{my_module, '_', '_'}, 100, [{scope, local}]. dbg module is built-in and can use that, but it doesn't have rate limiting so it can kill a busy node if you trace the wrong thing (it floods you with messages). So noticed what it was doing and noticed that a particular operation was taking way too long. It was because it was doing an O(n) operation instead of O(1) on each message. On a smaller scale it wasn't noticeable but when got to 1M plus instances it was bringing everything down. After solving the problem to test it, I compiled a .beam file locally, scp-ed to test machine, hot-patched it (code:load_abs("/tmp/my_module")) and noticed that everything was working fine. On whether to pick a message queue vs regular processes. It depends. They are very different. Regular processes are easier, simpler and cheap. But they are not persistent. So perhaps if your messages are like "add $1M to my account" and sender wants to just send the messages and not worry about acknowledging it. Then you'd want something with very good persistence guarantees.
- sitkack 10y agoThis isn't backpressure, this is forcing synchronicity by using a queue depth of 1. For low throughput applications this is fine, but the latency will kill you on the long run and vastly underutilize available resources. Backpressure should be proportional to the amount of work you want to prevent. And that backpressure should propagate up the event stream to limit the whole capacity of the subgraph. The slowest chain governs the speed of the whole system so that it can retain functionality instead of falling over. Which relates to other properties of the system * work stealing, processes handle their own dynamic capacity * prioritization, we can automatically guage what to NOT do * feedback
- rdtsc 10y agoYou can think of synchronicity as the most basic form of backpressure. The goal is to avoid flooding the target when source sends at a too high of a rate. But that was a quick fix. Eventually I implemented something else. Notice that in my case there was a also a bug, so it wasn't just due to normal hardware limits. One other simplistic hack I found is to force the source to send every Nth message synchronously. Say it sends 10k messages as a cast, then if target is too slow and has queued them up, source sends 10001st as a call So it is forced to wait for target to process them before sending the next. (Sorry reverting to Erlang/OTP terminology here, cast mean send without acknowledgement, call means wait for a response).
- sitkack 10y agoAgree, but that synchronous kludge needs to be temporary. I too have used a similar hack, launching waves of tasks/messages and waiting with a timeout, possibly terminating the slow ones and then starting again. The ones that are slow multiple times can be put into a longer, bounded queue.