3 ms·
From a quick skim of it I don't see any mention of the real difficulty I have encountered with the Actor model in practice: managing imbalanced actor workloads.
by zmmmmm 9y ago
From a quick skim of it I don't see any mention of the real difficulty I have encountered with the Actor model in practice: managing imbalanced actor workloads.
If you consider a simple chain of actors that do some work and then send on a result to a next actor that then does some more work:
A -> B -> C -> D
Suppose something (anything) happens that make messages to B take slightly longer to process than the rate at which A is sending them. The result is that B's message inbox expands indefinitely. One of two things will happen depending on the queuing model: either A will block on send because B refuses to accept another message, or B will simply run out of memory from too many pending messages.
This of course, is just the linear scenario. Consider a complex network where there may even be cycles (something D does could somehow indirectly result in a message coming back to A). You've essentially got a completely unstable, unpredictable system. The only way to avoid it is to ensure that every actor is massively over-resourced so that it can never fall behind on processing its messages.
Of course, the second stage of this is that you start designing scalable actor "pools" where more than one actor can service messages and you can scale the size of the pool up and down. But this is at the cost of adding even more dynamic, unpredictable behaviour, and to some extent you just lost the simplicity you were trying to gain by using Actors in the first place.
I'm curious if any thoughts from more experienced folks about how to handle these issues!
- ramchip 9y agoThere’s a lot of literature on this in the Erlang world. This post is a very thorough summary of the various solutions available: https://ferd.ca/handling-overload.html https://ferd.ca/handling-overload.html I don’t find that kind of fully async processing pipeline very idiomatic TBH, at least in Erlang - more typically, every request would be its own actor, and do blocking requests to other actors that own shared resources.
- zmmmmm 9y agoThank you for the insightful comments. That link looks perfect (down to the point of having a diagram with a->b->c->d just like mine!). The idea of each request being its own actor being more idiomatic is intriguing. Naively though does it really solve the problem? Don't you just end up with an overflow of actors now instead of an overflow of one actor's inbox?
- lliamander 9y agoIn Erlang the actors are lightweight processes are much smaller than OS threads. I wouldn't worry.
- zmmmmm 9y agoRight, but in my scenario (and I appreciate perhaps this means I'm "doing it wrong"), the messages themselves have a fairly heavy payload (around 800 bytes). Given that, whether I'm spinning up an actor or a message, if it can't be processed quickly it's consuming a non trivial chunk of memory.
- lliamander 9y agoI wouldn't say you are doing anything wrong (or, at least I couldn't say that for certain without knowing more about your problem domain :) ). However, I also don't think you have to worry about the message payloads being too large. In Erlang, normally objects are copied between processes. However, for large objects (>64 bytes) they are put in a shared heap, and only the pointers are copied between processes[0]. [0] https://hamidreza-s.github.io/erlang%20garbage%20collection%20memory%20layout%20soft%20realtime/2015/08/24/erlang-garbage-collection-details-and-why-it-matters.html https://hamidreza-s.github.io/erlang%20garbage%20collection%...
- ramchip 9y agoYou can put a bound on the number of actors under a given supervisor. That being said the main reason an Erlang program will use an actor per request is not to prevent overload, it’s for fault tolerance; in your a -> b -> c -> d example a bug (e.g. uncaught exception) triggered by one request that happens in process c can cause all requests in flight to fail, while with a process per request there’s only one affected. It does mean you may have thousands of actors rather than four or so, but that’s not a problem since Erlang is built and optimized for that kind of workload. On a platform where actors are OS threads, it may make sense to use a different approach.
- Skinney 9y agoSpin up a new actor for every message? Actor's are cheap, right? And don't form cycles.
- blatyo 9y agoThe solution for this in Elixir is to use GenStage actors, which provides backpressure by allowing downstream processes to ask for data to process. Check out this video which describes it pretty well: https://youtu.be/srtMWzyqdp8?t=1027 https://youtu.be/srtMWzyqdp8?t=1027
- javadocmd 9y agoIt is a non-trivial problem, but one common enough that I would suggest any Actor library worth its salt would include mechanisms to tackle it. (Here's Akka talking about back-pressure support in Streams, for example. https://doc.akka.io/docs/akka/2.5.11/stream/stream-flows-and-basics.html?language=scala#back-pressure-explained https://doc.akka.io/docs/akka/2.5.11/stream/stream-flows-and...) Akka also provides infrastructure for actor pooling and routing. I don't see the manifestation of this problem as forfeiting the point. You could on the opposite extreme write one monolithic application that does everything: web interface, application logic, and data storage. It's readily apparent that for most use-cases, the benefits of separating, say, your web and database software will more than outweigh the added complexity cost of running two applications and routing requests between them. (And we don't anymore balk at running pools of web servers in front of our databases.) Continue to chop your application into smaller segments and of course you'll see diminishing returns. The application just has to be complex enough to raise the expectation of benefit from segmentation.
- ex3ndr 9y agoIsn't it a wellknown and solvable problem with help of backpressure?