5 ms·
What Kind of Asynchronous Is Right for You?
- deleted 3y ago[deleted]
- pphysch 3y ago> Choose event-driven communication wherever you can. When designing a new system, start with events. This article advocates for event-driven systems, but elides their subtle but major downside over request-response: they make your control flow "non-linear" and very difficult to reason about and analyze statically. That is, when an event fires, what happens in response? Well, any bit of code in the entire system could be triggered by it. Events are powerful, but very much a double-edged sword, similar to GOTO.
- erhaetherth 3y agoWe have this problem at my work. Some components listen for events, but where did they come from? I have no idea, I didn't write the entire stack, and there isn't an easy way to trace it. Similarly, if I send an event (and this may be JS/DOM specific), who is going to catch it, and are they going to prevent it from bubbling up to other listeners (potentially breaking them)?
- hinkley 3y agoI had a logic bug in some new code this week and was reminded that in JavaScript, if you emit an Error and there are no handlers, it bubbles back up to the emitter as an exception. Someone’s notion that they didn’t want errors to go unhandled. So I had to rewrite the logic so the conditional for reporting the error was inside the listener instead of around it.
- lutzh 3y agoThanks for your feedback! Of course we all have our "bubble", we make our own experiences, and if we share our learnings, they might not always match what other people have experienced. In practice, I feel a statement like "any event could trigger any bit of code in the entire system" a bit exaggerated. When I think of event driven systems, I think of business applications, where a large number of microservices interoperates mainly over Kafka. So the services that are triggered by an event are already limited to those subscribing to a specific Kafka topic. Of course if you follow the chain of subsequent events (i.e. service A emits an event, with leads to service B emitting events, with lead to service C and D emitting events..), you might not be aware for every event what'll happen way downstream. But you can turn this into an architectural strength, as it leads to better decoupling and separation of concerns. On another note, as you describe GOTO as a double sided sword, now I really wonder what the positive side of GOTO is..
- pphysch 3y agoEvent-driven communication is appropriate for implementing hooks for external consumers. "Firing an event" is roughly equivalent to "running an undefined set of procedures in an undefined order". That's useful, because consumer behavior is usually undefined, or at least far outside the scope of a single software system. Your system firing the events should assume the size of that set is 0. That is, it should not rely on its own events for its internal control flow, why introduce what is essentially UB? That would be like a webserver querying its own HTTP endpoints instead of its local database for the same data. Abusing external interfaces for internal purposes. Wasteful and difficult to trace.
- roeles 3y agoI guess erlang is an example of keeping your flow linear, albeit at the cost of searching through the incoming event queue and thus reordering of events.
- iudqnolq 3y agoCan you clarify? I haven't seen people peek at messages without consuming them in Erlang. Even so I don't see how this gets you reordering.
- toast0 3y agoI'm not quite sure, but I think what they're getting at is that when you send a message and immediately wait for the reply, that's generally a selective receive: in the optimized case, it inspects all inbound messages from after you created your outbound message; in the non optimized case, it looks through the whole process mailbox before waiting for new messages. Either way, it's processing some inbound messages before others for a process that is getting messages for requests as well as responses. Personally, I have seen ocassional attempts to prioritize inbound queues, but it gets unwieldy quickly: you end up either reading nearly the whole mailbox on each iteration, or having to queue non-priority messages yourself, or do terrible things in erts code to add the ability to send to the front of a mailbox.
- roeles 3y agoAs far as I understand, the receive[1] call performs pattern matching on messages in the queue. If no match is found, the call blocks until any of the patterns match. If a match is found, the first match is returned. Since you perform this pattern matching, this means that messages may not be processed in the order they were received. [1] https://www.erlang.org/blog/message-passing/ https://www.erlang.org/blog/message-passing/
- hinkley 3y agoIncluding the bit that fired the original event. Event loops have happened in every event driven system I’ve ever encountered. The original proponents having sworn that nothing like that would happen, and yet it did. Code is a collaboration. Everyone is putting their ideas into it. And some days our best ideas aren’t very good. Systems that require constant vigilance always unravel in the end, and event based systems require constant vigilance. With an event system, if the loop happens infrequently enough, it will only result in a slight decrease in throughput of the system, and so the system will continue to work for hours, days, even weeks. And once a bug has survived for more than a few weeks it becomes difficult to triage.
- ttymck 3y ago> a failure of one service does not cascade and degrade the functionality of all other services using it. This always struck me as a blatantly disingenuous generalization. What are the two services being described here? It matters very much. If my product is an email platform, and the "API service" accepted the "send email" request by enqueuing a "send email" message for the "email sender service", if the "email sender service" is down, it absolutely does degrade the experience of API service users: "your email is sent" vs "your email will be sent as soon as we're able" is two very different promises. I think I'd like to work for the company who doesn't care about the difference in those two promises. It'd be very easy to just implement the worst possible solution and provide the flakiest possible experience. Of course, I haven't seemed to find those employers yet in my career. Everyone wants reliability and performance and everything else that sounds nice, all without specifying how the product should really behave. Where are the companies where I can implement these brilliant solutions without the downsides?
- jerrygenser 3y agoI think it's clearly better to have the email eventually sent than the case where a completely synchronous service is down resulting in no email being sent
- iudqnolq 3y agoUnder certain circumstances you might want to put backpressure on the email emitter. Under certain circumstances you might want to hard error. If I'm trying to reset my password and your UI says to check my email but I don't receive an email I'm gonna be annoyed. My first guess will be some kind of intermittent delivery issue, so I'll probably send a new request to your UI. If this is important to me then after an hour or so I might open a support ticket. I'll have wasted a bunch of time checking to see if I've finally got the email. I'd much rather have gotten an error and the retried when your status page indicates the issue is resolved.
- lutzh 3y agoThanks for your feedback! Delayed e-mail can have of course have many other reasons than the service supposed to send it being down. But let's go with your example. Let's say you have a LoginService, and a MessagingService, and if you request a password reset, the LoginService sends a ResetRequested event. Now if the MessagingService is down, you'd still get told to check your inbox, although the e-mail hasn't even been sent. That's a bad user experience. Agreed. Your suggested solution is to make the communication between LoginService and MessagingService not event driven. So if the MessagingService is down, you can "hard error". But how much better is the experience then, really? As a user, you request the reset, and after a few seconds (you're already annoyed and checking if your internet connection is down), you get an error message that you can't reset your password right now. I'd argue that option is equally unacceptable. What you need to do, is to make the MessagingService better. In an event driven system, each participating service makes a promise. The MessagingService must give service level guarantees, it must be built and scaled to be able to process all the events it's listening to. If it has an error, not you should get the error message - you are the last person able to do something about it - but someone who fixes it. If you want a better user experience, don't build unreliable services and then apologize for the inconvenience. Built better (more reliable, more scalable) software. In the end, that's what event-driven architecture is about.
- continuational 3y agoAs a user, I always need a response, even if it's just the dreaded "we'll get back to you ASAP".
- lutzh 3y agoAgreed! For the user facing bit, I consider REST or some other form of HTTP-based a given. As a (human) user, I want to base my communication on commands and queries, and do want responses! But the services that generate those responses should not have runtime dependencies on other services. Those backend services should interoperate in an event-driven way.
- bullen 3y agoHow this article does not mention SSE, comet or chunking escapes me. What does their definition of event-driven really look like in practice. Nobody has a clue. Here is the ideal event driven system, it's async-to-async: https://github.com/tinspin/rupy/wiki/Fuse https://github.com/tinspin/rupy/wiki/Fuse The example is not working because I had to shut down the services for multiple reasons, but the high level of it is that you use 4 (potentially different) threads to do one request/response middle man transaction. That way you have _zero_ io-wait or thread idling. I'm surprised nobody has copied this approach since I invented it 10 years ago. I understand why though you need your entire chain to be async and that means rewriting everything and that is a big risk when it's hard to debug. But if you succeed you can build something that is 10x perf/watt than all other implementations. Which is going to be important when interest rates go higher and crash our entire industry.
- deleted 3y ago[deleted]
- anacrolix 3y agowtf is this shit
- hinkley 3y agoYou have the same problem that a number of my coworkers have had: there comes a point where someone is so bad at explaining themselves that you can’t tell if they have a really brilliant idea, or if they are so used to complecting things that they’ve taken something straightforward and wrapped it up in a baroque package. Sometimes such people succeed anyway, but typically when they find themselves surrounded with people suffering from Impostor Syndrome. Which is to say, people who think, “this is to complex for me, I must be too dumb” and not, “this is to complex for me, it must be too dumb” The problem with most really exceptional ideas is that, once said out loud, they sound completely obvious to most everyone else. If they come out early in a design process, people think you haven’t contributed much at all. They only see the value if the suggestions come after a plan has already started to wind itself around the axles. At which point the people suffering from Sunk Cost may push back because we worked so hard to solve this problem the hard way we’d feel foolish going back now. If you want people to understand what you’re saying, do the work. Maybe try reading some Feynman. Stop using contractions in your examples. Use bigger charts, and use charting types people in industry actually use, instead of inventing your own. Pull out a thesaurus and find more accurate words (but still at an 5-8th grade reading level). Software made of complicated ideas doesn’t scale on the human side. Because eventually you’re staring at fifty “simple” ideas that interact in very complex patterns. If the ideas start out more complex than necessary, you run out of runway (similarly if you try to dumb everything down with “All Xs are Ys” absolutism you end up with a hundred ideas instead of fifty, and did by the same sword). As simple as possible, but no simpler. And if you can’t explain your ideas to college freshman, you don’t understand the domain as well as you think you do.
- samsquire 3y agoThank you for such an article. * In some systems I want to reply ASAP because a message is on the hot path, so the response is acknowledgement of persisting data and an fsync to assure the request was saved and maybe an identifier so the progress of the request can be checked. I might want to send some work to background processes. * I feel systems lack end-to-end responsiveness. If you're polling with REST to see if something is finished then there's a problem with your design chosen by the kind of async you chose. * I feel async/sync nonblocking/blocking push/pull is orthogonal to everything in your system design and everything falls out from that decision. If only there was a way to map between the three. I am working in this space: I want async/sync to be something that can be seamlessly switched between and handled by the infrastructural plumbing and you can focus on the business logic. * I think systems design is actually a tree for performance reasons similar to Rust communicating between nodes is slow as it introduces round trip times.
- jefc1111 3y agoNice article. My favourite bit is '"No" is better!'. Going to use that one plenty.
- Hardliner66 3y agoFirst of all, I hate that the author pitches event-driven as something different than message passing. It's the same thing, just without reply. And if you need a reply or not is irrelevant to the usage of message passing. It's also dependent on the use case if you want/need a reply or not. Do I want to authenticate a user? Then a reply might be useful. Do I send performance metrics to a server? I might not need to know if that worked or not, depending on what's defined.