10 ms·

I often hear the argument in favor of event-driven architecture that you can work on one part of a system in isolation without having to consider the other part

I often hear the argument in favor of event-driven architecture that you can work on one part of a system in isolation without having to consider the other parts, and then I get assigned some task which requires me to consider the entire system operation, now with events that are harder to trace than function calls would have been. Now when people argue “because decoupling,” I hear, “You don’t get as much notification that you just broke a downstream system.”

moandcompany 2y ago
Integration tests?
- worik 2y ago
  ...happen too late
hobs 2y ago
I think generally a lot of these types of problems were actually had by folks who grew out of single node systems and had a lot of interesting ideas to solve problems that were new in those domains, GIVEN they've already solved the stateful domain problems as well. When you've never grown out of a single node domain but you do event driven "because scaling" or whatever, you've shot yourself in the foot amazingly hard.
- The_Colonel 2y ago
  Yes, events, async, eventual consistency, decoupling represent a difficult/complex solution for some hard problems encountered when scaling high. But people often forget there are trade-offs to everything and if you don't have these hard problems, you're giving yourself only headaches. My pet-peeve is "decoupling" - it's treated as holy with only benefits and no downsides. But it's actually again a level of complexity - unless you need it, tightly coupled code will be easier to write, read, debug etc.
scubbo 2y ago
Amen. Event-driven architecture makes it easier to bury your head in the sand, and harder to implement an actually-working feature.
trevor-e 2y ago
Like anything it can be abused and sometimes folks go overboard with turning everything into an event. However, when done right, it is really amazing to work with. As an event producer as long as you follow reasonable backwards-compatibility best practices then you should be pretty safe from breaking things downstream. As a consumer, follow defensive programming and allow for idempotency in case you need to reprocess an event. Pretty straightforward once you get the hang of things.
- gnat 2y ago
  Do you have pointers to such best practices? Gratefully received etc.
  - trevor-e 2y ago
    Design for things to be easily failable. It should be trivial to have failed messages to go to a DLQ and then reprocess them later on, say after a bug fix. Only make additive changes, don't change existing fields. For enums it's up to the consumer to ensure they don't fail when a new case is added. Be very careful with including data (especially time/expiry stuff) in the message too. If you need to reprocess the event several hours later then it may no longer work or be stale. Rather than include the data in the message itself, we would include the database ID and then have the consumer query for that entry.
  - mvc 2y ago
    Rich Hickey's talk about "growth" (as opposed to change) of software systems is a good one for this. Tldr: ok to add things. Not ok to remove things or change things
- majormajor 2y ago
  > As an event producer as long as you follow reasonable backwards-compatibility best practices then you should be pretty safe from breaking things downstream. That can protect you from "downstream can't even read the message anymore" but it doesn't help you with the much more common "downstream isn't doing the right thing with the message anymore" problem. Schema evolution is kinda like schema'd RPC calls vs plain JSON: it will protect you from "oops, we sent eventId instead of event_id" type of errors, but won't prevent you from making logical errors. In a larger org, this can turn into delayed-discovery nightmares. A synchronous API call could give you back an error response and alert your immediately to something being wrong. The system notifies you directly. A downstream event consumer may fail in ways entirely off of your team's radar. The downstream team starts getting alerts. Whether or not those alerts make it immediately obvious to them that it's your fault... that depends on a bunch of factors.
NicoJuicy 2y ago
And that's why a SAGA describes that flow. Don't take it into consideration and you're fucked. Source: previous "seniors" didn't take it into consideration, they left
Osiris 2y ago
What I like about event driven is that you don't even need to know if anyone is listening to or cares about your event. And as a consumer, many independent tasks can be triggered by the same event. I'm working on a system right now and because of events, it's very easy for me to write a handler for when a certain type of record is created in the database. My feature depends on knowing that new record was made so we can send some emails and do other things. The people that wrote the code that creates the record, didn't have to do anything to support the feature. But I agree that it's not the right solution for every problem. But there are certain problems it solves really well.
- zdragnar 2y ago
  > you don't even need to know if anyone is listening to or cares about your event. Right up until you need to change something about the event because the business logic it represents has changed. Then you suddenly need to track down all the systems that have been relying on it, including that one that nobody knows anything about and always forgets exists because some guy decided to implement the service in erlang and nobody who ever touched it even works at the company anymore.
  - friendzis 2y ago
    I really dislike this argument, because it puts the duty of managing dependencies and requirements directly on code. This is organizational issue! First, if your event (or whatever) changes enough that there are inter-component breakages it means engineering requirements must have changed and tracing dependencies of requirements is organizational thing. Second, you either do trunk based development and constantly break downstream or do leaf based development and have constantly out of date core dependencies. In any case, that's release version management, which is again organizational thing.
  - sharlos201068 2y ago
    How is that any different for an API-driven architecture? You'd need to track down all consumers of your API you're wanting to make a breaking change to.
junto 2y ago
You need to improve your telemetry to feel the benefits. I can trace all the way through multiple services easily on a simple detailed flame graph in our systems. https://www.datadoghq.com/knowledge-center/distributed-tracing/flame-graph/ https://www.datadoghq.com/knowledge-center/distributed-traci... Unless you have a single monolith, you’re going to face issues with versioning whether it’s event based or API based. In each case you can usually add new properties to a message, but you can’t remove properties or change their types. If you need that, create a new version. The author does a lot of videos on the event sourcing topic. Event driven I get. It works well in several applications I’ve helped to build over the last 15 years. But event sourcing? I truly don’t get it. Yeah I get it’s nice in terms of auditing to see every change to an entity and who made it, or replay up to to change x on y date, but that really is a niche requirement.
- scubbo 2y ago
  > I can trace all the way through multiple services easily on a simple detailed flame graph in our systems I'm not sure what point is being made here. It's good that you can do that - but are you implying that that's not possible in an API-driven system?
- okr 2y ago
  A niche requirement? There are big accounting firms who organize payrolls, where the changes that you mentioned are an important part of their business. There are also other companies, which do the typical snapshot and roll up to the current time, when they start the services, that need the data without having access to the database.
  - nithril 2y ago
    You don't need event sourcing to organize payrolls event "at scale".
- YZF 2y ago
  There are many examples of event sourcing but perhaps the most classical one is the bank account. It's not just about auditing, it's also about transactionality and atomicity. If you want to withdraw $5 from your account, the traditional approach of locking, updating everything, unlocking (or in other words wrapping everything in a transaction) doesn't scale as well as the notion that you just record the transaction (event). Implementation-wise this withdrawal can involve, updating two accounts and updating the audit/account transaction logs. We also want this to scale since our bank has millions of customers all operating more or less concurrently. A distributed log (like Kafka) is easy to scale and easy to reason about. You just insert the transaction record and you have a distributed system that will scale and is easy to reason about. Another driver/flavour for something like event sourcing is what some might call state-based or state-oriented programming. That is instead of modifying state directly you are synchronizing state via events. This lets you e.g. code state machines around those that can lead, again, to easier to reason about (and test) code.
mrkeen 2y ago
> now with events that are harder to trace than function calls would have been I don't know how this could be true. Events are things - nouns which can be backed-up, replicated, stored, queried, rendered, indexed and searched over.
- sattoshi 2y ago
  How is it not true? Instead of tracking data and function calls over a unified stacktrace, you track Things and Messages over databases, queues, and logs —- none of which you can trivially attach a debugger to. I generally like event-driven architecture, but I need to admit that debuggability is sacrificed where it matters most.
- lmm 2y ago
  There's no "find usages" for events, and it becomes harder to find out why something didn't happen. A function call can't simply "not return" - in the worst case you get an exception, or a stuck thread in the caller that will show up in a stack dump. But downstream event processing can very easily just not happen, for one of many different reasons, and out-of-the-box it's often difficult to investigate.
  - soulofmischief 2y ago
    In JavaScript, const myEvent = 'myEvent', target = new EventTarget() target.on( myEvent, () => { console.log( "It's easy to introspect well-organized code." ) }) target.dispatchEvent( new Event( myEvent ))
    - lmm 2y ago
      Yeah, good luck remembering to do that up-front for every event handler. You missed one? Whoops, enjoy the information you wanted silently not being there when you need it.
      - soulofmischief 2y ago
        Remembering to do what? Properly maintain a list of constants and enums to use throughout my application? That's not something I have to remember or forget, it's a simple habit that is as natural as importing and referencing a function. As a general rule, numbers and string literals should never be hardcoded. Internalizing this should be a base expectation of any high-performing team member.
pyrale 2y ago
> now with events that are harder to trace than function calls Same issue as microservices: there are people who want to use the paradigm but not do the investment in monitoring/tooling.

moandcompany 2y ago

Integration tests?

worik 2y ago
...happen too late

hobs 2y ago

I think generally a lot of these types of problems were actually had by folks who grew out of single node systems and had a lot of interesting ideas to solve problems that were new in those domains, GIVEN they've already solved the stateful domain problems as well. When you've never grown out of a single node domain but you do event driven "because scaling" or whatever, you've shot yourself in the foot amazingly hard.

The_Colonel 2y ago
Yes, events, async, eventual consistency, decoupling represent a difficult/complex solution for some hard problems encountered when scaling high. But people often forget there are trade-offs to everything and if you don't have these hard problems, you're giving yourself only headaches. My pet-peeve is "decoupling" - it's treated as holy with only benefits and no downsides. But it's actually again a level of complexity - unless you need it, tightly coupled code will be easier to write, read, debug etc.

scubbo 2y ago

Amen. Event-driven architecture makes it easier to bury your head in the sand, and harder to implement an actually-working feature.

trevor-e 2y ago

Like anything it can be abused and sometimes folks go overboard with turning everything into an event. However, when done right, it is really amazing to work with. As an event producer as long as you follow reasonable backwards-compatibility best practices then you should be pretty safe from breaking things downstream. As a consumer, follow defensive programming and allow for idempotency in case you need to reprocess an event. Pretty straightforward once you get the hang of things.

gnat 2y ago
Do you have pointers to such best practices? Gratefully received etc.
- trevor-e 2y ago
  Design for things to be easily failable. It should be trivial to have failed messages to go to a DLQ and then reprocess them later on, say after a bug fix. Only make additive changes, don't change existing fields. For enums it's up to the consumer to ensure they don't fail when a new case is added. Be very careful with including data (especially time/expiry stuff) in the message too. If you need to reprocess the event several hours later then it may no longer work or be stale. Rather than include the data in the message itself, we would include the database ID and then have the consumer query for that entry.
- mvc 2y ago
  Rich Hickey's talk about "growth" (as opposed to change) of software systems is a good one for this. Tldr: ok to add things. Not ok to remove things or change things
majormajor 2y ago
> As an event producer as long as you follow reasonable backwards-compatibility best practices then you should be pretty safe from breaking things downstream. That can protect you from "downstream can't even read the message anymore" but it doesn't help you with the much more common "downstream isn't doing the right thing with the message anymore" problem. Schema evolution is kinda like schema'd RPC calls vs plain JSON: it will protect you from "oops, we sent eventId instead of event_id" type of errors, but won't prevent you from making logical errors. In a larger org, this can turn into delayed-discovery nightmares. A synchronous API call could give you back an error response and alert your immediately to something being wrong. The system notifies you directly. A downstream event consumer may fail in ways entirely off of your team's radar. The downstream team starts getting alerts. Whether or not those alerts make it immediately obvious to them that it's your fault... that depends on a bunch of factors.

NicoJuicy 2y ago

And that's why a SAGA describes that flow. Don't take it into consideration and you're fucked. Source: previous "seniors" didn't take it into consideration, they left

Osiris 2y ago

What I like about event driven is that you don't even need to know if anyone is listening to or cares about your event. And as a consumer, many independent tasks can be triggered by the same event. I'm working on a system right now and because of events, it's very easy for me to write a handler for when a certain type of record is created in the database. My feature depends on knowing that new record was made so we can send some emails and do other things. The people that wrote the code that creates the record, didn't have to do anything to support the feature. But I agree that it's not the right solution for every problem. But there are certain problems it solves really well.

zdragnar 2y ago
> you don't even need to know if anyone is listening to or cares about your event. Right up until you need to change something about the event because the business logic it represents has changed. Then you suddenly need to track down all the systems that have been relying on it, including that one that nobody knows anything about and always forgets exists because some guy decided to implement the service in erlang and nobody who ever touched it even works at the company anymore.
- friendzis 2y ago
  I really dislike this argument, because it puts the duty of managing dependencies and requirements directly on code. This is organizational issue! First, if your event (or whatever) changes enough that there are inter-component breakages it means engineering requirements must have changed and tracing dependencies of requirements is organizational thing. Second, you either do trunk based development and constantly break downstream or do leaf based development and have constantly out of date core dependencies. In any case, that's release version management, which is again organizational thing.
- sharlos201068 2y ago
  How is that any different for an API-driven architecture? You'd need to track down all consumers of your API you're wanting to make a breaking change to.

junto 2y ago

You need to improve your telemetry to feel the benefits. I can trace all the way through multiple services easily on a simple detailed flame graph in our systems. https://www.datadoghq.com/knowledge-center/distributed-tracing/flame-graph/ https://www.datadoghq.com/knowledge-center/distributed-traci... Unless you have a single monolith, you’re going to face issues with versioning whether it’s event based or API based. In each case you can usually add new properties to a message, but you can’t remove properties or change their types. If you need that, create a new version. The author does a lot of videos on the event sourcing topic. Event driven I get. It works well in several applications I’ve helped to build over the last 15 years. But event sourcing? I truly don’t get it. Yeah I get it’s nice in terms of auditing to see every change to an entity and who made it, or replay up to to change x on y date, but that really is a niche requirement.

scubbo 2y ago
> I can trace all the way through multiple services easily on a simple detailed flame graph in our systems I'm not sure what point is being made here. It's good that you can do that - but are you implying that that's not possible in an API-driven system?
okr 2y ago
A niche requirement? There are big accounting firms who organize payrolls, where the changes that you mentioned are an important part of their business. There are also other companies, which do the typical snapshot and roll up to the current time, when they start the services, that need the data without having access to the database.
- nithril 2y ago
  You don't need event sourcing to organize payrolls event "at scale".
YZF 2y ago
There are many examples of event sourcing but perhaps the most classical one is the bank account. It's not just about auditing, it's also about transactionality and atomicity. If you want to withdraw $5 from your account, the traditional approach of locking, updating everything, unlocking (or in other words wrapping everything in a transaction) doesn't scale as well as the notion that you just record the transaction (event). Implementation-wise this withdrawal can involve, updating two accounts and updating the audit/account transaction logs. We also want this to scale since our bank has millions of customers all operating more or less concurrently. A distributed log (like Kafka) is easy to scale and easy to reason about. You just insert the transaction record and you have a distributed system that will scale and is easy to reason about. Another driver/flavour for something like event sourcing is what some might call state-based or state-oriented programming. That is instead of modifying state directly you are synchronizing state via events. This lets you e.g. code state machines around those that can lead, again, to easier to reason about (and test) code.

mrkeen 2y ago

> now with events that are harder to trace than function calls would have been I don't know how this could be true. Events are things - nouns which can be backed-up, replicated, stored, queried, rendered, indexed and searched over.

sattoshi 2y ago
How is it not true? Instead of tracking data and function calls over a unified stacktrace, you track Things and Messages over databases, queues, and logs —- none of which you can trivially attach a debugger to. I generally like event-driven architecture, but I need to admit that debuggability is sacrificed where it matters most.
lmm 2y ago
There's no "find usages" for events, and it becomes harder to find out why something didn't happen. A function call can't simply "not return" - in the worst case you get an exception, or a stuck thread in the caller that will show up in a stack dump. But downstream event processing can very easily just not happen, for one of many different reasons, and out-of-the-box it's often difficult to investigate.
- soulofmischief 2y ago
  In JavaScript, const myEvent = 'myEvent', target = new EventTarget() target.on( myEvent, () => { console.log( "It's easy to introspect well-organized code." ) }) target.dispatchEvent( new Event( myEvent ))
  - lmm 2y ago
    Yeah, good luck remembering to do that up-front for every event handler. You missed one? Whoops, enjoy the information you wanted silently not being there when you need it.
    - soulofmischief 2y ago
      Remembering to do what? Properly maintain a list of constants and enums to use throughout my application? That's not something I have to remember or forget, it's a simple habit that is as natural as importing and referencing a function. As a general rule, numbers and string literals should never be hardcoded. Internalizing this should be a base expectation of any high-performing team member.

pyrale 2y ago

> now with events that are harder to trace than function calls Same issue as microservices: there are people who want to use the paradigm but not do the investment in monitoring/tooling.