5 ms·

There's something about event driven architectures that captures our engineering minds and makes our imaginations run wild. At face value it seems like the Gran

There's something about event driven architectures that captures our engineering minds and makes our imaginations run wild. At face value it seems like the Grand Unifying Theory of software engineering, a Grand Unifying Architecture. Everything is perfectly decoupled and only reads and writes to an event bus. Perfect uniformity, everything just reads and writes events. Perfect open-closed semantics, I can completely add new functionality without touching any existing components just by having it read from the event bus. However, like the grand unifying theories of physics so far, it just doesn't match reality. I have yet to witness a system that fully embraces event driven architecture that isn't a complete nightmare to read, understand, debug and modify. Yet we seem unable to shake the idea that it is a panacea. It captures the minds of each new generation of engineers and gets implemented in the technology of the era. For me it was applying the observable pattern to everything in Java. Now it's setting up a bunch of microservices that only communicate through Kafka. Maybe I'm just not a true Scotsman but this pattern has anecdotally never worked well in anything I've had to work with. I would think very hard before applying it as a pattern in my code let alone as the driving force of my architecture. That's just my experience and 2 cents.

fallous 7y ago
I think the level of abstraction applied to "event" is key in determining whether the architecture is successful (or even appropriate). If you attempt to implement at a granular level (got click on button!) then it turns into a giant mess because you're essentially reinventing the logical flow and event loop of programming via a message bus, which is a pretty terrible idea. If instead you operate at the business logic level of abstraction it can result in a much more coherent and easily-extended system, especially since that level also sees the most change from executives or operations. At that level it's also easier to integrate with third-party services. It is certainly not a cure for all that ails you, despite the claims of some snake-oil salesmen to the contrary.
ivix 7y ago
In general I tend to agree. However really the issue is one of correctly defining the granularity of events that the system should be aware of. After all, even a completely synchronous webapp is event driven, it just so happens to only care about a single event, a request. The problems come about when code which can all be synchronous becomes unnecessarily complicated and spread across multiple microservices. Microservices are generally a solution to people scaling issues, not technical ones. If you have significantly more services than teams, you are probably doing something wrong.
m463 7y ago
The thing is -- computers ARE event driven at the low levels. Problem is most systems are designed to completely hide the event-driven nature of things. The "nightmare to read, understand, debug and modify" is happily abstracted away and we have a nice safe environment to work with. ... and it then becomes much harder to support parallelism, error handling and responsiveness. I think there may be a more nuanced solution.
- delusional 7y ago
  I'm curious here. In my mind the computer is basically imperitive. It processes a series of instructions. It then happens to process multiple of such streams at once, but it's still essentially serial. How is it event driven?
  - iainmerrick 7y ago
    I suspect they’re confusing “event driven” with “interrupt driven”.
    - m463 7y ago
      Yes, interrupts and events are distinct. Would it be clearer to say sequential vs asynchronous? Or to say that interrupts are one form of event, but not all events involve interrupts.
  - dalore 7y ago
    Most computers have had more than one core and even additional cpu's like a gpu. Also things on the bus usually have their own processors. So things are happening at the same time, it might tick at the same rate that the bus allows but they communicate by events like interrupts. As far as I understand it.
    - deleted 7y ago
      [deleted]
  - m463 7y ago
shados 7y ago
If you do it at a small scale, where a team of a handful engineers is responsible for following/maintaining/debugging countless distinct systems that are only linked through Kafka, its insane. if you have hundreds or thousands of engineers in more teams than you can count, and each team is responsible for a few event producers or consumers and have the time and resources to dedicate to be experts about their own systems and their interfaces/boundaries, it's an amazing way to scale your organization. It really does work.
- kingdomcome50 7y ago
  I understand what you are saying here. But... I am having trouble determining the implication. It seems like you are saying that an EDA is only useful when no one has to understand the system in its entirety? That can't possibly be useful can it? The implication is that an EDA is inappropriate if I have a system with 1000 services and 5 engineers, but amazing if I have 500 engineers? Isn't the purpose of a technical architecture to optimize for the former (assuming engineering is a cost-center)? The former (small-scale) version you describe is really just an optimization for the latter (large-scale) version you describe a la "We have simplified our systems so much that we no longer need as many engineers to maintain it". In a way, your comment serves to substantiate the point of its parent in that it asserts that choosing an EDA requires more engineers per service. I must be misunderstanding your point.
  - bcrosby95 7y ago
    > Isn't the purpose of a technical architecture to optimize for the former (assuming engineering is a cost-center)? The companies that developed these things are employing hundreds or thousands of engineers. So, yeah, they are optimizing for engineering costs when you have thousands of engineers. They are building systems that cannot be fully understood by a single person. If you don't have thousands of engineers, their solutions to their problem won't necessarily be your solution to your problem. Complex beasts like microservices and event driven architecture are about enabling huge engineering departments to make changes to huge systems without constantly stepping on eachothers toes. Ignoring headcount, they are actually very inefficient because systems designed like this necessarily has tons of duplicated effort across the organization. Those inefficiencies are made up by reducing the amount of communication required between thousands of people and hundreds of teams to change and maintain the system. But if your department isn't huge then the communication overhead is less than the duplicated effort, which tends to make these designs a poor choice. These sorts of things usually do not scale down. If you have 1,000 services and 5 engineers there's a 99.999% chance you're doing it wrong. 5 engineers is not even enough employees to manage, monitor, and build the requisite tooling to make efficient use of a system distributed across 1,000 services, much less build and understand the system as a whole.
chadcmulligan 7y ago
My experience is similar - the problem becomes when an event is triggered - where did it come from, oh it was from this event which was from this event and so on. Where events shine in my experience is in GUI's - the onClick event then does some code. So my rule of thumb is only one level of events and it comes from an external source - user, network etc.
snovv_crash 7y ago
Event driven architecture is like GOTOs but with data instead of execution flow. Throw your data to the wind, someone somewhere will handle it (I hope).
dkarl 7y ago
In my experience it's a bit the opposite; the domains I've worked in have always have events, whether given that name or not, and the systems for handling these events are usually coded much more optimistically and naively at first before people break down and start migrating parts of the system to run in event-driven style for performance and reliability reasons. People who are aware of event-driven architecture tend to make a smoother transition and create systems that make sense and are easier to work with. People who get there accidentally, forced every inch by successive performance and reliability bugs, end up with a hodge-podge that in retrospect is a poorly designed event-driven system. But that's just what I've seen in my experience. I've seen damage from people being ignorant of event-driven architecture or being in denial about how their systems are evolving; you've seen damage from people being overeager to use it. Probably in your shoes I would have seen the same things you have. One thing I've given up on seeing is the content of the linked article.
- m463 7y ago
  you can try reading https://www.fullhn.com/ https://www.fullhn.com/ while it's on the front page
- safety-third 7y ago
  In my experience, much more damage has been done by people who implement EDA and fancy cloud architecture in general. Something like 95% of applications are simple CRUD with a minuscule amount of custom domain logic. Doing anything beyond a sanely laid out monolith for these is resume padding and potentially project killing. I think people just don't want to admit that their job is validation, presentation, and shuffling data in and out of a database.
  - dryan76 7y ago
    I think this is the key: EDA should be used to communicate between different applications. If it is used within an application it points to over-engineering.
  - dkarl 7y ago
    Yeah, but very quickly in my experience the simple REST-based CRUD service starts waking up every 15 minutes to post a batch of rows from the database to a different REST-based CRUD service (or to itself if you have a monolith), and when that gets slow it gets scaled by increasing the wake-up frequency, and then someone starts writing batching logic and trying to scale it horizontally. Then external services start getting mixed in (and, if you're doing B2B, customer services) and you have to deal with their performance and reliability issues. Or maybe not; maybe you're Reddit and can get huge while still being essentially a CRUD app with a couple of UI front ends.
erik_landerholm 7y ago
I couldn't agree more. It's like the pot of gold at the end of a rainbow.
patsplat 7y ago
It works extremely well for ux development ;-)
Arelius 7y ago
Ohh my god this. My current project has this highly overengineered (with C++ and templates no less) event system that persists. Despite being one of the biggest complaints of our customers. We continue to think that the event system is "just one more" small tweak away from solving all of our problems.
kazinator 7y ago
> Everything is perfectly decoupled And every damned stack trace in the debugger goes a couple of frames up into a "receive_event" function, and then the trail goes cold.

fallous 7y ago

I think the level of abstraction applied to "event" is key in determining whether the architecture is successful (or even appropriate). If you attempt to implement at a granular level (got click on button!) then it turns into a giant mess because you're essentially reinventing the logical flow and event loop of programming via a message bus, which is a pretty terrible idea. If instead you operate at the business logic level of abstraction it can result in a much more coherent and easily-extended system, especially since that level also sees the most change from executives or operations. At that level it's also easier to integrate with third-party services. It is certainly not a cure for all that ails you, despite the claims of some snake-oil salesmen to the contrary.

ivix 7y ago

In general I tend to agree. However really the issue is one of correctly defining the granularity of events that the system should be aware of. After all, even a completely synchronous webapp is event driven, it just so happens to only care about a single event, a request. The problems come about when code which can all be synchronous becomes unnecessarily complicated and spread across multiple microservices. Microservices are generally a solution to people scaling issues, not technical ones. If you have significantly more services than teams, you are probably doing something wrong.

m463 7y ago

The thing is -- computers ARE event driven at the low levels. Problem is most systems are designed to completely hide the event-driven nature of things. The "nightmare to read, understand, debug and modify" is happily abstracted away and we have a nice safe environment to work with. ... and it then becomes much harder to support parallelism, error handling and responsiveness. I think there may be a more nuanced solution.

delusional 7y ago
I'm curious here. In my mind the computer is basically imperitive. It processes a series of instructions. It then happens to process multiple of such streams at once, but it's still essentially serial. How is it event driven?
- iainmerrick 7y ago
  I suspect they’re confusing “event driven” with “interrupt driven”.
  - m463 7y ago
    Yes, interrupts and events are distinct. Would it be clearer to say sequential vs asynchronous? Or to say that interrupts are one form of event, but not all events involve interrupts.
- dalore 7y ago
  Most computers have had more than one core and even additional cpu's like a gpu. Also things on the bus usually have their own processors. So things are happening at the same time, it might tick at the same rate that the bus allows but they communicate by events like interrupts. As far as I understand it.
  - deleted 7y ago
    [deleted]
- m463 7y ago

shados 7y ago

If you do it at a small scale, where a team of a handful engineers is responsible for following/maintaining/debugging countless distinct systems that are only linked through Kafka, its insane. if you have hundreds or thousands of engineers in more teams than you can count, and each team is responsible for a few event producers or consumers and have the time and resources to dedicate to be experts about their own systems and their interfaces/boundaries, it's an amazing way to scale your organization. It really does work.

kingdomcome50 7y ago
I understand what you are saying here. But... I am having trouble determining the implication. It seems like you are saying that an EDA is only useful when no one has to understand the system in its entirety? That can't possibly be useful can it? The implication is that an EDA is inappropriate if I have a system with 1000 services and 5 engineers, but amazing if I have 500 engineers? Isn't the purpose of a technical architecture to optimize for the former (assuming engineering is a cost-center)? The former (small-scale) version you describe is really just an optimization for the latter (large-scale) version you describe a la "We have simplified our systems so much that we no longer need as many engineers to maintain it". In a way, your comment serves to substantiate the point of its parent in that it asserts that choosing an EDA requires more engineers per service. I must be misunderstanding your point.
- bcrosby95 7y ago
  > Isn't the purpose of a technical architecture to optimize for the former (assuming engineering is a cost-center)? The companies that developed these things are employing hundreds or thousands of engineers. So, yeah, they are optimizing for engineering costs when you have thousands of engineers. They are building systems that cannot be fully understood by a single person. If you don't have thousands of engineers, their solutions to their problem won't necessarily be your solution to your problem. Complex beasts like microservices and event driven architecture are about enabling huge engineering departments to make changes to huge systems without constantly stepping on eachothers toes. Ignoring headcount, they are actually very inefficient because systems designed like this necessarily has tons of duplicated effort across the organization. Those inefficiencies are made up by reducing the amount of communication required between thousands of people and hundreds of teams to change and maintain the system. But if your department isn't huge then the communication overhead is less than the duplicated effort, which tends to make these designs a poor choice. These sorts of things usually do not scale down. If you have 1,000 services and 5 engineers there's a 99.999% chance you're doing it wrong. 5 engineers is not even enough employees to manage, monitor, and build the requisite tooling to make efficient use of a system distributed across 1,000 services, much less build and understand the system as a whole.

chadcmulligan 7y ago

My experience is similar - the problem becomes when an event is triggered - where did it come from, oh it was from this event which was from this event and so on. Where events shine in my experience is in GUI's - the onClick event then does some code. So my rule of thumb is only one level of events and it comes from an external source - user, network etc.

snovv_crash 7y ago

Event driven architecture is like GOTOs but with data instead of execution flow. Throw your data to the wind, someone somewhere will handle it (I hope).

dkarl 7y ago

In my experience it's a bit the opposite; the domains I've worked in have always have events, whether given that name or not, and the systems for handling these events are usually coded much more optimistically and naively at first before people break down and start migrating parts of the system to run in event-driven style for performance and reliability reasons. People who are aware of event-driven architecture tend to make a smoother transition and create systems that make sense and are easier to work with. People who get there accidentally, forced every inch by successive performance and reliability bugs, end up with a hodge-podge that in retrospect is a poorly designed event-driven system. But that's just what I've seen in my experience. I've seen damage from people being ignorant of event-driven architecture or being in denial about how their systems are evolving; you've seen damage from people being overeager to use it. Probably in your shoes I would have seen the same things you have. One thing I've given up on seeing is the content of the linked article.

m463 7y ago
you can try reading https://www.fullhn.com/ https://www.fullhn.com/ while it's on the front page
safety-third 7y ago
In my experience, much more damage has been done by people who implement EDA and fancy cloud architecture in general. Something like 95% of applications are simple CRUD with a minuscule amount of custom domain logic. Doing anything beyond a sanely laid out monolith for these is resume padding and potentially project killing. I think people just don't want to admit that their job is validation, presentation, and shuffling data in and out of a database.
- dryan76 7y ago
  I think this is the key: EDA should be used to communicate between different applications. If it is used within an application it points to over-engineering.
- dkarl 7y ago
  Yeah, but very quickly in my experience the simple REST-based CRUD service starts waking up every 15 minutes to post a batch of rows from the database to a different REST-based CRUD service (or to itself if you have a monolith), and when that gets slow it gets scaled by increasing the wake-up frequency, and then someone starts writing batching logic and trying to scale it horizontally. Then external services start getting mixed in (and, if you're doing B2B, customer services) and you have to deal with their performance and reliability issues. Or maybe not; maybe you're Reddit and can get huge while still being essentially a CRUD app with a couple of UI front ends.

erik_landerholm 7y ago

I couldn't agree more. It's like the pot of gold at the end of a rainbow.

patsplat 7y ago

It works extremely well for ux development ;-)

Arelius 7y ago

Ohh my god this. My current project has this highly overengineered (with C++ and templates no less) event system that persists. Despite being one of the biggest complaints of our customers. We continue to think that the event system is "just one more" small tweak away from solving all of our problems.

kazinator 7y ago

> Everything is perfectly decoupled And every damned stack trace in the debugger goes a couple of frames up into a "receive_event" function, and then the trail goes cold.