32 ms·

Event-Driven Architecture

160 points by pradeepl 7y ago

iamspoilt 7y ago
It seems like the post on event driven architecture couldn't handle the click event load from Hacker News and is throwing a "Database Error".
- dantodor 7y ago
  Yup, the pipelines are down and CQRS is suffering :D
  - ralphael 7y ago
    shame as I like the content and this could distract from it
  - NicoJuicy 7y ago
    Cached db query ? :P Totally fine in Infrastructure
  - ak39 7y ago
    That’s a saga for another time.
  - erik_seaberg 7y ago
    HN traffic is coming from the wrong side of the RPC/streaming boundary. Everyone could get by with a lot less hardware if HTTP GETs were queued instead of bursts of realtime load.
- ralphael 7y ago
  shame as I like the content and this could distract from it
- Nuzzerino 7y ago
  This is tangential but I'm reminded of the ontology people who constantly bash RDF but have trouble providing good arguments or links to documentation as to why it's a bad thing. Ontological tech helps with organization of conceptual data so it would be an easy ask for them. Moral of the story is there's a lot of information being preached by those who either haven't or won't put that knowledge into practice first. It would have been nice to hear about any projects the author could point to and explain how the architecture helped it succeed, but that is absent from this post.
  - aledalgrande 7y ago
    Totally agree, a lot of people preaching just after reading books and blog posts and running a minimal hobby project. But I am not surprised. Nothing is inherently bad, unless you use it for the wrong purpose.
  - socceroos 7y ago
    The banking system
- claytongulick 7y ago
  Given what I've seen in practice with companies that try to follow this architecture, I can't say I'm very surprised. This type of architecture is really only suitable for a narrow problem domain, imho. Most people that I've seen attempt it don't understand all of the edge cases that arise from it. For example, not getting immediate success or failure status from an API call, and having to constantly poll to see what the status of a submitted message is.
  - nickserv 7y ago
    I think an event based system can apply to a wide variety of domains, but that the consequences need to be understood up front. This means doing things differently than the traditional top down controller approach with synchronous responses. For example, an asynchronous API should have a way of pushing notifications to the caller, and the caller then has a way of processing these returns.
- _sbrk 7y ago
  Heh heh heh .. yep: "Error establishing a database connection"
- nawgszy 7y ago
  While I definitely respect seizing the snark opportunity.. I feel like I see loads of websites crash from HN / Reddit hug of death regardless of architecture, so I don't think I would necessarily consider this a failure of the architecture
  - tjoff 7y ago
    Maybe says more about current architectures. It is hard to imagine any reason as to why a normal site/blog couldn't cope with this. Now since whatever solution you pick is going to be an overengineered mess it's easy to understand each in isolation, most of us don't have the time/energy to optimize for this. And that is fine! But it is kinda sad that the quick and easy approach is not a statically generated site. There is absolutely no reason why that should be any harder. Yet for some reason we don't value that.
    - hombre_fatal 7y ago
      We don't value it when 99% of blogs never reach a level of traffic, beyond rare spikes, where they need to introduce a cache (like pre-generated files) in front of their dynamically-generated website. The blog operator likely doesn't even notice it, and when they do + are savvy enough, they just install one of the various Wordpress caching/static plugins. And we do value it when downtime loses us business, something that's not likely to apply to our personal blogs. Maybe Wordpress should enable some low-risk caching by default, but maybe it's not worth it since most installs never get traffic and caching is confusing especially to the non-tech-savvy.
      - tjoff 7y ago
        My point is that we probably should value simplicity itself. And as a bonus things like this wouldn't happen.
- organsnyder 7y ago
  I realize this is mostly snark, but is there any evidence that the website is hosted on a stack that is event-driven? Most likely it's just hosted on WordPress or something.
ch_sm 7y ago
Here is a cached version of the article: https://webcache.googleusercontent.com/search?q=cache:https://pradeeploganathan.com/architecture/event-driven-architecture/ https://webcache.googleusercontent.com/search?q=cache:https:...
ak39 7y ago
This is just an overly elaborate concept of a centralized (or commonly accessible table) of states that other independent systems can use for their own “triggers”. What seems like a lifetime ago, we achieved this with IBM MQSeries allowing us to send events from a Microsoft transaction server (MTS) object to a COBOL program that listened for the events to insert records into an order entry system that was on an AS/400 DB2 db. Maybe an Event Bus concept carries all the expected “good design aspects” of this pattern. But it is just another example of an integration concept - but please can we stop with evangelizing this as though it ought to be the linchpin or necessary aspect of a good modern system you are starting with! Edit: this design concept 25 years ago was the best we had considering the zero alternatives for guaranteeing that events don’t get lost etc. IOW: this design itself was a compromise for absence of a cohesive and unified database (what the Event Bus crew and the Microservices crowd would now call a monolith)!
- idiocratic 7y ago
  I think this is just a good overview of the architecture and its characteristics. I'm not sure anyone is trying to evangelize anything.
jeremycw 7y ago
There's something about event driven architectures that captures our engineering minds and makes our imaginations run wild. At face value it seems like the Grand Unifying Theory of software engineering, a Grand Unifying Architecture. Everything is perfectly decoupled and only reads and writes to an event bus. Perfect uniformity, everything just reads and writes events. Perfect open-closed semantics, I can completely add new functionality without touching any existing components just by having it read from the event bus. However, like the grand unifying theories of physics so far, it just doesn't match reality. I have yet to witness a system that fully embraces event driven architecture that isn't a complete nightmare to read, understand, debug and modify. Yet we seem unable to shake the idea that it is a panacea. It captures the minds of each new generation of engineers and gets implemented in the technology of the era. For me it was applying the observable pattern to everything in Java. Now it's setting up a bunch of microservices that only communicate through Kafka. Maybe I'm just not a true Scotsman but this pattern has anecdotally never worked well in anything I've had to work with. I would think very hard before applying it as a pattern in my code let alone as the driving force of my architecture. That's just my experience and 2 cents.
- fallous 7y ago
  I think the level of abstraction applied to "event" is key in determining whether the architecture is successful (or even appropriate). If you attempt to implement at a granular level (got click on button!) then it turns into a giant mess because you're essentially reinventing the logical flow and event loop of programming via a message bus, which is a pretty terrible idea. If instead you operate at the business logic level of abstraction it can result in a much more coherent and easily-extended system, especially since that level also sees the most change from executives or operations. At that level it's also easier to integrate with third-party services. It is certainly not a cure for all that ails you, despite the claims of some snake-oil salesmen to the contrary.
- ivix 7y ago
  In general I tend to agree. However really the issue is one of correctly defining the granularity of events that the system should be aware of. After all, even a completely synchronous webapp is event driven, it just so happens to only care about a single event, a request. The problems come about when code which can all be synchronous becomes unnecessarily complicated and spread across multiple microservices. Microservices are generally a solution to people scaling issues, not technical ones. If you have significantly more services than teams, you are probably doing something wrong.
stephenwilcock 7y ago
When I follow the link to the article I get an "Error establishing a database connection" message. Delicious irony.
- pradeepl 7y ago
  yes, this is ironical. I was away and thanks for the HN hug of death my blog went down :-). My blog provider throttled it down severely and I was able to restore it from backups just now. I blog occasionally to put down notes on what ever I am working on or thinking through, in the hope that maybe it would help some one or myself in the future. Apologies if I wasted your time.
teddyh 7y ago
Ubuntu’s “Upstart” init system used an event-based design, which was criticized by the creator of systemd, thus: — [Upstart]'s main feature is its event-based approach: starting and stopping of processes is bound to "events" happening in the system, where an "event" can be a lot of different things, such as: a network interfaces becomes available or some other software has been started. Upstart does service serialization via these events: if the syslog-started event is triggered this is used as an indication to start D-Bus since it can now make use of Syslog. And then, when dbus-started is triggered, NetworkManager is started, since it may now use D-Bus, and so on. One could say that this way the actual logical dependency tree that exists and is understood by the admin or developer is translated and encoded into event and action rules: every logical "a needs b" rule that the administrator/developer is aware of becomes a "start a when b is started" plus "stop a when b is stopped". In some way this certainly is a simplification: especially for the code in Upstart itself. However I would argue that this simplification is actually detrimental. First of all, the logical dependency system does not go away, the person who is writing Upstart files must now translate the dependencies manually into these event/action rules (actually, two rules for each dependency). So, instead of letting the computer figure out what to do based on the dependencies, the user has to manually translate the dependencies into simple event/action rules. Also, because the dependency information has never been encoded it is not available at runtime, effectively meaning that an administrator who tries to figure our why something happened, i.e. why a is started when b is started, has no chance of finding that out. Furthermore, the event logic turns around all dependencies, from the feet onto their head. Instead of minimizing the amount of work (which is something that a good init system should focus on, as pointed out in the beginning of this blog story), it actually maximizes the amount of work to do during operations. Or in other words, instead of having a clear goal and only doing the things it really needs to do to reach the goal, it does one step, and then after finishing it, it does all steps that possibly could follow it. Or to put it simpler: the fact that the user just started D-Bus is in no way an indication that NetworkManager should be started too (but this is what Upstart would do). It's right the other way round: when the user asks for NetworkManager, that is definitely an indication that D-Bus should be started too (which is certainly what most users would expect, right?). A good init system should start only what is needed, and that on-demand. Either lazily or parallelized and in advance. However it should not start more than necessary, particularly not everything installed that could use that service. Finally, I fail to see the actual usefulness of the event logic. It appears to me that most events that are exposed in Upstart actually are not punctual in nature, but have duration: a service starts, is running, and stops. A device is plugged in, is available, and is plugged out again. A mount point is in the process of being mounted, is fully mounted, or is being unmounted. A power plug is plugged in, the system runs on AC, and the power plug is pulled. Only a minority of the events an init system or process supervisor should handle are actually punctual, most of them are tuples of start, condition, and stop. This information is again not available in Upstart, because it focuses in singular events, and ignores durable dependencies. […] — http://0pointer.net/blog/projects/systemd http://0pointer.net/blog/projects/systemd
AndrewKemendo 7y ago
I'm consistently surprised at the negative comments on EDA on Hacker News because there are so many examples of major organizations successfully implementing and running EDA at scale. Here are a few examples: Uber: - https://eng.uber.com/ureplicator/ https://eng.uber.com/ureplicator/ - https://eng.uber.com/reliable-reprocessing/ https://eng.uber.com/reliable-reprocessing/ Google: - https://cloud.google.com/blog/products/gcp/implementing-an-event-driven-architecture-on-serverless-the-smart-parking-story https://cloud.google.com/blog/products/gcp/implementing-an-e... Twilio: - https://signal.twilio.com/2017/sf/sessions/18530/building-robust-and-scalable-data-pipelines-with-kafka https://signal.twilio.com/2017/sf/sessions/18530/building-ro... Stripe: - https://stripe.com/blog/canonical-log-lines https://stripe.com/blog/canonical-log-lines My hypothesis about why this is, is that most organizations probably don't need EDA yet. They don't have that many data producers and consumers and don't have HA and other requirements that drive the need, so implementing it is overkill, and so their experiences have been bad.
- mst 7y ago
  "Trying to implement google-scale things when you're tiny" seems to be an extremely common and understandably tempting antipattern.
  - nostrebored 7y ago
    But "trying to decouple, modularize, and choose the right tools for the job" for your applications seems applicable at any scale. EDA requires a change in testing methodology, in software design, and a bit of reading, but calling it a "google-scale thing" is pretty fallacious. The idea that a monolith is easier to maintain or that synchronous inter-component communication is easier to reason about also seems fallacious. I'd love to see a chart of developer's perceptions of event driven microservices and their personal operational expectations.
    - AndrewKemendo 7y ago
      Also worth noting that it is a new and different paradigm than most people are used to.
ykr1 7y ago
Says database error. Was that an event?
- justincredible 7y ago
  It was driven by the event: Submission
aabbcc1241 7y ago
For people cannot view the article, here's an archive: https://web.archive.org/web/20200210123446/https://pradeeploganathan.com/architecture/event-driven-architecture/ https://web.archive.org/web/20200210123446/https://pradeeplo...
rubiquity 7y ago
Web servers are event-driven architecture. They listen to events on the HTTP bus.
kazinator 7y ago
Event-driven architecture can be nasty. Ideally, you want the point of origin of an action, and the point of its execution, to be connected by a chain of function calls which can be traced in a debugger. Events are not easily traceable. An event is pulled from some event queue, and by the time that happens, the thing that put the event into the queue has long since buzzed off to do something else. That's just one problem. The other problem is how events can be subject to a routing, translating, duplicating and splitting labyrinth. Events can bi{tri,quad,...}furcate. Just because your code is processing the event over here doesn't mean you were the first to do so, or the last. I have a good idea! Let's process this event and re-inject it. Two years later, someone is debugging an event routing loop.

160 points by pradeepl 7y ago

It seems like the post on event driven architecture couldn't handle the click event load from Hacker News and is throwing a "Database Error".

dantodor 7y ago
Yup, the pipelines are down and CQRS is suffering :D
- ralphael 7y ago
  shame as I like the content and this could distract from it
- NicoJuicy 7y ago
  Cached db query ? :P Totally fine in Infrastructure
- ak39 7y ago
  That’s a saga for another time.
- erik_seaberg 7y ago
  HN traffic is coming from the wrong side of the RPC/streaming boundary. Everyone could get by with a lot less hardware if HTTP GETs were queued instead of bursts of realtime load.
ralphael 7y ago
shame as I like the content and this could distract from it
Nuzzerino 7y ago
This is tangential but I'm reminded of the ontology people who constantly bash RDF but have trouble providing good arguments or links to documentation as to why it's a bad thing. Ontological tech helps with organization of conceptual data so it would be an easy ask for them. Moral of the story is there's a lot of information being preached by those who either haven't or won't put that knowledge into practice first. It would have been nice to hear about any projects the author could point to and explain how the architecture helped it succeed, but that is absent from this post.
- aledalgrande 7y ago
  Totally agree, a lot of people preaching just after reading books and blog posts and running a minimal hobby project. But I am not surprised. Nothing is inherently bad, unless you use it for the wrong purpose.
- socceroos 7y ago
  The banking system
claytongulick 7y ago
Given what I've seen in practice with companies that try to follow this architecture, I can't say I'm very surprised. This type of architecture is really only suitable for a narrow problem domain, imho. Most people that I've seen attempt it don't understand all of the edge cases that arise from it. For example, not getting immediate success or failure status from an API call, and having to constantly poll to see what the status of a submitted message is.
- nickserv 7y ago
  I think an event based system can apply to a wide variety of domains, but that the consequences need to be understood up front. This means doing things differently than the traditional top down controller approach with synchronous responses. For example, an asynchronous API should have a way of pushing notifications to the caller, and the caller then has a way of processing these returns.
_sbrk 7y ago
Heh heh heh .. yep: "Error establishing a database connection"
nawgszy 7y ago
While I definitely respect seizing the snark opportunity.. I feel like I see loads of websites crash from HN / Reddit hug of death regardless of architecture, so I don't think I would necessarily consider this a failure of the architecture
- tjoff 7y ago
  Maybe says more about current architectures. It is hard to imagine any reason as to why a normal site/blog couldn't cope with this. Now since whatever solution you pick is going to be an overengineered mess it's easy to understand each in isolation, most of us don't have the time/energy to optimize for this. And that is fine! But it is kinda sad that the quick and easy approach is not a statically generated site. There is absolutely no reason why that should be any harder. Yet for some reason we don't value that.
  - hombre_fatal 7y ago
    We don't value it when 99% of blogs never reach a level of traffic, beyond rare spikes, where they need to introduce a cache (like pre-generated files) in front of their dynamically-generated website. The blog operator likely doesn't even notice it, and when they do + are savvy enough, they just install one of the various Wordpress caching/static plugins. And we do value it when downtime loses us business, something that's not likely to apply to our personal blogs. Maybe Wordpress should enable some low-risk caching by default, but maybe it's not worth it since most installs never get traffic and caching is confusing especially to the non-tech-savvy.
    - tjoff 7y ago
      My point is that we probably should value simplicity itself. And as a bonus things like this wouldn't happen.
organsnyder 7y ago
I realize this is mostly snark, but is there any evidence that the website is hosted on a stack that is event-driven? Most likely it's just hosted on WordPress or something.

ch_sm 7y ago

Here is a cached version of the article: https://webcache.googleusercontent.com/search?q=cache:https://pradeeploganathan.com/architecture/event-driven-architecture/ https://webcache.googleusercontent.com/search?q=cache:https:...

ak39 7y ago

This is just an overly elaborate concept of a centralized (or commonly accessible table) of states that other independent systems can use for their own “triggers”. What seems like a lifetime ago, we achieved this with IBM MQSeries allowing us to send events from a Microsoft transaction server (MTS) object to a COBOL program that listened for the events to insert records into an order entry system that was on an AS/400 DB2 db. Maybe an Event Bus concept carries all the expected “good design aspects” of this pattern. But it is just another example of an integration concept - but please can we stop with evangelizing this as though it ought to be the linchpin or necessary aspect of a good modern system you are starting with! Edit: this design concept 25 years ago was the best we had considering the zero alternatives for guaranteeing that events don’t get lost etc. IOW: this design itself was a compromise for absence of a cohesive and unified database (what the Event Bus crew and the Microservices crowd would now call a monolith)!

idiocratic 7y ago
I think this is just a good overview of the architecture and its characteristics. I'm not sure anyone is trying to evangelize anything.

jeremycw 7y ago

There's something about event driven architectures that captures our engineering minds and makes our imaginations run wild. At face value it seems like the Grand Unifying Theory of software engineering, a Grand Unifying Architecture. Everything is perfectly decoupled and only reads and writes to an event bus. Perfect uniformity, everything just reads and writes events. Perfect open-closed semantics, I can completely add new functionality without touching any existing components just by having it read from the event bus. However, like the grand unifying theories of physics so far, it just doesn't match reality. I have yet to witness a system that fully embraces event driven architecture that isn't a complete nightmare to read, understand, debug and modify. Yet we seem unable to shake the idea that it is a panacea. It captures the minds of each new generation of engineers and gets implemented in the technology of the era. For me it was applying the observable pattern to everything in Java. Now it's setting up a bunch of microservices that only communicate through Kafka. Maybe I'm just not a true Scotsman but this pattern has anecdotally never worked well in anything I've had to work with. I would think very hard before applying it as a pattern in my code let alone as the driving force of my architecture. That's just my experience and 2 cents.

fallous 7y ago
I think the level of abstraction applied to "event" is key in determining whether the architecture is successful (or even appropriate). If you attempt to implement at a granular level (got click on button!) then it turns into a giant mess because you're essentially reinventing the logical flow and event loop of programming via a message bus, which is a pretty terrible idea. If instead you operate at the business logic level of abstraction it can result in a much more coherent and easily-extended system, especially since that level also sees the most change from executives or operations. At that level it's also easier to integrate with third-party services. It is certainly not a cure for all that ails you, despite the claims of some snake-oil salesmen to the contrary.
ivix 7y ago
In general I tend to agree. However really the issue is one of correctly defining the granularity of events that the system should be aware of. After all, even a completely synchronous webapp is event driven, it just so happens to only care about a single event, a request. The problems come about when code which can all be synchronous becomes unnecessarily complicated and spread across multiple microservices. Microservices are generally a solution to people scaling issues, not technical ones. If you have significantly more services than teams, you are probably doing something wrong.

stephenwilcock 7y ago

When I follow the link to the article I get an "Error establishing a database connection" message. Delicious irony.

pradeepl 7y ago
yes, this is ironical. I was away and thanks for the HN hug of death my blog went down :-). My blog provider throttled it down severely and I was able to restore it from backups just now. I blog occasionally to put down notes on what ever I am working on or thinking through, in the hope that maybe it would help some one or myself in the future. Apologies if I wasted your time.

teddyh 7y ago

Ubuntu’s “Upstart” init system used an event-based design, which was criticized by the creator of systemd, thus: — [Upstart]'s main feature is its event-based approach: starting and stopping of processes is bound to "events" happening in the system, where an "event" can be a lot of different things, such as: a network interfaces becomes available or some other software has been started. Upstart does service serialization via these events: if the syslog-started event is triggered this is used as an indication to start D-Bus since it can now make use of Syslog. And then, when dbus-started is triggered, NetworkManager is started, since it may now use D-Bus, and so on. One could say that this way the actual logical dependency tree that exists and is understood by the admin or developer is translated and encoded into event and action rules: every logical "a needs b" rule that the administrator/developer is aware of becomes a "start a when b is started" plus "stop a when b is stopped". In some way this certainly is a simplification: especially for the code in Upstart itself. However I would argue that this simplification is actually detrimental. First of all, the logical dependency system does not go away, the person who is writing Upstart files must now translate the dependencies manually into these event/action rules (actually, two rules for each dependency). So, instead of letting the computer figure out what to do based on the dependencies, the user has to manually translate the dependencies into simple event/action rules. Also, because the dependency information has never been encoded it is not available at runtime, effectively meaning that an administrator who tries to figure our why something happened, i.e. why a is started when b is started, has no chance of finding that out. Furthermore, the event logic turns around all dependencies, from the feet onto their head. Instead of minimizing the amount of work (which is something that a good init system should focus on, as pointed out in the beginning of this blog story), it actually maximizes the amount of work to do during operations. Or in other words, instead of having a clear goal and only doing the things it really needs to do to reach the goal, it does one step, and then after finishing it, it does all steps that possibly could follow it. Or to put it simpler: the fact that the user just started D-Bus is in no way an indication that NetworkManager should be started too (but this is what Upstart would do). It's right the other way round: when the user asks for NetworkManager, that is definitely an indication that D-Bus should be started too (which is certainly what most users would expect, right?). A good init system should start only what is needed, and that on-demand. Either lazily or parallelized and in advance. However it should not start more than necessary, particularly not everything installed that could use that service. Finally, I fail to see the actual usefulness of the event logic. It appears to me that most events that are exposed in Upstart actually are not punctual in nature, but have duration: a service starts, is running, and stops. A device is plugged in, is available, and is plugged out again. A mount point is in the process of being mounted, is fully mounted, or is being unmounted. A power plug is plugged in, the system runs on AC, and the power plug is pulled. Only a minority of the events an init system or process supervisor should handle are actually punctual, most of them are tuples of start, condition, and stop. This information is again not available in Upstart, because it focuses in singular events, and ignores durable dependencies. […] — http://0pointer.net/blog/projects/systemd http://0pointer.net/blog/projects/systemd

AndrewKemendo 7y ago

I'm consistently surprised at the negative comments on EDA on Hacker News because there are so many examples of major organizations successfully implementing and running EDA at scale. Here are a few examples: Uber: - https://eng.uber.com/ureplicator/ https://eng.uber.com/ureplicator/ - https://eng.uber.com/reliable-reprocessing/ https://eng.uber.com/reliable-reprocessing/ Google: - https://cloud.google.com/blog/products/gcp/implementing-an-event-driven-architecture-on-serverless-the-smart-parking-story https://cloud.google.com/blog/products/gcp/implementing-an-e... Twilio: - https://signal.twilio.com/2017/sf/sessions/18530/building-robust-and-scalable-data-pipelines-with-kafka https://signal.twilio.com/2017/sf/sessions/18530/building-ro... Stripe: - https://stripe.com/blog/canonical-log-lines https://stripe.com/blog/canonical-log-lines My hypothesis about why this is, is that most organizations probably don't need EDA yet. They don't have that many data producers and consumers and don't have HA and other requirements that drive the need, so implementing it is overkill, and so their experiences have been bad.

mst 7y ago
"Trying to implement google-scale things when you're tiny" seems to be an extremely common and understandably tempting antipattern.
- nostrebored 7y ago
  But "trying to decouple, modularize, and choose the right tools for the job" for your applications seems applicable at any scale. EDA requires a change in testing methodology, in software design, and a bit of reading, but calling it a "google-scale thing" is pretty fallacious. The idea that a monolith is easier to maintain or that synchronous inter-component communication is easier to reason about also seems fallacious. I'd love to see a chart of developer's perceptions of event driven microservices and their personal operational expectations.
  - AndrewKemendo 7y ago
    Also worth noting that it is a new and different paradigm than most people are used to.

ykr1 7y ago

Says database error. Was that an event?

justincredible 7y ago
It was driven by the event: Submission

aabbcc1241 7y ago

For people cannot view the article, here's an archive: https://web.archive.org/web/20200210123446/https://pradeeploganathan.com/architecture/event-driven-architecture/ https://web.archive.org/web/20200210123446/https://pradeeplo...

rubiquity 7y ago

Web servers are event-driven architecture. They listen to events on the HTTP bus.

kazinator 7y ago

Event-driven architecture can be nasty. Ideally, you want the point of origin of an action, and the point of its execution, to be connected by a chain of function calls which can be traced in a debugger. Events are not easily traceable. An event is pulled from some event queue, and by the time that happens, the thing that put the event into the queue has long since buzzed off to do something else. That's just one problem. The other problem is how events can be subject to a routing, translating, duplicating and splitting labyrinth. Events can bi{tri,quad,...}furcate. Just because your code is processing the event over here doesn't mean you were the first to do so, or the last. I have a good idea! Let's process this event and re-inject it. Two years later, someone is debugging an event routing loop.