4 ms·

O’Reilly Building Event-Driven Microservices is a solid book to continue. I would not recommend Architecture Patterns with Python. Some patterns there are ques

O’Reilly Building Event-Driven Microservices is a solid book to continue. I would not recommend Architecture Patterns with Python. Some patterns there are questionable and introduce more complexity. Also, they talk about using Kafka and Service Bus in the book, but never mention an important issue you may face in real implementation: consistency in updating both a message bus and a database that usually is addressed with Outbox pattern https://microservices.io/patterns/data/transactional-outbox.html https://microservices.io/patterns/data/transactional-outbox....

skrtskrt 5y ago
Thank you!
skrtskrt 5y ago
What do you think of avoiding implementing Outbox by just having things like Kafka Connect subscribe to the table changes in the database?
- zeroc8 5y ago
  One thing would be that you then are just syncing database as opposed to sending domain events, aren't you?
  - skrtskrt 5y ago
    Good point, but you could have the sender/pusher process read from Kafka Connect and parse into a domain event before pushing. It sounds roundabout, but you're letting a battle-tested implementation handle the database-table-listening aspects with no special outbox implementation needed on the database write.
    - deleted 5y ago
      [deleted]
    - lmarcos 5y ago
      How do you handle domain events that are the composition of columns of different tables?
      - playing_colours 5y ago
        You can use views.
    - zeroc8 5y ago
      In our case it's not the easy to reconstruct domain events from CDC - in many cases, multiple events are affecting the same tables.
- playing_colours 5y ago
  It can be a problem if you care about schema compatibility for your events - and I would care about it in a multi-team microservices setup. Your database schema essentially becomes a contract, your service is not a "black box" anymore. You have locked yourself from the ability to change the schema freely. Also, the messages produced reflect database changes, not domain events. I would prefer to have an explicitly defined interface layer for consumption. At least, use database views as contracts with Kafka Connect.

skrtskrt 5y ago

Thank you!

skrtskrt 5y ago

What do you think of avoiding implementing Outbox by just having things like Kafka Connect subscribe to the table changes in the database?

zeroc8 5y ago
One thing would be that you then are just syncing database as opposed to sending domain events, aren't you?
- skrtskrt 5y ago
  Good point, but you could have the sender/pusher process read from Kafka Connect and parse into a domain event before pushing. It sounds roundabout, but you're letting a battle-tested implementation handle the database-table-listening aspects with no special outbox implementation needed on the database write.
  - deleted 5y ago
    [deleted]
  - lmarcos 5y ago
    How do you handle domain events that are the composition of columns of different tables?
    - playing_colours 5y ago
      You can use views.
  - zeroc8 5y ago
    In our case it's not the easy to reconstruct domain events from CDC - in many cases, multiple events are affecting the same tables.
playing_colours 5y ago
It can be a problem if you care about schema compatibility for your events - and I would care about it in a multi-team microservices setup. Your database schema essentially becomes a contract, your service is not a "black box" anymore. You have locked yourself from the ability to change the schema freely. Also, the messages produced reflect database changes, not domain events. I would prefer to have an explicitly defined interface layer for consumption. At least, use database views as contracts with Kafka Connect.