4 ms·

Designing Data Intensive Applications has treated me well, getting me through pretty much every system design interview I have had. I am going to move onto eit

Designing Data Intensive Applications has treated me well, getting me through pretty much every system design interview I have had. I am going to move onto either * O'Rielly Designing Distributed Systems or * O'Rielly Architecture Patterns with Python Could anyone recommend either one? I am mainly a Python dev (with no particular love for Python), but that doesn't necessarily make me want to use the Python book more. Having code examples I can grok immediately is nice, but I also enjoy the exercise of translating concepts to code myself.

aynyc 5y ago
Designing Distributed Systems This is more for devop folks, rather than creating your own distributed system.
playing_colours 5y ago
O’Reilly Building Event-Driven Microservices is a solid book to continue. I would not recommend Architecture Patterns with Python. Some patterns there are questionable and introduce more complexity. Also, they talk about using Kafka and Service Bus in the book, but never mention an important issue you may face in real implementation: consistency in updating both a message bus and a database that usually is addressed with Outbox pattern https://microservices.io/patterns/data/transactional-outbox.html https://microservices.io/patterns/data/transactional-outbox....
- skrtskrt 5y ago
  Thank you!
- skrtskrt 5y ago
  What do you think of avoiding implementing Outbox by just having things like Kafka Connect subscribe to the table changes in the database?
  - zeroc8 5y ago
    One thing would be that you then are just syncing database as opposed to sending domain events, aren't you?
    - skrtskrt 5y ago
      Good point, but you could have the sender/pusher process read from Kafka Connect and parse into a domain event before pushing. It sounds roundabout, but you're letting a battle-tested implementation handle the database-table-listening aspects with no special outbox implementation needed on the database write.
      - deleted 5y ago
        [deleted]
      - lmarcos 5y ago
        How do you handle domain events that are the composition of columns of different tables?
        playing_colours 5y ago
        You can use views.
      - zeroc8 5y ago
        In our case it's not the easy to reconstruct domain events from CDC - in many cases, multiple events are affecting the same tables.
  - playing_colours 5y ago
    It can be a problem if you care about schema compatibility for your events - and I would care about it in a multi-team microservices setup. Your database schema essentially becomes a contract, your service is not a "black box" anymore. You have locked yourself from the ability to change the schema freely. Also, the messages produced reflect database changes, not domain events. I would prefer to have an explicitly defined interface layer for consumption. At least, use database views as contracts with Kafka Connect.
nicohaenggi 5y ago
@skrtskrt: Sorry for hijacking this post, but is there a way to contact you directly (my email is in my profile)? Your approach to building Django apps closely resonates with me and I would love to discuss these concepts in more detail.

aynyc 5y ago

Designing Distributed Systems This is more for devop folks, rather than creating your own distributed system.

playing_colours 5y ago

O’Reilly Building Event-Driven Microservices is a solid book to continue. I would not recommend Architecture Patterns with Python. Some patterns there are questionable and introduce more complexity. Also, they talk about using Kafka and Service Bus in the book, but never mention an important issue you may face in real implementation: consistency in updating both a message bus and a database that usually is addressed with Outbox pattern https://microservices.io/patterns/data/transactional-outbox.html https://microservices.io/patterns/data/transactional-outbox....

skrtskrt 5y ago
Thank you!
skrtskrt 5y ago
What do you think of avoiding implementing Outbox by just having things like Kafka Connect subscribe to the table changes in the database?
- zeroc8 5y ago
  One thing would be that you then are just syncing database as opposed to sending domain events, aren't you?
  - skrtskrt 5y ago
    Good point, but you could have the sender/pusher process read from Kafka Connect and parse into a domain event before pushing. It sounds roundabout, but you're letting a battle-tested implementation handle the database-table-listening aspects with no special outbox implementation needed on the database write.
    - deleted 5y ago
      [deleted]
    - lmarcos 5y ago
      How do you handle domain events that are the composition of columns of different tables?
      - playing_colours 5y ago
        You can use views.
    - zeroc8 5y ago
      In our case it's not the easy to reconstruct domain events from CDC - in many cases, multiple events are affecting the same tables.
- playing_colours 5y ago
  It can be a problem if you care about schema compatibility for your events - and I would care about it in a multi-team microservices setup. Your database schema essentially becomes a contract, your service is not a "black box" anymore. You have locked yourself from the ability to change the schema freely. Also, the messages produced reflect database changes, not domain events. I would prefer to have an explicitly defined interface layer for consumption. At least, use database views as contracts with Kafka Connect.

nicohaenggi 5y ago

@skrtskrt: Sorry for hijacking this post, but is there a way to contact you directly (my email is in my profile)? Your approach to building Django apps closely resonates with me and I would love to discuss these concepts in more detail.