3 ms·

I agree that you can use Kafka as a raw event log - not necessarily a transaction log unless you are basically putting records into Kafka that don't need to be

I agree that you can use Kafka as a raw event log - not necessarily a transaction log unless you are basically putting records into Kafka that don't need to be transformed, which you can do but probably don't want/need to do. There are situations in which you want to replay raw events, but in most cases I think you want to replay actual transactions to your DB, so it makes more sense to log your progress as you commit using whatever consumes from Kafka rather than in Kafka itself. My main concern is that when all you have is a hammer, or you already have a hammer lying around, everything looks like a nail. Kafka can be great as a raw event stream, and yes you can store raw events forever, but are raw events really the source of truth for your business? If your workflow is, as I think is appropriate in almost all cases, Kafka->Consumer service->DB, why do you want to rely on Kafka when you have a consumer that can have better logic and custom handling regarding how you actually interpret your events? Moreover, why keep the data in Kafka when you can just plug it into a temporal DB from your service?

jackcodes 6y ago
To answer your last question; let’s say I have some shared state that multiple micro-services rely on and all need to fetch that state synchronously. My example is that you’re a loans company and almost everything is driven from the loan book. Reporting services need to know the active loan counts this month, invoicing services need to fetch the loan book to generate transaction lines, borrower services need to know whether the current user has any active loans. The standard wheel-and-spokes model would probably have a loan book service where all others call into it via HTTP requests, but you’ll have to build a query language on top of that REST API, or share DB access I suppose. Or in this model you would stream the loan book updates through Kafka and every consumer that needs a copy of the loan book can keep their own, query it against their own chosen materialised store in its native query language (some might just append to Mongo, or kSQL, some might always use event sourcing per-request). “select * from loans where user.id = current_user” becomes possible here rather than stitching the HTTP responses. Now I’m not saying one is right or wrong outright as you’re only trading a more native query interface with data consistency concerns, but this for me still feels like a valid architectural pattern and one I’d consider.
- moreaccountspls 6y ago
  If you have a bunch of services reaching into the same data store though, you've effectively built a distributed monolith. Which if that works for you, more power to you, but you lose most of the Conway's law advantages of services with that approach imo.
  - jackcodes 6y ago
    I agree with this completely. In some business applications DDD only gets you so far.

jackcodes 6y ago

To answer your last question; let’s say I have some shared state that multiple micro-services rely on and all need to fetch that state synchronously. My example is that you’re a loans company and almost everything is driven from the loan book. Reporting services need to know the active loan counts this month, invoicing services need to fetch the loan book to generate transaction lines, borrower services need to know whether the current user has any active loans. The standard wheel-and-spokes model would probably have a loan book service where all others call into it via HTTP requests, but you’ll have to build a query language on top of that REST API, or share DB access I suppose. Or in this model you would stream the loan book updates through Kafka and every consumer that needs a copy of the loan book can keep their own, query it against their own chosen materialised store in its native query language (some might just append to Mongo, or kSQL, some might always use event sourcing per-request). “select * from loans where user.id = current_user” becomes possible here rather than stitching the HTTP responses. Now I’m not saying one is right or wrong outright as you’re only trading a more native query interface with data consistency concerns, but this for me still feels like a valid architectural pattern and one I’d consider.

moreaccountspls 6y ago
If you have a bunch of services reaching into the same data store though, you've effectively built a distributed monolith. Which if that works for you, more power to you, but you lose most of the Conway's law advantages of services with that approach imo.
- jackcodes 6y ago
  I agree with this completely. In some business applications DDD only gets you so far.