5 ms·

>With Kafka, such a stream may record the history of your business for hundreds of years Do not do this. Kafka is not a database! Kafka should never be the sou

>With Kafka, such a stream may record the history of your business for hundreds of years Do not do this. Kafka is not a database! Kafka should never be the source of truth for your business. The source of truth should be in whatever consumes data from Kafka downstream when messages are committed as read. Why? Because in your middle layer you can do all the data normalization, sanity checking, processing, and interaction with a REAL database system downstream that can give you things like transactions, ACID, etc. Of course Confluent wants you to try to use Kafka as a DB, so your usage of it is very high and you pay for the top support package and they have you by the cajones, but that doesn't mean you should do that. You will miss out on all the benefits of using a real database, with what benefit? Having a simple client API?

sixdimensional 6y ago
So, I've been having a back and forth with a colleague on this and I'm genuinely interested in why you so strongly suggest this. For the record, I have good real world experience with all kinds of databases (relational, NoSQL, and even legacy multivalue and hierarchical ones), and I don't see why what you say has to be "always true". One way of looking at Kafka is that is an unbundled transaction log, nothing more or less, so it could be used to permanently store and replay transactional activity, if one wishes. Noting that, even most databases don't store an immutable, permanent transaction log (as they often grow to be huge and are truncated every so often, and tables are used as the current state). This article by Confluent seems to cover the topic (yes, recognizing it is written by the very vendor you suggest is trying to lock us in): https://www.confluent.io/blog/okay-store-data-apache-kafka/ https://www.confluent.io/blog/okay-store-data-apache-kafka/. Ok, so how about the idea of a persistent, immutable, never-ending transaction log (uhoh, sounds like blockchain now!)? Setting aside Kafka for now, what do you think about the basic design pattern? To me it sounds a bit like it could represent a temporal database in raw transactional log form. Why not? EDIT: After rereading your comment I see your main concern is using Kafka as a database management system (DBMS). I would agree, that's not what Kafka is for. But, I don't think Confluent intends that use case, do they? I look at it more as an unbundled single component that is very useful by itself, and is part of a more complex data platform/architecture (ex. Lambda or Kappa architecture).
- opportune 6y ago
  I agree that you can use Kafka as a raw event log - not necessarily a transaction log unless you are basically putting records into Kafka that don't need to be transformed, which you can do but probably don't want/need to do. There are situations in which you want to replay raw events, but in most cases I think you want to replay actual transactions to your DB, so it makes more sense to log your progress as you commit using whatever consumes from Kafka rather than in Kafka itself. My main concern is that when all you have is a hammer, or you already have a hammer lying around, everything looks like a nail. Kafka can be great as a raw event stream, and yes you can store raw events forever, but are raw events really the source of truth for your business? If your workflow is, as I think is appropriate in almost all cases, Kafka->Consumer service->DB, why do you want to rely on Kafka when you have a consumer that can have better logic and custom handling regarding how you actually interpret your events? Moreover, why keep the data in Kafka when you can just plug it into a temporal DB from your service?
  - jackcodes 6y ago
    To answer your last question; let’s say I have some shared state that multiple micro-services rely on and all need to fetch that state synchronously. My example is that you’re a loans company and almost everything is driven from the loan book. Reporting services need to know the active loan counts this month, invoicing services need to fetch the loan book to generate transaction lines, borrower services need to know whether the current user has any active loans. The standard wheel-and-spokes model would probably have a loan book service where all others call into it via HTTP requests, but you’ll have to build a query language on top of that REST API, or share DB access I suppose. Or in this model you would stream the loan book updates through Kafka and every consumer that needs a copy of the loan book can keep their own, query it against their own chosen materialised store in its native query language (some might just append to Mongo, or kSQL, some might always use event sourcing per-request). “select * from loans where user.id = current_user” becomes possible here rather than stitching the HTTP responses. Now I’m not saying one is right or wrong outright as you’re only trading a more native query interface with data consistency concerns, but this for me still feels like a valid architectural pattern and one I’d consider.
Joeri 6y ago
In a proper kafka setting you would use avro and a schema repository to have defined schemas for your events and have an ability to upgrade schemas automatically. For breaking changes you would use kafka streams to migrate the old topic into a new topic and then discard the old topic. Kafka has transactions, and with kafka streams you can create joined views that include those transactions. It’s a different way of working for sure, but for an event sourced system I think kafka is an excellent choice. In a sql database keeping the complete history of changes for years is much less practical than in kafka. What you shouldn’t do is look at kafka as a database. It’s a message log. You push messages in, you eventually read them back out. The key word is eventually. You can’t read your writes, and that makes building an interactive system on top hard.

sixdimensional 6y ago

So, I've been having a back and forth with a colleague on this and I'm genuinely interested in why you so strongly suggest this. For the record, I have good real world experience with all kinds of databases (relational, NoSQL, and even legacy multivalue and hierarchical ones), and I don't see why what you say has to be "always true". One way of looking at Kafka is that is an unbundled transaction log, nothing more or less, so it could be used to permanently store and replay transactional activity, if one wishes. Noting that, even most databases don't store an immutable, permanent transaction log (as they often grow to be huge and are truncated every so often, and tables are used as the current state). This article by Confluent seems to cover the topic (yes, recognizing it is written by the very vendor you suggest is trying to lock us in): https://www.confluent.io/blog/okay-store-data-apache-kafka/ https://www.confluent.io/blog/okay-store-data-apache-kafka/. Ok, so how about the idea of a persistent, immutable, never-ending transaction log (uhoh, sounds like blockchain now!)? Setting aside Kafka for now, what do you think about the basic design pattern? To me it sounds a bit like it could represent a temporal database in raw transactional log form. Why not? EDIT: After rereading your comment I see your main concern is using Kafka as a database management system (DBMS). I would agree, that's not what Kafka is for. But, I don't think Confluent intends that use case, do they? I look at it more as an unbundled single component that is very useful by itself, and is part of a more complex data platform/architecture (ex. Lambda or Kappa architecture).

opportune 6y ago
I agree that you can use Kafka as a raw event log - not necessarily a transaction log unless you are basically putting records into Kafka that don't need to be transformed, which you can do but probably don't want/need to do. There are situations in which you want to replay raw events, but in most cases I think you want to replay actual transactions to your DB, so it makes more sense to log your progress as you commit using whatever consumes from Kafka rather than in Kafka itself. My main concern is that when all you have is a hammer, or you already have a hammer lying around, everything looks like a nail. Kafka can be great as a raw event stream, and yes you can store raw events forever, but are raw events really the source of truth for your business? If your workflow is, as I think is appropriate in almost all cases, Kafka->Consumer service->DB, why do you want to rely on Kafka when you have a consumer that can have better logic and custom handling regarding how you actually interpret your events? Moreover, why keep the data in Kafka when you can just plug it into a temporal DB from your service?
- jackcodes 6y ago
  To answer your last question; let’s say I have some shared state that multiple micro-services rely on and all need to fetch that state synchronously. My example is that you’re a loans company and almost everything is driven from the loan book. Reporting services need to know the active loan counts this month, invoicing services need to fetch the loan book to generate transaction lines, borrower services need to know whether the current user has any active loans. The standard wheel-and-spokes model would probably have a loan book service where all others call into it via HTTP requests, but you’ll have to build a query language on top of that REST API, or share DB access I suppose. Or in this model you would stream the loan book updates through Kafka and every consumer that needs a copy of the loan book can keep their own, query it against their own chosen materialised store in its native query language (some might just append to Mongo, or kSQL, some might always use event sourcing per-request). “select * from loans where user.id = current_user” becomes possible here rather than stitching the HTTP responses. Now I’m not saying one is right or wrong outright as you’re only trading a more native query interface with data consistency concerns, but this for me still feels like a valid architectural pattern and one I’d consider.

Joeri 6y ago

In a proper kafka setting you would use avro and a schema repository to have defined schemas for your events and have an ability to upgrade schemas automatically. For breaking changes you would use kafka streams to migrate the old topic into a new topic and then discard the old topic. Kafka has transactions, and with kafka streams you can create joined views that include those transactions. It’s a different way of working for sure, but for an event sourced system I think kafka is an excellent choice. In a sql database keeping the complete history of changes for years is much less practical than in kafka. What you shouldn’t do is look at kafka as a database. It’s a message log. You push messages in, you eventually read them back out. The key word is eventually. You can’t read your writes, and that makes building an interactive system on top hard.