4 ms·

So, I've been having a back and forth with a colleague on this and I'm genuinely interested in why you so strongly suggest this. For the record, I have good re

• opportune 6y ago
  I agree that you can use Kafka as a raw event log - not necessarily a transaction log unless you are basically putting records into Kafka that don't need to be transformed, which you can do but probably don't want/need to do. There are situations in which you want to replay raw events, but in most cases I think you want to replay actual transactions to your DB, so it makes more sense to log your progress as you commit using whatever consumes from Kafka rather than in Kafka itself. My main concern is that when all you have is a hammer, or you already have a hammer lying around, everything looks like a nail. Kafka can be great as a raw event stream, and yes you can store raw events forever, but are raw events really the source of truth for your business? If your workflow is, as I think is appropriate in almost all cases, Kafka->Consumer service->DB, why do you want to rely on Kafka when you have a consumer that can have better logic and custom handling regarding how you actually interpret your events? Moreover, why keep the data in Kafka when you can just plug it into a temporal DB from your service?
  • jackcodes 6y ago
    To answer your last question; let’s say I have some shared state that multiple micro-services rely on and all need to fetch that state synchronously. My example is that you’re a loans company and almost everything is driven from the loan book. Reporting services need to know the active loan counts this month, invoicing services need to fetch the loan book to generate transaction lines, borrower services need to know whether the current user has any active loans. The standard wheel-and-spokes model would probably have a loan book service where all others call into it via HTTP requests, but you’ll have to build a query language on top of that REST API, or share DB access I suppose. Or in this model you would stream the loan book updates through Kafka and every consumer that needs a copy of the loan book can keep their own, query it against their own chosen materialised store in its native query language (some might just append to Mongo, or kSQL, some might always use event sourcing per-request). “select * from loans where user.id = current_user” becomes possible here rather than stitching the HTTP responses. Now I’m not saying one is right or wrong outright as you’re only trading a more native query interface with data consistency concerns, but this for me still feels like a valid architectural pattern and one I’d consider.
    • moreaccountspls 6y ago
      If you have a bunch of services reaching into the same data store though, you've effectively built a distributed monolith. Which if that works for you, more power to you, but you lose most of the Conway's law advantages of services with that approach imo.
      • jackcodes 6y ago
        I agree with this completely. In some business applications DDD only gets you so far.
• halbritt 6y ago
  In DBMS the "source of truth" is the log. Kafka can be used in exactly the same way, and in that sense, it can be the primary data store. Whether that's useful for a given use case is certainly debatable. IIRC, the New York Times uses Kafka as the primary storage for their entire archive.
  • muxator 6y ago
    > In DBMS the "source of truth" is the log that's only true in the (frequent) case of write-ahead logs. Some database engines may use Rollback logs, or undo logs.
    • halbritt 6y ago
      A log is a log. Maybe read "I <3 Logs" by Jay Kreps, one of the founders of Confluent and creators of kafka.
• yawaramin 6y ago
  > Ok, so how about the idea of a persistent, immutable, never-ending transaction log (uhoh, sounds like blockchain now!)? Setting aside Kafka for now, what do you think about the basic design pattern? To me it sounds a bit like it could represent a temporal database in raw transactional log form. Why not? Because nobody wants to be replaying events all the time to get their actual data. They want the data to be, well, materialized. Replaying events can be helpful if you need an audit trail but the systems which need that have mostly all evolved their own audit trail techniques, e.g. double-entry accounting. The people who invented these streaming event systems quickly realized that continually replaying events from the beginning would get absurdly expensive, so they even implemented 'checkpoint' events that snapshot the current state of the data every once in a while so that replays can start from (a hopefully recent) snapshot and finish quickly. At that point you have to encode the logic of how to roll up all events into the current state into your checkpointing code, which immediately enforces the notion of a global current state of the data, which is in fact what RDBMs solve anyway.
  • monadic2 6y ago
    RDBMs do not generally (ie with a generic interface) allow access to the backing event stream, though, meaning you’ll need to write this yourself when you need to sync changes across databases. There’s no free lunch.
    • yawaramin 6y ago
      Across different databases as in Postgres-to-Maria, sure. But people who decide to do that should know what they are getting into. But across different instances of say Postgres, it is rather simple.
      • gunnarmorling 6y ago
        Debezium (debezium.io) might be interesting for you; it provides open-source change data capture for a variety of databases; together with the right Kafka Connect sink connectors (or using it via Pulsar etc.), setting up a Postgres-to-Maria data pipeline is quite easy. (Disclaimer: I work on Debezium)