12 ms·

This notion of “stream-table duality” might be the most misleading, damaging idea floating around in software engineering today. Yes, you can turn a stream of e

• zok3102 6y ago
  Reminds me of that time when database vendors were overreaching to be message queues/brokers - OracleAQ, MSMQ, etc.
• math 6y ago
  At the moment you more or less need to write a DBMS in your app code, but I don't think that's the end state. I think what we're seeing the beginnings of something big - it just might not seem like it yet because it's the v1 / no where near complete version. I think having all your data in a single system (Kafka, KsqlDb, ..) that allows you to work with it in cross paradigm ways will turn out to be very compelling.
  • eternalban 6y ago
    > At the moment you more or less need to write a DBMS in your app code Since we're discussing misunderstandings in the community, it should be pointed out that a Database Management System (DBMS) is not merely a database, to say nothing of a data store. Oracle, Postgres, et al are genuine "DBMS". ATM you are very likely putting together a data store in your app code.
    • tomnipotent 6y ago
      You're conflating DBMS with RDBMS. I interpreted the OP as DBMS=database, which absolutely includes application code that stores and retrieves data in proprietary formats. Even a linked list mmap'd to disk can be a database, just maybe not a very good one.
      • eternalban 6y ago
        A RDBMS is simply a Relational Database Management System. A quick tour of CS history reveals ancient curiosities such as Hierarchical Database Management Systems. And there is more: https://www.studytonight.com/dbms/database-model.php https://www.studytonight.com/dbms/database-model.php
        • tomnipotent 6y ago
          There are many kinds of databases. Graph, key/value, relational, hierarchal etc. That doesn't change the fact that any app that writes code to store and retrieve data is creating their own database or using someone else's.
          • eternalban 6y ago
            There are many kinds of data models. A DBMS is a DBMS, and typically a specific DBMS supports a specific data model. A log structured file is at most a data store. To be fair this is a somewhat fuzzy categorization (DBMS, DB, Data Store) and it can cause confusion. An DBMS is a system, just like it says right on the acronym tin. A DBMS is a system that provides capabilities such DDL, DML, auxiliary processes, etc. to manage a database. There are various data models, e.g. a triple store, for databases, but the data model X is a orthogonal to XDBMS.
            • deleted 6y ago
              [deleted]
  • davedx 6y ago
    Meteor did some pretty interesting work with this by tailing mongodb’s oplog. Definitely still room for innovation.
• manigandham 6y ago
  That's why ksqlDB exists and handles all that for you, turning streams into tables that you can query.
  • georgewfraser 6y ago
    ksql does not solve any of the hard consistency or contention problems you will face if you attempt to use Kafka as a datastore. Consider the simplest possible example: you write an "update event" to a topic and then read a ksql view of that topic. The view may or may not yet reflect the update. This is called read-after-write consistency, and you will need to create it in your application code.
    • manigandham 6y ago
      It's not a real database and doesn't promise ACID though. I think it's fine with the understanding that it's an incremental eventually-consistent materialized view engine that works seamlessly with Kafka, especially if you're already a heavy user. Otherwise loading data into a real database/data warehouse and using ETL with triggers/views/aggregations is better if you need advanced querying rather than a simple stream to table transforms.
    • dkersten 6y ago
      The same thing happens if you write to an RDBMS master and then read from a read-only replica.
• epistasis 6y ago
  The point of that is for people trying to figure out why streams are a useful abstraction at all what's needed to make them useful are some sort of aggregation, and of course tabular state is a common end point. The article does not recommend writing this code yourself, it shows how to aggregate data into usable forms. So I think your concerns may be a bit overblown. If you think that ksqlDB or Kafka Streams, the tools shown in this blog post, are are at risk for what you warn, this comment would be a valid criticism. But it's clear that the article isn't advocating for people to write their own versions of that...
  • SpicyLemonZest 6y ago
    Yes, they're definitely at risk. ksqlDB does not appear to have transactions at all.
    • camdenreslink 6y ago
      The transaction boundary is the message/event itself (in the case of an event sourced system, which is what this would primarily be used with). Everything required to understand some event is contained within the payload. It is the atomic unit in this type of system.
• strictfp 6y ago
  I always found it ironic that you get most of this for free if you design your sql updates and save/query the transaction log and/or history. A lot of relational dbs have functionality for that. And if you don't want to use that, there's also products for this specifically, such as event store.
  • rclayton 6y ago
    That’s true, but you have to interpret the state changes from your table/row schema. Why not just update the DB and then publish an event?
    • deleted 6y ago
      [deleted]
    • jeffffff 6y ago
      because things can and will fail inbetween updating the db and publishing the event leading to inconsistency between your database and kafka. a better approach would be to update the db and then use something like debezium or maxwell to pull the updates from the db's bin log and publish them to kafka.
      • gunnarmorling 6y ago
        > update the DB and then publish an event It can make sense to do both, so to de-couple internal table structures from externally exposed events. Debezium supports you safely doing this via the transactional outbox pattern [1]. Events are inserts into an outbox table in the same database as the actual business data, both updated in one transaction. Debezium then can be used to capture the outbox events from the logs and send the events to Kafka. To me this hits the sweetspot for many use cases, as you avoid the potential inconsistencies of dual writes which you mention while still not exposing internal structures to external consumers (for other use cases, like updating a cache or search index on the contrary, CDC-ing the actual tables may be preferable). Disclaimer: I work on Debezium [1] https://debezium.io/documentation/reference/configuration/outbox-event-router.html https://debezium.io/documentation/reference/configuration/ou...
• Ozzie_osman 6y ago
  > This notion of “stream-table duality” might be the most misleading, damaging idea floating around in software engineering today. No. The notion of a "stream-table duality" is a powerful concept, that I've found can change the way any engineer thinks about how they are storing / retrieving data for the better (it's an old idea, rooted in Domain Driven Design, but for some reason a lot of engineers, myself included, still need to learn, or relearn, it). The notion that relying on a stream as the primary data persistence abstraction or mechanism in a production is the misleading part, at least for now. I'd argue Kafka pushes us in a direction that makes progress along that dimension, and you can apply it successfully with a lot of effort. But to match what you can get from a more traditional DBMS? The tech just isn't there (yet).
  • haggy 6y ago
    We've been relying on streams since the advent of write ahead logs but the complexity of streaming state transformations was re-formed into something that is easier to reason about. The most common form is a relational DB which is generally users asking questions against snapshots of the WAL. RDB's is a highly opinionated layer of abstraction on top of data streams (in the form of inserts and updates).
  • wpietri 6y ago
    Thanks, this is the take I was looking for. I've definitely seen table-focused projects that really struggle because they need to be thinking in stream-focused terms. That's especially true in the age of mobile hardware. E.g., you want to take your web-focused app and have it run on mobile devices with imperfect connectivity so people in the field can use it, table-only thinking is a pain. It's much easier to reconcile event streams than tables. Sure you can't expect a stream-oriented tool to do everything an SQL database does. But the reverse is true as well. If we threw out every tool because people unthinkingly used it for the wrong job, we wouldn't have many tools left.
• wenc 6y ago
  A quibble I have with the term "stream-table duality" is that it's not true duality. You can construct a state (table) from a stream, but you cannot do the reverse. You cannot deconstruct a table into its original stream because the WAL / CDC information is lost -- you can only create a new stream from the table. This means you lose all ability to retrieve previous states by replaying the stream. Information is lost. Duality in math is an overloaded term but it generally means you can go in either direction. This is not true here.
  • gen220 6y ago
    It depends on your data schema. If you store the data in a temporal form (every row has a transaction period), you can reconstruct a stream with ORDER BY LOWER(transaction history). One benefit of using the table as a source of truth this way is that you can run migrations on the entire “stream” if you will, and republish the events in order, in the new schema definition. Another benefit is that you can run SQL queries against your entire transaction history. Some (imo sketchy) tools exist that try to give you a SQL interface to Kafka, but that’s just bad. It’s much easier to have your cake and eat it in postgres than it is in Kafka, imo. Unless your data volume is absurd, in which case Kafka is your only realistic choice. Of course, in a “literal” sense you’re correct, it’s not possible. But from a practical point of view, if your bit empor all data implementation is stable, you can totally depend on your ability to produce streams from a database. We do this at my current place of work all the time.
  • mehrdadn 6y ago
    Just came here to post this comment. I feel like they must've known this when creating the terminology, so I can only imagine they called it "duality" because they wanted it to sound cool. But it's most definitely not duality and I would wager calling it that is actively harmful. Last thing you want is someone implementing a table under some misled impression that it can do everything a stream can!
    • eternalban 6y ago
      This Confluent blog is a rehash of earlier conceptual position statement of Jay Kreps, who is now the boss of the Confluent blogger. https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha... Pointing out (GP and yourself) that this is an 'asymmetric duality' is informative and valid.
• pgwhalen 6y ago
  >You will more or less have to write a full-fledged DBMS in your application code. Yes, but only if your data is highly relational, and needs to be operated on in that way (e.g. transactionally across records, needing joins to do anything useful, querying by secondary indices, etc.). Depending on your domain, this could be a large amount of your data, or not. Just like anything, Kafka and Kafka Streams are tools that can be used and misused. There is perhaps an exciting future where something like ksqlDB makes even highly relational data sit well in Kafka, but we're definitely not there yet, and I'm not terribly confident about getting there.
  • yawaramin 6y ago
    > Yes, but only if your data is highly relational, and needs to be operated on in that way (e.g. transactionally across records, needing joins to do anything useful, querying by secondary indices, etc.). Ah, you mean that if the data is in a normalized form. Relational data means, basically, data in tabular form. Tabular data can be normalized or not. > Depending on your domain, this could be a large amount of your data, or not. Assuming you are talking about data in normalized form, that's orthogonal to the domain. That's in fact the major breakthrough of relational databases–that data across domains can be modelled as relational and normalized. Now, in the relatively rarer cases that you don't care about normalizing data, a stream can be useful. But in the vast majority of cases IMHO, we should be looking at a proper relational database like Postgres before thinking about streams.
    • pgwhalen 6y ago
      I agree that _normalized_ is a more precise term for what I'm describing, I only used relational because contemporary discourse seems to favor that term in these kinds of discussions and seems to imply a high degree of normalization. >Assuming you are talking about data in normalized form, that's orthogonal to the domain. I don't agree with this though. Nothing about a software architecture is _ever_ orthogonal to the domain.
• pacala 6y ago
  Not super familiar with Kafka, but behind the scenes some RDBMS are log replay engines. Are there technical reasons to not slap a RDBMS frontend on top of a Kafka log and avoid dealing with raw streams in the application code? Edit: From the sibling comments, the answer is https://ksqldb.io https://ksqldb.io.
  • gen220 6y ago
    The technical reason is the inherently distributed and shared nature of Kafka setups makes writing an ACID-compliant frontend (basically?) impossible. If your application assumes ACID compliance, it will fail. Also, I haven’t looked into this one as directly, but I imagine that query latency compared to an RDBMS would be quite poor. IMO, most people reach for Kafka too soon, where an RDBMS with a proper schema that includes transaction and valid history would better suit the domain. It’s possible to write very efficient and ergonomic “append-only”, temporal data schemas in Postgres schemas.
    • hashhar 6y ago
      But there are a lot of usecases where ACID is not needed and we can do with eventually consistent states. Btw, you do get C and D.
• lmm 6y ago
  > However, during that process you will eventually confront every single hard problem that relational database management systems have faced for decades. However, during that process you will eventually confront every single hard problem that relational database management systems have faced for decades. You will more or less have to write a full-fledged DBMS in your application code. And you will probably not do a great job, and will end up with dirty reads, phantoms, and all the other symptoms of a buggy database. It's the same as moving from a framework to a library. At first, the framework seems great, it solves all your problems for you. But then as soon as you want to do something slightly different from the way your framework wanted you to do it, you're stuffed. Much better to use libraries that offer the same functionality that the framework had, but under your own control. If you use the sledgehammer of a traditional RDBMS you are practically guaranteed to get either lost updates, deadlocks, or both. If you take the time to understand your dataflow and write the stream -> table transformation explicitly, you front-load more of the work, but you get to a database that works properly more quickly. 99% of webapps don't use database transactions properly. Indeed during the LAMP golden age the most popular database didn't even have them. The most well-designed systems I've seen built on traditional databases used them like Kafka - they didn't read-modify-write, they had append-only table structures and transformed one table to compute the next one. At which point what is the database infrastructure buying you? If you're using a traditional database it's overwhelmingly likely that you're paying a significant performance and complexity cost for a bunch of functionality that you're not using. Given the limited state of true master-master HA in traditional databases, you're probably giving up a bunch of resilience for it as well. If Kafka had come first, everyone would see relational SQL databases as overengineered, overly-prescriptive, overhyped. There are niches where they're appropriate, but they're not a good general-purpose solution.
  • hilbertseries 6y ago
    I disagree pretty strongly with this. Things like indexes, uniqueness constraints, foreign keys, basically data schemas are hugely useful. There was a big movement for awhile in web dev, to use mongo instead of “the sledgehammer of an rbdms” , people learned over time that actually they wanted a database. Because you run into so many issues with querying and data consistency or maybe you want to do a join. All of these things become huge problems without a database. An eventually consistent log of data is not a replacement for a database. Can you imagine designing a simple crud app with a couple of tables and different products, but now do it with Kafka as your backing store. You’ve added so much more complexity to your application because you didn’t want to use a database.
• jholman 6y ago
  Recently I watched a 50-engineer startup allocate more than 50% of their engineering time for about two years to trying to cope with the consequences of using Kafka as their database, and eventually try to migrate off of it. The whole time I was wondering "but how could anyone have started down this path?!?" Apparently the primary reason they went out of business was sales-related, not purely technical, but if they hadn't used Kafka, they could have had 2x the feature velocity, or better yet 2x the runway, which might have let them survive and eventually thrive. Imagine, thinking you want a message bus as your primary database.
  • steve_adams_86 6y ago
    I worked with a short-lived startup which made this exact mistake. I suggested persisting important events to a postgres db, but I was shot down over and over. They were positive it would be fine to hold everything in kafka. There was this notion that kafka functioned fine as a long term data store with the correct configuration (I'm not arguing that), but there was no solution to the lack of... Well, everything postgres could offer. As you mentioned, the lack of velocity really dragged them down. Customers were constantly upset that x broke or y wasn't delivered yet, and each time it was clear that kafka wasn't helping them meet those needs. Kafka is awesome for what it does, but there are a ton of people out there using it for weird stuff.
    • konradb 6y ago
      Do you happen to know what sort of thinking leads a team down this path? It seems a fundamental mistake. Resume driven development?
      • afarrell 6y ago
        Insufficient asking "why?" 1. Why do we want to use Kafka? [insert thinking] So that we can do x yet avoid y. 2. Why is the customer asking for these features? [insert listening] So that they can do z and think of themselves as w. And then failure to remember and revisit that "why?".
      • 6y ago
• ypcx 6y ago
  I have recently analyzed Kafka as a message bus solution for our team. My take is that Kafka is a very mature and robust message system with clearly defined complexity and thus when used correctly, very dependable. The recent versions even have online rebalancing (but don't tell that to the Pulsar people). However "Kafka Streams" just make me shiver - I don't understand why someone would masquerade a project directly competing with other processing frameworks (e.g. Apache Flink or Spark) as being part of Kafka itself, when apart from utilizing Kafka's Producer and Consumer interfaces it really has nothing else in common with it. Streams is a giant, immature, complexity-hiding mess which as a responsible systems designer you have to run away from really, really fast.
• codeisawesome 6y ago
  Allow me to submit that this obsession is... Kafka-esque? :D
  • itestyourcode 6y ago
    Indeed https://kafkaesque.io/ https://kafkaesque.io/
• EamonnMR 6y ago
  It's worth noting that most managed Kafka solutions offer a service which will transform a stream into a database table so you can query it later. We've also built our own tooling to do it. We read from the Kafka stream directly when we're building a pipeline between microservices; for almost everything else we query the database. It works pretty well and we haven't found ourselves wishing we could write pseudo-queries on streams yet.
• hodgesrm 6y ago
  > This notion of “stream-table duality” might be the most misleading, damaging idea floating around in software engineering today. I disagree. Kafka is a log. That's half of a database. The other half is something that can checkpoint state. In analytics that's often a data warehouse. They are connected by the notion that the data warehouse is a view of the state of the log. This is one of a small number of fundamental concepts that come up again and again in data management. Since the beginning of the year I've talked to something like a dozen customers of my company who use exactly this pattern to implement large and very successful analytic systems. Practical example: you screw up aggregation in the data warehouse due to a system failure. What to do? Wipe aggregates from the last day, reset topic offsets, and replay from Kafka. This works because the app was designed to use Kafka as a replayable, external log. In analytic applications, the "database" is bigger than any single component. I agree there's a problem with the stream-table duality idea, but it's more that people don't understand the concept that's behind it. The same applies to transactions, another deceptively simple idea that underpins scalable systems. "What is a transaction and why are they useful?" has been my go-to interview question to test developers' technical knowledge for over 25 years. I'm still struck by how many people cannot answer it.
• mikekchar 6y ago
  I work on a legacy CouchDB application and the same problem exists in that area. CouchDB is actually really amazing and is wonderful for some types of applications. It really sucks if you want a relational DB. Keep in mind that I don't know anything about Kafka, but it seems like the same kind of thinking. You can think about your data as a log of events. Since you can always build the current state from the log of events, then you might as well just save the log. This works incredibly well when you treat your data as immutable and need to be able to see all of your history, or you want to be able to revert data to previous states. It also solves all sorts of problems when you want to do things in a distributed fashion. The application which fits this model well and that almost every develop is familiar with is source control (and specifically git). If that is the kind of thing that you want to do with your data, then it can be truly wonderful. If it's not what you want to do, then it will be the lodestone that drags you underwater to your death. The legacy application I work on is a sales system and actually it's not a bad area for this kind of data. You enter data. Modifications are essentially diffs. You can go back and forth through the DB to see how things have changed, when they were changed and often why they were changed. In many ways it is ideal for accounting data. But as you say, if you want a full fledged DBMS, this should not be your primary data store. It can act at the frontend for the data store -- with an eventually consistent DB that you use for querying (and you really have to be OK with that "eventually consistent" thing). But you don't want to build your entire app around it.
• dwenzek 6y ago
  > Kafka is a message broker. It’s not a database and it’s not close to being a database. Kafka is somehow a database, even if specialized on logs of events. For a similar discussion, I would refer to Jim Gray "Queues Are Databases" https://arxiv.org/pdf/cs/0701158.pdf https://arxiv.org/pdf/cs/0701158.pdf.
• dwenzek 6y ago
  I fully agree that this "stream-table duality" is misleading or at least quite vague. The diagram juxtaposing a chess board (the table) and a sequence of moves (the stream) seems enlightening but actually hides several issues. * An initial state is required to turn a stream into a table. * A set of rules is required to reject any move which would break the integrity constraints. * On the reverse direction, a table value cannot be turned into a stream. We need an history of table values for that. Hence, the duality, or more accurately the correspondence, is not between stream and table but between constrained-stream and table history. Even if these issues are not made explicit, the Confluence articles and the Kafka semantics are correct. * "A table provides mutable data". This highlights that stream-table-duality is relative to a sequence of table states. * "Another event with key bob arrives" is translated into an update ... to enforce an implicit uniqueness constraint on the key for the tables.
• eggsnbacon1 6y ago
  I worked on a project that used CQRS and Event Sourcing years ago. It was an unmitigated disaster, never made it out of prototype phase. By using Kafka (or anything else) as a "commit log" you've just resigned yourself to building the rest of the database around it. In a real RDBMS the commit log is just a small piece of maintaining consistency. Every time I've worked on a project where we handle our own "projections" (database tables) we ended up mired deep in the consistency concerns normally handled by our database. Whats so hard? Compaction is an evil problem. We figured we could "handle it later". Well it turns out maintaining an un-compactible commit log of every event even for testing data consumes many terabytes. This and "replays" sunk the project indirectly. Replays. The ES crowd pretends this is easy. It's not. If you have 3 months of events to replay every time you make certain schema changes you are screwed. Even if you application can handle 100X normal load, its going to take you over a day to restore using a replay. This also happened to us. Every time an instance of anything goes down, it has to replay all the events it cares about. If you have an outage of more than a few instances the thundering herd will take down your entire system no matter how much capacity you have. With a normal system, the Db always has the latest version of all the "projections" unless you're restoring from a backup, then it only has to restore a partial commit log replay, and only once. Data duplication. Turns out "projecting" everything everywhere is way worse on memory and performance than having the DB thats handling your commit log also store it all in one place. Who knew. Data corruption. If you get one bad event somewhere in your commit log you are cooked. This happens all the time from various bugs. We resorted to manually deleting these events, negating all the supposed advantages of an immutable commit log. We would have been fine if we let the db handle the commit log events. Questionable value of replays. You go through a ton of bs to get a system that can rewind to any point in time. When did we use this functionality? Never. We never found a use-case. When we abandoned ES we added Hibernate Auditing to some tables. Its relatively painless, transparent, and handled all of our use cases for replays