3 ms·

> If Kafka had come first, everyone would see relational SQL databases as overengineered, overly-prescriptive, overhyped. There are niches where they're appropr

> If Kafka had come first, everyone would see relational SQL databases as overengineered, overly-prescriptive, overhyped. There are niches where they're appropriate, but they're not a good general-purpose solution. Oh ho, there's some deep irony about calling a database over engineered when we're looking at kafka. I'm running a copy of sqlite right here, and it's pretty nice; it does queries and it's portable, it even comes as a single .c file that I can drop directly into a project. I use the same API for my mobile app. Where's the equivalent for kafka? No? I have to install java, and run several independent processes (forget embedded), and configure a cluster to... um... do what? send some messages? If I want to do more than that I have to add my own self implemented logic to handle everything. Different things you argue; but alas, that's the point; the point you made was that kafka would 'be it' if it had come first, but...since it cannot serve the purpose of a db in most contexts; you're just wrong. It's a different thing, and it might be suitable for some things, but I have deployed and managed (and watched others manage) kafka clusters, and I'll tell you a secret: Confluent sells managed kafka. They have a compelling product. You know why? ...because maintaining a kafka cluster is hard, hard work, and since that's the product confluent sells, they're not about to fix it. Ever. Now, if we're talking about event-streaming vs tables; now there's some interesting thoughts about what a simple single node embedded event streaming system might look like (hint: absolutely nothing like kafka), and what an alternate history where that became popular might have been... ...but Kafka, this kafka, would never have filled that role, and even now, it's highly dubious it fills that role in most cases; even, even if we acknowledge that event streaming could fill that role, if it was implemented in a different system.

lmm 6y ago
> Different things you argue; but alas, that's the point; the point you made was that kafka would 'be it' if it had come first, but...since it cannot serve the purpose of a db in most contexts; you're just wrong. SQLite can't serve the purpose of a db in most contexts either; not only can it not be accessed concurrently from multiple hosts, it can't safely be accessed from different hosts at all. It can't even concurrently be accessed from multiple processes on the same host. If we both took a list of general-purpose database use cases, and raced to implement them via Kafka or via SQLite, I guarantee the Kafka person would finish a lot quicker. > You know why? ...because maintaining a kafka cluster is hard, hard work, and since that's the product confluent sells, they're not about to fix it. Ever. Maintaining a Kafka cluster sucks. So does maintaining a Galera cluster (I've done both), and I'll bet the same is true for any other proper HA (master-master) datastore. Comparing Kafka against "SQL databases" is a category error, sure. But in the big use case that everyone talks about, that all the big-name SQL databases focus on - as a backend for a network service that runs continuously where you don't want a single point of failure - those big-name SQL databases are overengineered and underwhelming compared to Kafka, and in most other use cases an SQL database is overengineered compared to a (notional) similarly-designed event-sourcing system for use with the same sort of problems.
- skyde 6y ago
  it seem funny to compare kafka to sql for a OLTP use case. The main benefit of SQL are index and Transactions. If you don’t need them you might as well use a file system directly but if you need it, there is not much alternative :-)
  - lmm 6y ago
    Filesystems are a pain in various ways. The only decent HA option is AFS and it has performance issues and general awkwardness. In a lot of programming languages the filesystem APIs are surprisingly bad. And it's still really easy to have lost updates etc.
    - skyde 6y ago
      if you care about lost updates you should use transaction or at least an API that support optimistic concurrency control. Posix File system API don’t offer much for concurrency control other than lock. In my line of work we use HDFS or AWS S3 as “filesystem” so we are not using the OS file access API at all and most File are append only by a single server.
      - lmm 6y ago
        I don't understand - you suggested using the filesystem, but filesystems don't have good concurrency control. S3's concurrency handling is even worse than filesystems. Something like Kafka is a much better way to do a stream of commands that need to be transformed to eventually produce some kind of result (which is really almost all workloads).
        skyde 6y ago
        No this was a misunderstanding I said if you need (concurrency control/transaction and index) use a real database not a filesystem, since it’s the whole reason database exist. But if you don’t need it there are plenty of good distributed file system that will not eat your data.

lmm 6y ago

> Different things you argue; but alas, that's the point; the point you made was that kafka would 'be it' if it had come first, but...since it cannot serve the purpose of a db in most contexts; you're just wrong. SQLite can't serve the purpose of a db in most contexts either; not only can it not be accessed concurrently from multiple hosts, it can't safely be accessed from different hosts at all. It can't even concurrently be accessed from multiple processes on the same host. If we both took a list of general-purpose database use cases, and raced to implement them via Kafka or via SQLite, I guarantee the Kafka person would finish a lot quicker. > You know why? ...because maintaining a kafka cluster is hard, hard work, and since that's the product confluent sells, they're not about to fix it. Ever. Maintaining a Kafka cluster sucks. So does maintaining a Galera cluster (I've done both), and I'll bet the same is true for any other proper HA (master-master) datastore. Comparing Kafka against "SQL databases" is a category error, sure. But in the big use case that everyone talks about, that all the big-name SQL databases focus on - as a backend for a network service that runs continuously where you don't want a single point of failure - those big-name SQL databases are overengineered and underwhelming compared to Kafka, and in most other use cases an SQL database is overengineered compared to a (notional) similarly-designed event-sourcing system for use with the same sort of problems.

skyde 6y ago
it seem funny to compare kafka to sql for a OLTP use case. The main benefit of SQL are index and Transactions. If you don’t need them you might as well use a file system directly but if you need it, there is not much alternative :-)
- lmm 6y ago
  Filesystems are a pain in various ways. The only decent HA option is AFS and it has performance issues and general awkwardness. In a lot of programming languages the filesystem APIs are surprisingly bad. And it's still really easy to have lost updates etc.
  - skyde 6y ago
    if you care about lost updates you should use transaction or at least an API that support optimistic concurrency control. Posix File system API don’t offer much for concurrency control other than lock. In my line of work we use HDFS or AWS S3 as “filesystem” so we are not using the OS file access API at all and most File are append only by a single server.
    - lmm 6y ago
      I don't understand - you suggested using the filesystem, but filesystems don't have good concurrency control. S3's concurrency handling is even worse than filesystems. Something like Kafka is a much better way to do a stream of commands that need to be transformed to eventually produce some kind of result (which is really almost all workloads).
      - skyde 6y ago
        No this was a misunderstanding I said if you need (concurrency control/transaction and index) use a real database not a filesystem, since it’s the whole reason database exist. But if you don’t need it there are plenty of good distributed file system that will not eat your data.