8 ms·

Kafka Spawns Open-Source KarelDB

123 points by lelf 7y ago

netfl0 7y ago
Can anyone explain why this might be exciting? (genuinely curious)
- thanatos_dem 7y ago
  Ever wanted a database with all the operational complexity of Kafka, and more? Well now you can have it! It’s exciting as someone more in the SRE side, because you’d need to pay be an obscene amount to support this.
  - slowmovintarget 7y ago
    It's like the bulked-up Cartman of key-value stores. Edit: Nope, it's a relational DB
  - diehunde 7y ago
    Isn't that "side" just a bunch of clicks on a wizard on some cloud provider?
- solatic 7y ago
  There are companies that fit the following operational niche: * Need to run Kafka (for domain-specific reasons) * Need a relational database, but not necessarily a large or performant one A project that lets you "buy a Kafka cluster, get the relational database free" is strictly simpler and cheaper to operate (in theory...) compared to operating both a Kafka cluster and a traditional relational database.
- adamnemecek 7y ago
  It uses kafka as an operational primitive. Kafka is somewhat complicated but it also solves real problems. I can see a fast, distributed db built on top of kafka.
  - qaq 7y ago
    There fast distributed DBs and they are generally built on top of distributed ordered key-value stores. Many of them do not require to run external components like zookeeper. What is the advantage of running one on top of Kafka?
    - Godel_unicode 7y ago
      The main benefit I have heard is that some people already have Kafka.
      - qaq 7y ago
        in what sense is it a benefit?
- random023987 7y ago
  If you want an asymmetric persistence layer where all writes are documents placed in a Kafka topic, and all reads are via relational queries. It's a distributed, eventually consistent, highly available ledger that can be queried with SQL.
  - Scarbutt 7y ago
    Is not eventually consistent, it uses https://omid.incubator.apache.org/ https://omid.incubator.apache.org/.
- manigandham 7y ago
  It's exciting in the sense that you can apply all these open-source projects to create a very slick lightweight relational database that can replicate changes globally across unlimited instances by using a Kafka topic as the WAL log. Don't actually use this in production. Use a real database that does what you want, and use the native replication features. If you need to replicate to heterogenous or alternative sources or just want a stream of all changes then you can use Kafka with any number of existing database connectors that automatically read and export changes.
amdelamar 7y ago
So, KarelDB makes use of KCache (among other OSS). Which so happens that Kafka uses. But KarelDB does not use Kafka and vice versa. Did I get that right? EDIT: After reading the GitHub README, it does appear KarelDB makes use of Kafka. Its unclear to me if its running kafka brokers underneath, or depends on an existing cluster.
slowmovintarget 7y ago
https://yokota.blog/2019/09/23/building-a-relational-database-using-kafka/ https://yokota.blog/2019/09/23/building-a-relational-databas...
derefr 7y ago
I've been looking into Kafka lately, because I have a need for "the thing that Kafka is in practice" (not a message queue per se; rather, an indefinitely-durable event store that supports message-queuing API semantics for consumers.) And, while Kafka does do this, its infrastructure needs are kind of ridiculous when your producers 1. produce events in clean batches, and 2. could re-produce those events, in the exact same batches, if prompted. Ideally, rather than having to basically have a whole second DBMS in the form of a Kafka cluster (stateful nodes with big HDDs, carefully backed up, needing repair when any die), what I would love is a Kafka-like "event store with MQ semantics", which is in turn backed by an object store. Basically, like Datomic's segment-storage architecture; or like WAL-E in its use-case as an intermediary for Postgres replication. Unlike either of those, though, MQ log-segments don't need any additional metadata (e.g. checkpoints, indices), so they can just be chucked straight in object storage, and MQ nodes can base their view of the world entirely on what happens to be in the object-storage bucket. Of course, this kind of system only makes sense if you have no need for durability of new messages (because e.g. the producer still has the data that went into them, and can resend them), only old messages. Which is kind of an unusual requirement, I suppose, so I don't really expect anyone else to go building this sort of thing for me.
- random023987 7y ago
  > I would love is an event store You know you can put a key with a null value onto a topic. If your backend object store gives you data by key, this should give you exactly what you want. Later on, you can put a small subset of the data in value to assist with filtering or caching (if you want).
  - derefr 7y ago
    I'm not clear on what you're trying to suggest, here. I would like a system that presents itself the same as a Kafka broker, to both producers and consumers (who are just working with regular messages and topics, not object-handles), but where the MQ broker they're talking to is internally just a shared-nothing stateful write-through WAL proxy for an object store. (Sort of like how, with actual logging, you could think of the local rsyslog daemon on an instance as a "shared-nothing stateful WAL-writeback-caching proxy for an object store." Except rsyslog lives on the producer, whereas this write-through WAL proxy would live on the broker.) In such a design, "partitions" and partition ownership would only apply to the newest segment of the log; all older segments would be globally-available, because as soon as a log-segment is finalized, it would get pushed to the object store and become an object. Every node would then be doing Hierarchical Storage Management, with a local LRU cache of segments, populated just-in-time in response to requests, by the object store. (This is what I meant by the comparison to Datomic's storage architecture.) A lot of the technical parts of this could be achieved by just FUSE-mounting an S3 bucket to a directory on each Kafka node, and convincing the node to move log segments into that directory once they're done. But Kafka would still be wasting time and resources replicating segments between nodes that already now "have" them; and also, without further hacking, nodes wouldn't realize that they now also "have" 99% of the segments of every log potentially-available to serve. A system that was built this way from the ground up would end up a lot simpler than Kafka.
atombender 7y ago
Discussed here previously: https://news.ycombinator.com/item?id=21048822 https://news.ycombinator.com/item?id=21048822
mcepl 7y ago
Will the next version be called Josef? (to continue line of the famous authors from Prague)
AtlasBarfed 7y ago
... How does it do distributed joins? Which just don't scale in the AP sense, and really can you trust a distributed join in CP? Unless your data and the data you are joining are strictly shard-local. But that's also not a distributed join.
- manigandham 7y ago
  It's not a distributed database. It's just a single-node key/value store where all updates are sent to and then read from a Kafka topic, with relational querying and schema management handled by various layered Apache projects. It's a useful tool/model to follow if your data fits on a single node and you want easy replication across all instances.
  - AtlasBarfed 7y ago
    Wow, seems like a lot of work when Postgres can just be setup. But if someone needs it why not.
    - manigandham 7y ago
      This isn't a real product, it's an experiment showing off what you can do. Kafka is a distributed log, you can use it as replication log if you want and there are tons of production-ready connectors to all kinds of databases already. Anyone who needs non-native replication (perhaps across different databases or sources) should just use those.
manigandham 7y ago
KCache is an in-memory Java cache where all writes are sent to a Kafka topic and then applied locally by any instances listening to that topic, providing an eventually-consistent cache. Great for scaling a small dataset that can fit on a single-node with easy replication and a full log of all updates in the Kafka topic. KarelDB adds relational querying, schema/table management, transactions, and a SQL interface on top of KCache by using various Apache open-source projects (Calcite, Omid, Avatica, Avro). Here's a blog post from the author that has more details: https://yokota.blog/2019/09/23/building-a-relational-database-using-kafka/ https://yokota.blog/2019/09/23/building-a-relational-databas... A typical database (like Postgres) or cache (like Redis) has a WAL where all writes are applied and this log is also used for replicas. KCache and KarelDB are interesting experiments in doing this from scratch using open-source components, although using Kafka as a durable log of system-agnostic changes is already very common. In fact, it's probably the most used feature of Kafka Connect with various database connectors and changefeed streams.
- ako 7y ago
  Sounds like Kareldb would make a good alternative to rocksdb, when combining tables and streams in Kafka streaming?

123 points by lelf 7y ago

netfl0 7y ago

Can anyone explain why this might be exciting? (genuinely curious)

thanatos_dem 7y ago
Ever wanted a database with all the operational complexity of Kafka, and more? Well now you can have it! It’s exciting as someone more in the SRE side, because you’d need to pay be an obscene amount to support this.
- slowmovintarget 7y ago
  It's like the bulked-up Cartman of key-value stores. Edit: Nope, it's a relational DB
- diehunde 7y ago
  Isn't that "side" just a bunch of clicks on a wizard on some cloud provider?
solatic 7y ago
There are companies that fit the following operational niche: * Need to run Kafka (for domain-specific reasons) * Need a relational database, but not necessarily a large or performant one A project that lets you "buy a Kafka cluster, get the relational database free" is strictly simpler and cheaper to operate (in theory...) compared to operating both a Kafka cluster and a traditional relational database.
adamnemecek 7y ago
It uses kafka as an operational primitive. Kafka is somewhat complicated but it also solves real problems. I can see a fast, distributed db built on top of kafka.
- qaq 7y ago
  There fast distributed DBs and they are generally built on top of distributed ordered key-value stores. Many of them do not require to run external components like zookeeper. What is the advantage of running one on top of Kafka?
  - Godel_unicode 7y ago
    The main benefit I have heard is that some people already have Kafka.
    - qaq 7y ago
      in what sense is it a benefit?
random023987 7y ago
If you want an asymmetric persistence layer where all writes are documents placed in a Kafka topic, and all reads are via relational queries. It's a distributed, eventually consistent, highly available ledger that can be queried with SQL.
- Scarbutt 7y ago
  Is not eventually consistent, it uses https://omid.incubator.apache.org/ https://omid.incubator.apache.org/.
manigandham 7y ago
It's exciting in the sense that you can apply all these open-source projects to create a very slick lightweight relational database that can replicate changes globally across unlimited instances by using a Kafka topic as the WAL log. Don't actually use this in production. Use a real database that does what you want, and use the native replication features. If you need to replicate to heterogenous or alternative sources or just want a stream of all changes then you can use Kafka with any number of existing database connectors that automatically read and export changes.

amdelamar 7y ago

So, KarelDB makes use of KCache (among other OSS). Which so happens that Kafka uses. But KarelDB does not use Kafka and vice versa. Did I get that right? EDIT: After reading the GitHub README, it does appear KarelDB makes use of Kafka. Its unclear to me if its running kafka brokers underneath, or depends on an existing cluster.

slowmovintarget 7y ago

https://yokota.blog/2019/09/23/building-a-relational-database-using-kafka/ https://yokota.blog/2019/09/23/building-a-relational-databas...

derefr 7y ago

I've been looking into Kafka lately, because I have a need for "the thing that Kafka is in practice" (not a message queue per se; rather, an indefinitely-durable event store that supports message-queuing API semantics for consumers.) And, while Kafka does do this, its infrastructure needs are kind of ridiculous when your producers 1. produce events in clean batches, and 2. could re-produce those events, in the exact same batches, if prompted. Ideally, rather than having to basically have a whole second DBMS in the form of a Kafka cluster (stateful nodes with big HDDs, carefully backed up, needing repair when any die), what I would love is a Kafka-like "event store with MQ semantics", which is in turn backed by an object store. Basically, like Datomic's segment-storage architecture; or like WAL-E in its use-case as an intermediary for Postgres replication. Unlike either of those, though, MQ log-segments don't need any additional metadata (e.g. checkpoints, indices), so they can just be chucked straight in object storage, and MQ nodes can base their view of the world entirely on what happens to be in the object-storage bucket. Of course, this kind of system only makes sense if you have no need for durability of new messages (because e.g. the producer still has the data that went into them, and can resend them), only old messages. Which is kind of an unusual requirement, I suppose, so I don't really expect anyone else to go building this sort of thing for me.

random023987 7y ago
> I would love is an event store You know you can put a key with a null value onto a topic. If your backend object store gives you data by key, this should give you exactly what you want. Later on, you can put a small subset of the data in value to assist with filtering or caching (if you want).
- derefr 7y ago
  I'm not clear on what you're trying to suggest, here. I would like a system that presents itself the same as a Kafka broker, to both producers and consumers (who are just working with regular messages and topics, not object-handles), but where the MQ broker they're talking to is internally just a shared-nothing stateful write-through WAL proxy for an object store. (Sort of like how, with actual logging, you could think of the local rsyslog daemon on an instance as a "shared-nothing stateful WAL-writeback-caching proxy for an object store." Except rsyslog lives on the producer, whereas this write-through WAL proxy would live on the broker.) In such a design, "partitions" and partition ownership would only apply to the newest segment of the log; all older segments would be globally-available, because as soon as a log-segment is finalized, it would get pushed to the object store and become an object. Every node would then be doing Hierarchical Storage Management, with a local LRU cache of segments, populated just-in-time in response to requests, by the object store. (This is what I meant by the comparison to Datomic's storage architecture.) A lot of the technical parts of this could be achieved by just FUSE-mounting an S3 bucket to a directory on each Kafka node, and convincing the node to move log segments into that directory once they're done. But Kafka would still be wasting time and resources replicating segments between nodes that already now "have" them; and also, without further hacking, nodes wouldn't realize that they now also "have" 99% of the segments of every log potentially-available to serve. A system that was built this way from the ground up would end up a lot simpler than Kafka.

atombender 7y ago

Discussed here previously: https://news.ycombinator.com/item?id=21048822 https://news.ycombinator.com/item?id=21048822

mcepl 7y ago

Will the next version be called Josef? (to continue line of the famous authors from Prague)

AtlasBarfed 7y ago

... How does it do distributed joins? Which just don't scale in the AP sense, and really can you trust a distributed join in CP? Unless your data and the data you are joining are strictly shard-local. But that's also not a distributed join.

manigandham 7y ago
It's not a distributed database. It's just a single-node key/value store where all updates are sent to and then read from a Kafka topic, with relational querying and schema management handled by various layered Apache projects. It's a useful tool/model to follow if your data fits on a single node and you want easy replication across all instances.
- AtlasBarfed 7y ago
  Wow, seems like a lot of work when Postgres can just be setup. But if someone needs it why not.
  - manigandham 7y ago
    This isn't a real product, it's an experiment showing off what you can do. Kafka is a distributed log, you can use it as replication log if you want and there are tons of production-ready connectors to all kinds of databases already. Anyone who needs non-native replication (perhaps across different databases or sources) should just use those.

manigandham 7y ago

KCache is an in-memory Java cache where all writes are sent to a Kafka topic and then applied locally by any instances listening to that topic, providing an eventually-consistent cache. Great for scaling a small dataset that can fit on a single-node with easy replication and a full log of all updates in the Kafka topic. KarelDB adds relational querying, schema/table management, transactions, and a SQL interface on top of KCache by using various Apache open-source projects (Calcite, Omid, Avatica, Avro). Here's a blog post from the author that has more details: https://yokota.blog/2019/09/23/building-a-relational-database-using-kafka/ https://yokota.blog/2019/09/23/building-a-relational-databas... A typical database (like Postgres) or cache (like Redis) has a WAL where all writes are applied and this log is also used for replicas. KCache and KarelDB are interesting experiments in doing this from scratch using open-source components, although using Kafka as a durable log of system-agnostic changes is already very common. In fact, it's probably the most used feature of Kafka Connect with various database connectors and changefeed streams.

ako 7y ago
Sounds like Kareldb would make a good alternative to rocksdb, when combining tables and streams in Kafka streaming?