5 ms·

Looks like this also uses Zookeeper. Does anyone know of a simple streaming log system / database? Like, SQLite3 for streaming? I'm using this for personal proj

Looks like this also uses Zookeeper. Does anyone know of a simple streaming log system / database? Like, SQLite3 for streaming? I'm using this for personal projects more and more, and the solutions I see in this space are always big, distributed and hard to setup and keep running. I've been using a simple file format that just writes each message out sequentially in a simple format like: [event-id][event-type][event-size][event-bytes]. And there's a small TCP server that speaks Redis protocol to support remote access. But it's not really production code, rather something I hacked together over a couple weeks in the evening.

zcw100 7y ago
https://nats.io/ https://nats.io/
- manigandham 7y ago
  Note that NATS on it's own is just a pub/sub system and doesn't have any queuing or persistence. There is no "log". You need the NATS Streaming server for that which implements persistence while communicating over NATS.
  - afrodc_ 7y ago
    I was looking into https://github.com/liftbridge-io/liftbridge https://github.com/liftbridge-io/liftbridge. I believe he was a maintainer of NATS streaming and wanted to make something lightweight and Kafka-esque.
    - corford 7y ago
      Liftbridge looks cool but right now you need use their protobuf defs and cook your own client lib if you want bindings outside of go.
    - manigandham 7y ago
      Liftbridge is an experiment and has no real production users. NATS Streaming is already fine for lightweight single-host usage. If you really need to scale to multiple servers then I recommend skipping NATS Streaming and going straight to Kafka or Apache Pulsar.
      - dmix 7y ago
        Have any more lighter-weight recommendations that fit the queuing and persistence category?
        manigandham 7y ago
        RabbitMQ is another option. Clustering is fragile, but it's fine if you just need a single-node: https://www.rabbitmq.com/ https://www.rabbitmq.com/ There are also proprietary systems like Solace: https://solace.com/ https://solace.com/ Other than that, there are embedded queuing/log libraries or you can just dump the messages into a key/value store or write them out to a file yourself.
elvinyung 7y ago
Just curious, have you looked into Redis streams?
manigandham 7y ago
If you're already using the Redis protocol then Redis v5 has Streams as a first-class data structure. It's fast, has consumer groups (like Kafka), and also supports individual message acknowledgement.
- atombender 7y ago
  Is Redis Streams backed by disk? How does it perform when the size of the stream exceeds available RAM? Also, what's Redis' multi-node failover/high availability story these days (with streams)? Last I heard, it wasn't that great [1], but it's been a while. [1] https://aphyr.com/posts/283-jepsen-redis https://aphyr.com/posts/283-jepsen-redis
  - manigandham 7y ago
    Redis is an in-memory data store that has different options for persistence (snapshots + oplog) but it's not designed to persist every operation immediately. All data structures are covered including Streams. Redis keeps the entire working set in RAM so it'll start dropping writes or freeze if you run out of memory. This is where the simplicity and speed comes from and is a fundamental limitation. There's a simple replica system that works well but failover switching is the problem and requires a separate process or running the Redis Sentinel. There's also Redis Cluster but that just shards the keyspace and doesn't offer any scalability with a single key or stream, and is still hard to manage with the same failover issues. The OP asked for a single-node option so I suggested Redis, if you need a serious messaging cluster then I recommend Kafka or Pulsar instead.
    - atombender 7y ago
      Thanks. That confirms pretty much the picture I had in my head of how Redis works these days. It's frustrating that there's no obvious middle ground this and Kafka and Pulsar, both of which are memory-hungry JVM apps with multiple external dependencies. Both require ZooKeeper; Pulsar also requires BookKeeper. None of these components are operationally simple. I'm a fan of NATS itself, but NATS Streaming's clustering design leaves a lot to be desired. In particular, it punts on failover/HA, asking you to instead run an SQL database or shared file system that provides this. (An obvious low-maintenance option here would be CockroachDB, but NATS doesn't support it.)
corford 7y ago
Redis streams?
Xorlev 7y ago
https://github.com/xorlev/slogd https://github.com/xorlev/slogd This is a project of mine I use for some side projects for exactly that reason: I want streaming logs & pubsub semantics. It's basically Kafka xtra-lite. It doesn't have a super simple TCP protocol (though I wouldn't say no to a PR adding one!), it's dual-available over gRPC and HTTP (JSON API). Disclaimer: It's definitely one of my low-activity side projects and subject to change at any time.
james-mcelwain 7y ago
Kafka is moving towards eliminating the ZK dependency, while will make it easier to run as a single node if you don't care about HA.
- orthecreedence 7y ago
  Hey, is there a tracker somewhere for this? I've always been into trying Kafka, but I heard horror stories about ZK so having it removed would be nice.
graetzer 7y ago
Maybe the pub-sub pattern of https://zeromq.org/ https://zeromq.org/ could work for that

zcw100 7y ago

https://nats.io/ https://nats.io/

manigandham 7y ago
Note that NATS on it's own is just a pub/sub system and doesn't have any queuing or persistence. There is no "log". You need the NATS Streaming server for that which implements persistence while communicating over NATS.
- afrodc_ 7y ago
  I was looking into https://github.com/liftbridge-io/liftbridge https://github.com/liftbridge-io/liftbridge. I believe he was a maintainer of NATS streaming and wanted to make something lightweight and Kafka-esque.
  - corford 7y ago
    Liftbridge looks cool but right now you need use their protobuf defs and cook your own client lib if you want bindings outside of go.
  - manigandham 7y ago
    Liftbridge is an experiment and has no real production users. NATS Streaming is already fine for lightweight single-host usage. If you really need to scale to multiple servers then I recommend skipping NATS Streaming and going straight to Kafka or Apache Pulsar.
    - dmix 7y ago
      Have any more lighter-weight recommendations that fit the queuing and persistence category?
      - manigandham 7y ago
        RabbitMQ is another option. Clustering is fragile, but it's fine if you just need a single-node: https://www.rabbitmq.com/ https://www.rabbitmq.com/ There are also proprietary systems like Solace: https://solace.com/ https://solace.com/ Other than that, there are embedded queuing/log libraries or you can just dump the messages into a key/value store or write them out to a file yourself.

elvinyung 7y ago

Just curious, have you looked into Redis streams?

manigandham 7y ago

If you're already using the Redis protocol then Redis v5 has Streams as a first-class data structure. It's fast, has consumer groups (like Kafka), and also supports individual message acknowledgement.

atombender 7y ago
Is Redis Streams backed by disk? How does it perform when the size of the stream exceeds available RAM? Also, what's Redis' multi-node failover/high availability story these days (with streams)? Last I heard, it wasn't that great [1], but it's been a while. [1] https://aphyr.com/posts/283-jepsen-redis https://aphyr.com/posts/283-jepsen-redis
- manigandham 7y ago
  Redis is an in-memory data store that has different options for persistence (snapshots + oplog) but it's not designed to persist every operation immediately. All data structures are covered including Streams. Redis keeps the entire working set in RAM so it'll start dropping writes or freeze if you run out of memory. This is where the simplicity and speed comes from and is a fundamental limitation. There's a simple replica system that works well but failover switching is the problem and requires a separate process or running the Redis Sentinel. There's also Redis Cluster but that just shards the keyspace and doesn't offer any scalability with a single key or stream, and is still hard to manage with the same failover issues. The OP asked for a single-node option so I suggested Redis, if you need a serious messaging cluster then I recommend Kafka or Pulsar instead.
  - atombender 7y ago
    Thanks. That confirms pretty much the picture I had in my head of how Redis works these days. It's frustrating that there's no obvious middle ground this and Kafka and Pulsar, both of which are memory-hungry JVM apps with multiple external dependencies. Both require ZooKeeper; Pulsar also requires BookKeeper. None of these components are operationally simple. I'm a fan of NATS itself, but NATS Streaming's clustering design leaves a lot to be desired. In particular, it punts on failover/HA, asking you to instead run an SQL database or shared file system that provides this. (An obvious low-maintenance option here would be CockroachDB, but NATS doesn't support it.)

corford 7y ago

Redis streams?

Xorlev 7y ago

https://github.com/xorlev/slogd https://github.com/xorlev/slogd This is a project of mine I use for some side projects for exactly that reason: I want streaming logs & pubsub semantics. It's basically Kafka xtra-lite. It doesn't have a super simple TCP protocol (though I wouldn't say no to a PR adding one!), it's dual-available over gRPC and HTTP (JSON API). Disclaimer: It's definitely one of my low-activity side projects and subject to change at any time.

james-mcelwain 7y ago

Kafka is moving towards eliminating the ZK dependency, while will make it easier to run as a single node if you don't care about HA.

orthecreedence 7y ago
Hey, is there a tracker somewhere for this? I've always been into trying Kafka, but I heard horror stories about ZK so having it removed would be nice.

graetzer 7y ago

Maybe the pub-sub pattern of https://zeromq.org/ https://zeromq.org/ could work for that