6 ms·

Show HN: Liftbridge – Lightweight, fault-tolerant message streams

147 points by tylertreat 6y ago

kodablah 6y ago
I've been watching this with some interest as the successor of NATS streaming. One question I have is how does it fair in potential-high-latency, geographically distributed situations? E.g. what if I had dozens of servers at the edge worldwide wanting to receive events from one another?
- tylertreat 6y ago
  Good question. It's early days, so there's still a lot of work to be done around geo-replication/geo-aware. It depends on the specific use case, but there are a few approaches you could take. You could run a Liftbridge and NATS cluster in each DC with the NATS cluster connected to a global NATS supercluster. Alternatively, you could just run separate Liftbridge clusters and do replication between them (sort of like the MirrorMaker approach in Kafka, though the tooling here is still lacking for Liftbridge). Longer term, I'd like to take advantage of the recent work in NATS around superclusters and geo-aware subscribers to provide better geo-replication primitives directly inside Liftbridge (https://www.slideshare.net/nats_io/deploy-secure-and-scalable-services-across-kubernetes-clusters-with-nats https://www.slideshare.net/nats_io/deploy-secure-and-scalabl...).
- paddybyers 6y ago
  We've been experimenting with Liftbridge at a range of scales and workloads, and clusters that are distributed across a single region perform very well. However, it's not really suitable for operating a cluster distributed across multiple regions because the latency adversely impacts the bandwidth of the Raft protocol that's used for cluster-wide coordination; this impacts the rate at which streams can be created, and the rate at which changes to In-Sync Replica sets (ISRs) can be propagated. As suggested, a federation of clusters, each covering a single region, would be required.
tylertreat 6y ago
Author here. I started Liftbridge after working as a core committer on NATS and NATS Streaming (as well as my experience using Kafka in the past). In particular, implementing replication in NATS Streaming and drawing from that experience/lessons learned. Happy to answer any questions!
- tutfbhuf 6y ago
  Could you explain the differences between Liftbridge and Kafka?
  - tylertreat 6y ago
    Some of the key differences: - Written in Go rather than Java/Scala (big benefit here IMO is getting a small static binary rather than having to run a JVM) - Doesn't rely on Zookeeper - Is integrated with NATS (can extend NATS with Kafka-like semantics, but in the future will allow for abstracting NATS away entirely) - Uses gRPC for its API - Supports "wildcard topics" - e.g. a stream can listen to the topic "foo.*" and will receive messages published on "foo.bar", "foo.baz", "foo.qux", etc. - Allows for streams to be paused and subsequently resumed automatically when published to - on the roadmap is "auto-pausing" of sparsely used streams - Exposes an activity stream that allows you to respond to events such as streams being created, deleted, paused, etc. One advantage Kafka currently has is its consumer groups, which are on the roadmap for Liftbridge but not implemented yet. There's a more complete comparison available here: https://liftbridge.io/docs/feature-comparison.html https://liftbridge.io/docs/feature-comparison.html
    - DLA 6y ago
      And for exactly these reasons I absolutely LOVE this project. Big Go user. Finding this today was a pure gift. Thank you very much for making these excellent design choices and for building this! I hope to contribute in the future.
    - jaytaylor 6y ago
      For those not already familiar with NATS: https://docs.nats.io/nats-streaming-concepts/intro https://docs.nats.io/nats-streaming-concepts/intro
berkayozturk 6y ago
For those who are wondering the difference between Kafka/Pulsar and Liftbridge (from the docs): "Liftbridge was designed to bridge the gap between sophisticated but complex log-based messaging systems like Apache Kafka and Apache Pulsar and simpler, cloud-native solutions. There is no ZooKeeper or other unwieldy dependencies, no JVM, no complicated API or configuration, and client libraries are just gRPC. More importantly, Liftbridge aims to extend NATS with a durable, at-least-once delivery mechanism that upholds the NATS tenets of simplicity, performance, and scalability. Unlike NATS Streaming, it uses the core NATS protocol with optional extensions. This means it can be added to an existing NATS deployment to provide message durability with no code changes. The ultimate goal of Liftbridge is to provide a message-streaming solution with a focus on simplicity and usability."
- ignoramous 6y ago
  NATS' author, u/tylertreat, wrote a series of articles on how to build a distributed log that are pretty insightful: Part 1, Storage Mechanics: https://news.ycombinator.com/item?id=15983185 https://news.ycombinator.com/item?id=15983185 Part 2, Data Replication: https://news.ycombinator.com/item?id=16021876 https://news.ycombinator.com/item?id=16021876 Part 3, Message Delivery: https://news.ycombinator.com/item?id=16101223 https://news.ycombinator.com/item?id=16101223 Part 4, Trade-offs and Lessons: https://news.ycombinator.com/item?id=16181963 https://news.ycombinator.com/item?id=16181963 Part 5, Sketching a New System: https://news.ycombinator.com/item?id=16215788 https://news.ycombinator.com/item?id=16215788
  - atombender 6y ago
    I think you mean Liftbridge's author. NATS was created by Derek Collison [1], co-founder of CloudFoundry. [1] https://news.ycombinator.com/user?id=derekcollison https://news.ycombinator.com/user?id=derekcollison
kiwicopple 6y ago
This is very cool OP. Amazing work. For anyone looking for something similar, we are adding streams to Postgres - https://github.com/supabase/realtime https://github.com/supabase/realtime It isn’t an exact comparison but it has some of the features you see here too - you can use wildcards listen to: all changes in your database, schema-level, table-level changes, or row-level. The server is built with Elixir, so you can have thousands of listeners connected via web sockets. It’s not as lightweight as OP’s solution but PG is very familiar.
- davedx 6y ago
  This is really exciting, nice! We've just started using postgres for our new services and the other day I was just thinking to myself "wouldn't it be great if I could listen to certain database tables because I can then build a real-time data warehouse really easily". How far along is the implementation?
  - kiwicopple 6y ago
    Thanks for the feedback @davedx. Actually it's being used in production by a few companies already (not small companies either). We have a few things we want to do before it's "enterprise ready": - set up elixir clustering (i.e. autoscaling) - run benchmarks - fine-grained client auth - ability to listen to multiple databases > then build a real-time data warehouse really easily This is the nice thing about PG. It's an operational database that can handle the load of an analytics database. Easy choice! If you need a hand trying this out, message me: copple [at] supabase.io
- alexnewman 6y ago
  We made https://github.com/instructure/jsoncdc https://github.com/instructure/jsoncdc in rust which can do much of the same
fcanesin 6y ago
How does it compare in performance?
bamazizi 6y ago
The project has basically jumped from v0.0.1 to v1.0.0 within 4-5 months ... but how does it fare with reliability, throughput, ordering and deliveries? Is it PROD ready? are there any use-cases or project sizes this utility is NOT recommended for? i.e. sensitive events such as payment or ecomm related vs. web traffic noise logs

147 points by tylertreat 6y ago

I've been watching this with some interest as the successor of NATS streaming. One question I have is how does it fair in potential-high-latency, geographically distributed situations? E.g. what if I had dozens of servers at the edge worldwide wanting to receive events from one another?

tylertreat 6y ago
Good question. It's early days, so there's still a lot of work to be done around geo-replication/geo-aware. It depends on the specific use case, but there are a few approaches you could take. You could run a Liftbridge and NATS cluster in each DC with the NATS cluster connected to a global NATS supercluster. Alternatively, you could just run separate Liftbridge clusters and do replication between them (sort of like the MirrorMaker approach in Kafka, though the tooling here is still lacking for Liftbridge). Longer term, I'd like to take advantage of the recent work in NATS around superclusters and geo-aware subscribers to provide better geo-replication primitives directly inside Liftbridge (https://www.slideshare.net/nats_io/deploy-secure-and-scalable-services-across-kubernetes-clusters-with-nats https://www.slideshare.net/nats_io/deploy-secure-and-scalabl...).
paddybyers 6y ago
We've been experimenting with Liftbridge at a range of scales and workloads, and clusters that are distributed across a single region perform very well. However, it's not really suitable for operating a cluster distributed across multiple regions because the latency adversely impacts the bandwidth of the Raft protocol that's used for cluster-wide coordination; this impacts the rate at which streams can be created, and the rate at which changes to In-Sync Replica sets (ISRs) can be propagated. As suggested, a federation of clusters, each covering a single region, would be required.

tylertreat 6y ago

Author here. I started Liftbridge after working as a core committer on NATS and NATS Streaming (as well as my experience using Kafka in the past). In particular, implementing replication in NATS Streaming and drawing from that experience/lessons learned. Happy to answer any questions!

tutfbhuf 6y ago
Could you explain the differences between Liftbridge and Kafka?
- tylertreat 6y ago
  Some of the key differences: - Written in Go rather than Java/Scala (big benefit here IMO is getting a small static binary rather than having to run a JVM) - Doesn't rely on Zookeeper - Is integrated with NATS (can extend NATS with Kafka-like semantics, but in the future will allow for abstracting NATS away entirely) - Uses gRPC for its API - Supports "wildcard topics" - e.g. a stream can listen to the topic "foo.*" and will receive messages published on "foo.bar", "foo.baz", "foo.qux", etc. - Allows for streams to be paused and subsequently resumed automatically when published to - on the roadmap is "auto-pausing" of sparsely used streams - Exposes an activity stream that allows you to respond to events such as streams being created, deleted, paused, etc. One advantage Kafka currently has is its consumer groups, which are on the roadmap for Liftbridge but not implemented yet. There's a more complete comparison available here: https://liftbridge.io/docs/feature-comparison.html https://liftbridge.io/docs/feature-comparison.html
  - DLA 6y ago
    And for exactly these reasons I absolutely LOVE this project. Big Go user. Finding this today was a pure gift. Thank you very much for making these excellent design choices and for building this! I hope to contribute in the future.
  - jaytaylor 6y ago
    For those not already familiar with NATS: https://docs.nats.io/nats-streaming-concepts/intro https://docs.nats.io/nats-streaming-concepts/intro

berkayozturk 6y ago

For those who are wondering the difference between Kafka/Pulsar and Liftbridge (from the docs): "Liftbridge was designed to bridge the gap between sophisticated but complex log-based messaging systems like Apache Kafka and Apache Pulsar and simpler, cloud-native solutions. There is no ZooKeeper or other unwieldy dependencies, no JVM, no complicated API or configuration, and client libraries are just gRPC. More importantly, Liftbridge aims to extend NATS with a durable, at-least-once delivery mechanism that upholds the NATS tenets of simplicity, performance, and scalability. Unlike NATS Streaming, it uses the core NATS protocol with optional extensions. This means it can be added to an existing NATS deployment to provide message durability with no code changes. The ultimate goal of Liftbridge is to provide a message-streaming solution with a focus on simplicity and usability."

ignoramous 6y ago
NATS' author, u/tylertreat, wrote a series of articles on how to build a distributed log that are pretty insightful: Part 1, Storage Mechanics: https://news.ycombinator.com/item?id=15983185 https://news.ycombinator.com/item?id=15983185 Part 2, Data Replication: https://news.ycombinator.com/item?id=16021876 https://news.ycombinator.com/item?id=16021876 Part 3, Message Delivery: https://news.ycombinator.com/item?id=16101223 https://news.ycombinator.com/item?id=16101223 Part 4, Trade-offs and Lessons: https://news.ycombinator.com/item?id=16181963 https://news.ycombinator.com/item?id=16181963 Part 5, Sketching a New System: https://news.ycombinator.com/item?id=16215788 https://news.ycombinator.com/item?id=16215788
- atombender 6y ago
  I think you mean Liftbridge's author. NATS was created by Derek Collison [1], co-founder of CloudFoundry. [1] https://news.ycombinator.com/user?id=derekcollison https://news.ycombinator.com/user?id=derekcollison

kiwicopple 6y ago

This is very cool OP. Amazing work. For anyone looking for something similar, we are adding streams to Postgres - https://github.com/supabase/realtime https://github.com/supabase/realtime It isn’t an exact comparison but it has some of the features you see here too - you can use wildcards listen to: all changes in your database, schema-level, table-level changes, or row-level. The server is built with Elixir, so you can have thousands of listeners connected via web sockets. It’s not as lightweight as OP’s solution but PG is very familiar.

davedx 6y ago
This is really exciting, nice! We've just started using postgres for our new services and the other day I was just thinking to myself "wouldn't it be great if I could listen to certain database tables because I can then build a real-time data warehouse really easily". How far along is the implementation?
- kiwicopple 6y ago
  Thanks for the feedback @davedx. Actually it's being used in production by a few companies already (not small companies either). We have a few things we want to do before it's "enterprise ready": - set up elixir clustering (i.e. autoscaling) - run benchmarks - fine-grained client auth - ability to listen to multiple databases > then build a real-time data warehouse really easily This is the nice thing about PG. It's an operational database that can handle the load of an analytics database. Easy choice! If you need a hand trying this out, message me: copple [at] supabase.io
alexnewman 6y ago
We made https://github.com/instructure/jsoncdc https://github.com/instructure/jsoncdc in rust which can do much of the same

fcanesin 6y ago

How does it compare in performance?

bamazizi 6y ago

The project has basically jumped from v0.0.1 to v1.0.0 within 4-5 months ... but how does it fare with reliability, throughput, ordering and deliveries? Is it PROD ready? are there any use-cases or project sizes this utility is NOT recommended for? i.e. sensitive events such as payment or ecomm related vs. web traffic noise logs