6 ms·

Oh hell yeah! That's great news, tons of work went into this -- props to the contributors! I will take the opportunity to say that Kafka is kind of painful, wi

by cmckn 6y ago

Oh hell yeah! That's great news, tons of work went into this -- props to the contributors! I will take the opportunity to say that Kafka is kind of painful, with or without ZK. Check out NATS! [0]. It doesn't solve all the same problems, but is so much easier to use (during development especially) and can do a lot of the same things. [0]: https://docs.nats.io/whats_new_20 https://docs.nats.io/whats_new_20

deleted 6y ago
[deleted]
vorpalhex 6y ago
NATS isn't actually using a log structure though, it's a streaming message broker with a different set of consistency/delivery promises.
- cmckn 6y ago
  Correct; but I've seen many uses of Kafka that NATS could totally be used for. For example, load balancing across subscribers (use a NATS queue instead of a Kafka consumer group). NATS doesn't ever store messages persistently; but this might be fine for your application, and then you don't have to worry about setting 5 different config options to make sure Kafka actually frees up disk space like you expect it to ;) NATS also enables some unique patterns like request/reply via a "reply to" message header. Anyway, it's been a joy to use!
  - KptMarchewa 6y ago
    Sounds more like rabbitmq replacement than kafka
  - tamale 6y ago
    you also don't have to worry about those kinds of configuration gotchas if you use confluent cloud!
    - jimmyed 6y ago
      Only you have to be a billionaire to use the confluent cloud.
      - incognito_limey 6y ago
        So I guess pay as you go is too expensive for your org? Be curious to understand why you feel it's too expensive when it completely extrapolates any kafka management.
        stalller 6y ago
        relative to more mature Kafka cloud services, it's crazy expensive.
  - shaklee3 6y ago
    > NATS doesn't ever store messages persistently Not true. Both Nats streaming and the upcoming jetstream (core nats) do.
    - tsimionescu 6y ago
      It depends what you mean by 'persistently'. Normally NATS streaming will delete the messages after they have been delivered to all subscribers successsfully and some expiration time has passed.
      - shaklee3 6y ago
        The expiration time can be unlimited, just like kafka
    - cmckn 6y ago
      I'm aware, I was referring to the core binary, `nats-server`. NATS streaming server seems to be still receiving attention, but the client library (for Java) hasn't been committed to since 2019, so I'm not sure I'd build a new project with it. JetStream is out (as of this week, I believe) and is an optional module to `nats-server`, as you said. To the folks at Synadia -- I love NATS, but the naming and organization of these projects could use some work. What's with the `stan.*` repository names? Where did "jetstream" come from? Why is it baked into `nats-server` but `nats-streaming-server` isn't? Is `nats-streaming-server` on the back burner?
      - shaklee3 6y ago
        Stan is nats streaming. The clients don't have to be updated since they're forward compatible since jetstream. NATS streaming will be deprecated after jetstream is GA is my understanding. Is there a bug in the library you found?
simplecto 6y ago
Using Kafka only for ordered pub-sub is like ordering tap water at French Laundry.
- deleted 6y ago
  [deleted]
- Groxx 6y ago
  In that if you're already there and just want water, it's totally reasonable. It'll get you wet. If you went there for tap water, yeah, maybe there are better options.
yes_but_no 6y ago
I believe NATS Streaming[0] and upcoming Nats Jetstream[1] could be more relevant to those who looking for a Kafka alternative. They offer persistent messages, at-least-once delivery similar to Kafka. [0] https://docs.nats.io/nats-streaming-concepts/intro https://docs.nats.io/nats-streaming-concepts/intro [1] https://github.com/nats-io/jetstream https://github.com/nats-io/jetstream
- syscomet 6y ago
  [nb: I work for the main company behind NATS] Jetstream is GA with the 2.2.0 release. Folks who believe in waiting for "not .0" won't have to wait too much longer.
bruth 6y ago
There was a more recent 2.2 release with durable streams [0]. There is also a comparison page including Kafka [1]. [0]: https://docs.nats.io/whats_new_22 https://docs.nats.io/whats_new_22 [1]: https://docs.nats.io/compare-nats https://docs.nats.io/compare-nats
foobiekr 6y ago
NATS is not a replacement for any use of kafka that I think is actually a good fit.
- taywrobel 6y ago
  NATS by itself doesn’t handle the persistence side at all (by design). It’s at most once delivery, not at least once. That being said, have you checked out NATS Streaming Server? It’s effectively a first party client for NATS that gives it at least once semantics and persistence, and makes it much more applicable to use cases that are currently on Kafka. Docs here if you’re curious - https://docs.nats.io/nats-streaming-concepts/intro https://docs.nats.io/nats-streaming-concepts/intro
- SEJeff 6y ago
  Have you seen liftbridge, which was built ontop of nats to fit some of the more traditional kafka use cases? https://liftbridge.io/ https://liftbridge.io/
andrewmutz 6y ago
Does NATS support (or plan to support) log compaction? Without that, it's very hard to replace Kafka.
- cmckn 6y ago
  I"m not sure I understand your question; NATS streaming server (built on top of NATS) supports persistence to disk, a raft group, a SQL table, etc. and it appears the various storage implementations have mechanisms [0] to delete or compress old data. That being said, I don't think this is what differentiates the two systems, the guarantees they do/don't make are likely what will make the decision for your project. [0]: https://github.com/nats-io/nats-streaming-server/blob/master/stores/store.go#L71 https://github.com/nats-io/nats-streaming-server/blob/master...
- shaklee3 6y ago
  Yes, jetstream allows setting max messages to 1 in a stream.
gantengx 6y ago
Is there any message ordering guarantee in NATS? With Kafka you can achieve this by using keyed message and messages in the same partition will always be ordered Disclosure: I work for Confluent
- cmckn 6y ago
  Not the same guarantees, no: > messages from a given single publisher will be delivered to all eligible subscribers in the order in which they were originally published. There are no guarantees of message delivery order amongst multiple publishers. https://docs.nats.io/faq#does-nats-offer-any-guarantee-of-message-ordering https://docs.nats.io/faq#does-nats-offer-any-guarantee-of-me...
- nuharaf 6y ago
  I believe with jetstream, message in stream is ordered as they are written. Jetstream have a concept of consumer, (in the broker itself, not client), which can consume a subset of the stream, filtered by message subject.
- staticassertion 6y ago
  I've never really understood the appeal of ordered messages. You end up splitting your data across partitions anyways for parallelism, so who cares? What systems out there require strictly ordered data? It seems like any design that requires something like that is going to be extremely brittle.
  - geodel 6y ago
    Exactly. We ran into same issue with Kafka. If one needs ordered messages for some reason, Kafka is pretty much useless for this.
  - mrkeen 6y ago
    > You end up splitting your data across partitions anyways Messages are ordered within partitions. > What systems out there require strictly ordered data? It seems like any design that requires something like that is going to be extremely brittle. TCP/IP ?
SEJeff 6y ago
And if you want closer kafka semantics built on top of nats, check out liftbridge: https://liftbridge.io/ https://liftbridge.io/
manigandham 6y ago
NATS is just ephemeral at-most-once pub/sub. It needs NATS Streaming or the new Jetstream for at-least-once persisted data and still has different semantics. Apache Pulsar offers the same distributed log offering with a fundamentally better architecture, but Kafka has closed most of the gaps now and has far more integrations and a bigger ecosystem.

by cmckn 6y ago

deleted 6y ago

[deleted]

vorpalhex 6y ago

NATS isn't actually using a log structure though, it's a streaming message broker with a different set of consistency/delivery promises.

cmckn 6y ago
Correct; but I've seen many uses of Kafka that NATS could totally be used for. For example, load balancing across subscribers (use a NATS queue instead of a Kafka consumer group). NATS doesn't ever store messages persistently; but this might be fine for your application, and then you don't have to worry about setting 5 different config options to make sure Kafka actually frees up disk space like you expect it to ;) NATS also enables some unique patterns like request/reply via a "reply to" message header. Anyway, it's been a joy to use!
- KptMarchewa 6y ago
  Sounds more like rabbitmq replacement than kafka
- tamale 6y ago
  you also don't have to worry about those kinds of configuration gotchas if you use confluent cloud!
  - jimmyed 6y ago
    Only you have to be a billionaire to use the confluent cloud.
    - incognito_limey 6y ago
      So I guess pay as you go is too expensive for your org? Be curious to understand why you feel it's too expensive when it completely extrapolates any kafka management.
      - stalller 6y ago
        relative to more mature Kafka cloud services, it's crazy expensive.
- shaklee3 6y ago
  > NATS doesn't ever store messages persistently Not true. Both Nats streaming and the upcoming jetstream (core nats) do.
  - tsimionescu 6y ago
    It depends what you mean by 'persistently'. Normally NATS streaming will delete the messages after they have been delivered to all subscribers successsfully and some expiration time has passed.
    - shaklee3 6y ago
      The expiration time can be unlimited, just like kafka
  - cmckn 6y ago
    I'm aware, I was referring to the core binary, `nats-server`. NATS streaming server seems to be still receiving attention, but the client library (for Java) hasn't been committed to since 2019, so I'm not sure I'd build a new project with it. JetStream is out (as of this week, I believe) and is an optional module to `nats-server`, as you said. To the folks at Synadia -- I love NATS, but the naming and organization of these projects could use some work. What's with the `stan.*` repository names? Where did "jetstream" come from? Why is it baked into `nats-server` but `nats-streaming-server` isn't? Is `nats-streaming-server` on the back burner?
    - shaklee3 6y ago
      Stan is nats streaming. The clients don't have to be updated since they're forward compatible since jetstream. NATS streaming will be deprecated after jetstream is GA is my understanding. Is there a bug in the library you found?

simplecto 6y ago

Using Kafka only for ordered pub-sub is like ordering tap water at French Laundry.

deleted 6y ago
[deleted]
Groxx 6y ago
In that if you're already there and just want water, it's totally reasonable. It'll get you wet. If you went there for tap water, yeah, maybe there are better options.

yes_but_no 6y ago

I believe NATS Streaming[0] and upcoming Nats Jetstream[1] could be more relevant to those who looking for a Kafka alternative. They offer persistent messages, at-least-once delivery similar to Kafka. [0] https://docs.nats.io/nats-streaming-concepts/intro https://docs.nats.io/nats-streaming-concepts/intro [1] https://github.com/nats-io/jetstream https://github.com/nats-io/jetstream

syscomet 6y ago
[nb: I work for the main company behind NATS] Jetstream is GA with the 2.2.0 release. Folks who believe in waiting for "not .0" won't have to wait too much longer.

bruth 6y ago

There was a more recent 2.2 release with durable streams [0]. There is also a comparison page including Kafka [1]. [0]: https://docs.nats.io/whats_new_22 https://docs.nats.io/whats_new_22 [1]: https://docs.nats.io/compare-nats https://docs.nats.io/compare-nats

foobiekr 6y ago

NATS is not a replacement for any use of kafka that I think is actually a good fit.

taywrobel 6y ago
NATS by itself doesn’t handle the persistence side at all (by design). It’s at most once delivery, not at least once. That being said, have you checked out NATS Streaming Server? It’s effectively a first party client for NATS that gives it at least once semantics and persistence, and makes it much more applicable to use cases that are currently on Kafka. Docs here if you’re curious - https://docs.nats.io/nats-streaming-concepts/intro https://docs.nats.io/nats-streaming-concepts/intro
SEJeff 6y ago
Have you seen liftbridge, which was built ontop of nats to fit some of the more traditional kafka use cases? https://liftbridge.io/ https://liftbridge.io/

andrewmutz 6y ago

Does NATS support (or plan to support) log compaction? Without that, it's very hard to replace Kafka.

cmckn 6y ago
I"m not sure I understand your question; NATS streaming server (built on top of NATS) supports persistence to disk, a raft group, a SQL table, etc. and it appears the various storage implementations have mechanisms [0] to delete or compress old data. That being said, I don't think this is what differentiates the two systems, the guarantees they do/don't make are likely what will make the decision for your project. [0]: https://github.com/nats-io/nats-streaming-server/blob/master/stores/store.go#L71 https://github.com/nats-io/nats-streaming-server/blob/master...
shaklee3 6y ago
Yes, jetstream allows setting max messages to 1 in a stream.

gantengx 6y ago

Is there any message ordering guarantee in NATS? With Kafka you can achieve this by using keyed message and messages in the same partition will always be ordered Disclosure: I work for Confluent

cmckn 6y ago
Not the same guarantees, no: > messages from a given single publisher will be delivered to all eligible subscribers in the order in which they were originally published. There are no guarantees of message delivery order amongst multiple publishers. https://docs.nats.io/faq#does-nats-offer-any-guarantee-of-message-ordering https://docs.nats.io/faq#does-nats-offer-any-guarantee-of-me...
nuharaf 6y ago
I believe with jetstream, message in stream is ordered as they are written. Jetstream have a concept of consumer, (in the broker itself, not client), which can consume a subset of the stream, filtered by message subject.
staticassertion 6y ago
I've never really understood the appeal of ordered messages. You end up splitting your data across partitions anyways for parallelism, so who cares? What systems out there require strictly ordered data? It seems like any design that requires something like that is going to be extremely brittle.
- geodel 6y ago
  Exactly. We ran into same issue with Kafka. If one needs ordered messages for some reason, Kafka is pretty much useless for this.
- mrkeen 6y ago
  > You end up splitting your data across partitions anyways Messages are ordered within partitions. > What systems out there require strictly ordered data? It seems like any design that requires something like that is going to be extremely brittle. TCP/IP ?

SEJeff 6y ago

And if you want closer kafka semantics built on top of nats, check out liftbridge: https://liftbridge.io/ https://liftbridge.io/

manigandham 6y ago

NATS is just ephemeral at-most-once pub/sub. It needs NATS Streaming or the new Jetstream for at-least-once persisted data and still has different semantics. Apache Pulsar offers the same distributed log offering with a fundamentally better architecture, but Kafka has closed most of the gaps now and has far more integrations and a bigger ecosystem.