4 ms·

Queues like Kafka can be great for a certain class of solution, but I see them over used. The justification is usually a sort of handwave about scaling, but I

Queues like Kafka can be great for a certain class of solution, but I see them over used. The justification is usually a sort of handwave about scaling, but I tend to be skeptical of that argument. I remember in the early 2000's writing a system that handled all of the consumer equipment orders and returns for Verizon, pick pack and ship and advanced billing integration with the ILEC phone bill, credit systems, IVR, etc... On a PII class server running SQL server stored procedures and a .net UI. In 2009, I was processing a TB per month of 837 claims on even worse hardware. Not that long ago, I saw a post on HN where a Casio calculator was handling the front page traffic. My point is that even modest hardware has the capability to handle massive loads when the software is designed properly. Kafka and it's ilk are great for certain use cases, but it's normally a mistake to start out with it unless you're really sure you know what you're doing. CQRS and data-bus/event sourced systems can be tricky to get right, and you lose a lot when you go to async processing, especially when submitting transactions from a user-facing app. Inevitably you have to create some sort of polling or server push mechanism to update the UI, and the number of edge cases that can bite you are vast.

nickpeterson 4y ago
This has always been true. Some tight C code running on a raspberry pi could probably handle most of the data needs for ordinary businesses today. But to acknowledge this reality makes CIO's sweat ;)
tstrimple 4y ago
What? Calling Kafka a queue seems to be a fundamental misunderstanding what it is as a tool. Why would you be polling in Kafka for real-time updates to a UI? That's not necessary at all, a topic subscriber will automatically pick up new messages and can send them via web socket. Kafka doesn't mean databases go away, so there's absolutely zero reason for any sort of issues submitting transactions from a user-facing app. It's almost like you're not really talking about Kafka.
- ramenmeal 4y ago
  He mentioned "or server push mechanism", aka websockets. I don't think they're trying to call kafka a queue, they're just talking about asynchronous architectures.
xupybd 4y ago
I'm about to start implementing this sort of thing myself. We have a handful of webservices, an online store, a customer design tool and some third party connections to customers and suppliers. Then we have a factory of machines and operator stations that need to fetch and display information. My thinking was to use something like Kafka, leaning towards NATS, as a big pub sub pipe. I'm not looking for scaling. We are slow and don't need real time. What I need is uniformity. I don't want this connection to interface through FTP while that one is over a REST API. I'd prefer one connection for our main system and little interfaces to connect everything to the same pipe.
- codegangsta 4y ago
  Disclaimer: NATS maintainer NATS definitely tries to solve for that end to end experience, where you can have NATS simultaneously driving websocket connections as well as backend connections in a secure and performant way. I’ve personally had a ton of fun making interactive applications with NATS, it scales reliably and is really fun to work with
paulmd 4y ago
> Queues like Kafka can be great for a certain class of solution, but I see them over used. > The justification is usually a sort of handwave about scaling, but I tend to be skeptical of that argument. actually this sort of solution is a great fit to GPGPU, where each thread has some kind of a "window" and will perform N aligned accesses in a stepwise fashion to compute its particular data. Aligned/coalesced accesses and "re-computation" are very cheap in GPGPU and that often means you lean on re-computation rather than notarization or other "stateful" approaches. I think something like this would be a great fit to AVX-512 probably.
EdwardDiego 4y ago
Kafka's not a message queue, it's a distributed log. The main advantage of Kafka is that you can: 1. Write a shitload of data to it fast 2. Experience a cluster failure or degradation with low risk of data loss 3. Retain absolute ordering, with caveats, if you're careful. 4. Run large numbers of concurrent consumers without imposing significant overhead on the cluster That's pretty much it. I've made a good career the last few years out of Kafka, but there are _so_ many companies using it that don't need it, and what strikes me as passing odd is when I've been consulting on their Kafka adoption and tell them that, they don't want to hear it. Which is really weird IMO because I stand to make far more money by encouraging them to use it unnecessarily because... ...Kafka is a complex solution to a complicated problem, and often people using it don't have the sheer scale of data that would necessitate taking on that complexity. > In 2009, I was processing a TB per month of 837 claims on even worse hardware. Kafka starts to shine when you're creating X TiB of data a day, and would like not to lose it. It's best mentally modelled as a series of rather resilient tubes, that way people can stop thinking it's just a super duper scalable message queue, and they can just drop it in as a black box and get all the semantics they're used to from ActiveMQ / RabbitMQ etc. Even Pulsar, which tries to emulate MQ semantics, isn't a drop-in replacement, at all. And it has even more moving parts than Kafka.
- Kinrany 4y ago
  Thoughts on NATS?
  - codegangsta 4y ago
    Disclaimer: I work on NATS and am employed by Synadia, maintainer of NATS. NATS is much more lightweight than the alternatives, and has a much simpler operation story and can run on low resource hardware very easily. Kafka fits a very specific use case, and NATS tries to solve a much larger set of distributed computing use cases, including micro-services, distributed KV, streaming, pub/sub, object store and more. It’s a very different philosophy, but can be very powerful when it’s deployed as a utility inside an organization. We’ve seen so many success stories, especially companies moving to the edge, we continue to see NATS used as a common transport for all kinds of use cases in finance, IoT, edge, Industry 4.0, and Retail
lmm 4y ago
You don't need to scale for performance, but you do for reliability. SQL databases still tend to rely on having one server that never goes down, or at best having a replica and a semi-manual "promotion" procedure. Kafka does what a datastore should and nothing more. It's what databases should have been. High performance is a consequence of that good design, but the good design is valuable in itself. I would always start with Kafka. There are so many ways to shoot yourself in the foot with an SQL database that will bite you later, sometimes in a way that may corrupt your data permanently (e.g. read-modify-write logic), and you'll almost certainly end up having to change your data architecture to the one you'd've started with if you were using Kafka. Or you can just throw it in Kafka from day 1 and a lot of bad ideas just become impossible because the system gives you no way to do them. I was also working on phone systems in the 2000's that ran at "web scale". We used MySQL, but we essentially built Kafka on top of MySQL - tables for every intermediate step in the process, and a way for the next step in the processing pipeline to mark which rows it had started/finished consuming.
oweiler 4y ago
I agree that Kafka is overused. But IMHO Kafka's strength is not scaling but decoupling. Services can publish data without other interested parties being online. If you need this, is obviously another question.
- happymellon 4y ago
  That is true of most pub/sub systems though. Kafka has a lot of other "features" which is what makes it overkill for most situations.
happymellon 4y ago
All the times I've used Kafka, it's felt like the microservices confusion. A technical solution that can be taken to the extreme, when really it's a solution to a political problem. If we break our application into small enough chunks then we start running into internal consistency issues, so we will introduce Kafka to resolve this. I'm dealing with it at the moment where my customer decided to break their crud application to CUD and R because "best practice". Now they have Kafka synchronising the two applications because it's a terrible design.
- bicijay 4y ago
  You are talking about CQRS, which is far from a "terrible design", maybe not right for your customer?
  - happymellon 4y ago
    I understand that it might help some people, and I am referring to my clients desire as terrible rather than the whole concept. However I have yet to actually see it used without it increasing complexity without demonstrable improvements. I see that Martin Fowler agrees. https://martinfowler.com/bliki/CQRS.html https://martinfowler.com/bliki/CQRS.html It falls into the "you are not Google" problem solving hole. I'm dealing with a financial institution doing this. CQRS should be an evolution if you find it complex to model, use, and scale not the initial implementation. Now we have another location to lose data.
edude03 4y ago
> Queues like Kafka Kafka is a distributed log. A sibling comment to this mentions this as well, but I'm repeating it because I actually think that's were most people using kafka go wrong. You _can_ use kafka as a queue, but it'll would feel like using a drill to hammer a nail, would kind of work but would make you think the tool just sucks. > The justification is usually a sort of handwave about scaling, but I tend to be skeptical of that argument. Kafka does help with scaling in so far as you can independently scale the number of producers and consumers but I find kafka the most useful in its ability to allow multiple consumers to process the same message(s) at different rates. For example let's say you have an app that's producing messages like "{foo: 1, bar: 2 }" and you need to count how many foos there are and how many bars there are. You could queue the messages, then have 1 service that does both, or you could use kafka and have a FooCounter and BarCounter service. They each process the same message(s) but ignore the part of the payload they don't care about. While this is really trivial example, I've used, and have seen this pattern used for processing actual logs for example.

nickpeterson 4y ago

This has always been true. Some tight C code running on a raspberry pi could probably handle most of the data needs for ordinary businesses today. But to acknowledge this reality makes CIO's sweat ;)

tstrimple 4y ago

What? Calling Kafka a queue seems to be a fundamental misunderstanding what it is as a tool. Why would you be polling in Kafka for real-time updates to a UI? That's not necessary at all, a topic subscriber will automatically pick up new messages and can send them via web socket. Kafka doesn't mean databases go away, so there's absolutely zero reason for any sort of issues submitting transactions from a user-facing app. It's almost like you're not really talking about Kafka.

ramenmeal 4y ago
He mentioned "or server push mechanism", aka websockets. I don't think they're trying to call kafka a queue, they're just talking about asynchronous architectures.

xupybd 4y ago

I'm about to start implementing this sort of thing myself. We have a handful of webservices, an online store, a customer design tool and some third party connections to customers and suppliers. Then we have a factory of machines and operator stations that need to fetch and display information. My thinking was to use something like Kafka, leaning towards NATS, as a big pub sub pipe. I'm not looking for scaling. We are slow and don't need real time. What I need is uniformity. I don't want this connection to interface through FTP while that one is over a REST API. I'd prefer one connection for our main system and little interfaces to connect everything to the same pipe.

codegangsta 4y ago
Disclaimer: NATS maintainer NATS definitely tries to solve for that end to end experience, where you can have NATS simultaneously driving websocket connections as well as backend connections in a secure and performant way. I’ve personally had a ton of fun making interactive applications with NATS, it scales reliably and is really fun to work with

paulmd 4y ago

> Queues like Kafka can be great for a certain class of solution, but I see them over used. > The justification is usually a sort of handwave about scaling, but I tend to be skeptical of that argument. actually this sort of solution is a great fit to GPGPU, where each thread has some kind of a "window" and will perform N aligned accesses in a stepwise fashion to compute its particular data. Aligned/coalesced accesses and "re-computation" are very cheap in GPGPU and that often means you lean on re-computation rather than notarization or other "stateful" approaches. I think something like this would be a great fit to AVX-512 probably.

EdwardDiego 4y ago

Kafka's not a message queue, it's a distributed log. The main advantage of Kafka is that you can: 1. Write a shitload of data to it fast 2. Experience a cluster failure or degradation with low risk of data loss 3. Retain absolute ordering, with caveats, if you're careful. 4. Run large numbers of concurrent consumers without imposing significant overhead on the cluster That's pretty much it. I've made a good career the last few years out of Kafka, but there are _so_ many companies using it that don't need it, and what strikes me as passing odd is when I've been consulting on their Kafka adoption and tell them that, they don't want to hear it. Which is really weird IMO because I stand to make far more money by encouraging them to use it unnecessarily because... ...Kafka is a complex solution to a complicated problem, and often people using it don't have the sheer scale of data that would necessitate taking on that complexity. > In 2009, I was processing a TB per month of 837 claims on even worse hardware. Kafka starts to shine when you're creating X TiB of data a day, and would like not to lose it. It's best mentally modelled as a series of rather resilient tubes, that way people can stop thinking it's just a super duper scalable message queue, and they can just drop it in as a black box and get all the semantics they're used to from ActiveMQ / RabbitMQ etc. Even Pulsar, which tries to emulate MQ semantics, isn't a drop-in replacement, at all. And it has even more moving parts than Kafka.

Kinrany 4y ago
Thoughts on NATS?
- codegangsta 4y ago
  Disclaimer: I work on NATS and am employed by Synadia, maintainer of NATS. NATS is much more lightweight than the alternatives, and has a much simpler operation story and can run on low resource hardware very easily. Kafka fits a very specific use case, and NATS tries to solve a much larger set of distributed computing use cases, including micro-services, distributed KV, streaming, pub/sub, object store and more. It’s a very different philosophy, but can be very powerful when it’s deployed as a utility inside an organization. We’ve seen so many success stories, especially companies moving to the edge, we continue to see NATS used as a common transport for all kinds of use cases in finance, IoT, edge, Industry 4.0, and Retail

lmm 4y ago

You don't need to scale for performance, but you do for reliability. SQL databases still tend to rely on having one server that never goes down, or at best having a replica and a semi-manual "promotion" procedure. Kafka does what a datastore should and nothing more. It's what databases should have been. High performance is a consequence of that good design, but the good design is valuable in itself. I would always start with Kafka. There are so many ways to shoot yourself in the foot with an SQL database that will bite you later, sometimes in a way that may corrupt your data permanently (e.g. read-modify-write logic), and you'll almost certainly end up having to change your data architecture to the one you'd've started with if you were using Kafka. Or you can just throw it in Kafka from day 1 and a lot of bad ideas just become impossible because the system gives you no way to do them. I was also working on phone systems in the 2000's that ran at "web scale". We used MySQL, but we essentially built Kafka on top of MySQL - tables for every intermediate step in the process, and a way for the next step in the processing pipeline to mark which rows it had started/finished consuming.

oweiler 4y ago

I agree that Kafka is overused. But IMHO Kafka's strength is not scaling but decoupling. Services can publish data without other interested parties being online. If you need this, is obviously another question.

happymellon 4y ago
That is true of most pub/sub systems though. Kafka has a lot of other "features" which is what makes it overkill for most situations.

happymellon 4y ago

All the times I've used Kafka, it's felt like the microservices confusion. A technical solution that can be taken to the extreme, when really it's a solution to a political problem. If we break our application into small enough chunks then we start running into internal consistency issues, so we will introduce Kafka to resolve this. I'm dealing with it at the moment where my customer decided to break their crud application to CUD and R because "best practice". Now they have Kafka synchronising the two applications because it's a terrible design.

bicijay 4y ago
You are talking about CQRS, which is far from a "terrible design", maybe not right for your customer?
- happymellon 4y ago
  I understand that it might help some people, and I am referring to my clients desire as terrible rather than the whole concept. However I have yet to actually see it used without it increasing complexity without demonstrable improvements. I see that Martin Fowler agrees. https://martinfowler.com/bliki/CQRS.html https://martinfowler.com/bliki/CQRS.html It falls into the "you are not Google" problem solving hole. I'm dealing with a financial institution doing this. CQRS should be an evolution if you find it complex to model, use, and scale not the initial implementation. Now we have another location to lose data.

edude03 4y ago

> Queues like Kafka Kafka is a distributed log. A sibling comment to this mentions this as well, but I'm repeating it because I actually think that's were most people using kafka go wrong. You _can_ use kafka as a queue, but it'll would feel like using a drill to hammer a nail, would kind of work but would make you think the tool just sucks. > The justification is usually a sort of handwave about scaling, but I tend to be skeptical of that argument. Kafka does help with scaling in so far as you can independently scale the number of producers and consumers but I find kafka the most useful in its ability to allow multiple consumers to process the same message(s) at different rates. For example let's say you have an app that's producing messages like "{foo: 1, bar: 2 }" and you need to count how many foos there are and how many bars there are. You could queue the messages, then have 1 service that does both, or you could use kafka and have a FooCounter and BarCounter service. They each process the same message(s) but ignore the part of the payload they don't care about. While this is really trivial example, I've used, and have seen this pattern used for processing actual logs for example.