9 ms·

Not just startups. I interviewed with a large financial institution recently and they had their heart set on Kafka. From what I learnt , they did not need Kaf

Not just startups. I interviewed with a large financial institution recently and they had their heart set on Kafka. From what I learnt , they did not need Kafka -- the message order was not important, there was only one producer and one consumer and the volume of data was not extra-ordinary -- but they wanted to be on Kafka as that was the enterprise direction. I might have come across as an old fart when I asked the why they could not use their existing MQ infrastructure.

igetspam 5y ago
We've been using Kafka for a while. We're thinking about ditching it for something AMQP. For all it's bells and whistles, it's just not really all that exciting. We're probably spending 5x more on Kafka than we could something much simpler.
- jdmichal 5y ago
  We're doing the same. Actively moving away from Kafka, which was only ever used as a message queue anyway, to an actual message queue. We might move back at some point in the future to rebuild an actual event-driven processing model. But for now, it's not worth the additional complexity for the scale and systems we actually have.
  - himoacs 5y ago
    Which product are you moving to?
- EdwardDiego 5y ago
  Kafka is only exciting when you're wanting to stream a metric shit ton of data without things falling over. If you're using it, you're constrained by its engineering decisions, so you need to be sure it's a worthwhile tradeoff.
- himoacs 5y ago
  Just to clarify...AMQP is a protocol so you will need a broker that supports it. The two options I see for you are RabbitMQ and Solace. Both support AMQP.
  - marcelm72 5y ago
    Apache Pulsar seems to have decent AMQP support also.
mrweasel 5y ago
Not that I'm not impressed by Kafka and its stability, performance and scalability, but I see the same behaviour from our customers. They specifically want Kafka, there's no real reason other than they need a queue, which Kafka actively states that it's not. At that point it gets really tricky to reason with the developers about why they might be better served by something else. Generally speaking it's not much of an issue, because Kafka will deal with workloads just fine, it's just weird. I have seen one customer use Kafka as a database, that works less well. We do see the same with Kubernetes. The developers pick Kubernetes and at that point it's to late. They specifically want Kubernetes even if you could more easily solve the problem with Nomad, Docker-Compose, plain old VMs or EC2, depending on the problem.
- mumblemumble 5y ago
  I'm about to pick Kubernetes even though a different solution would theoretically be a much better fit for my needs. This is entirely because some other tools I'm looking at play nicely with Kubernetes out of the box. If I picked something else, I'd end up writing my own glue code. I wonder if something similar is happening with Kafka?
  - nobleach 5y ago
    My last gig was Kubernetes, and aside from all the hate it gets here, (You're not Facebook, you don't need Facebook scale) it was a very pleasant experience. So pleasant in fact that when I moved on to my next job (Amazon EC2 VMs) it was pretty painful. They were running an old version of Amazon Linux and hadn't been updated in years. The versions of some runtimes were impossible to update due to GLibc being out of date. Our immediate answer was, can we at least get to a Docker solution? ECS/Fargate provided a nice middle ground. But I'll admit. Once you start getting into running multiple replicas, it's nice to have the other stuff that Kubernetes affords you.
    - snowzach 5y ago
      I pick Kubernetes because I want to manage software and not manage servers. I think it gets a lot of hate because people look at helm charts that are designed to support all possible software configurations and they are quite confusing. If you break it down to the basic pieces of configuration it's not really any more complicated than say a docker-compose file. Just a little more verbose.
bencyoung 5y ago
The main thing Kafka is great at is scaling out via partitions with it's natural stickyness, while maintaining excellent ordering guarantees. This means it's really easy to create a load of reliable in-memory processing where each thread processes a single partition of data. When scaling out on a standard message queue you generally have greedy consumers which means you can't assume stickyness, or have to create your own partitioning structures. It makes it great for realtime apps... If there were cheaper alternatives I think people would use them but it does definitely have powerful benefits.
BeefWellington 5y ago
It's the same with any new shiny technology it seems. The specific thing I have experience with is in analytics/relational databases. Suddenly around 8-10 years ago it became imperative for every client I was dealing with to migrate their RDBMSes to Hadoop/Hive setups, even when their largest dataset was only about 120M rows denormalized. They were trading three servers (primary, backup, DR) for sometimes 15 to 20. Queries that MSSQL was handling sub-second were suddenly taking 45s on Hive. It was utter madness and was as far as I can tell driven by good salespeople, FOMO, and the feeling of importance of being able to say your company is running Big Data(TM?). I saw maybe one implementation (of dozens) that actually stayed in use for any length of time.
derefr 5y ago
What would you recommend as a "pipe with MQ-ish semantics" to use when "the message order was not important, there was only one producer and one consumer and the volume of data was not extra-ordinary", but you do still want the core premise of an MQ — reliable message delivery in the face of faults in the producer and/or consumer? We actually have a use-case that exactly matches this. One service makes [stuff], the other service consumes [stuff], both services are "immutable infrastructure" with no local stable state storage, and [stuff] is individually too small and frequent to be affordable with IaaS managed-MQ per-message costs — but batching messages into reasonable chunks before send means potentially losing up-to-a-batch worth of messages if the producer dies.
- zo1 5y ago
  Not OP, but I would recommend RabbitMQ. It's surprisingly simple to setup and if your load isn't huge it'll hum along just fine without clusters other bells.

igetspam 5y ago

We've been using Kafka for a while. We're thinking about ditching it for something AMQP. For all it's bells and whistles, it's just not really all that exciting. We're probably spending 5x more on Kafka than we could something much simpler.

jdmichal 5y ago
We're doing the same. Actively moving away from Kafka, which was only ever used as a message queue anyway, to an actual message queue. We might move back at some point in the future to rebuild an actual event-driven processing model. But for now, it's not worth the additional complexity for the scale and systems we actually have.
- himoacs 5y ago
  Which product are you moving to?
EdwardDiego 5y ago
Kafka is only exciting when you're wanting to stream a metric shit ton of data without things falling over. If you're using it, you're constrained by its engineering decisions, so you need to be sure it's a worthwhile tradeoff.
himoacs 5y ago
Just to clarify...AMQP is a protocol so you will need a broker that supports it. The two options I see for you are RabbitMQ and Solace. Both support AMQP.
- marcelm72 5y ago
  Apache Pulsar seems to have decent AMQP support also.

mrweasel 5y ago

Not that I'm not impressed by Kafka and its stability, performance and scalability, but I see the same behaviour from our customers. They specifically want Kafka, there's no real reason other than they need a queue, which Kafka actively states that it's not. At that point it gets really tricky to reason with the developers about why they might be better served by something else. Generally speaking it's not much of an issue, because Kafka will deal with workloads just fine, it's just weird. I have seen one customer use Kafka as a database, that works less well. We do see the same with Kubernetes. The developers pick Kubernetes and at that point it's to late. They specifically want Kubernetes even if you could more easily solve the problem with Nomad, Docker-Compose, plain old VMs or EC2, depending on the problem.

mumblemumble 5y ago
I'm about to pick Kubernetes even though a different solution would theoretically be a much better fit for my needs. This is entirely because some other tools I'm looking at play nicely with Kubernetes out of the box. If I picked something else, I'd end up writing my own glue code. I wonder if something similar is happening with Kafka?
- nobleach 5y ago
  My last gig was Kubernetes, and aside from all the hate it gets here, (You're not Facebook, you don't need Facebook scale) it was a very pleasant experience. So pleasant in fact that when I moved on to my next job (Amazon EC2 VMs) it was pretty painful. They were running an old version of Amazon Linux and hadn't been updated in years. The versions of some runtimes were impossible to update due to GLibc being out of date. Our immediate answer was, can we at least get to a Docker solution? ECS/Fargate provided a nice middle ground. But I'll admit. Once you start getting into running multiple replicas, it's nice to have the other stuff that Kubernetes affords you.
  - snowzach 5y ago
    I pick Kubernetes because I want to manage software and not manage servers. I think it gets a lot of hate because people look at helm charts that are designed to support all possible software configurations and they are quite confusing. If you break it down to the basic pieces of configuration it's not really any more complicated than say a docker-compose file. Just a little more verbose.

bencyoung 5y ago

The main thing Kafka is great at is scaling out via partitions with it's natural stickyness, while maintaining excellent ordering guarantees. This means it's really easy to create a load of reliable in-memory processing where each thread processes a single partition of data. When scaling out on a standard message queue you generally have greedy consumers which means you can't assume stickyness, or have to create your own partitioning structures. It makes it great for realtime apps... If there were cheaper alternatives I think people would use them but it does definitely have powerful benefits.

BeefWellington 5y ago

It's the same with any new shiny technology it seems. The specific thing I have experience with is in analytics/relational databases. Suddenly around 8-10 years ago it became imperative for every client I was dealing with to migrate their RDBMSes to Hadoop/Hive setups, even when their largest dataset was only about 120M rows denormalized. They were trading three servers (primary, backup, DR) for sometimes 15 to 20. Queries that MSSQL was handling sub-second were suddenly taking 45s on Hive. It was utter madness and was as far as I can tell driven by good salespeople, FOMO, and the feeling of importance of being able to say your company is running Big Data(TM?). I saw maybe one implementation (of dozens) that actually stayed in use for any length of time.

derefr 5y ago

What would you recommend as a "pipe with MQ-ish semantics" to use when "the message order was not important, there was only one producer and one consumer and the volume of data was not extra-ordinary", but you do still want the core premise of an MQ — reliable message delivery in the face of faults in the producer and/or consumer? We actually have a use-case that exactly matches this. One service makes [stuff], the other service consumes [stuff], both services are "immutable infrastructure" with no local stable state storage, and [stuff] is individually too small and frequent to be affordable with IaaS managed-MQ per-message costs — but batching messages into reasonable chunks before send means potentially losing up-to-a-batch worth of messages if the producer dies.

zo1 5y ago
Not OP, but I would recommend RabbitMQ. It's surprisingly simple to setup and if your load isn't huge it'll hum along just fine without clusters other bells.