4 ms·

This post completely ignores fault tolerance. The reason Kafka requires you to have three machines isn’t because it is somehow magical, but because that allows

This post completely ignores fault tolerance. The reason Kafka requires you to have three machines isn’t because it is somehow magical, but because that allows it to effectively recover from machine-level failure without data loss (Depending on how your producers are configured, of course). Saying this approach is even “a bit like Kafka” is incredibly weak - if he is trying to do semi-durable message queueing, fine, but instead he consistently attempts to pitch his solution as mostly on par with Kafka. In the end he has created Kafka minus the utility.

monstrado 8y ago
Exactly, and if you /really/ wanted to, you could run a one node Kafka "cluster" using data replication of one instead of three for topics. I totally get the fun of creating something from scratch, but you introduce far more complexity and burden to your stack when you create something that needs the durability of a file system.
jasebell 8y ago
It does ignore fault tolerance yes I agree, because right now I don't need it. When you're bootstrapping this stuff you want to be as lean as you can. The plan is to move to Kafka, which I do use a lot btw, when there's a case to move to that level of throughput. If I had my way now I'd use Kafka, my wallet on the other hand disagrees. I know the reason for needing three machines for a Kafka cluster. And I'm certainly not pitching Durable Queue as a Kafka alternative unless, as in my case, the throughput will be so low in the initial stages as not to warrant a full Kafka cluster. I could have run Kafka on a single node..... The "a bit like Kafka", weak yeah it is but the solution posted presented a fairly quick way to establish a queue and decouple messaging from the running application, that was the whole point.
- vinceve 8y ago
  Ever looked at Azure Event hubs? We’re usung this as a Kafka alternative because it’s way cheaper actually (for xx million number of messages with a throughput of 4 MBs per second its 75 €) then setting up a full Kafka and keeping it running? Or do you see disadvantages on that technology? I’m not trolling or so, just curious.
  - jasebell 8y ago
    No I haven't but I appreciate the headsup. It's been a few years since I even looked at anything on Azure in all honesty.

monstrado 8y ago

Exactly, and if you /really/ wanted to, you could run a one node Kafka "cluster" using data replication of one instead of three for topics. I totally get the fun of creating something from scratch, but you introduce far more complexity and burden to your stack when you create something that needs the durability of a file system.

jasebell 8y ago

It does ignore fault tolerance yes I agree, because right now I don't need it. When you're bootstrapping this stuff you want to be as lean as you can. The plan is to move to Kafka, which I do use a lot btw, when there's a case to move to that level of throughput. If I had my way now I'd use Kafka, my wallet on the other hand disagrees. I know the reason for needing three machines for a Kafka cluster. And I'm certainly not pitching Durable Queue as a Kafka alternative unless, as in my case, the throughput will be so low in the initial stages as not to warrant a full Kafka cluster. I could have run Kafka on a single node..... The "a bit like Kafka", weak yeah it is but the solution posted presented a fairly quick way to establish a queue and decouple messaging from the running application, that was the whole point.

vinceve 8y ago
Ever looked at Azure Event hubs? We’re usung this as a Kafka alternative because it’s way cheaper actually (for xx million number of messages with a throughput of 4 MBs per second its 75 €) then setting up a full Kafka and keeping it running? Or do you see disadvantages on that technology? I’m not trolling or so, just curious.
- jasebell 8y ago
  No I haven't but I appreciate the headsup. It's been a few years since I even looked at anything on Azure in all honesty.