4 ms·

Sure, if you're working on a small homelab with minimal to no processing volume. The second you approach any kind of scale, this falls apart and/or you end up

by fatal94 10mo ago

Sure, if you're working on a small homelab with minimal to no processing volume. The second you approach any kind of scale, this falls apart and/or you end up with a more expensive and worse version of Kafka.

devnull3 10mo ago
I think there is a wide spectrum between small-homelab and google scale. I was surprised how far sqlite goes with some sharding on modern SSDs for those in-between scale services/saas
- fatal94 10mo ago
  What you're doing is fine for a homelab, or learning. But barring any very specific reason other than just not liking Kafka, its bad. The second that pattern needs to be fanned out to support even 50+ producers/consumers, the overhead and complexity needed to manage already-solved problems becomes a very bad design choice. Kafka already solves this problem and gives me message durability, near infinite scale out, sharding, delivery guarantees, etc out of the box. I do not care to develop, reshard databases or production-alize this myself.
  - CyberDildonics 10mo ago
    sqlite can do 40,000 transactions per second, that's going to be a lot more than 'homelab' (home lab). Not everything needs to be big and complicated.
  - NewJazz 10mo ago
    Some people don't and won't need 50+ producers/consumers for a long while, if ever. Rewriting the code at that point may be less costly than operating Kafka in the interim. Kafka is also has a higher potential for failure than sqlite.
    - fatal94 10mo ago
      Ofc, and not everybody needs or cares for all the features Kafka has. Then use another known and tested messaging system. Use NATS or ZMQ. Or any cloud native pubsub system My main point is, I have zero interest in creating novel solutions to a solved problem. It just artificially increases the complexity of my work and the learning curve for contributors.
    - School-Cotton 10mo ago
      Okay, then those people don’t have to use Kafka. What is your point?
      - NewJazz 10mo ago
        I was responding to someone who was responding to someone that wasn't using Kafka telling them to use Kafka. What's yours?
raverbashing 10mo ago
"Any kind of scale" No, there's a long way of better and more straightforward solutions than the simple SELECT (SELECT * from EVENTS where TIMESTAMP > LAST_TS LIMIT 50) for example

by fatal94 10mo ago

devnull3 10mo ago

I think there is a wide spectrum between small-homelab and google scale. I was surprised how far sqlite goes with some sharding on modern SSDs for those in-between scale services/saas

fatal94 10mo ago
What you're doing is fine for a homelab, or learning. But barring any very specific reason other than just not liking Kafka, its bad. The second that pattern needs to be fanned out to support even 50+ producers/consumers, the overhead and complexity needed to manage already-solved problems becomes a very bad design choice. Kafka already solves this problem and gives me message durability, near infinite scale out, sharding, delivery guarantees, etc out of the box. I do not care to develop, reshard databases or production-alize this myself.
- CyberDildonics 10mo ago
  sqlite can do 40,000 transactions per second, that's going to be a lot more than 'homelab' (home lab). Not everything needs to be big and complicated.
- NewJazz 10mo ago
  Some people don't and won't need 50+ producers/consumers for a long while, if ever. Rewriting the code at that point may be less costly than operating Kafka in the interim. Kafka is also has a higher potential for failure than sqlite.
  - fatal94 10mo ago
    Ofc, and not everybody needs or cares for all the features Kafka has. Then use another known and tested messaging system. Use NATS or ZMQ. Or any cloud native pubsub system My main point is, I have zero interest in creating novel solutions to a solved problem. It just artificially increases the complexity of my work and the learning curve for contributors.
  - School-Cotton 10mo ago
    Okay, then those people don’t have to use Kafka. What is your point?
    - NewJazz 10mo ago
      I was responding to someone who was responding to someone that wasn't using Kafka telling them to use Kafka. What's yours?

raverbashing 10mo ago

"Any kind of scale" No, there's a long way of better and more straightforward solutions than the simple SELECT (SELECT * from EVENTS where TIMESTAMP > LAST_TS LIMIT 50) for example