3 ms·

Being an optimist here. Maybe they needed a change. If Kafka out performs their custom messaging system. Why not?

• sidlls 8y ago
  Most of the deployments of Kafka I've seen have been desperate attempts to work around poorly designed data architecture. You shouldn't need Kafka to handle a few hundred MB or a few GB of data per day. Twitter's scale is way beyond that, though and I can easily see the case for it there.
  • nemothekid 8y ago
    Interesting by Kafka here, do you mean Kafka specfically or any sort of event queue (+ RabbitMQ/NSQ too)? And what would you suggest instead for an event processing pipeline?
    • manigandham 8y ago
      Redis 5.0 Streams feature is a great alternative.
  • Xorlev 8y ago
    > You shouldn't need Kafka to handle a few hundred MB or a few GB of data per day. I wouldn't link the two problems (size, bad architecture). Kafka is often used to bail out poorly designed data architectures, but it doesn't matter whether the volume is small or large at that point: it's still a pile of crap. Even for smaller cases it's a nice way to decouple producers from multiple consumers. I wouldn't suggest building Kafka infrastructure if you don't also have at least one bigger use-case, but if it's there it's convenient. Especially if you need replay or the ability to consume the same messages for days afterwards.
  • badpun 8y ago
    The case for Kafka I've seen the most is "decoupling". I.e. two pieces of a system, both of which are often owned by one team, are talking to each other over Kafka because this way they are decoupled, which keeps the architecture clean. So, to make the system simpler, we make it more complicated... I was never sold on this.