3 ms·

> There are various event sources we consume data from, and they are produced in different platforms and storage systems, such as Hadoop, Vertica, Manhattan dis

> There are various event sources we consume data from, and they are produced in different platforms and storage systems, such as Hadoop, Vertica, Manhattan distributed databases, Kafka, Twitter Eventbus, GCS, BigQuery, and PubSub. I'm surprised that they mentioned Twitter EventBus, I thought that they were migrating away from that to Apache Kafka entirely. [1] Mind you, they've got a lot of tech going on, so not surprising if it's still present in legacy systems. Fun fact, if you dig into the architecture of Twitter EventBus, it will seem awfully familiar to the architecture of Apache Pulsar (storage separated from brokers, storage based on Bookkeeper), and that's no coincidence, as Sijie Guo, CEO of StreamNative developed EventBus at Twitter (also a main dev of BookKeeper). StreamNative is to Pulsar as Confluent is to Kafka. And the reasons that Twitter moved from EventBus to Kafka also apply to Pulsar, which is worth keeping in mind the next time an HN commenter proclaims "Le roi (Kafka) est mort, vive le roi! (Pulsar)". [1]: https://blog.twitter.com/engineering/en_us/topics/insights/2018/twitters-kafka-adoption-story https://blog.twitter.com/engineering/en_us/topics/insights/2...

ntonozzi 5y ago
They use the term EventBus, but really it's a thin EventBus client library that uses Kafka under the hood. I think all EventBus clients at Twitter have been migrated to use Kafka.
- EdwardDiego 5y ago
  Ah, cheers, that makes sense, I didn't see it mentioned further in the architecture.
abraxas 5y ago
Their whole platform sounds like an unholy mess by the sound of that. A big pile of dung that just accreted over time and they never cleaned out their technical debt. I just left a job like that a few weeks ago. It was such a nightmare I took a slight paycut.
- EdwardDiego 5y ago
  Don't all code bases that are continually responding to changing business needs and/or scaling demands fated to become big piles of dung? (Or as we all politely call it "legacy").

ntonozzi 5y ago

They use the term EventBus, but really it's a thin EventBus client library that uses Kafka under the hood. I think all EventBus clients at Twitter have been migrated to use Kafka.

EdwardDiego 5y ago
Ah, cheers, that makes sense, I didn't see it mentioned further in the architecture.

abraxas 5y ago

Their whole platform sounds like an unholy mess by the sound of that. A big pile of dung that just accreted over time and they never cleaned out their technical debt. I just left a job like that a few weeks ago. It was such a nightmare I took a slight paycut.

EdwardDiego 5y ago
Don't all code bases that are continually responding to changing business needs and/or scaling demands fated to become big piles of dung? (Or as we all politely call it "legacy").