7 ms·

Twitter’s Kafka adoption story

137 points by sciurus 8y ago

stevewilhelm 8y ago
Did they review other Pub/Sub solutions? I'm thinking of NATS in particular.
- nevi-me 8y ago
  If they're looking for a mature solution, Kafka would be more in that line. Plus beyond pubsub, Kafka is well supported as a streaming data source in the Hadoop-esque world
- temuze 8y ago
  I've been a big fan of Apache Pulsar lately. I like that they separate storage into a separate layer.
  - sciurus 8y ago
    Interestingly, Twitter's in-house system did that too, but they now seem to think it was a mistake because it increased latency.
    - airfreak 8y ago
      It all depends. For example with Apache Pulsar, tailing readers are served from an in-memory cache in the serving layer (the Pulsar brokers) and only catch-up readers end up having to be served from the storage layer (Apache BookKeeper). This is a little different from DistributedLog which always required going to BookKeeper for reads. Apache BookKeeper can add additional latency to catch-up readers, on top of the extra hop, because the data of multiple topics are combined into each ledger. This means that we lose some performance from sequential reads. This is mitigated in BookKeeper by writing to disk in batches and sorting a ledger by topic so messages of the same topic are found together, but it still involves more jumping around on disk. Also, BookKeeper allows the nice separation of disk IO. The read and write path are separate and can be served by different disks so you can scale your reads and writes separately to a certain extent. For all those reasons, I would have loved to have seen Twitter look at Apache Pulsar and compare performance profiles with Apache Kafka.
      - anhldbk 8y ago
        Streamlio published their OpenMessaging benchmarks between Apache Kafka & Apache Pulsar here: https://streaml.io/pdf/Gigaom-Benchmarking-Streaming-Platforms.pdf https://streaml.io/pdf/Gigaom-Benchmarking-Streaming-Platfor...
- andonisus 8y ago
  We us NATS (and STAN) for real-time IoT metrics and reporting with a high cardinality of subscriptions. The throughput from a single NATS box is very nice (we currently send 100K+ messages/sec using a single EC2 instance running NATS).
- tveita 8y ago
  It would be very interesting to see a comparison of Pulsar or LogDevice on Twitter scale workloads.
  - majidazimi 8y ago
    Both Pulsar and LogDevice solve the problem of far-away-consumers with kind of segmenting the log and storing each segment in separate machine to reduce disk seeks and page cache pollution (when c1 reads head of the log and c2 reads tail). So technically they are better than Kafka. But as the article mentions, they just used SSDs to get around this issue in Kafka. By looking at article the only valid reason to switch to Kafka seems to be KStream.
    - manigandham 8y ago
      Pulsar has a much better storage system that scales independently so latency stays low regardless of consumer offset. It can also now tier to S3/cloud storage natively and it has many other features like supporting millions of topics and per-message acknowledgements.
asdfasafasdf 8y ago
Twitter loves migrations. "We will continuously keep an eye on different messaging and streaming systems in the ecosystem and ensure that our team is making the right decision for our customers and Twitter, even if it’s a hard decision" Rather than setting industry direction, they are inclined to changing underlying platform every few years, which brings a lot of challenges. May be it creates work for platform teams. But, its not useful to do this often.
- Jaruzel 8y ago
  Twitter (the company) always comes across as constantly bailing out their leaky ship, and then running around plugging holes with bits of scrap that 'might' just fit. There never seems to be any sort of proper roadmap that you'd expect for a tech company with such a large user-base.
  - bmiller2 8y ago
    To be fair, they are operating at a scale probably unmatched anywhere (Facebook?). There isn’t exactly a well trodden path for them to fall back on.
- zukzuk 8y ago
  Their migration-to-Scala-that-wasn't being another great example.
  - tormeh 8y ago
    What happened? A quick Google search returned no promising headlines.
  - aphexairlines 8y ago
    From a 2018 presentation: "Most Twitter services are written in Scala" https://downloads.ctfassets.net/oxjq45e8ilak/6eh2A72b4IyWsWOIcig4K0/cbb664566fe86672d92ddfb210623920/Chris_Thalinger_Twitter_s_quest_for_a_wholly_Graal_runtime.pdf https://downloads.ctfassets.net/oxjq45e8ilak/6eh2A72b4IyWsWO...
  - 8y ago
majidazimi 8y ago
I submitted a question in previous discussion link[0] but I didn't get an answer by author. But hopefully someone will answer here: How did you solve producer fencing mechanism which was built into Bookkeeper with Kafka (Assume I need to limit only one client to be able to write to a topic)? [0] https://news.ycombinator.com/item?id=18559284 https://news.ycombinator.com/item?id=18559284
- Pixelstime 8y ago
  Sorry for that I didn't understand your question fully, I suggest you to ask question in the slack[link] of BookKeeper community. [link]http://bookkeeper.apache.org/community/slack/ http://bookkeeper.apache.org/community/slack/
  - majidazimi 8y ago
    Sorry for my bad English. Here is a better explanation: In Bookkeeper, one producer can fence the current active producer from appending to log. If majority of Bookkeeper nodes respond with OK to a FENCE request, then the current producer can not append to the log. This helps implementing master-slave database replication quiet trivial. Master keeps appending to log, and slaves keep reading from log. In case master fails, one of the slaves sends a FENCE request to Bookkeeper nodes which reliably blocks current Mal-functioning master from appending to db log. However in Kafka, this isn't a native operation. Any client can push to a topic. There is no easy way to safely do this. I was looking forward to see how Twitter team actually handled this specific use case.
scarcely 8y ago
Slightly off topic, but I feel that when naming one's technical creation it's best to avoid using the last names of famous people especially when that person has become so famous it's become standard to refer to him/her simply by last name. Just google "Kafka" and check out the first page of search results and you'll see what I mean. (No, googling "Franz Kafka" isn't the same)
- will_pseudonym 8y ago
  I agree completely. Similarly, I have no idea why the creators of GIMP and CockroachDB (Spencer Kimball and Peter Mattis, and Ben Darnell for the latter) thought those names were the best possible options.
  - slededit 8y ago
    CockroachDB at least refers to their marketing line of "almost impossible to kill/take down" and the way it scales to many servers. There are few less gross insects they could have used though like Bees or Ants.
    - andrewflnr 8y ago
      Bees and ants don't have anywhere near the same rep for invincibility. Only one of those animals is a stereotypical nuclear apocalypse survivor.
      - slededit 8y ago
        They should have went with Water Bears. It can survive even in space. Plus they sound cute (but look scary).
        mjibson 8y ago
        We have a conference room named Tardigrade.
manigandham 8y ago
Why didn't they just run EventBus storage and serving layers on the same machine if they wanted to get better resource utilization?
- jbs40 8y ago
  There seems to be a fundamental technical misunderstanding of these systems in the blog post. Unless you're running in the (generally quite dangerous) scenario where replication factor = 1, both EventBus and Kafka have a network hop (Kafka to the replica and EventBus to the storage layer) and JVM traversal, so the argument that Kafka is more efficient there doesn't make sense. As you say, it would be trivial to run EventBus storage and serving on the same nodes if the network hop were an issue. The argument that a decoupled storage layer costs more than co-locating storage and serving also seems dubious. The cluster is basically doing the same amount of work when you decouple those layers, it's just separating them into components that can be optimized for specific roles. So if you pick the right hardware/instances for each layer and tune for that, you really shouldn't need more hardware. Otherwise we'd all still be running full-stack applications on mainframes to minimize the number of servers. :)
  - manigandham 8y ago
    Yes, agreed. It seemed odd because Kafka isn't split into layers so there's no other option, and it confuses resource utilization for number of servers.
will_pseudonym 8y ago
When I saw the word Kafka in relation to Twitter, I immediately thought about Twitter's recent changes to their terms of service [0] which polices language used on the platform. "Franz Kafka (1883-1924) was a Czech-born German-language writer whose surreal fiction vividly expressed the anxiety, alienation, and powerlessness of the individual in the 20th century. Kafka's work is characterized by nightmarish settings in which characters are crushed by nonsensical, blind authority. Thus, the word Kafkaesque is often applied to bizarre and impersonal administrative situations where the individual feels powerless to understand or control what is happening." [1] [0] https://help.twitter.com/en/rules-and-policies/hateful-conduct-policy https://help.twitter.com/en/rules-and-policies/hateful-condu... [1] https://www.merriam-webster.com/dictionary/Kafkaesque https://www.merriam-webster.com/dictionary/Kafkaesque

137 points by sciurus 8y ago

stevewilhelm 8y ago

Did they review other Pub/Sub solutions? I'm thinking of NATS in particular.

nevi-me 8y ago
If they're looking for a mature solution, Kafka would be more in that line. Plus beyond pubsub, Kafka is well supported as a streaming data source in the Hadoop-esque world
temuze 8y ago
I've been a big fan of Apache Pulsar lately. I like that they separate storage into a separate layer.
- sciurus 8y ago
  Interestingly, Twitter's in-house system did that too, but they now seem to think it was a mistake because it increased latency.
  - airfreak 8y ago
    It all depends. For example with Apache Pulsar, tailing readers are served from an in-memory cache in the serving layer (the Pulsar brokers) and only catch-up readers end up having to be served from the storage layer (Apache BookKeeper). This is a little different from DistributedLog which always required going to BookKeeper for reads. Apache BookKeeper can add additional latency to catch-up readers, on top of the extra hop, because the data of multiple topics are combined into each ledger. This means that we lose some performance from sequential reads. This is mitigated in BookKeeper by writing to disk in batches and sorting a ledger by topic so messages of the same topic are found together, but it still involves more jumping around on disk. Also, BookKeeper allows the nice separation of disk IO. The read and write path are separate and can be served by different disks so you can scale your reads and writes separately to a certain extent. For all those reasons, I would have loved to have seen Twitter look at Apache Pulsar and compare performance profiles with Apache Kafka.
    - anhldbk 8y ago
      Streamlio published their OpenMessaging benchmarks between Apache Kafka & Apache Pulsar here: https://streaml.io/pdf/Gigaom-Benchmarking-Streaming-Platforms.pdf https://streaml.io/pdf/Gigaom-Benchmarking-Streaming-Platfor...
andonisus 8y ago
We us NATS (and STAN) for real-time IoT metrics and reporting with a high cardinality of subscriptions. The throughput from a single NATS box is very nice (we currently send 100K+ messages/sec using a single EC2 instance running NATS).
tveita 8y ago
It would be very interesting to see a comparison of Pulsar or LogDevice on Twitter scale workloads.
- majidazimi 8y ago
  Both Pulsar and LogDevice solve the problem of far-away-consumers with kind of segmenting the log and storing each segment in separate machine to reduce disk seeks and page cache pollution (when c1 reads head of the log and c2 reads tail). So technically they are better than Kafka. But as the article mentions, they just used SSDs to get around this issue in Kafka. By looking at article the only valid reason to switch to Kafka seems to be KStream.
  - manigandham 8y ago
    Pulsar has a much better storage system that scales independently so latency stays low regardless of consumer offset. It can also now tier to S3/cloud storage natively and it has many other features like supporting millions of topics and per-message acknowledgements.

asdfasafasdf 8y ago

Twitter loves migrations. "We will continuously keep an eye on different messaging and streaming systems in the ecosystem and ensure that our team is making the right decision for our customers and Twitter, even if it’s a hard decision" Rather than setting industry direction, they are inclined to changing underlying platform every few years, which brings a lot of challenges. May be it creates work for platform teams. But, its not useful to do this often.

Jaruzel 8y ago
Twitter (the company) always comes across as constantly bailing out their leaky ship, and then running around plugging holes with bits of scrap that 'might' just fit. There never seems to be any sort of proper roadmap that you'd expect for a tech company with such a large user-base.
- bmiller2 8y ago
  To be fair, they are operating at a scale probably unmatched anywhere (Facebook?). There isn’t exactly a well trodden path for them to fall back on.
zukzuk 8y ago
Their migration-to-Scala-that-wasn't being another great example.
- tormeh 8y ago
  What happened? A quick Google search returned no promising headlines.
- aphexairlines 8y ago
  From a 2018 presentation: "Most Twitter services are written in Scala" https://downloads.ctfassets.net/oxjq45e8ilak/6eh2A72b4IyWsWOIcig4K0/cbb664566fe86672d92ddfb210623920/Chris_Thalinger_Twitter_s_quest_for_a_wholly_Graal_runtime.pdf https://downloads.ctfassets.net/oxjq45e8ilak/6eh2A72b4IyWsWO...
- 8y ago

majidazimi 8y ago

I submitted a question in previous discussion link[0] but I didn't get an answer by author. But hopefully someone will answer here: How did you solve producer fencing mechanism which was built into Bookkeeper with Kafka (Assume I need to limit only one client to be able to write to a topic)? [0] https://news.ycombinator.com/item?id=18559284 https://news.ycombinator.com/item?id=18559284

Pixelstime 8y ago
Sorry for that I didn't understand your question fully, I suggest you to ask question in the slack[link] of BookKeeper community. [link]http://bookkeeper.apache.org/community/slack/ http://bookkeeper.apache.org/community/slack/
- majidazimi 8y ago
  Sorry for my bad English. Here is a better explanation: In Bookkeeper, one producer can fence the current active producer from appending to log. If majority of Bookkeeper nodes respond with OK to a FENCE request, then the current producer can not append to the log. This helps implementing master-slave database replication quiet trivial. Master keeps appending to log, and slaves keep reading from log. In case master fails, one of the slaves sends a FENCE request to Bookkeeper nodes which reliably blocks current Mal-functioning master from appending to db log. However in Kafka, this isn't a native operation. Any client can push to a topic. There is no easy way to safely do this. I was looking forward to see how Twitter team actually handled this specific use case.

scarcely 8y ago

Slightly off topic, but I feel that when naming one's technical creation it's best to avoid using the last names of famous people especially when that person has become so famous it's become standard to refer to him/her simply by last name. Just google "Kafka" and check out the first page of search results and you'll see what I mean. (No, googling "Franz Kafka" isn't the same)

will_pseudonym 8y ago
I agree completely. Similarly, I have no idea why the creators of GIMP and CockroachDB (Spencer Kimball and Peter Mattis, and Ben Darnell for the latter) thought those names were the best possible options.
- slededit 8y ago
  CockroachDB at least refers to their marketing line of "almost impossible to kill/take down" and the way it scales to many servers. There are few less gross insects they could have used though like Bees or Ants.
  - andrewflnr 8y ago
    Bees and ants don't have anywhere near the same rep for invincibility. Only one of those animals is a stereotypical nuclear apocalypse survivor.
    - slededit 8y ago
      They should have went with Water Bears. It can survive even in space. Plus they sound cute (but look scary).
      - mjibson 8y ago
        We have a conference room named Tardigrade.

manigandham 8y ago

Why didn't they just run EventBus storage and serving layers on the same machine if they wanted to get better resource utilization?

jbs40 8y ago
There seems to be a fundamental technical misunderstanding of these systems in the blog post. Unless you're running in the (generally quite dangerous) scenario where replication factor = 1, both EventBus and Kafka have a network hop (Kafka to the replica and EventBus to the storage layer) and JVM traversal, so the argument that Kafka is more efficient there doesn't make sense. As you say, it would be trivial to run EventBus storage and serving on the same nodes if the network hop were an issue. The argument that a decoupled storage layer costs more than co-locating storage and serving also seems dubious. The cluster is basically doing the same amount of work when you decouple those layers, it's just separating them into components that can be optimized for specific roles. So if you pick the right hardware/instances for each layer and tune for that, you really shouldn't need more hardware. Otherwise we'd all still be running full-stack applications on mainframes to minimize the number of servers. :)
- manigandham 8y ago
  Yes, agreed. It seemed odd because Kafka isn't split into layers so there's no other option, and it confuses resource utilization for number of servers.

will_pseudonym 8y ago

When I saw the word Kafka in relation to Twitter, I immediately thought about Twitter's recent changes to their terms of service [0] which polices language used on the platform. "Franz Kafka (1883-1924) was a Czech-born German-language writer whose surreal fiction vividly expressed the anxiety, alienation, and powerlessness of the individual in the 20th century. Kafka's work is characterized by nightmarish settings in which characters are crushed by nonsensical, blind authority. Thus, the word Kafkaesque is often applied to bizarre and impersonal administrative situations where the individual feels powerless to understand or control what is happening." [1] [0] https://help.twitter.com/en/rules-and-policies/hateful-conduct-policy https://help.twitter.com/en/rules-and-policies/hateful-condu... [1] https://www.merriam-webster.com/dictionary/Kafkaesque https://www.merriam-webster.com/dictionary/Kafkaesque