7 ms·

Twitter loves migrations. "We will continuously keep an eye on different messaging and streaming systems in the ecosystem and ensure that our team is making th

Twitter loves migrations. "We will continuously keep an eye on different messaging and streaming systems in the ecosystem and ensure that our team is making the right decision for our customers and Twitter, even if it’s a hard decision" Rather than setting industry direction, they are inclined to changing underlying platform every few years, which brings a lot of challenges. May be it creates work for platform teams. But, its not useful to do this often.

Jaruzel 8y ago
Twitter (the company) always comes across as constantly bailing out their leaky ship, and then running around plugging holes with bits of scrap that 'might' just fit. There never seems to be any sort of proper roadmap that you'd expect for a tech company with such a large user-base.
- bmiller2 8y ago
  To be fair, they are operating at a scale probably unmatched anywhere (Facebook?). There isn’t exactly a well trodden path for them to fall back on.
zukzuk 8y ago
Their migration-to-Scala-that-wasn't being another great example.
- tormeh 8y ago
  What happened? A quick Google search returned no promising headlines.
- aphexairlines 8y ago
  From a 2018 presentation: "Most Twitter services are written in Scala" https://downloads.ctfassets.net/oxjq45e8ilak/6eh2A72b4IyWsWOIcig4K0/cbb664566fe86672d92ddfb210623920/Chris_Thalinger_Twitter_s_quest_for_a_wholly_Graal_runtime.pdf https://downloads.ctfassets.net/oxjq45e8ilak/6eh2A72b4IyWsWO...
- speedkills 8y ago
  Are you implying Scala is not the majority of Twitter's code?
outworlder 8y ago
Or maybe, just maybe, engineering is keeping tech debt in check and replacing pieces that are working, but not working well, with better pieces. Also, unless we are working at a similar scale, I don't think we are qualified to comment on their engineering challenges.
- hinkley 8y ago
  Engineering is keeping your knives sharp and your tools clean. One of the oft unspoken things in software is that we let the talent do things that are prima facie suboptimal for the organization in order to keep them here and keep them vibrant. Some minor boondoggles I used to think harshly about make more sense in hindsight when I think about my own mental processes when I leave a place or even when I’m just feeling demotivated. I love to solve a problem and be done with it. Probably more than most. But... you have to keep moving, and it’s probably even true that there can be no sacred cows.
capkutay 8y ago
I'm curious if there's some conflicting, underlying motives from their senior level technology folks. They had a game changing tool in Apache Storm...once Nathan Marz (creator of Storm) left twitter, another senior engineer took over that group and created his OWN variant of Twitter Storm called Heron. After much engineering PR and github fanfare [0], couple years later he leaves to start his own company to do 'enterprise heron' [1]. So is the goal here to use something that works well for their end-users? Or is it to create a shiny new tool that they can spin off into a new company? 0: https://news.ycombinator.com/item?id=11770758 https://news.ycombinator.com/item?id=11770758 1: https://architecht.io/streamlio-founders-on-why-the-world-needs-a-new-streaming-data-platform-bdbb43c3f886?gi=ec98fbc31a6f https://architecht.io/streamlio-founders-on-why-the-world-ne...
- fnbr 8y ago
  The joke that I've heard is "performance (review) driven development". If the way to get promoted is to launch a shiny new product, then your most senior people will be the best at finding shiny new products to launch, even if that's not the right technical decision to make.
  - blackaspen 8y ago
    "Promobird" and "Promobird V2" (and so on).
  - yxhuvud 8y ago
    Closely related to CV driven development.
  - Benjammer 8y ago
    "Oh but our senior engineers would _never_ make such a selfish choice to make themselves look better at the expense of the company, everyone here _believes_ in the mission."
- snaky 8y ago
  There is resume driven development for blue collar coders masses. But this needs its' own name.
monksy 8y ago
Being an optimist here. Maybe they needed a change. If Kafka out performs their custom messaging system. Why not?
- sidlls 8y ago
  Most of the deployments of Kafka I've seen have been desperate attempts to work around poorly designed data architecture. You shouldn't need Kafka to handle a few hundred MB or a few GB of data per day. Twitter's scale is way beyond that, though and I can easily see the case for it there.
  - nemothekid 8y ago
    Interesting by Kafka here, do you mean Kafka specfically or any sort of event queue (+ RabbitMQ/NSQ too)? And what would you suggest instead for an event processing pipeline?
    - manigandham 8y ago
      Redis 5.0 Streams feature is a great alternative.
  - Xorlev 8y ago
    > You shouldn't need Kafka to handle a few hundred MB or a few GB of data per day. I wouldn't link the two problems (size, bad architecture). Kafka is often used to bail out poorly designed data architectures, but it doesn't matter whether the volume is small or large at that point: it's still a pile of crap. Even for smaller cases it's a nice way to decouple producers from multiple consumers. I wouldn't suggest building Kafka infrastructure if you don't also have at least one bigger use-case, but if it's there it's convenient. Especially if you need replay or the ability to consume the same messages for days afterwards.
  - badpun 8y ago
    The case for Kafka I've seen the most is "decoupling". I.e. two pieces of a system, both of which are often owned by one team, are talking to each other over Kafka because this way they are decoupled, which keeps the architecture clean. So, to make the system simpler, we make it more complicated... I was never sold on this.
shuzchen 8y ago
From the article: "For single consumer use cases, we saw a 68% resource savings, and for fanout cases with multiple consumers we saw a 75% resource savings." At twitter's server spend it makes perfect sense to make such an investment.

Jaruzel 8y ago

Twitter (the company) always comes across as constantly bailing out their leaky ship, and then running around plugging holes with bits of scrap that 'might' just fit. There never seems to be any sort of proper roadmap that you'd expect for a tech company with such a large user-base.

bmiller2 8y ago
To be fair, they are operating at a scale probably unmatched anywhere (Facebook?). There isn’t exactly a well trodden path for them to fall back on.

zukzuk 8y ago

Their migration-to-Scala-that-wasn't being another great example.

tormeh 8y ago
What happened? A quick Google search returned no promising headlines.
aphexairlines 8y ago
From a 2018 presentation: "Most Twitter services are written in Scala" https://downloads.ctfassets.net/oxjq45e8ilak/6eh2A72b4IyWsWOIcig4K0/cbb664566fe86672d92ddfb210623920/Chris_Thalinger_Twitter_s_quest_for_a_wholly_Graal_runtime.pdf https://downloads.ctfassets.net/oxjq45e8ilak/6eh2A72b4IyWsWO...
speedkills 8y ago
Are you implying Scala is not the majority of Twitter's code?

outworlder 8y ago

Or maybe, just maybe, engineering is keeping tech debt in check and replacing pieces that are working, but not working well, with better pieces. Also, unless we are working at a similar scale, I don't think we are qualified to comment on their engineering challenges.

hinkley 8y ago
Engineering is keeping your knives sharp and your tools clean. One of the oft unspoken things in software is that we let the talent do things that are prima facie suboptimal for the organization in order to keep them here and keep them vibrant. Some minor boondoggles I used to think harshly about make more sense in hindsight when I think about my own mental processes when I leave a place or even when I’m just feeling demotivated. I love to solve a problem and be done with it. Probably more than most. But... you have to keep moving, and it’s probably even true that there can be no sacred cows.

capkutay 8y ago

I'm curious if there's some conflicting, underlying motives from their senior level technology folks. They had a game changing tool in Apache Storm...once Nathan Marz (creator of Storm) left twitter, another senior engineer took over that group and created his OWN variant of Twitter Storm called Heron. After much engineering PR and github fanfare [0], couple years later he leaves to start his own company to do 'enterprise heron' [1]. So is the goal here to use something that works well for their end-users? Or is it to create a shiny new tool that they can spin off into a new company? 0: https://news.ycombinator.com/item?id=11770758 https://news.ycombinator.com/item?id=11770758 1: https://architecht.io/streamlio-founders-on-why-the-world-needs-a-new-streaming-data-platform-bdbb43c3f886?gi=ec98fbc31a6f https://architecht.io/streamlio-founders-on-why-the-world-ne...

fnbr 8y ago
The joke that I've heard is "performance (review) driven development". If the way to get promoted is to launch a shiny new product, then your most senior people will be the best at finding shiny new products to launch, even if that's not the right technical decision to make.
- blackaspen 8y ago
  "Promobird" and "Promobird V2" (and so on).
- yxhuvud 8y ago
  Closely related to CV driven development.
- Benjammer 8y ago
  "Oh but our senior engineers would _never_ make such a selfish choice to make themselves look better at the expense of the company, everyone here _believes_ in the mission."
snaky 8y ago
There is resume driven development for blue collar coders masses. But this needs its' own name.

monksy 8y ago

Being an optimist here. Maybe they needed a change. If Kafka out performs their custom messaging system. Why not?

sidlls 8y ago
Most of the deployments of Kafka I've seen have been desperate attempts to work around poorly designed data architecture. You shouldn't need Kafka to handle a few hundred MB or a few GB of data per day. Twitter's scale is way beyond that, though and I can easily see the case for it there.
- nemothekid 8y ago
  Interesting by Kafka here, do you mean Kafka specfically or any sort of event queue (+ RabbitMQ/NSQ too)? And what would you suggest instead for an event processing pipeline?
  - manigandham 8y ago
    Redis 5.0 Streams feature is a great alternative.
- Xorlev 8y ago
  > You shouldn't need Kafka to handle a few hundred MB or a few GB of data per day. I wouldn't link the two problems (size, bad architecture). Kafka is often used to bail out poorly designed data architectures, but it doesn't matter whether the volume is small or large at that point: it's still a pile of crap. Even for smaller cases it's a nice way to decouple producers from multiple consumers. I wouldn't suggest building Kafka infrastructure if you don't also have at least one bigger use-case, but if it's there it's convenient. Especially if you need replay or the ability to consume the same messages for days afterwards.
- badpun 8y ago
  The case for Kafka I've seen the most is "decoupling". I.e. two pieces of a system, both of which are often owned by one team, are talking to each other over Kafka because this way they are decoupled, which keeps the architecture clean. So, to make the system simpler, we make it more complicated... I was never sold on this.

shuzchen 8y ago

From the article: "For single consumer use cases, we saw a 68% resource savings, and for fanout cases with multiple consumers we saw a 75% resource savings." At twitter's server spend it makes perfect sense to make such an investment.