6 ms·

I'm curious if there's some conflicting, underlying motives from their senior level technology folks. They had a game changing tool in Apache Storm...once Nath

I'm curious if there's some conflicting, underlying motives from their senior level technology folks. They had a game changing tool in Apache Storm...once Nathan Marz (creator of Storm) left twitter, another senior engineer took over that group and created his OWN variant of Twitter Storm called Heron. After much engineering PR and github fanfare [0], couple years later he leaves to start his own company to do 'enterprise heron' [1]. So is the goal here to use something that works well for their end-users? Or is it to create a shiny new tool that they can spin off into a new company? 0: https://news.ycombinator.com/item?id=11770758 https://news.ycombinator.com/item?id=11770758 1: https://architecht.io/streamlio-founders-on-why-the-world-needs-a-new-streaming-data-platform-bdbb43c3f886?gi=ec98fbc31a6f https://architecht.io/streamlio-founders-on-why-the-world-ne...

fnbr 8y ago
The joke that I've heard is "performance (review) driven development". If the way to get promoted is to launch a shiny new product, then your most senior people will be the best at finding shiny new products to launch, even if that's not the right technical decision to make.
- blackaspen 8y ago
  "Promobird" and "Promobird V2" (and so on).
- yxhuvud 8y ago
  Closely related to CV driven development.
- Benjammer 8y ago
  "Oh but our senior engineers would _never_ make such a selfish choice to make themselves look better at the expense of the company, everyone here _believes_ in the mission."
snaky 8y ago
There is resume driven development for blue collar coders masses. But this needs its' own name.
manigandham 8y ago
Streamlio is at least working on Apache Pulsar, which is a much better alternative to Kafka.
- dominotw 8y ago
  better?
  - manigandham 8y ago
    Yes, in basically every way. The website has a good overview unless you have specific questions: https://pulsar.apache.org/docs/en/concepts-overview/ https://pulsar.apache.org/docs/en/concepts-overview/
    - kasey_junk 8y ago
      I really like Pulsar and it definitely has overlap for some use cases with Kafka but its important to understand that they are 2 different kinds of systems. Pulsar is a classic Pub/Sub system. It offers persistence but the basic concepts are what you'd find in most any Pub/Sub system. Kafka is a distributed log (data structure, not tracing) system. Its basic concepts are what you would expect around something that is persisting events. In many ways its better to think about Kafka as an event datastore and not a messaging system. In particular, Kafka is much better in use cases where you want to provide all events to many different consumers on their own time frames, over and over again. Use cases like building distributed caches and replication formats will likely fit Kafka better than Pulsar. Also, Kafka has much higher industry adoption. You are much more likely to have third party support for Kafka than for Pulsar. Also each has a seperate Apache project dependency. Zookeeper has... warts, but they are well known and operating Zookeeper clusters is a much more common skillset than operating BookKeeper. Again, I really like Pulsar but claiming it is better in basically every way is just false.
      - manigandham 8y ago
        I don't see the differences that you list. Both systems produce and subscribe to named topics, with multiple consumers which can read and re-read at their own pace, with data sharded by key to scale, and configurable retention. Pub/sub is the interface to the underlying log storage in both, and you can layer an event sourcing, service bus, CEP, MQ, or any other messaging buzzword on top. However, Pulsar goes further than Kafka. It supports millions of topics, multi-tenant namespacing, more consumer options (exclusive, shared/group), per-message acknowledgements instead of a single offset, non-persistent topics for broadcast or ephemeral messaging, geo-replication, tiering to cloud storage (useful for that event store), and a functions platform for lightweight processing. Pulsar also uses Zookeeper for coordination and has official Kubernetes deployments for easy operations. You're right that Kafka has much wider industry usage, and perhaps that is a significant advantage for most, but the constant announcement of yet another suite of helper tools by every major company only seems to show that it's not quite polished. Other than community and integrations marketshare, I cannot think of any functional area where Kafka is better than Pulsar. Can you name any?
abalone 8y ago
Speaking of underlying motives... you strongly imply that Heron was not a significant improvement on Storm and doesn’t “work well for end users.” Without evidence. Why? Re: motives, “Heron grew out of real concerns Twitter was having with its large deployment of Storm topologies. These included difficulties with profiling and reasoning about Storm workers when scaled at the data level and at a topology level, the static nature of resource allocation in comparison to a system that runs on Mesos or YARN, lack of back-pressure support, and more.” “Although Twitter could have adopted Apache Spark or Apache Flink, that would have involved rewriting all of Twitter's existing code.”[1] [1] https://www.infoworld.com/article/3078134/analytics/had-it-with-apache-storm-heron-swoops-to-the-rescue.html https://www.infoworld.com/article/3078134/analytics/had-it-w...
- capkutay 8y ago
  i never said it doesn't work well for end users. I just question the intentions when they do a huge PR campaign for an internal project...then the leader of that project leaves to start his own company shortly after. And no...more PR about heron is not going to convince me that 'it's a huge improvement over Kafka'. Kafka has massive adoption, and almost no Kafka users I've come across have expressed a burning desire for Heron or Pulsar.

fnbr 8y ago

The joke that I've heard is "performance (review) driven development". If the way to get promoted is to launch a shiny new product, then your most senior people will be the best at finding shiny new products to launch, even if that's not the right technical decision to make.

blackaspen 8y ago
"Promobird" and "Promobird V2" (and so on).
yxhuvud 8y ago
Closely related to CV driven development.
Benjammer 8y ago
"Oh but our senior engineers would _never_ make such a selfish choice to make themselves look better at the expense of the company, everyone here _believes_ in the mission."

snaky 8y ago

There is resume driven development for blue collar coders masses. But this needs its' own name.

manigandham 8y ago

Streamlio is at least working on Apache Pulsar, which is a much better alternative to Kafka.

dominotw 8y ago
better?
- manigandham 8y ago
  Yes, in basically every way. The website has a good overview unless you have specific questions: https://pulsar.apache.org/docs/en/concepts-overview/ https://pulsar.apache.org/docs/en/concepts-overview/
  - kasey_junk 8y ago
    I really like Pulsar and it definitely has overlap for some use cases with Kafka but its important to understand that they are 2 different kinds of systems. Pulsar is a classic Pub/Sub system. It offers persistence but the basic concepts are what you'd find in most any Pub/Sub system. Kafka is a distributed log (data structure, not tracing) system. Its basic concepts are what you would expect around something that is persisting events. In many ways its better to think about Kafka as an event datastore and not a messaging system. In particular, Kafka is much better in use cases where you want to provide all events to many different consumers on their own time frames, over and over again. Use cases like building distributed caches and replication formats will likely fit Kafka better than Pulsar. Also, Kafka has much higher industry adoption. You are much more likely to have third party support for Kafka than for Pulsar. Also each has a seperate Apache project dependency. Zookeeper has... warts, but they are well known and operating Zookeeper clusters is a much more common skillset than operating BookKeeper. Again, I really like Pulsar but claiming it is better in basically every way is just false.
    - manigandham 8y ago
      I don't see the differences that you list. Both systems produce and subscribe to named topics, with multiple consumers which can read and re-read at their own pace, with data sharded by key to scale, and configurable retention. Pub/sub is the interface to the underlying log storage in both, and you can layer an event sourcing, service bus, CEP, MQ, or any other messaging buzzword on top. However, Pulsar goes further than Kafka. It supports millions of topics, multi-tenant namespacing, more consumer options (exclusive, shared/group), per-message acknowledgements instead of a single offset, non-persistent topics for broadcast or ephemeral messaging, geo-replication, tiering to cloud storage (useful for that event store), and a functions platform for lightweight processing. Pulsar also uses Zookeeper for coordination and has official Kubernetes deployments for easy operations. You're right that Kafka has much wider industry usage, and perhaps that is a significant advantage for most, but the constant announcement of yet another suite of helper tools by every major company only seems to show that it's not quite polished. Other than community and integrations marketshare, I cannot think of any functional area where Kafka is better than Pulsar. Can you name any?

abalone 8y ago

Speaking of underlying motives... you strongly imply that Heron was not a significant improvement on Storm and doesn’t “work well for end users.” Without evidence. Why? Re: motives, “Heron grew out of real concerns Twitter was having with its large deployment of Storm topologies. These included difficulties with profiling and reasoning about Storm workers when scaled at the data level and at a topology level, the static nature of resource allocation in comparison to a system that runs on Mesos or YARN, lack of back-pressure support, and more.” “Although Twitter could have adopted Apache Spark or Apache Flink, that would have involved rewriting all of Twitter's existing code.”[1] [1] https://www.infoworld.com/article/3078134/analytics/had-it-with-apache-storm-heron-swoops-to-the-rescue.html https://www.infoworld.com/article/3078134/analytics/had-it-w...

capkutay 8y ago
i never said it doesn't work well for end users. I just question the intentions when they do a huge PR campaign for an internal project...then the leader of that project leaves to start his own company shortly after. And no...more PR about heron is not going to convince me that 'it's a huge improvement over Kafka'. Kafka has massive adoption, and almost no Kafka users I've come across have expressed a burning desire for Heron or Pulsar.