6 ms·

Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing. I feel like this is a bit overboard. And this is before we talk about

• virgilp 8y ago
  > Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing. Well, no, you went too far. Kafka is primarily used for communication & data transport, by most people (can be used in other ways, and it has the KafkaStreams library that enables you do to some computation on said data - but it is, primarily, a transport & communication mechanism; also maybe storage, if you squint right) Spark and Flink might be similar on first sight, but if you look a bit closer you realize Spark is primarily geared towards batch workloads, and Flink towards realtime. Sure, you can do micro-batch in Spark and pretend that's realtime stream processing, but the focus of it is fairly clear - as is the focus of Flink. So both have legitimate rights to exist. I'm not sure about the others, haven't used them. There may be indeed consistent overlap - but I'm sure they are different approaches. What's wrong with that?
  • meritt 8y ago
    On the contrary, I didn't go far enough. I didn't talk about Apache Gearpump, NiFi, Beam, Ignite, or Trident. I know there are subtle differences in each specific technology, and that's probably the same justifications used to support the thesis when building yet another very similar framework. > What's wrong with that? I believe it drastically reduces adoption of these tools because many of us avoid what appears to be bandwagon technologies as we don't want to consciously add layers of future technical debt when a majority of these projects will be abandoned.
    • virgilp 8y ago
      I said "you went to far" to claim Kafka, Spark and Flink do basically the same thing. It's reasonable to use all 3 of them in the same team - so they clearly don't do the same thing. > I believe it drastically reduces adoption of these tools As does any competition. Tons of smartphone makers = less adoption for any one of them - and many will close down. Still not a bad thing.
      • rakoo 8y ago
        > As does any competition. Those are open-source frameworks, which is antithetical to competition. The whole point of open-source is that the contributions of one entity can benefit everyone, while in this situation, having so many frameworks "dilutes" the efforts because the same problems need to be solved in each framework. Smartphone makers aren't here to share their technical advances with everyone, they're here to make money. I'm not saying we should purposefully agree on killing all but one framework, but the ideal situation is if one or two come out at the top as being just the best in their category so that everyone just use them, and all efforts converge towards those.
        • virgilp 8y ago
          I don't think that's how it works; "best" is sooo hard to define. Humans are very diverse, and have different needs/preferences. Also what is best today may not be best tomorrow (not because of the system itself, but because of changes in the environment), so it pays off to have "sub-par" systems evolve in parallel with different slightly different goals or implementations. I don't see "open-source" as being antithetical to competition, at all. In fact, e.g. Databricks only exists because of Spark. There is competition in the open-source world too, and IMO that's a good thing.
        • PurpleRamen 8y ago
          Since when is there no competition and diversity in open source? Isn't too much choice usually a major complain with linux, programming languages, frameworks, etc?
• powerapple 8y ago
  ... I am using Apache Pulsar to replace current Kafka setup * Apache Pulsar is an open-source distributed pub-sub messaging system
  • continuations 8y ago
    What compelled you to replace Kafka with Pulsar?
    • dvlsg 8y ago
      Not OP, but I'm keeping a close eye on pulsar. Im not sure I'd want to replace an existing / working Kafka setup with it, but if pulsar catches on, I would very strongly consider using it for newer projects. I haven't used pulsar extensively in production for anything, but all the setup and testing I've done on my own was a pleasure. Hoping pulsar continues to grow.
• m_mueller 8y ago
  isn't this just a sign of a still young field (data science)? maybe we'll soon start to enter a consolidation phase.
• asavinov 8y ago
  > Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing. Yes, conceptually they are very similar. If youu want something radically new then check out Bistro Streams: https://github.com/asavinov/bistro https://github.com/asavinov/bistro
• mxmxm 8y ago
  Not quite. All of these are open-source projects but they are very different in many aspects: * Apache Flink Sophisticated stream processing framework with focus on robustness (managed memory) and correctness (exactly-once semantics) * Apache Flume Tailored towards log data. * Apache Storm First stream processing framework. Legacy. * Apache Samza Only used at LinkedIn. Tight to Hadoop's YARN. * Apache Spark Only great in batch processing. * Apache Apex Dead project. Tight to Hadoop's YARN. * Apache Kafka A distributed message queue with simple stream processing built on top via the Confluent Platform.
  • dankohn1 8y ago
    Nice list. There are also non-Apache projects like NATS, hosted by CNCF. Here are the 19 streaming and messaging projects and products that CNCF is tracking: https://landscape.cncf.io/category=streaming-messaging&format=card-mode&grouping=category https://landscape.cncf.io/category=streaming-messaging&forma...
  • michaelmior 8y ago
    > Only used at LinkedIn. And Intuit, Uber, Netflix, VMWare, .... https://cwiki.apache.org/confluence/display/SAMZA/Powered+By https://cwiki.apache.org/confluence/display/SAMZA/Powered+By
• buboard 8y ago
  Solving problems of the internet behemoths -> creating opportunity to be acquired.
• rsynnott 8y ago
  In the beginning, there was Hadoop, which was just a MapReduce clone, more or less. And then there was Hadoop 2, which was kind of meant to do everything. And that didn't really work out, so there was specialisation. I don't really see the problem. Many of these do different things, others do similar things different ways. Some are essentially dead ends (I doubt there are many new users of Storm, say). This all seems pretty normal.
  • justicezyx 8y ago
    No Hadoop does more than MapReduce, much more. Primarily: MapReduce does not have a distributed file-like block storage.
    • bunderbunder 8y ago
      OK, so, to be more precise: Hadoop MapReduce does what Google MapReduce does, and Hadoop Distributed File System does what the Google Filesystem does. And Apache Spark does the same thing as that other Google product whose name I forget. And Drill vs Dremel, etc etc etc. I think that the OP is plenty right for the purposes of the point they were making. And I think (from a non-Googler's perspective) that it's a worthwhile point, since the FOSS ecosystem has since evolved along similar lines to Google's internal tools.
      • erik_seaberg 8y ago
        Apache Spark does what Google FlumeJava does, express a chain of map/reduce steps in a DSL (and inline away some intermediate results). Apache Flume is unrelated except for the confusing name.
• diehunde 8y ago
  > Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing. That's like saying C++, Java, C#, Scala, Python, Clojure... all do basically the same thing
• lazzlazzlazz 8y ago
  As a frequent user of almost half the programs you've listed... I couldn't disagree more. They're completely different. I'm surprised to see this level of misunderstanding posted with such confidence.
  • kevinconaway 8y ago
    It makes you wonder about some of the other insightful sounding top comments on topics that you aren't as well versed in