14 ms·

Apache Heron: A realtime, distributed, fault-tolerant stream processing engine

JasonFruit 5y ago
I read this as a realtime, distributed, fault-tolerant steam engine, which was hard to imagine.
- conjectures 5y ago
  Tbh I would be more excited about that headline.
doteka 5y ago
How many distributed stream processing engines is the Apache foundation planning to collect? At this point it seems like there’s more projects that do this (if you squint a bit), than companies with a serious usecase for that type of architecture.
- jarym 5y ago
  I wonder this too - I guess enough software engineering teams had similar problems with no solutions and each set about building their own? Not sure how else this proliferation happened.
- aasasd 5y ago
  Apparently it's another one from Twitter.
  - EdwardDiego 5y ago
    At least they had the decency to not open source their Bookkeeper backed pub/sub system when they gave up on it and switched to Kafka... ...then some of the employees involved moved to Yahoo, built it again, and then open sourced it as Pulsar. Then moved onto form a Confluent v2 to sell their Kafka v2 (now with even more Zookeeper!)
  - mumblemumble 5y ago
    I had a problem, so I invented Summingbird, and now I can relatively inexpensively invent new problems on an annual basis.
- elric 5y ago
  I work for a company that could greatly benefit from an out-of-the-box distributed stream processing engine (we've been rolling our own for over a decade). At this point, it's pretty much impossible to pick one. All similar Apache tools have similar looking web pages, promising similar benefits, similar use cases, etc. What are the differentiating factors? At which point does it make sense to pick Heron over one of the others? Or vice versa?
  - himoacs 5y ago
    I recommend checking out Solace. They have been in business for 20 years. It's not open source though but packs all the enterprise features you would need.
    - jsight 5y ago
      https://xkcd.com/927/ https://xkcd.com/927/
    - haik90 5y ago
      Had a lot trouble running solace on vm and docker, spend a lot time try to find root cause memory leak (happen every few months) I still don't get why they use VPN terms for event broker
      - himoacs 5y ago
        That's a shame. I work at Solace so shoot me a message and I will show you how to set it up if you are interested. Solace's first product was hardware appliances which are still used for high throughput and low latency usecases. Concept of VPN was used to set up isolated virtual brokers so different teams can have their own environments on a shared hardware appliance. The concept was ported over to software as well and is extremely useful in an enterprise environment. It allows different teams to have their own virtual brokers but not have to pay for or manage multiple brokers.
        Nullabillity 5y ago
- BenoitP 5y ago
  The long term trend for data access is having a reactive component; stream processing engines allow you do have that. On the other hand databases have a huge lock-in power (been trying to strangle an Oracle myself for over 5 years now). It is lucrative to be in the database business. I'd say that every project with a claim of some improvement can, and should try to establish itself in the market; and that having it join the Apache foundation is a great way to get some brand recognition on the cheap. ---- Also, Heron is not that new. It has been developed at Twitter, for replacing Storm IIRC.
  - skywhopper 5y ago
    But none of them spell out for what use cases they excel over the other options. Since they are all under the ASF banner, it’s impossible to know which is “better” for me. They are all just “great stream processing engines”. But surely they must have diverging properties for given use cases. But none of the pages even attempt to say how they differ. Just “try it out!!”
    - mumblemumble 5y ago
      Years ago, I found a scientific paper that evaluated them all on a fairly detailed rubric. As I recall, it turned out that they're all, in actual fact, crap stream processing engines. You just need to pick the one that's crap at things you don't need to do. (I could frame it in a more glass half full way, but I find that the pessimistic way of looking at it helps a lot with trimming down options when you have far, far, far too many options.)
      - brandmeyer 5y ago
        Citation needed. And not in the snarky way - I'd really like to read that paper :)
        mumblemumble 5y ago
        I've long since lost track of it. It would be 5 or 6 years out of date at this point, anyway, so probably not very useful for informing decisions anymore.
- ww520 5y ago
  It really comes down to the latency and throughput requirements. Projects are architected differently for different latency expectation and different throughput expectation. In event processing there's a continuum of expected latencies from batch processing to realtime. Batch processing is typically running reports over large volume of events (good for throughput). Hadoop is a good example. On the other end, sub-second realtime report is possible with Heron/Storm. Spark is kind of in the middle with hybrid mini-batching. Reportedly Twitter has used Heron/Storm to track word counts in all the tweets to find trending topics, where the latency between a new tweet coming in to the word counts updated over the whole network is in 100s milliseconds.
- phamilton4 5y ago
  I started using Apache Storm in 2012 and was really surprised at the number of Apache projects for stream processing back in 2015 or so when I did some research for moving off of storm. I remember Flink, Samza, Spark, Flume and NiFi(?) all being probably suitable options. I remember thinking exactly what you said! I guess there are enough users to keep the communities alive?
  - void_mint 5y ago
    *Enough companies reinventing the wheel to spinoff new Apache distributed stream processing libraries.
loremipsium 5y ago
hadoop, kafka, storm, spark, flink, samza, confluent This tastes an aweful lot like javascript framework hell
- sivakon 5y ago
  You forgot gearpump http://incubator.apache.org/projects/gearpump.html http://incubator.apache.org/projects/gearpump.html
  - querulous 5y ago
    and beam: beam.apache.org
  - ternaryoperator 5y ago
    It appears gearpump was retired in 2018
- EdwardDiego 5y ago
  - Hadoop is an ecosystem. - Kafka is a distributed log. - Storm, Samza & Flink are stream processing engines. - Spark is a Map/Reduce framework that uses memory to cache computations to provide some performance increase over other disk-based frameworks. It can also do some streaming computations if you squint hard enough. - Confluent is a company that sells an enterprise Kafka. Not really sure the comparison you made is apt.
  - deleted 5y ago
    [deleted]
  - shipp02 5y ago
    Kafka's homepage[1] advertises stream processing as a feature > Built-in Stream Processing > Process streams of events with joins, aggregations, filters, transformations, and more, using event-time and exactly-once processing. [1]:https://kafka.apache.org/ https://kafka.apache.org/
    - 5y ago
aynyc 5y ago
Sometimes, I feel like Apache foundation is like Thanos, collecting all the distributed engines and watch the IT world burn. From the beginning, Heron was envisioned as a new kind of stream processing system, built to meet the most demanding of technological requirements, to handle even the most massive of workloads, and to meet the needs of organizations of all sizes and degrees of complexity. Amongst these requirements: The ability to process billions of events per minute Extremely low end-to-end latency Predictable behavior regardless of scale and in the face of issue like extreme traffic spikes and pipeline congestion Simple administration, including: The ability to deploy on shared infrastructure Powerful monitoring capabilities Fine-grained configurability Easy debuggability I can't wait for my next startup interview. We have a requirement of 25 messages per hour, with 10KB per message, you think you can build the ingestion pipeline using Kafka and MongoDB on a 10 node M5d.24xlarge cluster?
- manishsharan 5y ago
  Not just startups. I interviewed with a large financial institution recently and they had their heart set on Kafka. From what I learnt , they did not need Kafka -- the message order was not important, there was only one producer and one consumer and the volume of data was not extra-ordinary -- but they wanted to be on Kafka as that was the enterprise direction. I might have come across as an old fart when I asked the why they could not use their existing MQ infrastructure.
  - igetspam 5y ago
    We've been using Kafka for a while. We're thinking about ditching it for something AMQP. For all it's bells and whistles, it's just not really all that exciting. We're probably spending 5x more on Kafka than we could something much simpler.
    - jdmichal 5y ago
      We're doing the same. Actively moving away from Kafka, which was only ever used as a message queue anyway, to an actual message queue. We might move back at some point in the future to rebuild an actual event-driven processing model. But for now, it's not worth the additional complexity for the scale and systems we actually have.
ubertaco 5y ago
Oh, is this that "next-gen Storm" project that Twitter built? Seems like they've finally given it to Apache, like they did with Storm after buying the company that built it. Based on a quick Wikipedia skim, looks like the answer is "yes". That explains this bullet point: > Heron is API compatible with Apache Storm and hence no code change is required for migration.
- deleted 5y ago
  [deleted]
- squarecog 5y ago
  Donated to Apache in 2018. https://blog.twitter.com/engineering/en_us/topics/open-source/2018/heron-donated-to-apache-software-foundation https://blog.twitter.com/engineering/en_us/topics/open-sourc... (and open-sourced in 2016, having started development in 2014).
bob1029 5y ago
If you are interested in hyper-scale event processing but you want to learn it from first principles, I strongly recommend following Martin Thompson's talks. Here is an example of one on cluster consensus: > https://www.youtube.com/watch?v=GFfLCGW_5-w https://www.youtube.com/watch?v=GFfLCGW_5-w and another on event log architecture: > https://www.youtube.com/watch?v=RlwO6CJbJjQ https://www.youtube.com/watch?v=RlwO6CJbJjQ After digging through all of this material and playing around with LMAX Disruptor & Raft, I have been able to develop a really good understanding of how to build these sorts of systems on my own. Fun constraints like "Only one thread actually mutates anything, and its the same one over and over" make for incredibly elegant implementation opportunities. Not having to constantly hunt down exotic thread-safe data structures means that you can focus on building actual value. Latency is the biggest devil you will dance with in this arena, so almost everything you do will be oriented around mitigating that effect. Latency both at the network and inside the CPU/memory/storage. It applies at every level.
- runT1ME 5y ago
  >Only one thread actually mutates anything, and its the same one over and over Sounds great in theory and usually results in fantastic average case, but you get a hot partition and suddenly you can't work share and things go south. It's not a perfect solution.
  - bob1029 5y ago
    If you have a single serialized+synchronous business context in which activities must be processed, you literally have no other option than to use a single physical core if you want to go fast. The trick is to minimize the raw amount of bytes that must enter into that synchronous context. Maybe the account disclosure PDFs can be referred to by some GUID token in AWS S3, but the actual decimal account balance/transaction facts should be included in the event data stream. One other option is to talk to the business and see if you can break their 1 gigantic synchronous context into multiple smaller ones that can progress independently.
  - random314 5y ago
    There is no working around Amdahls law unless you redesign your app.
rayrrr 5y ago
For once almost all commenters so far are thinking the same thing I am. Yet another stream processor from Apache. Might as well create the acronym now. YASP.
- EdwardDiego 5y ago
  Not so much "from" Apache, as "sent to Apache to die with dignity after the originating big company git bored of it"
dikei 5y ago
IMHO, if you need stream processing, start with Apache Flink. It not only offers a much easier to user API compare to Storm and Heron, but also has a superior execution model for time-based, exactly-once, stateful stream processing.
- pixelmonkey 5y ago
  Flink is pretty cool, but the Flink Python API and Beam Python APIs are pretty atrocious, and some of the higher-level APIs (like Flink SQL) are pretty hard to grok. I kicked the tires on Flink and tried to love it, but I couldn't get there. Storm (which is Heron's inspiration and is still better than Heron, IMO) is a lot simpler conceptually. But it's pretty clear Flink was built to avoid the need to combine Storm Streaming + Spark Batch for "Lambda Architecture" style setups.
  - uDontKnowMe 5y ago
    Python's cool, but "right tool for the job" and all that. Just dive in with the Java or Scala api's and you'll have a better time :-).
    - pixelmonkey 5y ago
      Thanks for the suggestion, but, pragmatically, it's not an option. My colleagues and I haven't written data processing, distributed systems, NLP/ML/datasci, or web tier code in Java since 2006, and we don't plan on starting now. We've used Python at scale on petabyte-sized production data, multi-billion-request API tiers, and high-concurrency low-latency data processing all the while. Java is the right tool for writing system-level code in some isolated contexts, but Python is the right tool for the job for a huge number of important use cases, with those use cases growing by the day. I love Java (and appreciate Kotlin and Clojure), but with parallel compute power cheaper by the day, Python's focus on code simplicity, open source ecosystem, and programmer happiness continues to win the day.
- manishsharan 5y ago
  If you are familiar with Flink, would you mind sharing your use cases? why Flink and not Kafka ? I have experience with Kafka but I have not had a reason to investigate Flink for my use cases.
cblconfederate 5y ago
For a nonprofit, apache seems to have an unhealthy affection for solving scaling problems that only a few giant rich companies have
- detaro 5y ago
  That criticism doesn't really make sense, given that Apache the non-profit isn't developing the software.
  - cblconfederate 5y ago
    But it is funding it
    - detaro 5y ago
      The Apache Foundation isn't funding development in any meaningful way, no.
      - uDontKnowMe 5y ago
        What does the Apache Foundation do/provide/give these projects if no funding is involved? Is it just like a governance thing like "if this project falls into an abandoned state, we'll oversee the process of transferring ownership to any interested and credible parties interested in taking over"?
        detaro 5y ago
        Mostly a governance thing, yes. "Community infrastructure, coordination and legal framework in a box".
- kristjansson 5y ago
  You’ve got the flow backwards. Giant rich companies contribute projects that solve their problems to the ASF
deleted 5y ago
[deleted]
pixelmonkey 5y ago
Wonder why this is getting posted today in particular? The quick summary here is that this was a clean-house rewrite of Apache Storm done by an internal team at Twitter. As an open source project history refresher, Apache Storm was originally built by a startup called Backtype, and the project was led by Nathan Marz, the technical founder of Backtype. Then, Backtype was acquired by Twitter, and Storm became a major component for large-scale stream processing (of tweets, tweet analytics, and other things) at Twitter. I wrote a summary of the "interesting bits" of Apache Storm here: https://blog.parse.ly/storm/ https://blog.parse.ly/storm/ However, at a certain point, Nathan Marz left Twitter, and a different group of engineers tried to rethink Storm inside Twitter. There was also a lot of work going on around Apache Mesos at the time. Heron is kind of a merger of their "rethinking" of Storm while also making it possible to manage Storm-like Heron clusters using Mesos. But, I don't think Heron really took off. Meanwhile, Storm got very, very stable in the 1.x series, and then had a clean-house rewrite from Clojure to Java in the 2.x series, mainly to improve performance even more. The last stable/major Storm release was in 2020. Storm provides a stream processing programming API, a multi-lang wire protocol, and a cluster management approach. But certain cluster computing problems can probably be better solved at the infrastructure layer today. (For example, Storm was developed before the whole container + docker + k8s focus in cloud ops.) That said, it's still a very powerful system; on my team, we process 75K+ events per second across hundreds of vCPU cores and thousands of Python processes with sub-second latencies by combining Storm and Kafka with our open source Python project, streamparse. https://github.com/Parsely/streamparse https://github.com/Parsely/streamparse The core problems Storm solves: modeling data processing as a computation graph; high-speed network communication between threads, processes, and nodes; message delivery guarantees and retry capabilities; tunable parallelism; built-in monitoring and logging; and much more. (Also, I'd be remiss if I didn't mention -- if you're interested in stream processing and distributed computing, we are hiring Python Data Engineers to work on a stack involving Storm, Spark, Kafka, Cassandra, etc.) -- https://www.parse.ly/careers/python_data_engineer https://www.parse.ly/careers/python_data_engineer
- AtlasBarfed 5y ago
  Is there a discussion as to the decision to dump clojure --> java for "performance reasons"? I'm not even a clojure user, but my impression was that it was pretty performant. I remember a discussion that they didn't even really need the JVM invokedynamic because they were doing pretty well without it, so that made me think it was close to pure JVM speed.
yamrzou 5y ago
I've worked with a bunch of stream processing engines a few years ago (Samza, Kafka Streaming, Spark Streaming, Storm and Flink), and did a comparison between them as part of my internship. IMO, Apache Flink is the most complete project for those use cases. It is well maintained and the devs are very helpful when asked on the mailing lists.
- uDontKnowMe 5y ago
  Totally agreed! Flink is way underrated.
  - saryant 5y ago
    Had it improved? We tried to ship a fairly large product on Flink a couple years ago and it was a nightmare. It’s threading model was guaranteed to blow up in your face and running it reliably on a noisy network was near impossible.
CSDude 5y ago
I really wonder what people actually use stream processing for, like very concrete examples. My best examples would only go far filtering a stream over a time window to compute an aggregate. My job does not require anything more, it's always basic ETL, but I really need to hear specific examples where it's useful for others. Been a long time fan of Apache Flink.
- thinkharderdev 5y ago
  Standard use cases for stream processing are: 1. Enriching event streams. Say you have a stream of log records with an IP address field. You want to enrich with a geo-location before sending the logs to Elasticsearch. 2. Windowed aggregation. Maybe you have an application that is emitting "login" events and you want to to detect login attempts from different IP addresses within X minutes of each other. 3. Joining multiple event streams. You have multiple different event streams and you want to join them together using some common join key (maybe session ID or something like that) to compute a metric that aggregates all of them. There are plenty of more esoteric use cases as well.
  - CSDude 5y ago
    They are the obvious ones, I'm looking for more advanced & specific ones.
    - infinite8s 5y ago
      Operational analytics when dealing with real world systems (transportation logistics, tracking machine states on factory floors, sensor data fusion, real-time operational dashboards for capital markets, etc).
- Mehdi2277 5y ago
  Streaming model training for recommendation systems like tiktok, facebook, youtube, etc. If you want models to learn quickly to new events allowing them to do better on very new content/users you will want to use a streaming pipeline. I have seen flink used for model training at large scale here. Model evaluation can also be done in a streaming manner with training and needs to be done parallel for model monitoring. The lowest latency model refresh time I've come across is ~5 minutes. Going lower is likely to be a lot of data transfers for model synchronization (sync the serving model with training model) for little value.
majormajor 5y ago
I would love to know more about how they do stateful stuff, since I haven't found anything that can compare to Flink there, but Flink has really poor quality of life stuff compared to some other options (e.g. Flink serialization setup pains vs Beam coders) but their docs kinda trail off here: > Non-idempotent stateful topologies are stateful topologies that do not apply processing logic along the model of "multiply by zero" and thus cannot provide effectively-once semantics. An example of a non-idempotent https://heron.incubator.apache.org/docs/heron-delivery-semantics#stateful-topologies https://heron.incubator.apache.org/docs/heron-delivery-seman... (I'm puzzled by their idempotent vs non-idempotent stateful topology description, because if something is mutating an internal state upon receiving events, it will likely be non-idempotent by design... unless they just mean "idempotent stateful" here to refer to keeping track of source/output position state and such.) (They also do say that can only support state storage in ZK or local FS, which feels like a likely non-starter compared to Flink for some of my use cases.)
- dikei 5y ago
  Flink serialization become much less painful when we switch every value type in our job to Avro. It requires a bit more code compare to using Java POJO or Scala case class, but you can actually add more field to existing data type without breaking anything now :)
Wonnk13 5y ago
sometimes I poke fun at the front-end folks and all the new frameworks they're constantly chasing. Lately it's been starting to feel the same for distributed systems. How many streaming engines are there now under Apache? Four?
- haggy 5y ago
  Depends how you group them. If we're talking battle tested then yea maybe four but I believe there are many more that don't have widespread adoption
fmakunbound 5y ago
Is it another corporate dumping at Apache or does Twitter seem to continue to support it as an open source project there?
karmasimida 5y ago
Anyone use this in production except Twitter? Just curious.
elchief 5y ago
So, AWS Pelican coming in 3 months?
biggestlou 5y ago
Everyone, please note that this is NOT an announcement of Heron entering the Apache Foundation. This happened several years ago, I believe in 2017 (when I was working on it).
deleted 5y ago
[deleted]
WC3w6pXxgGd 5y ago
Isn't this the same as Apache Kafka?
antpls 5y ago
Since no one mentioned it in the comments so far, please add Hazelcast Jet to the long list of stream processors : https://github.com/hazelcast/hazelcast-jet https://github.com/hazelcast/hazelcast-jet
aetherspawn 5y ago
Uhh, a triple-take: read this twice as Apache Heroin, and I’m not a druggy. Is it apt? Reader exercise.
diehunde 5y ago
I find odd so many people complaining about having too many alternatives for streaming systems. If you are not an advanced user of these systems of course you won't be able to choose the right one and it won't be your job anyways. Think of this, we have dozens of databases out there and there are new ones coming out every week and nobody complains about that. That's because there's way more people that understands the differences and the use cases so it makes sense.

I read this as a realtime, distributed, fault-tolerant steam engine, which was hard to imagine.

conjectures 5y ago
Tbh I would be more excited about that headline.

How many distributed stream processing engines is the Apache foundation planning to collect? At this point it seems like there’s more projects that do this (if you squint a bit), than companies with a serious usecase for that type of architecture.

jarym 5y ago
I wonder this too - I guess enough software engineering teams had similar problems with no solutions and each set about building their own? Not sure how else this proliferation happened.
aasasd 5y ago
Apparently it's another one from Twitter.
- EdwardDiego 5y ago
  At least they had the decency to not open source their Bookkeeper backed pub/sub system when they gave up on it and switched to Kafka... ...then some of the employees involved moved to Yahoo, built it again, and then open sourced it as Pulsar. Then moved onto form a Confluent v2 to sell their Kafka v2 (now with even more Zookeeper!)
- mumblemumble 5y ago
  I had a problem, so I invented Summingbird, and now I can relatively inexpensively invent new problems on an annual basis.
elric 5y ago
I work for a company that could greatly benefit from an out-of-the-box distributed stream processing engine (we've been rolling our own for over a decade). At this point, it's pretty much impossible to pick one. All similar Apache tools have similar looking web pages, promising similar benefits, similar use cases, etc. What are the differentiating factors? At which point does it make sense to pick Heron over one of the others? Or vice versa?
- himoacs 5y ago
  I recommend checking out Solace. They have been in business for 20 years. It's not open source though but packs all the enterprise features you would need.
  - jsight 5y ago
    https://xkcd.com/927/ https://xkcd.com/927/
  - haik90 5y ago
    Had a lot trouble running solace on vm and docker, spend a lot time try to find root cause memory leak (happen every few months) I still don't get why they use VPN terms for event broker
    - himoacs 5y ago
      That's a shame. I work at Solace so shoot me a message and I will show you how to set it up if you are interested. Solace's first product was hardware appliances which are still used for high throughput and low latency usecases. Concept of VPN was used to set up isolated virtual brokers so different teams can have their own environments on a shared hardware appliance. The concept was ported over to software as well and is extremely useful in an enterprise environment. It allows different teams to have their own virtual brokers but not have to pay for or manage multiple brokers.
      - Nullabillity 5y ago
BenoitP 5y ago
The long term trend for data access is having a reactive component; stream processing engines allow you do have that. On the other hand databases have a huge lock-in power (been trying to strangle an Oracle myself for over 5 years now). It is lucrative to be in the database business. I'd say that every project with a claim of some improvement can, and should try to establish itself in the market; and that having it join the Apache foundation is a great way to get some brand recognition on the cheap. ---- Also, Heron is not that new. It has been developed at Twitter, for replacing Storm IIRC.
- skywhopper 5y ago
  But none of them spell out for what use cases they excel over the other options. Since they are all under the ASF banner, it’s impossible to know which is “better” for me. They are all just “great stream processing engines”. But surely they must have diverging properties for given use cases. But none of the pages even attempt to say how they differ. Just “try it out!!”
  - mumblemumble 5y ago
    Years ago, I found a scientific paper that evaluated them all on a fairly detailed rubric. As I recall, it turned out that they're all, in actual fact, crap stream processing engines. You just need to pick the one that's crap at things you don't need to do. (I could frame it in a more glass half full way, but I find that the pessimistic way of looking at it helps a lot with trimming down options when you have far, far, far too many options.)
    - brandmeyer 5y ago
      Citation needed. And not in the snarky way - I'd really like to read that paper :)
      - mumblemumble 5y ago
        I've long since lost track of it. It would be 5 or 6 years out of date at this point, anyway, so probably not very useful for informing decisions anymore.
ww520 5y ago
It really comes down to the latency and throughput requirements. Projects are architected differently for different latency expectation and different throughput expectation. In event processing there's a continuum of expected latencies from batch processing to realtime. Batch processing is typically running reports over large volume of events (good for throughput). Hadoop is a good example. On the other end, sub-second realtime report is possible with Heron/Storm. Spark is kind of in the middle with hybrid mini-batching. Reportedly Twitter has used Heron/Storm to track word counts in all the tweets to find trending topics, where the latency between a new tweet coming in to the word counts updated over the whole network is in 100s milliseconds.
phamilton4 5y ago
I started using Apache Storm in 2012 and was really surprised at the number of Apache projects for stream processing back in 2015 or so when I did some research for moving off of storm. I remember Flink, Samza, Spark, Flume and NiFi(?) all being probably suitable options. I remember thinking exactly what you said! I guess there are enough users to keep the communities alive?
- void_mint 5y ago
  *Enough companies reinventing the wheel to spinoff new Apache distributed stream processing libraries.

loremipsium 5y ago

hadoop, kafka, storm, spark, flink, samza, confluent This tastes an aweful lot like javascript framework hell

sivakon 5y ago
You forgot gearpump http://incubator.apache.org/projects/gearpump.html http://incubator.apache.org/projects/gearpump.html
- querulous 5y ago
  and beam: beam.apache.org
- ternaryoperator 5y ago
  It appears gearpump was retired in 2018
EdwardDiego 5y ago
- Hadoop is an ecosystem. - Kafka is a distributed log. - Storm, Samza & Flink are stream processing engines. - Spark is a Map/Reduce framework that uses memory to cache computations to provide some performance increase over other disk-based frameworks. It can also do some streaming computations if you squint hard enough. - Confluent is a company that sells an enterprise Kafka. Not really sure the comparison you made is apt.
- deleted 5y ago
  [deleted]
- shipp02 5y ago
  Kafka's homepage[1] advertises stream processing as a feature > Built-in Stream Processing > Process streams of events with joins, aggregations, filters, transformations, and more, using event-time and exactly-once processing. [1]:https://kafka.apache.org/ https://kafka.apache.org/
  - 5y ago

aynyc 5y ago

Sometimes, I feel like Apache foundation is like Thanos, collecting all the distributed engines and watch the IT world burn. From the beginning, Heron was envisioned as a new kind of stream processing system, built to meet the most demanding of technological requirements, to handle even the most massive of workloads, and to meet the needs of organizations of all sizes and degrees of complexity. Amongst these requirements: The ability to process billions of events per minute Extremely low end-to-end latency Predictable behavior regardless of scale and in the face of issue like extreme traffic spikes and pipeline congestion Simple administration, including: The ability to deploy on shared infrastructure Powerful monitoring capabilities Fine-grained configurability Easy debuggability I can't wait for my next startup interview. We have a requirement of 25 messages per hour, with 10KB per message, you think you can build the ingestion pipeline using Kafka and MongoDB on a 10 node M5d.24xlarge cluster?

manishsharan 5y ago
Not just startups. I interviewed with a large financial institution recently and they had their heart set on Kafka. From what I learnt , they did not need Kafka -- the message order was not important, there was only one producer and one consumer and the volume of data was not extra-ordinary -- but they wanted to be on Kafka as that was the enterprise direction. I might have come across as an old fart when I asked the why they could not use their existing MQ infrastructure.
- igetspam 5y ago
  We've been using Kafka for a while. We're thinking about ditching it for something AMQP. For all it's bells and whistles, it's just not really all that exciting. We're probably spending 5x more on Kafka than we could something much simpler.
  - jdmichal 5y ago
    We're doing the same. Actively moving away from Kafka, which was only ever used as a message queue anyway, to an actual message queue. We might move back at some point in the future to rebuild an actual event-driven processing model. But for now, it's not worth the additional complexity for the scale and systems we actually have.

ubertaco 5y ago

Oh, is this that "next-gen Storm" project that Twitter built? Seems like they've finally given it to Apache, like they did with Storm after buying the company that built it. Based on a quick Wikipedia skim, looks like the answer is "yes". That explains this bullet point: > Heron is API compatible with Apache Storm and hence no code change is required for migration.

deleted 5y ago
[deleted]
squarecog 5y ago
Donated to Apache in 2018. https://blog.twitter.com/engineering/en_us/topics/open-source/2018/heron-donated-to-apache-software-foundation https://blog.twitter.com/engineering/en_us/topics/open-sourc... (and open-sourced in 2016, having started development in 2014).

bob1029 5y ago

If you are interested in hyper-scale event processing but you want to learn it from first principles, I strongly recommend following Martin Thompson's talks. Here is an example of one on cluster consensus: > https://www.youtube.com/watch?v=GFfLCGW_5-w https://www.youtube.com/watch?v=GFfLCGW_5-w and another on event log architecture: > https://www.youtube.com/watch?v=RlwO6CJbJjQ https://www.youtube.com/watch?v=RlwO6CJbJjQ After digging through all of this material and playing around with LMAX Disruptor & Raft, I have been able to develop a really good understanding of how to build these sorts of systems on my own. Fun constraints like "Only one thread actually mutates anything, and its the same one over and over" make for incredibly elegant implementation opportunities. Not having to constantly hunt down exotic thread-safe data structures means that you can focus on building actual value. Latency is the biggest devil you will dance with in this arena, so almost everything you do will be oriented around mitigating that effect. Latency both at the network and inside the CPU/memory/storage. It applies at every level.

runT1ME 5y ago
>Only one thread actually mutates anything, and its the same one over and over Sounds great in theory and usually results in fantastic average case, but you get a hot partition and suddenly you can't work share and things go south. It's not a perfect solution.
- bob1029 5y ago
  If you have a single serialized+synchronous business context in which activities must be processed, you literally have no other option than to use a single physical core if you want to go fast. The trick is to minimize the raw amount of bytes that must enter into that synchronous context. Maybe the account disclosure PDFs can be referred to by some GUID token in AWS S3, but the actual decimal account balance/transaction facts should be included in the event data stream. One other option is to talk to the business and see if you can break their 1 gigantic synchronous context into multiple smaller ones that can progress independently.
- random314 5y ago
  There is no working around Amdahls law unless you redesign your app.

rayrrr 5y ago

For once almost all commenters so far are thinking the same thing I am. Yet another stream processor from Apache. Might as well create the acronym now. YASP.

EdwardDiego 5y ago
Not so much "from" Apache, as "sent to Apache to die with dignity after the originating big company git bored of it"

dikei 5y ago

IMHO, if you need stream processing, start with Apache Flink. It not only offers a much easier to user API compare to Storm and Heron, but also has a superior execution model for time-based, exactly-once, stateful stream processing.

pixelmonkey 5y ago
Flink is pretty cool, but the Flink Python API and Beam Python APIs are pretty atrocious, and some of the higher-level APIs (like Flink SQL) are pretty hard to grok. I kicked the tires on Flink and tried to love it, but I couldn't get there. Storm (which is Heron's inspiration and is still better than Heron, IMO) is a lot simpler conceptually. But it's pretty clear Flink was built to avoid the need to combine Storm Streaming + Spark Batch for "Lambda Architecture" style setups.
- uDontKnowMe 5y ago
  Python's cool, but "right tool for the job" and all that. Just dive in with the Java or Scala api's and you'll have a better time :-).
  - pixelmonkey 5y ago
    Thanks for the suggestion, but, pragmatically, it's not an option. My colleagues and I haven't written data processing, distributed systems, NLP/ML/datasci, or web tier code in Java since 2006, and we don't plan on starting now. We've used Python at scale on petabyte-sized production data, multi-billion-request API tiers, and high-concurrency low-latency data processing all the while. Java is the right tool for writing system-level code in some isolated contexts, but Python is the right tool for the job for a huge number of important use cases, with those use cases growing by the day. I love Java (and appreciate Kotlin and Clojure), but with parallel compute power cheaper by the day, Python's focus on code simplicity, open source ecosystem, and programmer happiness continues to win the day.
manishsharan 5y ago
If you are familiar with Flink, would you mind sharing your use cases? why Flink and not Kafka ? I have experience with Kafka but I have not had a reason to investigate Flink for my use cases.

cblconfederate 5y ago

For a nonprofit, apache seems to have an unhealthy affection for solving scaling problems that only a few giant rich companies have

detaro 5y ago
That criticism doesn't really make sense, given that Apache the non-profit isn't developing the software.
- cblconfederate 5y ago
  But it is funding it
  - detaro 5y ago
    The Apache Foundation isn't funding development in any meaningful way, no.
    - uDontKnowMe 5y ago
      What does the Apache Foundation do/provide/give these projects if no funding is involved? Is it just like a governance thing like "if this project falls into an abandoned state, we'll oversee the process of transferring ownership to any interested and credible parties interested in taking over"?
      - detaro 5y ago
        Mostly a governance thing, yes. "Community infrastructure, coordination and legal framework in a box".
kristjansson 5y ago
You’ve got the flow backwards. Giant rich companies contribute projects that solve their problems to the ASF

deleted 5y ago

[deleted]

pixelmonkey 5y ago

Wonder why this is getting posted today in particular? The quick summary here is that this was a clean-house rewrite of Apache Storm done by an internal team at Twitter. As an open source project history refresher, Apache Storm was originally built by a startup called Backtype, and the project was led by Nathan Marz, the technical founder of Backtype. Then, Backtype was acquired by Twitter, and Storm became a major component for large-scale stream processing (of tweets, tweet analytics, and other things) at Twitter. I wrote a summary of the "interesting bits" of Apache Storm here: https://blog.parse.ly/storm/ https://blog.parse.ly/storm/ However, at a certain point, Nathan Marz left Twitter, and a different group of engineers tried to rethink Storm inside Twitter. There was also a lot of work going on around Apache Mesos at the time. Heron is kind of a merger of their "rethinking" of Storm while also making it possible to manage Storm-like Heron clusters using Mesos. But, I don't think Heron really took off. Meanwhile, Storm got very, very stable in the 1.x series, and then had a clean-house rewrite from Clojure to Java in the 2.x series, mainly to improve performance even more. The last stable/major Storm release was in 2020. Storm provides a stream processing programming API, a multi-lang wire protocol, and a cluster management approach. But certain cluster computing problems can probably be better solved at the infrastructure layer today. (For example, Storm was developed before the whole container + docker + k8s focus in cloud ops.) That said, it's still a very powerful system; on my team, we process 75K+ events per second across hundreds of vCPU cores and thousands of Python processes with sub-second latencies by combining Storm and Kafka with our open source Python project, streamparse. https://github.com/Parsely/streamparse https://github.com/Parsely/streamparse The core problems Storm solves: modeling data processing as a computation graph; high-speed network communication between threads, processes, and nodes; message delivery guarantees and retry capabilities; tunable parallelism; built-in monitoring and logging; and much more. (Also, I'd be remiss if I didn't mention -- if you're interested in stream processing and distributed computing, we are hiring Python Data Engineers to work on a stack involving Storm, Spark, Kafka, Cassandra, etc.) -- https://www.parse.ly/careers/python_data_engineer https://www.parse.ly/careers/python_data_engineer

AtlasBarfed 5y ago
Is there a discussion as to the decision to dump clojure --> java for "performance reasons"? I'm not even a clojure user, but my impression was that it was pretty performant. I remember a discussion that they didn't even really need the JVM invokedynamic because they were doing pretty well without it, so that made me think it was close to pure JVM speed.

yamrzou 5y ago

I've worked with a bunch of stream processing engines a few years ago (Samza, Kafka Streaming, Spark Streaming, Storm and Flink), and did a comparison between them as part of my internship. IMO, Apache Flink is the most complete project for those use cases. It is well maintained and the devs are very helpful when asked on the mailing lists.

uDontKnowMe 5y ago
Totally agreed! Flink is way underrated.
- saryant 5y ago
  Had it improved? We tried to ship a fairly large product on Flink a couple years ago and it was a nightmare. It’s threading model was guaranteed to blow up in your face and running it reliably on a noisy network was near impossible.

CSDude 5y ago

I really wonder what people actually use stream processing for, like very concrete examples. My best examples would only go far filtering a stream over a time window to compute an aggregate. My job does not require anything more, it's always basic ETL, but I really need to hear specific examples where it's useful for others. Been a long time fan of Apache Flink.

thinkharderdev 5y ago
Standard use cases for stream processing are: 1. Enriching event streams. Say you have a stream of log records with an IP address field. You want to enrich with a geo-location before sending the logs to Elasticsearch. 2. Windowed aggregation. Maybe you have an application that is emitting "login" events and you want to to detect login attempts from different IP addresses within X minutes of each other. 3. Joining multiple event streams. You have multiple different event streams and you want to join them together using some common join key (maybe session ID or something like that) to compute a metric that aggregates all of them. There are plenty of more esoteric use cases as well.
- CSDude 5y ago
  They are the obvious ones, I'm looking for more advanced & specific ones.
  - infinite8s 5y ago
    Operational analytics when dealing with real world systems (transportation logistics, tracking machine states on factory floors, sensor data fusion, real-time operational dashboards for capital markets, etc).
Mehdi2277 5y ago
Streaming model training for recommendation systems like tiktok, facebook, youtube, etc. If you want models to learn quickly to new events allowing them to do better on very new content/users you will want to use a streaming pipeline. I have seen flink used for model training at large scale here. Model evaluation can also be done in a streaming manner with training and needs to be done parallel for model monitoring. The lowest latency model refresh time I've come across is ~5 minutes. Going lower is likely to be a lot of data transfers for model synchronization (sync the serving model with training model) for little value.

majormajor 5y ago

I would love to know more about how they do stateful stuff, since I haven't found anything that can compare to Flink there, but Flink has really poor quality of life stuff compared to some other options (e.g. Flink serialization setup pains vs Beam coders) but their docs kinda trail off here: > Non-idempotent stateful topologies are stateful topologies that do not apply processing logic along the model of "multiply by zero" and thus cannot provide effectively-once semantics. An example of a non-idempotent https://heron.incubator.apache.org/docs/heron-delivery-semantics#stateful-topologies https://heron.incubator.apache.org/docs/heron-delivery-seman... (I'm puzzled by their idempotent vs non-idempotent stateful topology description, because if something is mutating an internal state upon receiving events, it will likely be non-idempotent by design... unless they just mean "idempotent stateful" here to refer to keeping track of source/output position state and such.) (They also do say that can only support state storage in ZK or local FS, which feels like a likely non-starter compared to Flink for some of my use cases.)

dikei 5y ago
Flink serialization become much less painful when we switch every value type in our job to Avro. It requires a bit more code compare to using Java POJO or Scala case class, but you can actually add more field to existing data type without breaking anything now :)

Wonnk13 5y ago

sometimes I poke fun at the front-end folks and all the new frameworks they're constantly chasing. Lately it's been starting to feel the same for distributed systems. How many streaming engines are there now under Apache? Four?

haggy 5y ago
Depends how you group them. If we're talking battle tested then yea maybe four but I believe there are many more that don't have widespread adoption

fmakunbound 5y ago

Is it another corporate dumping at Apache or does Twitter seem to continue to support it as an open source project there?

karmasimida 5y ago

Anyone use this in production except Twitter? Just curious.

elchief 5y ago

So, AWS Pelican coming in 3 months?

biggestlou 5y ago

Everyone, please note that this is NOT an announcement of Heron entering the Apache Foundation. This happened several years ago, I believe in 2017 (when I was working on it).

deleted 5y ago

[deleted]

WC3w6pXxgGd 5y ago

Isn't this the same as Apache Kafka?

antpls 5y ago

Since no one mentioned it in the comments so far, please add Hazelcast Jet to the long list of stream processors : https://github.com/hazelcast/hazelcast-jet https://github.com/hazelcast/hazelcast-jet

aetherspawn 5y ago

Uhh, a triple-take: read this twice as Apache Heroin, and I’m not a druggy. Is it apt? Reader exercise.

diehunde 5y ago

I find odd so many people complaining about having too many alternatives for streaming systems. If you are not an advanced user of these systems of course you won't be able to choose the right one and it won't be your job anyways. Think of this, we have dozens of databases out there and there are new ones coming out every week and nobody complains about that. That's because there's way more people that understands the differences and the use cases so it makes sense.