12 ms·
Open-Sourcing Twitter Heron
- buremba 10y agoGreat news! Here is the paper if you're interested in Heron. http://dl.acm.org/citation.cfm?id=2742788&CFID=620516550&CFTOKEN=92648489 http://dl.acm.org/citation.cfm?id=2742788&CFID=620516550&CFT...
- mark242 10y agoHow does Heron compare to Spark streaming?
- jsmthrowaway 10y agoWell, the big one is that Heron is built to support real-time streaming while Spark Streaming is not, given its choice to use micro-batching. If latency matters, that matters to you. I'd appreciate if it were called Spark Microbatching, but we can't have everything.
- kod 10y agoThe amount of FUD thrown around about micro batching is really kind of silly. How many streaming analytics use cases are ok with the JVM, ok with 10s of millis of latency, but not ok with 100s of millis of latency?
- jsmthrowaway 10y agoRemove "analytics" from your statement and you'll arrive at the answer, because while analytics is driving the streaming space as we speak it is literally one of the most boring applications of streaming imaginable. I positively cannot get excited about engagement numbers from an event pipeline. Intrusion detection, fraud detection, about half of infrastructure monitoring tasks, event sourcing... You're also glossing over the impact of microbatching which looks more like seconds than milliseconds, as well as what microbatching does to your windowing abilities (such as having to double-process to work around the microbatch interval as applied to your desired windowing semantics). Microbatch latency immediately rules out several useful applications of streaming. I also didn't say anything about the JVM nor it being okay for my purposes. You did. I can back up my statements on streaming from Flink to Samza to Dataflow/Beam to Storm to MillWheel to Spark "Streaming" and back again because it has been my primary focus (literally thinking of nothing else) for a couple years. Please accuse someone else of FUD because that's a relatively veiled way to say "you don't know what you're talking about," and I assure you that you're (condescendingly) wrong. I think you're also coming at that angle from interpreting me as negative on Spark Streaming. Read carefully.
- kod 10y ago> You're also glossing over the impact of microbatching which looks more like seconds than milliseconds I've run spark streaming jobs at 250ms batch times. This comment you made, right here, is why I used the word FUD.
- mark242 10y agoOkay, the better question then would be, how does this compare with writing a processor on top of Akka streams?
- dswalter 10y agoI've seen a decent amount of complaining about the difficulty of supporting a kafka infrastructure. I'd be interested in thoughts from people who have used Heron as to how it is running it in production.
- elodina 10y agoWe open sourced Kafka support Spout, Bolt & Example Topology https://github.com/twitter/heron/pull/751 https://github.com/twitter/heron/pull/751 We run Kafka Mesos https://github.com/mesos/kafka https://github.com/mesos/kafka allows for as a service implementation for broker clusters for users and apps. Vagrant for running example https://github.com/elodina/heron/blob/v2/contrib/kafka9/vagrant/README.md https://github.com/elodina/heron/blob/v2/contrib/kafka9/vagr...
- itaifrenkel 10y agoDoes anyone know if Storm ShellBolt compatible with Heron, or is there a better way in Heron for running non-jvm bolts?
- zenlikethat 10y agoFeel like they should feature the link to the repo much more prominently in the article! For anyone curious, it's https://github.com/twitter/heron https://github.com/twitter/heron
- ghayes 10y agoAnd they built a site for the project with a good Getting Started section: http://twitter.github.io/heron/ http://twitter.github.io/heron/ Getting Started: http://twitter.github.io/heron/docs/getting-started/ http://twitter.github.io/heron/docs/getting-started/
- lovelearning 10y agoThey have dropped clojure completely. All the critical path messaging in Storm was done using clojure. Dropped netty too. The actual messaging ("stream manager") seems to be C++. Perhaps that explains the latency and CPU improvements. The architectural changes mentioned account for better cluster utilization, fault tolerance and back pressure implementation, but don't explain why raw streaming performance is so much better in Heron. Edit: Not so sure what they are using for networking. Seemed like cpp-netlib at first, but I don't think so now.
- JonathonW 10y agoLooks like a homegrown networking framework: https://github.com/twitter/heron/tree/master/heron/common/src/cpp/network https://github.com/twitter/heron/tree/master/heron/common/sr... Their default event loop implementation uses libevent, and they're using protobuf in some of their higher-level networking classes, but the networking code itself seems to pretty much be plain sockets (with a thin portability layer on top in a few places).
- abc_lisper 10y agoWonder how this compares to Storm 1.0
- abc_lisper 10y agoWhy am I being down-modded? When Twitter first announced Heron(~ a year ago), they compared it with a version of Storm available then. Since then Storm has improved it's latency, ability to scale-out, added back-pressure and in some cases it is 10x faster than the previous version of storm (0.10.0?). I was wondering how Heron compares to Storm's performance now.
- scaleout1 10y agoAs someone who has used Heron (along with MillWheel, Spark Streaming and Storm) I feel like this announcement is too late. The biggest thing Heron offer is raw scale but since they decided to use existing Storm API, it has the same shitty spout /bolt API that Storms offer. In contrast, Spark streaming/Flink/ Kafka Streaming are all offering map/flatmap/filter/sink based functional API. At twitter most teams used SummingBird on top of Heron to get the same functional API but summingbird didnt get a lot of traction outside twitter and I am not sure how actively maintained OSS version of summingbird is. Even if you bite the bullet and decide to use SB with Heron, you will still miss out on a lot of usecases as SB was mostly focused on doing read/transform/aggregate/write whereas most streaming problem that i have noticed outside of twitter involve doing read/transform/aggregate/decision/write. I suppose you can implement decisioning in SB but i havent seen it done. Comparing Heron to google millwheel is interesting because of the design choices they made. Heron support at least one and at most once message guarantees but at Twitter most job ran with acked turned off so it was at most once with acknowledged data loss ( they had a batchjob doing mop up work to pick up missing data). Google on the other hand implemented exactly once semantic by doing idempotent sinks/ watermarking and managing out of order messages plus deduping support. Since both Flink and Spark will be implementing Apache Beam (millwheel's predecessors) model, only reason I see someone picking heron instead of Flink/Spark is that they are operating at massive scale that flink/spark dont support yet
- eldenbishop 10y agoStorm is a low level system for managing (optionally) transactional multi-machine tasks. It makes no assumptions about what is being processed (ie. analytics, data transforms). The primitives you are talking about exist in the child project Trident which runs on top of storm. Storm itself is no more for analytics than a web-server. It is a lower level tool.
- weego 10y agoThe parent also ignored the time-to-process difference which is drastically lower in storm. It has its flaws but scale is not the only metric to use as a decider
- baby 10y agoI'm genuinely curious as to why new products are constantly written in Java. My experience with the language is far from pleasant. Is it because people actually like the language? Is it because there is no other alternative when it comes to solid development? ...?
- drinkabeer 10y agoI think predictable performance is a major reason. Also, there are loads of developers which are still using Java, so you don't need to worry about expanding your teams and teaching hordes of developers a new development setup.
- zintinio5 10y agoIt has good tooling, IDE support, lots of people know it, it's fast, relatively easy to write, accessible from Scala, Groovy, Clojure, etc. Java is a step removed from having to write a system in C or C++, but you don't need absolute control over every single detail (garbage collection is acceptable). Most development happens in Java, C#, or C++, just not on HN. There's something of an inertial effect with Java, where people continue to write Java because everyone else writes it (and thus it continually improves). What other platforms do you know of that have a similar track-record of performing in large-scale software?
- abritinthebay 10y agoIt's not bad once you strip out most of the crap that it tries to foist upon you. But honestly...? It's taught in schools a lot so more people are familiar with it. The greatest disservice to an entire generation of programmers is Java being standardized at Universities.
- EdSharkey 10y agoAs I understand it, Twitter is heavily invested in Java and the JVM. They maintain their own fork of the JVM that they tweak for low latency. As a starting point for speed and stability, the JVM is top of the line. I like coding with Java 8. There's lots of new syntactic sugar that let you cut down on all the boilerplate and cool new standard library features like Java 8 streams that let you do functional programming. The best part of Java 8 streams is they perform really well. You get more readable code along with better performance.
- ps4fanboy 10y agoIf twitter's stock price continues to fall maybe they will open source the entire stack.
- vemv 10y agoHearing about a Clojure project being dropped feels like a slap in the face. I only wonder if it was more of a political decision.