4 ms·

Twitter Heron: Stream Processing at Scale

49 points by mekishizufu 11y ago

djb_hackernews 11y ago
> All the Stream Managers in a topology connect to each other to form an O(k^2) connection network shouldn't that be !(k-1)? Or am I missing something about the architecture?
- deleted 11y ago
  [deleted]
- escaped_hn 11y ago
  Every stream manager is connected to every other stream manager. k managers = k^2 connections.
  - djb_hackernews 11y ago
    Yea I must be missing something here. If you have k=2 stream managers you have 1 connection, not 4 . Even if the connections weren't bidirectional you'd have 2 connections.
    - Horusiath 11y ago
      it's (k-1)^2 actually - each to each except self.
    - bmm6o 11y ago
      Parent is talking big-O and so is ignoring that it isn't exactly k^2. Depending on how you count (do 2 nodes require 2 connections or 1 bi-directional connection) it's k(k-1).
pixelmonkey 11y ago
"In the evaluation in section 7 of the paper, Heron beats Storm in every single metric." In every metric except perhaps the most important one for actual users: Storm is open source, Heron isn't. Also, the @TwitterOSS account indicated that there are "no plans to open source in the short-term": https://twitter.com/TwitterOSS/status/605460461761396736 https://twitter.com/TwitterOSS/status/605460461761396736
- frankmcsherry 11y ago
  The paper contains the motivation: "For our needs at Twitter, we needed a stream processing platform that was open-source, high-performance, scalable, and was compatible with the current Storm API." I guess 3/4 isn't bad.
wheaties 11y ago
Now I get it. It's not just "I didn't write this, therefore shit, so I need to rewrite it from scratch." This was a clear explanation in a bite sized chunk which illustrated the reasons for Heron. Thanks.
JSno 11y ago
is Heron written in Clojure too? Maybe lack of good Clj programmers made storm hard to maitain?
dang 11y ago
Is the paper online? If someone can produce a standard URL for it, we'll change the link in this post.
- rckclmbr 11y ago
  http://dl.acm.org/citation.cfm?id=2742788 http://dl.acm.org/citation.cfm?id=2742788
  - dang 11y ago
    Sorry, I meant the pdf. Is there a url for it that will work for everybody? The one I get when I click "pdf" on that ACM page seems unlikely.
    - frankmcsherry 11y ago
      SIGMOD 2015 proceedings are open access, meaning the conference has arranged with ACM to provide the content to pretty much everyone. That being said, the submission is to someone's (Adrian's) interpretation of the work, and is arguably more useful than just linking at the pdf itself.

49 points by mekishizufu 11y ago

> All the Stream Managers in a topology connect to each other to form an O(k^2) connection network shouldn't that be !(k-1)? Or am I missing something about the architecture?

deleted 11y ago
[deleted]
escaped_hn 11y ago
Every stream manager is connected to every other stream manager. k managers = k^2 connections.
- djb_hackernews 11y ago
  Yea I must be missing something here. If you have k=2 stream managers you have 1 connection, not 4 . Even if the connections weren't bidirectional you'd have 2 connections.
  - Horusiath 11y ago
    it's (k-1)^2 actually - each to each except self.
  - bmm6o 11y ago
    Parent is talking big-O and so is ignoring that it isn't exactly k^2. Depending on how you count (do 2 nodes require 2 connections or 1 bi-directional connection) it's k(k-1).

pixelmonkey 11y ago

"In the evaluation in section 7 of the paper, Heron beats Storm in every single metric." In every metric except perhaps the most important one for actual users: Storm is open source, Heron isn't. Also, the @TwitterOSS account indicated that there are "no plans to open source in the short-term": https://twitter.com/TwitterOSS/status/605460461761396736 https://twitter.com/TwitterOSS/status/605460461761396736

frankmcsherry 11y ago
The paper contains the motivation: "For our needs at Twitter, we needed a stream processing platform that was open-source, high-performance, scalable, and was compatible with the current Storm API." I guess 3/4 isn't bad.

wheaties 11y ago

Now I get it. It's not just "I didn't write this, therefore shit, so I need to rewrite it from scratch." This was a clear explanation in a bite sized chunk which illustrated the reasons for Heron. Thanks.

JSno 11y ago

is Heron written in Clojure too? Maybe lack of good Clj programmers made storm hard to maitain?

dang 11y ago

Is the paper online? If someone can produce a standard URL for it, we'll change the link in this post.

rckclmbr 11y ago
http://dl.acm.org/citation.cfm?id=2742788 http://dl.acm.org/citation.cfm?id=2742788
- dang 11y ago
  Sorry, I meant the pdf. Is there a url for it that will work for everybody? The one I get when I click "pdf" on that ACM page seems unlikely.
  - frankmcsherry 11y ago
    SIGMOD 2015 proceedings are open access, meaning the conference has arranged with ACM to provide the content to pretty much everyone. That being said, the submission is to someone's (Adrian's) interpretation of the work, and is arguably more useful than just linking at the pdf itself.