3 ms·

Dataflow Programming: Handling Huge Data Loads Without Adding Complexity

35 points by bergie 15y ago

Sukotto 15y ago
OP links to page 2 of the article. Here's page 1 http://drdobbs.com/go-parallel/article/231400148?pgno=1 http://drdobbs.com/go-parallel/article/231400148?pgno=1
alecco 15y ago
Beware of deceiving graphs. That system scales with many cores compared to itself. Erlang isn't easy to program. Depending on the case, you might be better off moving the critical part of the data processing to C or C++, perhaps using a relevant library. On many benchmarks Erlang is 10x slower than C. And more importantly for huge data, Erlang consumes 3x to 30x more memory. http://shootout.alioth.debian.org/u64q/benchmark.php?test=all&lang=hipe&lang2=gcc http://shootout.alioth.debian.org/u64q/benchmark.php?test=al... Edit: Relative to C/C++, Scala is somewhat better in cpu (still slower) but memory-wise it's even worse: http://shootout.alioth.debian.org/u64q/benchmark.php?test=all&lang=scala&lang2=gcc http://shootout.alioth.debian.org/u64q/benchmark.php?test=al...
- zokier 15y ago
  Dataflow as a pattern is not constrained to Scala or Erlang specifically. In fact, the "Further reading" article[1] is about dataflow programming in C++. [1]: http://drdobbs.com/cpp/224202533 http://drdobbs.com/cpp/224202533
  - alecco 15y ago
    Interesting. But then why didn't the author benchmark that too? Pretty graphs like that can confuse people not making a deep analysis of what's going on. There are no disclaimers or notes about it.
    - igouy 15y ago
      Figure 2 is a very ordinary line graph, and the text tells you exactly what it's intended to show - "good scalability as the compute resources increased". When I make mistakes reading, it is most often my fault for not paying attention and not reading carefully.
      - alecco 15y ago
        The point of the whole article is "proven" with that test/graph. It neither includes a simple scalar version nor versions on different languages. And they are recommending specific languages (Scala/Erlang).
  - bergie 15y ago
    I've made some implementations as well: Node.js https://github.com/bergie/noflo https://github.com/bergie/noflo PHP https://github.com/bergie/phpflo https://github.com/bergie/phpflo
- igouy 15y ago
  Beware of making mistakes reading. Figure 2 shows run times for "performance of a dataflow based framework" and the writer has already said Erlang uses the actor model not the dataflow model. Not Erlang. (Probably the Java based dataflow framework sold by the company that employs the writer.)
  - alecco 15y ago
    The article very clearly promotes Erlang and the actor model too: > The actor model appears much better suited to task-based > parallelism where concurrency is important. For example, > Erlang programs control telecom equipment with thousands > of unique requests per minute. Meanwhile, dataflow is > focused on data parallelism where single-job data > processing and analysis performance are the main > concerns. In a sense, they're slightly different > solutions for different domains, although conceptually > similar. [Citation needed] for all that, with a proper point of reference with C or C++, as per my original comment.

35 points by bergie 15y ago

OP links to page 2 of the article. Here's page 1 http://drdobbs.com/go-parallel/article/231400148?pgno=1 http://drdobbs.com/go-parallel/article/231400148?pgno=1

alecco 15y ago

Beware of deceiving graphs. That system scales with many cores compared to itself. Erlang isn't easy to program. Depending on the case, you might be better off moving the critical part of the data processing to C or C++, perhaps using a relevant library. On many benchmarks Erlang is 10x slower than C. And more importantly for huge data, Erlang consumes 3x to 30x more memory. http://shootout.alioth.debian.org/u64q/benchmark.php?test=all&lang=hipe&lang2=gcc http://shootout.alioth.debian.org/u64q/benchmark.php?test=al... Edit: Relative to C/C++, Scala is somewhat better in cpu (still slower) but memory-wise it's even worse: http://shootout.alioth.debian.org/u64q/benchmark.php?test=all&lang=scala&lang2=gcc http://shootout.alioth.debian.org/u64q/benchmark.php?test=al...

zokier 15y ago
Dataflow as a pattern is not constrained to Scala or Erlang specifically. In fact, the "Further reading" article[1] is about dataflow programming in C++. [1]: http://drdobbs.com/cpp/224202533 http://drdobbs.com/cpp/224202533
- alecco 15y ago
  Interesting. But then why didn't the author benchmark that too? Pretty graphs like that can confuse people not making a deep analysis of what's going on. There are no disclaimers or notes about it.
  - igouy 15y ago
    Figure 2 is a very ordinary line graph, and the text tells you exactly what it's intended to show - "good scalability as the compute resources increased". When I make mistakes reading, it is most often my fault for not paying attention and not reading carefully.
    - alecco 15y ago
      The point of the whole article is "proven" with that test/graph. It neither includes a simple scalar version nor versions on different languages. And they are recommending specific languages (Scala/Erlang).
- bergie 15y ago
  I've made some implementations as well: Node.js https://github.com/bergie/noflo https://github.com/bergie/noflo PHP https://github.com/bergie/phpflo https://github.com/bergie/phpflo
igouy 15y ago
Beware of making mistakes reading. Figure 2 shows run times for "performance of a dataflow based framework" and the writer has already said Erlang uses the actor model not the dataflow model. Not Erlang. (Probably the Java based dataflow framework sold by the company that employs the writer.)
- alecco 15y ago
  The article very clearly promotes Erlang and the actor model too: > The actor model appears much better suited to task-based > parallelism where concurrency is important. For example, > Erlang programs control telecom equipment with thousands > of unique requests per minute. Meanwhile, dataflow is > focused on data parallelism where single-job data > processing and analysis performance are the main > concerns. In a sense, they're slightly different > solutions for different domains, although conceptually > similar. [Citation needed] for all that, with a proper point of reference with C or C++, as per my original comment.