4 ms·

We run Flink for stream processing and Spark for batch processing, both of which are backed by Akka (and therefore the actor model).

• lostintangent 5y ago
  Very dumb question: when you say backed by Akka, what exactly do you mean? That you’ve configured a “binding” between Flint/Spark and Akka, which spins up actor instances (Java/Scala classes?) whenever some events occur? Or that Flint/Spark use Akka behind the scenes?
  • closeparen 5y ago
    They both use Akka behind the scenes. I don’t know the internal details exactly, but they’re both capable of showing you the processing pipeline you’ve asked for as a DAG visualization. I think what’s happening is something like each node on that DAG is an actor, receiving data from the previous node and sending it along to the next. You don’t write that though, you just write map/filter/etc.
    • trebecks 5y ago
      flink uses akka for control plane kind of stuff. data plane stuff, like piping data between tasks, doesn't. check out https://flink.apache.org/2019/06/05/flink-network-stack.html https://flink.apache.org/2019/06/05/flink-network-stack.html for some (maybe outdated) details.
• deleted 5y ago
  [deleted]