5 ms·

You're not going to get clean out-of-order processing semantics with any mode of Spark transformations. If you actually take the time to read the article, there

by takidau 11y ago

You're not going to get clean out-of-order processing semantics with any mode of Spark transformations. If you actually take the time to read the article, there's a section discussing the Java/Scala angle. The difference in code size is really secondary (though it is a difference). The difficulty in maintaining and evolving your pipeline over time using Spark is the main point, given the way important concepts become conflated with their API (any version of it). This all comes across much more clearly for those that actually take the time to read the words.

glogla 11y ago
> If you actually take the time to read the article, there's a section discussing the Java/Scala angle. They claim this isn't about the length of code, yet they select the most verbose way to use Spark and proudly display how long it is. I mean sure, lack of event-time based processing is known limitation of Spark (and a pretty annoying one - though it is supposed to be worked on) but there are ways to write about it without code made to look bad on purpose. EDIT: come to think of it, this whole article is "spark streaming can't do event time" written in thousands of words with contrived examples attached.
- vgt 11y ago
  I think both length of code is a side effect that results from the primary argument, and "cant do event time" is one of the symptoms. Neither is a primary argument in the blog post. The primary argument is demonstrated through color coding different logical bits, which end up being clearly portable and elegantly distinct in dataflow. This is demonstrated in two ways: 1. The "juicy value add" code that does the aggregation is labeled yellow, and doesn't change across all the samples with Dataflow. With Spark, it needs to be rewritten for every use case. Similarly, for all colors. 2. In Dataflow all the colors are separate. This makes expressing your logic easier. In Spark, the colors mix in dramatic ways with every demonstrated use case. As Tyler said, all this is described in the blog post itself, but I don't blame you for missing it, since it's a really long post :)
andrioni 11y ago
What do you think about Cloudera's Spark backend to Google Cloud Dataflow pipelines[1]? [1]: https://github.com/cloudera/spark-dataflow https://github.com/cloudera/spark-dataflow
- jkff 11y ago
  Hi, I'm also an engineer on the Dataflow team. This is a very exciting project! However it is still constrained by Spark's limitations outlined in this post in terms of what Dataflow pipelines it can run. For example, currently it doesn't support session windows, it windows based on processing time instead of event time, and it doesn't seem to support triggers (the word "trigger" doesn't appear in the code base). AFAIK Spark has plans to add support for event time, and then it will be possible to make this runner run more pipelines consistently with the semantics of the Dataflow model. It is also likely that the capabilities of this runner will expand greatly as https://wiki.apache.org/incubator/BeamProposal https://wiki.apache.org/incubator/BeamProposal progresses.

by takidau 11y ago

glogla 11y ago

> If you actually take the time to read the article, there's a section discussing the Java/Scala angle. They claim this isn't about the length of code, yet they select the most verbose way to use Spark and proudly display how long it is. I mean sure, lack of event-time based processing is known limitation of Spark (and a pretty annoying one - though it is supposed to be worked on) but there are ways to write about it without code made to look bad on purpose. EDIT: come to think of it, this whole article is "spark streaming can't do event time" written in thousands of words with contrived examples attached.

vgt 11y ago
I think both length of code is a side effect that results from the primary argument, and "cant do event time" is one of the symptoms. Neither is a primary argument in the blog post. The primary argument is demonstrated through color coding different logical bits, which end up being clearly portable and elegantly distinct in dataflow. This is demonstrated in two ways: 1. The "juicy value add" code that does the aggregation is labeled yellow, and doesn't change across all the samples with Dataflow. With Spark, it needs to be rewritten for every use case. Similarly, for all colors. 2. In Dataflow all the colors are separate. This makes expressing your logic easier. In Spark, the colors mix in dramatic ways with every demonstrated use case. As Tyler said, all this is described in the blog post itself, but I don't blame you for missing it, since it's a really long post :)

andrioni 11y ago

What do you think about Cloudera's Spark backend to Google Cloud Dataflow pipelines[1]? [1]: https://github.com/cloudera/spark-dataflow https://github.com/cloudera/spark-dataflow

jkff 11y ago
Hi, I'm also an engineer on the Dataflow team. This is a very exciting project! However it is still constrained by Spark's limitations outlined in this post in terms of what Dataflow pipelines it can run. For example, currently it doesn't support session windows, it windows based on processing time instead of event time, and it doesn't seem to support triggers (the word "trigger" doesn't appear in the code base). AFAIK Spark has plans to add support for event time, and then it will be possible to make this runner run more pipelines consistently with the semantics of the Dataflow model. It is also likely that the capabilities of this runner will expand greatly as https://wiki.apache.org/incubator/BeamProposal https://wiki.apache.org/incubator/BeamProposal progresses.