5 ms·

What about using Apache Beam[0] for an abstracting over these stream processing frameworks? In a recent Software Engineering Daily podcast[1], Frances Perry rec

What about using Apache Beam[0] for an abstracting over these stream processing frameworks? In a recent Software Engineering Daily podcast[1], Frances Perry recommends using Flink for the data flow model. 0: http://beam.incubator.apache.org/ http://beam.incubator.apache.org/ 1: http://softwareengineeringdaily.com/2016/08/19/apache-beam-with-frances-perry/ http://softwareengineeringdaily.com/2016/08/19/apache-beam-w...

nl 10y ago
The problem with Beam is that it is an abstraction layer. This sounds like a stupid comment, but when the underlying services are all evolving as quickly as they do in this space programming against an abstraction layer means you need to wait for (often sorely needed) new features. (Frances Perry is an engineer on Beam at Google, so it would be surprising if she recommended against it)
- agibsonccc 10y ago
  Do you have examples of such features? Most production deployments of these kinds of things don't upgrade THAT often. You have a point that abstraction layers can limit you since it's another compatibility layer, but looking at what's done in actual practice I don't see it being that bad. If anything, it should be evaluated on a case by case basis. Flink is actually better tech overall for my use case, but a lot of customers want spark streaming since it's already installed. Having beam where we can do both is kinda nice.
  - nl 10y ago
    We had to abandon Cloudera (and Horton Works) Hadoop distributions because they didn't ship R support for Spark quickly enough (in 1.3 or 1.2 or whatever version that was). We jumped from 1.5 to 1.6 because of algorithms in MLLib (although that turned out to be a bit of a disappointment).
    - agibsonccc 10y ago
      Most organizations I see tend to just wait...this sounds more like 1 off to me. You're at a unique employer if the data scientists have that much power..
- cle 10y ago
  Also, these distributed processing platforms are extraordinarily leaky (for anything non-trivial, you really need to have a good low-level understanding of your system in order to tune/debug/optimize/implement). There are definitely other benefits of a unified API, but making it seamless to switch underlying implementations is probably not one...

nl 10y ago

The problem with Beam is that it is an abstraction layer. This sounds like a stupid comment, but when the underlying services are all evolving as quickly as they do in this space programming against an abstraction layer means you need to wait for (often sorely needed) new features. (Frances Perry is an engineer on Beam at Google, so it would be surprising if she recommended against it)

agibsonccc 10y ago
Do you have examples of such features? Most production deployments of these kinds of things don't upgrade THAT often. You have a point that abstraction layers can limit you since it's another compatibility layer, but looking at what's done in actual practice I don't see it being that bad. If anything, it should be evaluated on a case by case basis. Flink is actually better tech overall for my use case, but a lot of customers want spark streaming since it's already installed. Having beam where we can do both is kinda nice.
- nl 10y ago
  We had to abandon Cloudera (and Horton Works) Hadoop distributions because they didn't ship R support for Spark quickly enough (in 1.3 or 1.2 or whatever version that was). We jumped from 1.5 to 1.6 because of algorithms in MLLib (although that turned out to be a bit of a disappointment).
  - agibsonccc 10y ago
    Most organizations I see tend to just wait...this sounds more like 1 off to me. You're at a unique employer if the data scientists have that much power..
cle 10y ago
Also, these distributed processing platforms are extraordinarily leaky (for anything non-trivial, you really need to have a good low-level understanding of your system in order to tune/debug/optimize/implement). There are definitely other benefits of a unified API, but making it seamless to switch underlying implementations is probably not one...