To be a bit more future-proof you should give Apache Beam a try. The same code should (theoretically) work with any of the supported runners[0] and so you could

To be a bit more future-proof you should give Apache Beam a try. The same code should (theoretically) work with any of the supported runners[0] and so you could deploy it on top of the most suitable framework/technology for your specific workload. Moreover, at this point, the community has several examples of how to add an additional runner. Edit: I didn't see you mentioned Beam in your second salvo :) [0]: https://beam.apache.org/documentation/runners/capability-matrix/ https://beam.apache.org/documentation/runners/capability-mat...