3 ms·
Hi! Dataflow team member here. On maturity: Dataflow is built by the people who over the past 12 years created MapReduce, FlumeJava and FlumeC++, Pregel and Mi
by jkff 11y ago
Hi! Dataflow team member here.
On maturity: Dataflow is built by the people who over the past 12 years created MapReduce, FlumeJava and FlumeC++, Pregel and Millwheel, based on the sum of experience obtained from all of these. It shares most of the back-end stack (work scheduling, pipeline optimization, fault tolerance etc.) with FlumeJava and FlumeC++ for batch jobs and Millwheel for streaming jobs, all of which are extensively used inside Google for data processing, and it shares a lot of the Java SDK with FlumeJava.
On usage: the blog post that announced General Availability of Dataflow lists a few public customer testimonials: http://googlecloudplatform.blogspot.com/2015/08/Announcing-General-Availability-of-Google-Cloud-Dataflow-and-Cloud-Pub-Sub.html http://googlecloudplatform.blogspot.com/2015/08/Announcing-G... . Quite a few blog posts by other users can also be found via https://www.reddit.com/r/dataflow https://www.reddit.com/r/dataflow .
Happy to answer additional questions. If I forget to check on this thread, feel free to ask on dataflow-feedback@google.com or on StackOverflow with tag google-cloud-dataflow - we constantly monitor these and usually answer everybody.