A 19 hour, 100-job Hadoop pipeline! Are you running up against the limits of your architecture, or do you think there's room to grow? "The MapReduce pipeline s

A 19 hour, 100-job Hadoop pipeline! Are you running up against the limits of your architecture, or do you think there's room to grow? "The MapReduce pipeline starts to process data as soon as the data is available. It’s triggered by the condition jobs which periodically check if the data is available on S3." What do you mean by that? What's the trigger condition, when the number of logged events reaches a threshold? I was a little surprised to hear it's dynamically triggered, and not on a scheduled clock. Cool stuff, thanks for posting it.