Pachyderm (github.com/pachyderm/pachyderm) is a great "medium data" tool. I'm biased of course because I'm one of the creators, but we build in data pipelining,

Pachyderm (github.com/pachyderm/pachyderm) is a great "medium data" tool. I'm biased of course because I'm one of the creators, but we build in data pipelining, data versioning, and you can use whatever languages or libraries you want because we containerize the whole workload. You example above would actually work perfectly. You can literally use grep in a distributed fashion in our system. One of our example pipelines is using grep and awk to do log filtering/aggregation or word count.