3 ms·

Reminds me quite a bit of https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can

• ILMostro7 8y ago
  Bash 5.0? Parallel execution
  • lmilcin 8y ago
    Counting unique occurences isn't easy if you are given data the way it was described in the article and you have a requirement for exact count. But there are things you can do to help if you are willing to redesign it a bit. You could partition the data up front, as it is being written. Then you could send partitions to be processed in parallel (and you still could do this with the shell command!)
• zxcmx 8y ago
  There is a nearly-as-simple solution involving lambda (which would not require running a kafka cluster) but meh.