3 ms·
Actually, piping data between standard Unix tools can be extremely efficient. This example is working on gigabytes rather than terabytes of data, but memory ut
by paulmd 11y ago
Actually, piping data between standard Unix tools can be extremely efficient. This example is working on gigabytes rather than terabytes of data, but memory utilization is basically limited to just buffers, so you could definitely scale that to terabytes without a problem.
http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html http://aadrake.com/command-line-tools-can-be-235x-faster-tha...