2 ms·
Back in the day, optimizing this would be about parallel IO and some map-reduce processing. Data sharded on a bunch of nodes, each effectively doing "sort | uni
by saltcured 1y ago
Back in the day, optimizing this would be about parallel IO and some map-reduce processing. Data sharded on a bunch of nodes, each effectively doing "sort | uniq -c" and then doing a merge of those sorted counts.
And then there would be countless arguments about whether you have to count the time it takes to stage the data into the cluster as part of the task completion benchmark...
- noamteyssier 1y agoI think you'd still need to go through that if you were really optimizing both `sort` and `uniq` working with their constraints. What I'm really optimizing here is the functional equivalent of `sort | uniq -c | sort -n`