2 ms·
If you find yourself piping any combination of cut, grep, sed, uniq (and likely others I'm missing) together, you can probably do it all in awk. If you can guar
by sgarland 4y ago
If you find yourself piping any combination of cut, grep, sed, uniq (and likely others I'm missing) together, you can probably do it all in awk. If you can guarantee usage of gawk, you can add sort to that list (tbf you can also implement any algorithm you want in awk, but arguably at that point you're wasting time) - and it's also worth noting that you can dedupe in awk _without_ having sorted input, albeit at the cost of storing all unique lines in memory.
Pipes are great because they enable you to trivially send data between programs, and they're terrible for the same reason. While the execution time on modern computers for the average data size isn't noticeable, on larger datasets or repeated execution, it absolutely is. If you don't have to pipe, don't.
- Rediscover 4y ago> ...dedup in awk... I most recently did that an hour ago, and a few hours ago, pretty much every day. awk '! x[$0]++' foo