4 ms·

'sort | uniq -c | sort -n' is an interesting pipeline. It will always work and does a great job with large cardinality data on low memory systems. However, if

'sort | uniq -c | sort -n' is an interesting pipeline. It will always work and does a great job with large cardinality data on low memory systems. However, if you have the ram, or know the data set has a low cardinality (like, http status codes or filesnames instead of ip addresses) then something that works in memory will be much more efficient. I threw 144,000,000 'hello' and 'world' into a file: justin@box:~$ ls -lh words -rw-r--r-- 1 justin justin 824M Jan 7 15:21 words justin@box:~$ wc -l words 144000000 words justin@box:~$ time (sort <words|uniq -c) 72000000 hello 72000000 world real 0m22.831s user 0m32.999s sys 0m4.675s Compared to doing it in memory with awk: justin@box:~$ time awk '{words[$1]++} END {for (w in words) printf("%s %d\n", w, words[w])}' < words hello 72000000 world 72000000 real 0m10.639s user 0m9.736s sys 0m0.876s so, half the time and 1/3 the cpu.

tuldia 7y ago
This is because in the first example you are invoking two programs. The first one sort the content of the file, the second count how many lines are equal. While in the awk example it is creating a hash table with all words and incrementing by the key and then printing. There is no sorting plus printing may be buffered.
- justinsaccount 7y ago
  Thanks for explaining my own comment to me.
  - tuldia 7y ago
    Not explaining, trying to tell that you are comparing apples to oranges and making a conclusion based on that. Also, you don't need to spawn a subshell nor feed sort via stdin in the first example :)
    - cperciva 7y ago
      He's comparing apples to oranges and reaching the conclusion that... yes, apples and oranges are different things. He's quite aware of this, and even points out the tradeoff -- `sort | uniq -c` still works if your dataset doesn't fit into RAM.
      - tuldia 7y ago
        Counting "words" vs sorting+counting "lines" are a complete different thing. Heck, forget about RAM, the output of both programs don't even match. That awk is pretty efficient and fast is no surprise ;)
crystaldev 7y ago
All of your examples work in memory.
- justinsaccount 7y ago
  Not exactly. sort (at least GNU sort) will end up doing external merge sort on temporary files if you give it more data than you have memory. Which, if you give it 100GB of 5 different strings, ends up being a huge waste.
  - tuldia 7y ago
    Not only GNU sort, but also postgresql, mysql and many more... Please, "huge waste"? How do you sort something that does not fit in memory?
    - justinsaccount 7y ago
      Are you being difficult on purpose? I posted a comment on how 'sort | uniq -c | sort -n' is an interesting and very capable pipeline, but often misused and slower than other alternatives. > you are comparing Yes, I am comparing two methods of accomplishing the same thing. That is how comparing things works. > Please, "huge waste"? How do you sort something that does not fit in memory? Note how the full sentence included "if you give it 100GB of 5 different strings". If your input is 100GB of 5 different strings, then the hash table will easily fit in memory, and sorting the entire data set only to pass it to 'uniq -c' is indeed a 'huge waste'. There are tons of large data sets that only have a small number of unique values in particular fields. protocols, ports, http status codes, hour of the day, etc. 'sort | uniq -c | sort -n' will work for all of them, but not nearly as efficient a hash table.
      - tuldia 7y ago
        > Are you being difficult on purpose? Programming is about paying the bare minimum attention to the details. > [...] two methods of accomplishing the same thing [...] Absolutelly not. one prints: 72000000 hello 72000000 world the other hello 72000000 world 72000000 Now try both examples against a file with more than one column to understand what I'm talking about ;)

tuldia 7y ago

This is because in the first example you are invoking two programs. The first one sort the content of the file, the second count how many lines are equal. While in the awk example it is creating a hash table with all words and incrementing by the key and then printing. There is no sorting plus printing may be buffered.

justinsaccount 7y ago
Thanks for explaining my own comment to me.
- tuldia 7y ago
  Not explaining, trying to tell that you are comparing apples to oranges and making a conclusion based on that. Also, you don't need to spawn a subshell nor feed sort via stdin in the first example :)
  - cperciva 7y ago
    He's comparing apples to oranges and reaching the conclusion that... yes, apples and oranges are different things. He's quite aware of this, and even points out the tradeoff -- `sort | uniq -c` still works if your dataset doesn't fit into RAM.
    - tuldia 7y ago
      Counting "words" vs sorting+counting "lines" are a complete different thing. Heck, forget about RAM, the output of both programs don't even match. That awk is pretty efficient and fast is no surprise ;)

crystaldev 7y ago

All of your examples work in memory.

justinsaccount 7y ago
Not exactly. sort (at least GNU sort) will end up doing external merge sort on temporary files if you give it more data than you have memory. Which, if you give it 100GB of 5 different strings, ends up being a huge waste.
- tuldia 7y ago
  Not only GNU sort, but also postgresql, mysql and many more... Please, "huge waste"? How do you sort something that does not fit in memory?
  - justinsaccount 7y ago
    Are you being difficult on purpose? I posted a comment on how 'sort | uniq -c | sort -n' is an interesting and very capable pipeline, but often misused and slower than other alternatives. > you are comparing Yes, I am comparing two methods of accomplishing the same thing. That is how comparing things works. > Please, "huge waste"? How do you sort something that does not fit in memory? Note how the full sentence included "if you give it 100GB of 5 different strings". If your input is 100GB of 5 different strings, then the hash table will easily fit in memory, and sorting the entire data set only to pass it to 'uniq -c' is indeed a 'huge waste'. There are tons of large data sets that only have a small number of unique values in particular fields. protocols, ports, http status codes, hour of the day, etc. 'sort | uniq -c | sort -n' will work for all of them, but not nearly as efficient a hash table.
    - tuldia 7y ago
      > Are you being difficult on purpose? Programming is about paying the bare minimum attention to the details. > [...] two methods of accomplishing the same thing [...] Absolutelly not. one prints: 72000000 hello 72000000 world the other hello 72000000 world 72000000 Now try both examples against a file with more than one column to understand what I'm talking about ;)