I'm getting 2 minutes and 29.98 seconds, with my 2017 MacBook Pro with 8 cores + 16 GiB ram. I'm using GNU tools, could you share more details on how you're ge

I'm getting 2 minutes and 29.98 seconds, with my 2017 MacBook Pro with 8 cores + 16 GiB ram. I'm using GNU tools, could you share more details on how you're getting 12 seconds? $ time awk 'BEGIN{for(i=0; i<3100000; i+=1) for(y=0; y<10; y+= 1) print i "@gmail.com"}' > email30m.sorted awk > email30m.sorted 8.29s user 0.82s system 97% cpu 9.352 total $ ls -lh email30m.sorted 522M Mar 8 17:14 email30m I didn't have your email30m dataset, so I fabricated one, mine is smaller at 522M and has less entropy, since 'm assuming everyone uses gmail & uses numbers for usernames. :) $ time shuf email30m.sorted > email30m shuf email30m.sorted > email30m 15.58s user 1.69s system 98% cpu 17.561 total I also shuffled the dataset, because that would be cheating. $ wc -l email30m 31000000 email30m $ time sort email30m | uniq > email30m.uniq sort email30m 873.55s user 4.29s system 585% cpu 2:29.98 total uniq > email30m.uniq 4.10s user 0.34s system 2% cpu 2:29.98 total $ wc -l email30m.uniq 3100000 email30m.uniq I followed what you did, but it took way longer when running on my machine. $ time sort email30m.sorted | uniq > email30m.sorted.uniq sort email30m.sorted 413.75s user 3.66s system 468% cpu 1:29.04 total uniq > email30m.sorted.uniq 4.11s user 0.40s system 5% cpu 1:29.04 total And just out of curiosity, it takes 1 minute 29 seconds to sort & uniq the presorted dataset.