3 ms·

Thanks for running these statistics. Your experience is certainly different from mine. Is your Python actually PyPy? Here's what I observed: I had a simple tex

by rspeer 8y ago

Thanks for running these statistics. Your experience is certainly different from mine. Is your Python actually PyPy? Here's what I observed: I had a simple text-processing tool I needed that I call "countmerge", which just merges adjacent lines with the same key and adds up their corresponding values. I needed to run it on a lot of large files. I first wrote it in Python, where it was a significant bottleneck compared to the steps that came before it (split, sort, uniq -c). Eventually I rewrote it in Rust [1], and it was at least 5 times faster, at the expense of a fair amount more low-level code. But then rewriting it in awk [2] turned out to be as fast as what I wrote in Rust, possibly inconclusively faster. [1] https://github.com/rspeer/countmerge https://github.com/rspeer/countmerge [2] https://gist.github.com/rspeer/60c87dca1ab550326f8bd6d086452613 https://gist.github.com/rspeer/60c87dca1ab550326f8bd6d086452...

sametmax 8y ago
Ok, I generated a file similar to your test case >>> with open('data.txt', 'w') as f: ... for l in string.ascii_uppercase: ... for x in range(0, random.randint(1, 100000)): ... f.write('Key {}\t{}\n'.format(l, random.randint(0, 100))) With this script: import sys old_key = total = 0 for line in sys.stdin: key, value = line.split('\t') if old_key != key: old_key = key total = 0 print(key, value, end="") total += int(value) I get: $ <data.txt time python3 test.py Key A 2 Key B 87 Key C 58 Key D 64 Key E 29 Key F 25 Key G 2 Key H 74 Key I 17 Key J 37 Key K 97 Key L 77 Key M 19 Key N 74 Key O 33 Key P 61 Key Q 67 Key R 23 Key S 4 Key T 70 Key U 25 Key V 15 Key W 35 Key X 17 Key Y 31 Key Z 18 1.03user 0.01system 0:01.05elapsed 99%CPU (0avgtext+0avgdata 9564maxresident)k 0inputs+0outputs (0major+1100minor)pagefaults 0swaps But I can't manage to get the awk version working. It only prints one line on Ubuntu 16.04: $ <data.txt awk -f ./countmerge.awk Key 0 So I can't check it.

by rspeer 8y ago

sametmax 8y ago

Ok, I generated a file similar to your test case >>> with open('data.txt', 'w') as f: ... for l in string.ascii_uppercase: ... for x in range(0, random.randint(1, 100000)): ... f.write('Key {}\t{}\n'.format(l, random.randint(0, 100))) With this script: import sys old_key = total = 0 for line in sys.stdin: key, value = line.split('\t') if old_key != key: old_key = key total = 0 print(key, value, end="") total += int(value) I get: $ <data.txt time python3 test.py Key A 2 Key B 87 Key C 58 Key D 64 Key E 29 Key F 25 Key G 2 Key H 74 Key I 17 Key J 37 Key K 97 Key L 77 Key M 19 Key N 74 Key O 33 Key P 61 Key Q 67 Key R 23 Key S 4 Key T 70 Key U 25 Key V 15 Key W 35 Key X 17 Key Y 31 Key Z 18 1.03user 0.01system 0:01.05elapsed 99%CPU (0avgtext+0avgdata 9564maxresident)k 0inputs+0outputs (0major+1100minor)pagefaults 0swaps But I can't manage to get the awk version working. It only prints one line on Ubuntu 16.04: $ <data.txt awk -f ./countmerge.awk Key 0 So I can't check it.