I was curious about the performance difference between these shell scripts and sqlite3, single threaded but fast and hardened database. I don't have forever, s

I was curious about the performance difference between these shell scripts and sqlite3, single threaded but fast and hardened database. I don't have forever, so I modified the linked-to scripts to make only 12,345 customer records (two orders of magnitude slower). I wrote a python importer to sqlite, which is largely CPU bound on the python side -- go would be much faster, but I wanted to stick with his other tools. 20 lines of code. Times for 'setting up the data': in his case, catting everything into one file. In mine, splitting the data up into rows and inserting into sqlite: #His Data Prep time ls | parallel -m -j 32 "cat {} >> ../transactions_cat" real 0m1.218s #My Data Prep time pypy import.py real 1m54s #His Count unique products (Note that as written it overcounts) time cut -d ' ' -f 5- ../transactions_cat | tr -d \" | tr ',' '\n' | parallel --pipe --block 1M sort -u | sort -u | wc -l real 1m52s #My Count Unique products time sqlite3 transaction.db "select count(distinct item) from txs" real 0m3.033s #Transactions per day #(his script -- note that as written it adds up the wrong columns) real 0m8.976s #My transactions per day. time sqlite3 transaction.db ' select substr(date, 1, 10) as day, count(*) from txs group by day' real 0m8.625s #Transactions per store month # I used the single threaded oneliner from the article, he noted it was fastest. time cut -d ' ' -f 2,3,5 transactions.csv | tr -d '[A-Za-z\"/\- ]' | awk '{print (substr($1,1,5)"-"substr($1,6,6)), length(substr($1,14))+1}' | awk '{a[$1]+=$2;}END{for(i in a)print i" "a[i];}' | sort real 0m50.886s #Mine time sqlite3 transaction.db ' select substr(date, 1, 7) as month, store, count(*) from txs group by month, store' real 0m16.908s In each case except for import sqlite is faster. To me the biggest win is not just the speed or the added expressivity (it's a real difference though for many of these queries), but the simplicity with which correctness can be assessed. Two of his three scripts did not do what they said on the tin, and the third one (which seems to), had a large disclaimer on it that he wasn't sure he'd properly optimized it.