4 ms·

thanks for this. am looking at the benchmarks. how do I get huge.csv? Don't see how to fetch or generate

mattewong 4y ago
FYI I ran on the worldcities data at https://github.com/petewarden/dstkdata https://github.com/petewarden/dstkdata (credit to xsv for choosing that dataset) against https://github.com/BurntSushi/xsv https://github.com/BurntSushi/xsv and https://github.com/liquidaty/zsv https://github.com/liquidaty/zsv (full disclosure: I am one of the zsv authors). Here's what I got. fastest to slowest: zsv (0.07), xsv (0.16), goawk (0.42), python (~1.6) Obviously, does not tell the whole story as this test was limited to "count" and an interpreted language is expected to always be slower compared to a precompiled command, but, it might be relevant to a user deciding what tool to use. Also, might be instructive as to room for improvement in the go code (or possibly the go code could use the c lib)-- I note that even if the goawk command is '{}' the runtime is still about the same. full results: --- goawk: 1000001 7000007 real 0m0.435s user 0m0.435s sys 0m0.031s 1000001 7000007 real 0m0.413s user 0m0.419s sys 0m0.024s 1000001 7000007 real 0m0.425s user 0m0.430s sys 0m0.024s xsv: 1000000 real 0m0.157s user 0m0.141s sys 0m0.013s 1000000 real 0m0.156s user 0m0.141s sys 0m0.012s 1000000 real 0m0.158s user 0m0.142s sys 0m0.013s zsv: 1000000 real 0m0.066s user 0m0.053s sys 0m0.010s 1000000 real 0m0.077s user 0m0.060s sys 0m0.012s 1000000 real 0m0.069s user 0m0.056s sys 0m0.010s python: 1000001 7000007 real 0m1.589s user 0m1.553s sys 0m0.026s 1000001 7000007 real 0m1.583s user 0m1.550s sys 0m0.025s 1000001 7000007 real 0m2.122s user 0m1.675s sys 0m0.037s --- The script for this was: --- echo 'goawk:' (time goawk -i csv '{ w+=NF } END { print NR, w }' < worldcitiespop_mil.csv) 2>&1 | xargs (time goawk -i csv '{ w+=NF } END { print NR, w }' < worldcitiespop_mil.csv) 2>&1 | xargs (time goawk -i csv '{ w+=NF } END { print NR, w }' < worldcitiespop_mil.csv) 2>&1 | xargs echo 'xsv:' (time xsv count < worldcitiespop_mil.csv) 2>&1 | xargs (time xsv count < worldcitiespop_mil.csv) 2>&1 | xargs (time xsv count < worldcitiespop_mil.csv) 2>&1 | xargs echo 'zsv:' (time zsv count < worldcitiespop_mil.csv) 2>&1 | xargs (time zsv count < worldcitiespop_mil.csv) 2>&1 | xargs (time zsv count < worldcitiespop_mil.csv) 2>&1 | xargs echo 'python:' (time python3 count.py < worldcitiespop_mil.csv) 2>&1 | xargs (time python3 count.py < worldcitiespop_mil.csv) 2>&1 | xargs (time python3 count.py < worldcitiespop_mil.csv) 2>&1 | xargs
- benhoyt 4y ago
  Thanks for that! zsv looks amazing. Yeah, it's definitely going to whip GoAWK, what with SIMD parsing and careful memory handling. I've done a couple of basic things for GoAWK's CSV performance, but haven't profiled or looked at allocation bottlenecks (absolute performance definitely wasn't my first focus). Yeah, sorry about huge.csv -- I found it online somewhere originally by searching for something like "large csv example", but can't for the life of me find it now. It's a monster 1.5GB file with 286 columns including quoted fields (whereas worldcitiespop only has a few columns and it doesn't look like it has quoted fields). I can upload to a file transfer service and send a link to you if you want ... though I should really update my benchmarks to use an easily-downloadable file instead.
  - mattewong 4y ago
    Not a problem at all. Nice work with goawk!
- benhoyt 4y ago
  I've now updated the benchmarks to avoid huge.csv: the write benchmarks now write a big 1GB, 20-column CSV and the read benchmarks use that same CSV: https://github.com/benhoyt/goawk/commit/07eb4505a7f64cffceeb7d8579acc70ce3e96689 https://github.com/benhoyt/goawk/commit/07eb4505a7f64cffceeb...

FYI I ran on the worldcities data at https://github.com/petewarden/dstkdata https://github.com/petewarden/dstkdata (credit to xsv for choosing that dataset) against https://github.com/BurntSushi/xsv https://github.com/BurntSushi/xsv and https://github.com/liquidaty/zsv https://github.com/liquidaty/zsv (full disclosure: I am one of the zsv authors). Here's what I got. fastest to slowest: zsv (0.07), xsv (0.16), goawk (0.42), python (~1.6) Obviously, does not tell the whole story as this test was limited to "count" and an interpreted language is expected to always be slower compared to a precompiled command, but, it might be relevant to a user deciding what tool to use. Also, might be instructive as to room for improvement in the go code (or possibly the go code could use the c lib)-- I note that even if the goawk command is '{}' the runtime is still about the same. full results: --- goawk: 1000001 7000007 real 0m0.435s user 0m0.435s sys 0m0.031s 1000001 7000007 real 0m0.413s user 0m0.419s sys 0m0.024s 1000001 7000007 real 0m0.425s user 0m0.430s sys 0m0.024s xsv: 1000000 real 0m0.157s user 0m0.141s sys 0m0.013s 1000000 real 0m0.156s user 0m0.141s sys 0m0.012s 1000000 real 0m0.158s user 0m0.142s sys 0m0.013s zsv: 1000000 real 0m0.066s user 0m0.053s sys 0m0.010s 1000000 real 0m0.077s user 0m0.060s sys 0m0.012s 1000000 real 0m0.069s user 0m0.056s sys 0m0.010s python: 1000001 7000007 real 0m1.589s user 0m1.553s sys 0m0.026s 1000001 7000007 real 0m1.583s user 0m1.550s sys 0m0.025s 1000001 7000007 real 0m2.122s user 0m1.675s sys 0m0.037s --- The script for this was: --- echo 'goawk:' (time goawk -i csv '{ w+=NF } END { print NR, w }' < worldcitiespop_mil.csv) 2>&1 | xargs (time goawk -i csv '{ w+=NF } END { print NR, w }' < worldcitiespop_mil.csv) 2>&1 | xargs (time goawk -i csv '{ w+=NF } END { print NR, w }' < worldcitiespop_mil.csv) 2>&1 | xargs echo 'xsv:' (time xsv count < worldcitiespop_mil.csv) 2>&1 | xargs (time xsv count < worldcitiespop_mil.csv) 2>&1 | xargs (time xsv count < worldcitiespop_mil.csv) 2>&1 | xargs echo 'zsv:' (time zsv count < worldcitiespop_mil.csv) 2>&1 | xargs (time zsv count < worldcitiespop_mil.csv) 2>&1 | xargs (time zsv count < worldcitiespop_mil.csv) 2>&1 | xargs echo 'python:' (time python3 count.py < worldcitiespop_mil.csv) 2>&1 | xargs (time python3 count.py < worldcitiespop_mil.csv) 2>&1 | xargs (time python3 count.py < worldcitiespop_mil.csv) 2>&1 | xargs

benhoyt 4y ago
Thanks for that! zsv looks amazing. Yeah, it's definitely going to whip GoAWK, what with SIMD parsing and careful memory handling. I've done a couple of basic things for GoAWK's CSV performance, but haven't profiled or looked at allocation bottlenecks (absolute performance definitely wasn't my first focus). Yeah, sorry about huge.csv -- I found it online somewhere originally by searching for something like "large csv example", but can't for the life of me find it now. It's a monster 1.5GB file with 286 columns including quoted fields (whereas worldcitiespop only has a few columns and it doesn't look like it has quoted fields). I can upload to a file transfer service and send a link to you if you want ... though I should really update my benchmarks to use an easily-downloadable file instead.
- mattewong 4y ago
  Not a problem at all. Nice work with goawk!
benhoyt 4y ago
I've now updated the benchmarks to avoid huge.csv: the write benchmarks now write a big 1GB, 20-column CSV and the read benchmarks use that same CSV: https://github.com/benhoyt/goawk/commit/07eb4505a7f64cffceeb7d8579acc70ce3e96689 https://github.com/benhoyt/goawk/commit/07eb4505a7f64cffceeb...