3 ms·
That's written in Python and uses the "agate" library which uses Python's built-in "csv" module. I did a couple of simple benchmarks against Python's csv module
by benhoyt 4y ago
That's written in Python and uses the "agate" library which uses Python's built-in "csv" module. I did a couple of simple benchmarks against Python's csv module in the article: https://benhoyt.com/writings/goawk-csv/#performance https://benhoyt.com/writings/goawk-csv/#performance (Go/GoAWK is a bit more than 3x as fast)
I also did a quick test using csvcut to pull out a single field, and compared it to GoAWK. Looks like GoAWK is about 4x as fast here:
$ time csvcut -c agency_id huge.csv >/dev/null
real 0m25.977s
user 0m25.240s
sys 0m0.424s
$ time goawk -i csv -H -o csv '{ print @"agency_id" }' huge.csv >/dev/null
real 0m6.584s
user 0m7.434s
sys 0m0.480s
- rgoulter 4y agoI meant more in terms of user/developer experience. I imagine there's some use case where AWK will be able to come up with output that csvkit can't. But for simple cases, csvkit's invocation is easier to remember.
- benhoyt 4y agoAh, true -- that would be a good comparison to do, or section to add the the docs (csv.md in the repo). I'll add it to my to-do list!