4 ms·

I'm not totally sure why csvgrep csvsort csvcut csvjoin are needed as replacements for grep sort, cut or join? The syntax doesn't seem noticeably clearer? And

I'm not totally sure why csvgrep csvsort csvcut csvjoin are needed as replacements for grep sort, cut or join? The syntax doesn't seem noticeably clearer? And - without testing - I presume csvkit in Python is a bit slower than the GNU coreutils in C?

peterwwillis 12y ago
They're not replacements for those tools. They're new tools that work with a specific tabular data format.
- Malarkey73 12y ago
  The data I come across is a mix of comma, tab, and space delimited though predominantly what the commenter above would call "line oriented". So I personally just don't see the benefit from a less generalised version of coreutil tools ... but fair does I'm sure there are plenty who have use for a more specific tool.
  - dragonwriter 12y ago
    > So I personally just don't see the benefit from a less generalised version of coreutil tools Others have pointed out specific examples, but the general reason (which I don't think I've seen explicitly articulated) is that CSV isn't strictly line oriented -- for intsance, records can span multiple "physical" lines, because line breaks within quoted strings are part of the field value, not separate records -- so line-oriented tools don't work with CSV generally (they may if you happen to have CSV where nothing inconsistent with a line-oriented view happens to be done, but that's luck not something that is generally true of CSV.)
  - peterwwillis 12y ago
    I'm an experienced Unix admin, and I actually wrote my own versions of these csv tools long ago instead of using coreutils. I had a lot of different datasets to with with, and it turned it there were so many inconsistencies in the CSV data that I needed a tool that could handle it all. It also became super simple to extract and munge data with options for tabular data. With two tiny commands I can exclude rows, include rows, sed columns, merge tables, and export new tables, all with just csv files. So it's pretty handy.
icebraining 12y ago
Grep understands text files, not CSV syntax. They're not equivalent. For example, the following CSV document has two lines, but grep treats them as three: 1,2,5,"Hello" 3,4,5,"How are you doing?"
- dbro 12y ago
  That's correct, and as you illustrate it's the possibility to have newlines and commas inside quoted fields that complicates things for grep/awk/cut/etc. So instead of making a more complex version of tools like grep, we can make the data simple for these tools to understand. That's what https://github.com/dbro/csvquote https://github.com/dbro/csvquote does. It can be run in a pipeline before the grep stage, and allow grep/cut/awk/... to work with unambiguous field and record delimiters. Then it can restore the newlines and commas inside the quoted fields at the end of the pipeline.
teleclimber 12y ago
And don't forget AWK! I've been playing with these old school tools lately on medium sized tabular data files and I've been impressed by the performance and capabilities.
acdha 12y ago
> The syntax doesn't seem noticeably clearer? From the very first example: csvgrep -c 1 -m ILLINOIS using coreutils grep you'd need something like this: grep -E '^ILLINOIS,' 2009.csv Oh, wait, that doesn't work because the file was last saved by something which quotes it. grep -E '^"?ILLINOIS"?,' except that would produce unexpected results with 'ILLINOIS"' so it really needs to be something like: grep -E '^(["]{0,1})ILLINOIS\1,' Bear in mind that this is the simplest possible case and doesn't even touch on issues like quoting in the shell or needing to handle files which have embedded separators as data values (imagine what happens when our erstwhile grep data-miner needs to check the stats for `WASHINGTON, DISTRICT OF COLUMBIA`…). In all but the most trivial cases it's safer and easier to use tools designed for the job. csvkit also has the very nice property that it's callable as a Python library so when you outgrow a simple shell processing pipeline you could migrate your notes / scripts to a full Python program without having to retest everything related to basic file I/O, unicode, etc. which you would otherwise need to do when switching readers. (Bear in mind that the author works in data journalism – the target user is not a grizzled Unix expert but someone who has a bunch of CSV files and a full time job which is not shell scripting) > And - without testing - I presume csvkit in Python is a bit slower than the GNU coreutils in C? Perhaps but it'd be unlikely for it to be noticeable for n less than millions on a remotely modern system – the Python regular expression engine has quite decent performance and if that became an issue, PyPy will even JIT them for you. In the very few cases I've seen where there is a noticeable difference it was always because the Python version was decoding Unicode and the shell tools were running with LC_ALL=C, which meant that corrupt data made it further before being caught and, in some cases, either failing to match all of the records or subtly quietly things by not extracting all of a combined character, etc. For the target use-case, however, this is likely all to be many, many orders of magnitude less than the time most people would spend debugging regex patterns.
- Malarkey73 12y ago
  I take you're point and those other well made points about line continuations below -- but you're over egging it here as cut -d, -f1 2010.csv | grep ILLINOIS works just fine for this data. I emphasise again - I take your overall point though.

peterwwillis 12y ago

They're not replacements for those tools. They're new tools that work with a specific tabular data format.

Malarkey73 12y ago
The data I come across is a mix of comma, tab, and space delimited though predominantly what the commenter above would call "line oriented". So I personally just don't see the benefit from a less generalised version of coreutil tools ... but fair does I'm sure there are plenty who have use for a more specific tool.
- dragonwriter 12y ago
  > So I personally just don't see the benefit from a less generalised version of coreutil tools Others have pointed out specific examples, but the general reason (which I don't think I've seen explicitly articulated) is that CSV isn't strictly line oriented -- for intsance, records can span multiple "physical" lines, because line breaks within quoted strings are part of the field value, not separate records -- so line-oriented tools don't work with CSV generally (they may if you happen to have CSV where nothing inconsistent with a line-oriented view happens to be done, but that's luck not something that is generally true of CSV.)
- peterwwillis 12y ago
  I'm an experienced Unix admin, and I actually wrote my own versions of these csv tools long ago instead of using coreutils. I had a lot of different datasets to with with, and it turned it there were so many inconsistencies in the CSV data that I needed a tool that could handle it all. It also became super simple to extract and munge data with options for tabular data. With two tiny commands I can exclude rows, include rows, sed columns, merge tables, and export new tables, all with just csv files. So it's pretty handy.

icebraining 12y ago

Grep understands text files, not CSV syntax. They're not equivalent. For example, the following CSV document has two lines, but grep treats them as three: 1,2,5,"Hello" 3,4,5,"How are you doing?"

dbro 12y ago
That's correct, and as you illustrate it's the possibility to have newlines and commas inside quoted fields that complicates things for grep/awk/cut/etc. So instead of making a more complex version of tools like grep, we can make the data simple for these tools to understand. That's what https://github.com/dbro/csvquote https://github.com/dbro/csvquote does. It can be run in a pipeline before the grep stage, and allow grep/cut/awk/... to work with unambiguous field and record delimiters. Then it can restore the newlines and commas inside the quoted fields at the end of the pipeline.

teleclimber 12y ago

And don't forget AWK! I've been playing with these old school tools lately on medium sized tabular data files and I've been impressed by the performance and capabilities.

acdha 12y ago

> The syntax doesn't seem noticeably clearer? From the very first example: csvgrep -c 1 -m ILLINOIS using coreutils grep you'd need something like this: grep -E '^ILLINOIS,' 2009.csv Oh, wait, that doesn't work because the file was last saved by something which quotes it. grep -E '^"?ILLINOIS"?,' except that would produce unexpected results with 'ILLINOIS"' so it really needs to be something like: grep -E '^(["]{0,1})ILLINOIS\1,' Bear in mind that this is the simplest possible case and doesn't even touch on issues like quoting in the shell or needing to handle files which have embedded separators as data values (imagine what happens when our erstwhile grep data-miner needs to check the stats for `WASHINGTON, DISTRICT OF COLUMBIA`…). In all but the most trivial cases it's safer and easier to use tools designed for the job. csvkit also has the very nice property that it's callable as a Python library so when you outgrow a simple shell processing pipeline you could migrate your notes / scripts to a full Python program without having to retest everything related to basic file I/O, unicode, etc. which you would otherwise need to do when switching readers. (Bear in mind that the author works in data journalism – the target user is not a grizzled Unix expert but someone who has a bunch of CSV files and a full time job which is not shell scripting) > And - without testing - I presume csvkit in Python is a bit slower than the GNU coreutils in C? Perhaps but it'd be unlikely for it to be noticeable for n less than millions on a remotely modern system – the Python regular expression engine has quite decent performance and if that became an issue, PyPy will even JIT them for you. In the very few cases I've seen where there is a noticeable difference it was always because the Python version was decoding Unicode and the shell tools were running with LC_ALL=C, which meant that corrupt data made it further before being caught and, in some cases, either failing to match all of the records or subtly quietly things by not extracting all of a combined character, etc. For the target use-case, however, this is likely all to be many, many orders of magnitude less than the time most people would spend debugging regex patterns.

Malarkey73 12y ago
I take you're point and those other well made points about line continuations below -- but you're over egging it here as cut -d, -f1 2010.csv | grep ILLINOIS works just fine for this data. I emphasise again - I take your overall point though.