4 ms·

Alas, I don't think this works with the standard Unix tools, which is the main way I process tab-delimited text. Changing the field delimiter to whatever you wa

by mjn 13y ago

Alas, I don't think this works with the standard Unix tools, which is the main way I process tab-delimited text. Changing the field delimiter to whatever you want is fine, since nearly everything takes that as a parameter. But newline as record separator is assumed by nearly everything (both in the standard set of tools, and in the very useful Google additions found in http://code.google.com/p/crush-tools/ http://code.google.com/p/crush-tools/). Google's defaults are ASCII (or UTF-8) 0xfe for the field separator, and '\n' for the record separator. I guess that's a bit safer than tabs, but the kind of data I put in TSV really shouldn't have embedded tabs in a field... and I check to make sure it doesn't, because they're likely to cause unexpected problems down the line. Generally I want all my fields to be either numeric data, or UTF-8 strings without formatting characters. Not to mention that one of the advantages of using a text record format at all is that you can view it using standard text viewers.

sheetjs 13y ago
Awk lets you set both: $ echo -n "1,2,3|4,5|6|7,8,9,0" | awk 'BEGIN{FS=","; RS="|"} {print NF, $0}' 3 1,2,3 2 4,5 1 6 4 7,8,9,0 In fact, you can also specify the output delimiters as well: $ echo -n "1,2,3|4,5|6|7,8,9,0" | awk 'BEGIN{FS=","; RS="|";OFS="foo";ORS="bar"} {print NF, $0}' 3foo1,2,3bar2foo4,5bar1foo6bar4foo7,8,9,0bar
- groovy2shoes 13y ago
  Yup. It wasn't fun to type (using Ctrl-V in bash to input the raw control characters), but it works fine with the ASCII separators as well: $ echo -n 'a^_1^^b^_2^^c^_3^^' |awk 'BEGIN{FS="^_"; RS="^^"} {print $1": "$2}' a: 1 b: 2 c: 3
sigil 13y ago
> Alas, I don't think this works with the standard Unix tools. It kind of works with standard unix tools. cut -d$'\37' -f ... sort -t$'\37' -k ... join -t$'\37' ... Those will parse ASCII-31-separated fields. But records are still newline separated, no way to change that AFAIK, short of running everything through tr '\036' '\n' first. Which defeats the purpose of choosing "weird" delimiters in the first place. (Also note that the $'\..' syntax is bash-specific and doesn't exist in POSIX sh.)

by mjn 13y ago

sheetjs 13y ago

Awk lets you set both: $ echo -n "1,2,3|4,5|6|7,8,9,0" | awk 'BEGIN{FS=","; RS="|"} {print NF, $0}' 3 1,2,3 2 4,5 1 6 4 7,8,9,0 In fact, you can also specify the output delimiters as well: $ echo -n "1,2,3|4,5|6|7,8,9,0" | awk 'BEGIN{FS=","; RS="|";OFS="foo";ORS="bar"} {print NF, $0}' 3foo1,2,3bar2foo4,5bar1foo6bar4foo7,8,9,0bar

groovy2shoes 13y ago
Yup. It wasn't fun to type (using Ctrl-V in bash to input the raw control characters), but it works fine with the ASCII separators as well: $ echo -n 'a^_1^^b^_2^^c^_3^^' |awk 'BEGIN{FS="^_"; RS="^^"} {print $1": "$2}' a: 1 b: 2 c: 3

sigil 13y ago

> Alas, I don't think this works with the standard Unix tools. It kind of works with standard unix tools. cut -d$'\37' -f ... sort -t$'\37' -k ... join -t$'\37' ... Those will parse ASCII-31-separated fields. But records are still newline separated, no way to change that AFAIK, short of running everything through tr '\036' '\n' first. Which defeats the purpose of choosing "weird" delimiters in the first place. (Also note that the $'\..' syntax is bash-specific and doesn't exist in POSIX sh.)