3 ms·

Why use Tab separated values instead of Comma separated values? Is it possible to just use some random character, maybe even from different charset, to act as

Why use Tab separated values instead of Comma separated values? Is it possible to just use some random character, maybe even from different charset, to act as delimiter for importing/exporting tabular data?

whatupmd 8y ago
The argument is here [http://www.catb.org/esr/writings/taoup/html/ch05s02.html http://www.catb.org/esr/writings/taoup/html/ch05s02.html]: This format is to Unix what CSV (comma-separated value) format is under Microsoft Windows and elsewhere outside the Unix world. CSV (fields separated by commas, double quotes used to escape commas, no continuation lines) is rarely found under Unix. In fact, the Microsoft version of CSV is a textbook example of how not to design a textual file format. Its problems begin with the case in which the separator character (in this case, a comma) is found inside a field. The Unix way would be to simply escape the separator with a backslash, and have a double escape represent a literal backslash. This design gives us a single special case (the escape character) to check for when parsing the file, and only a single action when the escape is found (treat the following character as a literal). The latter conveniently not only handles the separator character, but gives us a way to handle the escape character and newlines for free. CSV, on the other hand, encloses the entire field in double quotes if it contains the separator. If the field contains double quotes, it must also be enclosed in double quotes, and the individual double quotes in the field must themselves be repeated twice to indicate that they don't end the field. The bad results of proliferating special cases are twofold. First, the complexity of the parser (and its vulnerability to bugs) is increased. Second, because the format rules are complex and underspecified, different implementations diverge in their handling of edge cases. Sometimes continuation lines are supported, by starting the last field of the line with an unterminated double quote — but only in some products! Microsoft has incompatible versions of CSV files between its own applications, and in some cases between different versions of the same application (Excel being the obvious example here).
sn41 8y ago
I too don't see why tsv is better than csv. That said, as many people have pointed out, commas have legitimate uses in fields like Address of a locality etc. and this could cause a problem in scripting. About exporting using random characters, I have found that ~ (tilde) is almost never seen in actual data, hence it can be used as a field separator in most cases. Another safe choice is the | (pipe) symbol. (In Windows, you can change the field separator in the "Region Settings" - a rather weird place to have that setting.)
- frou_dh 8y ago
  Tab is in the Control Character block of ASCII so that alone makes it strictly more appropriate
- zengargoyle 8y ago
  Tilde is used in path names too much (~user), pipe is a logical OR. The best I've found is ^ (caret) which rarely shows up in real life.
em500 8y ago
There's a bunch of ASCII characters (28 - 31) specifically meant to be used as delimiters, but they never caught on. But even if they did, you still always need to handle the case where one might legitimately want to use the delimiter character inside a field value.
- kickingvegas 8y ago
  Wow, TIL this actually exists. Mind is blown on how they never got adopted outside of folks thinking (foolishly) that these characters were not "plain" text.
- srean 8y ago
  This is one of my disappointments -- the fact that unit and field separators did not catch on.
sevensor 8y ago
There are absurdly many corner cases either way. For my money the best representation for an Excel spreadsheet is xlsx. There are several perfectly capable open source xlsx reading and writing libraries, not to mention Excel itself. (At this point, Excel is pretty much the only reason I can think of for wanting to install Windows.) Using xlsx rules out vim of course, and much as I'm a fan of vim I think it's plain foolishness to endorse it as a spreadsheet program. Vim is fine for ad-hoc viewing and quick-and-dirty edits, especially if the file is full of numbers and uncomplicated text, but it's really easy to break a csv that has fancy escaping in it.

whatupmd 8y ago

The argument is here [http://www.catb.org/esr/writings/taoup/html/ch05s02.html http://www.catb.org/esr/writings/taoup/html/ch05s02.html]: This format is to Unix what CSV (comma-separated value) format is under Microsoft Windows and elsewhere outside the Unix world. CSV (fields separated by commas, double quotes used to escape commas, no continuation lines) is rarely found under Unix. In fact, the Microsoft version of CSV is a textbook example of how not to design a textual file format. Its problems begin with the case in which the separator character (in this case, a comma) is found inside a field. The Unix way would be to simply escape the separator with a backslash, and have a double escape represent a literal backslash. This design gives us a single special case (the escape character) to check for when parsing the file, and only a single action when the escape is found (treat the following character as a literal). The latter conveniently not only handles the separator character, but gives us a way to handle the escape character and newlines for free. CSV, on the other hand, encloses the entire field in double quotes if it contains the separator. If the field contains double quotes, it must also be enclosed in double quotes, and the individual double quotes in the field must themselves be repeated twice to indicate that they don't end the field. The bad results of proliferating special cases are twofold. First, the complexity of the parser (and its vulnerability to bugs) is increased. Second, because the format rules are complex and underspecified, different implementations diverge in their handling of edge cases. Sometimes continuation lines are supported, by starting the last field of the line with an unterminated double quote — but only in some products! Microsoft has incompatible versions of CSV files between its own applications, and in some cases between different versions of the same application (Excel being the obvious example here).

sn41 8y ago

I too don't see why tsv is better than csv. That said, as many people have pointed out, commas have legitimate uses in fields like Address of a locality etc. and this could cause a problem in scripting. About exporting using random characters, I have found that ~ (tilde) is almost never seen in actual data, hence it can be used as a field separator in most cases. Another safe choice is the | (pipe) symbol. (In Windows, you can change the field separator in the "Region Settings" - a rather weird place to have that setting.)

frou_dh 8y ago
Tab is in the Control Character block of ASCII so that alone makes it strictly more appropriate
zengargoyle 8y ago
Tilde is used in path names too much (~user), pipe is a logical OR. The best I've found is ^ (caret) which rarely shows up in real life.

em500 8y ago

There's a bunch of ASCII characters (28 - 31) specifically meant to be used as delimiters, but they never caught on. But even if they did, you still always need to handle the case where one might legitimately want to use the delimiter character inside a field value.

kickingvegas 8y ago
Wow, TIL this actually exists. Mind is blown on how they never got adopted outside of folks thinking (foolishly) that these characters were not "plain" text.
srean 8y ago
This is one of my disappointments -- the fact that unit and field separators did not catch on.

sevensor 8y ago

There are absurdly many corner cases either way. For my money the best representation for an Excel spreadsheet is xlsx. There are several perfectly capable open source xlsx reading and writing libraries, not to mention Excel itself. (At this point, Excel is pretty much the only reason I can think of for wanting to install Windows.) Using xlsx rules out vim of course, and much as I'm a fan of vim I think it's plain foolishness to endorse it as a spreadsheet program. Vim is fine for ad-hoc viewing and quick-and-dirty edits, especially if the file is full of numbers and uncomplicated text, but it's really easy to break a csv that has fancy escaping in it.