5 ms·

It makes me cry how much time has been invested in formats like CSV's, TSV's, etc. ASCII (and UTF-8) has characters reserved for column, row, and even group sep

It makes me cry how much time has been invested in formats like CSV's, TSV's, etc. ASCII (and UTF-8) has characters reserved for column, row, and even group separation. Just use them and save a lot of pain.

jefftk 7y ago
While that might have been a good idea if we'd started a while ago, at this point CSV/TSV are so much more established that using commas or tabs doesn't actually save pain.
zachrose 7y ago
But wouldn’t you have to account for the possibility that those separators exist in the content your working with, putting you back at square one in terms of escaping?
- kec 7y ago
  That's entirely the point: 0x1C and 0x1E should never actually appear in "normal" text unless someone has explicitly put them there, which is not necessarily true of , \t or \n.
mdellavo 7y ago
using csv and tools like grep and join is very fast for large datasets
- lgas 7y ago
  Using the reserved separator characters wouldn't change that.
- perl4ever 7y ago
  I always used cut, sort, and uniq a lot too.
dev_dull 7y ago
We’ve discussed this at length and I’m squarely in the DO NOT camp. The drawbacks of using non-readable meta characters exceeds the benefits: 1. TSV can be read and imported by almost everything. 2. People can add and adjust TSV files from any editor. 3. What’s the way to insert meta characters again In VIM? And now nano? Argh I’ll just try and copy and paste it. Ugh that doesn’t work. Just use CSV/TSV folks. Anything more complicated and reach for a better serialization format (json, yaml) and not a better delimiter.
- cbsmith 7y ago
  Like a lot of things, #1 is only partly true, particularly if you have embedded tabs. The number of times I've had to deal with parsing errors because of embedded carriage returns, commas, tabs, etc., sometimes costing millions of dollars is just... upsetting. It's like JSON... we say everything can parse it, but really what we've got is some approximation that will come back to haunt us... and once you've done all the work to make sure everything is precise and correct, you'd really have saved time if you'd just used the tools we already had in the first place.
  - mruts 7y ago
    Dealing with tabs and carriage returns has sometimes cost you millions of dollars? If so, that sounds like a great story and I would love to hear it!
    - jlgaddis 7y ago
      Especially if (as it sounds) it's happened more than once!
- cbsmith 7y ago
  2. More accurately, people can screw up TSV files from any editor. Have you not seen embedded spaces used instead of tabs and totally mess everyone up?
- cbsmith 7y ago
  3. If you don't know how to use your editor, maybe you should learn. If your editor can't insert even all the characters in basic ASCII, it's not an editor.
xashor 7y ago
We took the wrong path quite some time ago. Most of the POSIX utils like sort, cut, grep only support \n (or sometimes \0) as row separator.
- jlgaddis 7y ago
  Are you aware of $IFS?
  - xashor 7y ago
    I think $IFS is only used by the shell (with read and expansions), not by e.g. sort. It is so at least on my system.
perl4ever 7y ago
For some reason, in some circles, it seems to be semi standard to use þ (0xFE aka thorn) as the delimiter and a paragraph symbol (0x14 aka DC4 aka ^T) as the separator. The latter is not to be confused with 0xB6. Anyway, these character are presumably not going to occur in ordinary text.
- PeterisP 7y ago
  All of "upper ascii half" can occur in ordinary text in "pre-Unicode" encodings. 0xFE is a good example - you may get a customer or employee from Iceland with that character in their name (e.g. https://en.wikipedia.org/wiki/Haf%C3%BE%C3%B3r_J%C3%BAl%C3%ADus_Bj%C3%B6rnsson https://en.wikipedia.org/wiki/Haf%C3%BE%C3%B3r_J%C3%BAl%C3%A...), or data in cyrillic cp1251 or koi8-r enconding where 0xFE also represents characters that you'll encounter in surnames, etc.
  - perl4ever 7y ago
    It would be escaped in that case.
dagenix 7y ago
But, there is nothing to stop your values from containing these characters. So, you still have to escape your input. And once you've done that, you might as well just use csv / tsv.
- cbsmith 7y ago
  Actually, the whole point of those characters is that they cannot be used in values.
jstimpfle 7y ago
How do you do (for example) column names or types with ASCII only? How do you extend ASCII?
- kps 7y ago
  ESC is the ASCII character for code extension. ISO 2022 aka ECMA-35¹ standardizes how to use it in general, but the only thing that really caught on is a subset of the terminal control extensions (ISO 6429 aka ECMA-48² aka ‘ANSI’). In the hypothetical alternate universe where people use ASCII-based structured data, ISO 2022 would have standard sequences for such metadata. If you were rolling your own today, you'd probably wrap the metadata in Application Program Command (ESC _ … ESC \) or Start Of String (ESC X … ESC \) or one of the private-use sequences. ¹ https://www.ecma-international.org/publications/standards/Ecma-035.htm https://www.ecma-international.org/publications/standards/Ec... ² https://www.ecma-international.org/publications/standards/Ecma-048.htm https://www.ecma-international.org/publications/standards/Ec...

jefftk 7y ago

While that might have been a good idea if we'd started a while ago, at this point CSV/TSV are so much more established that using commas or tabs doesn't actually save pain.

zachrose 7y ago

But wouldn’t you have to account for the possibility that those separators exist in the content your working with, putting you back at square one in terms of escaping?

kec 7y ago
That's entirely the point: 0x1C and 0x1E should never actually appear in "normal" text unless someone has explicitly put them there, which is not necessarily true of , \t or \n.

mdellavo 7y ago

using csv and tools like grep and join is very fast for large datasets

lgas 7y ago
Using the reserved separator characters wouldn't change that.
perl4ever 7y ago
I always used cut, sort, and uniq a lot too.

dev_dull 7y ago

We’ve discussed this at length and I’m squarely in the DO NOT camp. The drawbacks of using non-readable meta characters exceeds the benefits: 1. TSV can be read and imported by almost everything. 2. People can add and adjust TSV files from any editor. 3. What’s the way to insert meta characters again In VIM? And now nano? Argh I’ll just try and copy and paste it. Ugh that doesn’t work. Just use CSV/TSV folks. Anything more complicated and reach for a better serialization format (json, yaml) and not a better delimiter.

cbsmith 7y ago
Like a lot of things, #1 is only partly true, particularly if you have embedded tabs. The number of times I've had to deal with parsing errors because of embedded carriage returns, commas, tabs, etc., sometimes costing millions of dollars is just... upsetting. It's like JSON... we say everything can parse it, but really what we've got is some approximation that will come back to haunt us... and once you've done all the work to make sure everything is precise and correct, you'd really have saved time if you'd just used the tools we already had in the first place.
- mruts 7y ago
  Dealing with tabs and carriage returns has sometimes cost you millions of dollars? If so, that sounds like a great story and I would love to hear it!
  - jlgaddis 7y ago
    Especially if (as it sounds) it's happened more than once!
cbsmith 7y ago
2. More accurately, people can screw up TSV files from any editor. Have you not seen embedded spaces used instead of tabs and totally mess everyone up?
cbsmith 7y ago
3. If you don't know how to use your editor, maybe you should learn. If your editor can't insert even all the characters in basic ASCII, it's not an editor.

xashor 7y ago

We took the wrong path quite some time ago. Most of the POSIX utils like sort, cut, grep only support \n (or sometimes \0) as row separator.

jlgaddis 7y ago
Are you aware of $IFS?
- xashor 7y ago
  I think $IFS is only used by the shell (with read and expansions), not by e.g. sort. It is so at least on my system.

perl4ever 7y ago

For some reason, in some circles, it seems to be semi standard to use þ (0xFE aka thorn) as the delimiter and a paragraph symbol (0x14 aka DC4 aka ^T) as the separator. The latter is not to be confused with 0xB6. Anyway, these character are presumably not going to occur in ordinary text.

PeterisP 7y ago
All of "upper ascii half" can occur in ordinary text in "pre-Unicode" encodings. 0xFE is a good example - you may get a customer or employee from Iceland with that character in their name (e.g. https://en.wikipedia.org/wiki/Haf%C3%BE%C3%B3r_J%C3%BAl%C3%ADus_Bj%C3%B6rnsson https://en.wikipedia.org/wiki/Haf%C3%BE%C3%B3r_J%C3%BAl%C3%A...), or data in cyrillic cp1251 or koi8-r enconding where 0xFE also represents characters that you'll encounter in surnames, etc.
- perl4ever 7y ago
  It would be escaped in that case.

dagenix 7y ago

But, there is nothing to stop your values from containing these characters. So, you still have to escape your input. And once you've done that, you might as well just use csv / tsv.

cbsmith 7y ago
Actually, the whole point of those characters is that they cannot be used in values.

jstimpfle 7y ago

How do you do (for example) column names or types with ASCII only? How do you extend ASCII?

kps 7y ago
ESC is the ASCII character for code extension. ISO 2022 aka ECMA-35¹ standardizes how to use it in general, but the only thing that really caught on is a subset of the terminal control extensions (ISO 6429 aka ECMA-48² aka ‘ANSI’). In the hypothetical alternate universe where people use ASCII-based structured data, ISO 2022 would have standard sequences for such metadata. If you were rolling your own today, you'd probably wrap the metadata in Application Program Command (ESC _ … ESC \) or Start Of String (ESC X … ESC \) or one of the private-use sequences. ¹ https://www.ecma-international.org/publications/standards/Ecma-035.htm https://www.ecma-international.org/publications/standards/Ec... ² https://www.ecma-international.org/publications/standards/Ecma-048.htm https://www.ecma-international.org/publications/standards/Ec...