6 ms·

I recently learned via https://news.ycombinator.com/item?id=31257248 https://news.ycombinator.com/item?id=31257248 that ASCII has the idea of records and fields

I recently learned via https://news.ycombinator.com/item?id=31257248 https://news.ycombinator.com/item?id=31257248 that ASCII has the idea of records and fields ever since. It's just not used, but workaround CSV. No improvement of CSV handling will ever improved on that.

cb321 4y ago
See the "Conventions for lossless conversion to TSV" on the current version of https://en.wikipedia.org/wiki/Tab-separated_values https://en.wikipedia.org/wiki/Tab-separated_values . There are really only 3 chars to escape - the escape char, and 2 delimiters - to make everything easy to deal with (even binary data in fields - what "lossless" means here).
belter 4y ago
Discussed here before, not everybody was seeing only positives. "ASCII Delimited Text – Not CSV or TAB delimited text": https://news.ycombinator.com/item?id=7474600 https://news.ycombinator.com/item?id=7474600
- Chris2048 4y ago
  A big problem is tooling - what they currently support, and the fact that excel will f*ck up anything. I suppose the best way would be a suite of tools to edit them that are compatible with existing editors? TBH a great value-add for adoption would be versioning s.t. you can see (and revert) when other tools mess up your files.
RhysU 4y ago
Could one get away with regular AWK using these combined with a streaming to/from CSV converter?
- cb321 4y ago
  Not just "regular awk", but "regular most things", like head, tail, grep, etc. The "things" just need to be 8-bit/NUL handling clean and the data could even have binary fields. I mean, if there is a library like Python/Go/etc. that you especially trust to write a streaming converter then that is a fine approach, but otherwise see both https://news.ycombinator.com/item?id=31352517 https://news.ycombinator.com/item?id=31352517 and https://news.ycombinator.com/item?id=31352704 https://news.ycombinator.com/item?id=31352704 (at least!)
- mmcgaha 4y ago
  Yes and I suspect that this is why it was never added to AWK. I,and I am sure most people, have an AWK filter to transform their csv or whatever format to a format that AWK can use with an appropriate FS and RS. Their "but why" section should really go into more detail about why filters are not 100% if your data has the possibility of containing your preferred line/record separator. But real world, I have never had a situation where a csv-to-awk filter (written in awk of course) did not work.
andylynch 4y ago
Closely related- I remain mystified as to why FIX Protocol chose to use control characters to separate fields, but used SOH rather than RS/US or the like.
- kevin_thibedeau 4y ago
  Because of EBCDIC.
carapace 4y ago
I'm just gonna leave this here. https://git.sr.ht/~sforman/sv/tree/master/item/sv.py https://git.sr.ht/~sforman/sv/tree/master/item/sv.py (Python code for ASCII-separated values. TL;DR: it's so stupid simple that I feel any objection to using this format has just got to be wrong. It's like "Hey, we don't have to keep stabbing ourselves in the face." ... "But how will we use our knives?" See? It's like that.)
- a1369209993 4y ago
  > any objection to using this format How do I edit the data file in a text editor? More generally, how do I edit the entire data file, including field/record/etc seperators, in a editor that can display only characters that are valid content of data fields? (Not that CSV (barring de-facto-nonstandard extensions like rfc4180's `"b CRLF bb"` and `"b""bb"`) is any good for this either, of course.)
  - CRConrad 4y ago
    > > any objection to using this format > How do I edit the data file in a text editor? Use a non-sucky text editor. For example, AFAICR NotePad++ displays those characters. Windows doesn't have any ultra-easy way to input them, but within NP++ you can copy-paste them.
    - a1369209993 4y ago
      > For example, AFAICR NotePad++ displays those characters. Great! Now go back to the other half of the requirement: how do I store all characters that NotePad++ displays in the content of a data field?
      - CRConrad 4y ago
        These delimiters, you mean? You don't. They're delimiters, they never go in a field. Bam, problem solved. (Or rather, problem didn't exist in the first place.)
  - 4y ago

cb321 4y ago

See the "Conventions for lossless conversion to TSV" on the current version of https://en.wikipedia.org/wiki/Tab-separated_values https://en.wikipedia.org/wiki/Tab-separated_values . There are really only 3 chars to escape - the escape char, and 2 delimiters - to make everything easy to deal with (even binary data in fields - what "lossless" means here).

belter 4y ago

Discussed here before, not everybody was seeing only positives. "ASCII Delimited Text – Not CSV or TAB delimited text": https://news.ycombinator.com/item?id=7474600 https://news.ycombinator.com/item?id=7474600

Chris2048 4y ago
A big problem is tooling - what they currently support, and the fact that excel will f*ck up anything. I suppose the best way would be a suite of tools to edit them that are compatible with existing editors? TBH a great value-add for adoption would be versioning s.t. you can see (and revert) when other tools mess up your files.

RhysU 4y ago

Could one get away with regular AWK using these combined with a streaming to/from CSV converter?

cb321 4y ago
Not just "regular awk", but "regular most things", like head, tail, grep, etc. The "things" just need to be 8-bit/NUL handling clean and the data could even have binary fields. I mean, if there is a library like Python/Go/etc. that you especially trust to write a streaming converter then that is a fine approach, but otherwise see both https://news.ycombinator.com/item?id=31352517 https://news.ycombinator.com/item?id=31352517 and https://news.ycombinator.com/item?id=31352704 https://news.ycombinator.com/item?id=31352704 (at least!)
mmcgaha 4y ago
Yes and I suspect that this is why it was never added to AWK. I,and I am sure most people, have an AWK filter to transform their csv or whatever format to a format that AWK can use with an appropriate FS and RS. Their "but why" section should really go into more detail about why filters are not 100% if your data has the possibility of containing your preferred line/record separator. But real world, I have never had a situation where a csv-to-awk filter (written in awk of course) did not work.

andylynch 4y ago

Closely related- I remain mystified as to why FIX Protocol chose to use control characters to separate fields, but used SOH rather than RS/US or the like.

kevin_thibedeau 4y ago
Because of EBCDIC.

carapace 4y ago

I'm just gonna leave this here. https://git.sr.ht/~sforman/sv/tree/master/item/sv.py https://git.sr.ht/~sforman/sv/tree/master/item/sv.py (Python code for ASCII-separated values. TL;DR: it's so stupid simple that I feel any objection to using this format has just got to be wrong. It's like "Hey, we don't have to keep stabbing ourselves in the face." ... "But how will we use our knives?" See? It's like that.)

a1369209993 4y ago
> any objection to using this format How do I edit the data file in a text editor? More generally, how do I edit the entire data file, including field/record/etc seperators, in a editor that can display only characters that are valid content of data fields? (Not that CSV (barring de-facto-nonstandard extensions like rfc4180's `"b CRLF bb"` and `"b""bb"`) is any good for this either, of course.)
- CRConrad 4y ago
  > > any objection to using this format > How do I edit the data file in a text editor? Use a non-sucky text editor. For example, AFAICR NotePad++ displays those characters. Windows doesn't have any ultra-easy way to input them, but within NP++ you can copy-paste them.
  - a1369209993 4y ago
    > For example, AFAICR NotePad++ displays those characters. Great! Now go back to the other half of the requirement: how do I store all characters that NotePad++ displays in the content of a data field?
    - CRConrad 4y ago
      These delimiters, you mean? You don't. They're delimiters, they never go in a field. Bam, problem solved. (Or rather, problem didn't exist in the first place.)
- 4y ago