4 ms·

ASCII has had field and record separators since like, forever. Wish we had kept using those.

by djbusby 3y ago

josefx 3y ago
Does it have nesting operators? I want to embedd ASCII within my ASCII fields. So I can have a table within my table.
- lolive 3y ago
  A manager: I wish I could have a CSV inside my CSV. Any sane person: NO!
  - josefx 3y ago
    base64 encoded fields it is.
    - lolive 3y ago
      That was the right answer. You are hired!
  - bruce511 3y ago
    Well you might be wrong, but EDI in general and HL7 specifically allow 3 levels of "fields in fields in field". As long as your parser copes, and as long as you have appropriate structures to import into, its no big deal.
    - iainmerrick 3y ago
      So now when I'm exporting data I need to know what nesting level it's going to live at so I can generate the correct separators? I really think that might be the worst idea I've heard for a while!
      - bruce511 3y ago
        It's not toooo bad :) But it's very much a "thing" in the real word. It's called EDI, and it's been around for a long time.
  - wiredfool 3y ago
    Been there, done the Interop.
  - icedchai 3y ago
    About 10 years ago, I worked at a place where we were embedding both XML and JSON in CSV fields. Then there are always the people who can't generate a valid CSV due to escaping issues... Nothing is ever simple.
- vaylian 3y ago
  It kind of does. See `man ascii` * FS (0x1C) file separator * GS (0x1D) group separator * RS (0x1E) record separator * US (0x1F) unit separator I've never seen these in the wild though.
  - alex_suzuki 3y ago
    They‘re used a lot in barcodes, e.g. for delimiting the different fields of a driving license.
    - euroderf 3y ago
      That sounds like the premise for an utterly fascinating deep dive.
      - alex_suzuki 3y ago
        It‘s quite the rabbit hole, I can assure you.
  - adammarples 3y ago
    Not seeing them in the wild is good, it means they will work when you use them. The more they get used the more often you'll find they crop up in the text fields you're trying to separate and the whole csv escaping nightmare will continue.
    - Repulsion9513 3y ago
      If you've got delimited-text embedded inside your delimited-text, you've got a nightmare that escaping can't save you from. (obligatory https://knowyourmeme.com/memes/xzibit-yo-dawg https://knowyourmeme.com/memes/xzibit-yo-dawg)
  - dhagberg 3y ago
    They're definitely used in ACARS messages that go to every commercial airplane cockpit...
- ykonstant 3y ago
  The POSIX spec defines portable and non-portable ASCII characters, prudently placing the separators in the non-portable set. In order to nest tables, base64 (or whatever portable encoding) code the table into the field. This works much better, easier and more error-free than any escaping strategy. Regarding visibility in editors, if you are nesting tables I don't think you care too much about manual editing, but if you do, it is easy to setup vim/emacs/vscode to display the ASCII separators. I am being told even notepad++ can display those, there are no excuses.
lolive 3y ago
https://github.com/SixArm/usv/tree/main/doc/faq#why-use-control-picture-characters-rather-than-the-control-characters-themselves https://github.com/SixArm/usv/tree/main/doc/faq#why-use-cont...
- deleted 3y ago
  [deleted]
wombatpm 3y ago
No you don’t. It’s a holdover from when files were on tapes. The logic is all inverted too. Record separators were at the beginning of a record. Group and unit separators could then be nested. You really needed a spec for whatever document you were trying to makt
- rightbyte 3y ago
  It doesn't matter if the sentinel byte is after or before each record. Having it before is nice for syncing byte streams.
  - wombatpm 3y ago
    It matters when you trying to roundtrip the data through a text editor because existing tools balk at a 300MB file with a single line.
- Repulsion9513 3y ago
  You need a "spec" just the same for a CSV: does it have headers, is there a consistent column ordering, etc. Control characters work the exact same as commas and newlines, but you don't have to worry about escaping them because (unless your data is binary or recursive) the control characters won't show up in the data!
- jdougan 3y ago
  Do you have a reference to how this worked?

ASCII has had field and record separators since like, forever. Wish we had kept using those.

josefx 3y ago

Does it have nesting operators? I want to embedd ASCII within my ASCII fields. So I can have a table within my table.

lolive 3y ago
A manager: I wish I could have a CSV inside my CSV. Any sane person: NO!
- josefx 3y ago
  base64 encoded fields it is.
  - lolive 3y ago
    That was the right answer. You are hired!
- bruce511 3y ago
  Well you might be wrong, but EDI in general and HL7 specifically allow 3 levels of "fields in fields in field". As long as your parser copes, and as long as you have appropriate structures to import into, its no big deal.
  - iainmerrick 3y ago
    So now when I'm exporting data I need to know what nesting level it's going to live at so I can generate the correct separators? I really think that might be the worst idea I've heard for a while!
    - bruce511 3y ago
      It's not toooo bad :) But it's very much a "thing" in the real word. It's called EDI, and it's been around for a long time.
- wiredfool 3y ago
  Been there, done the Interop.
- icedchai 3y ago
  About 10 years ago, I worked at a place where we were embedding both XML and JSON in CSV fields. Then there are always the people who can't generate a valid CSV due to escaping issues... Nothing is ever simple.
vaylian 3y ago
It kind of does. See `man ascii` * FS (0x1C) file separator * GS (0x1D) group separator * RS (0x1E) record separator * US (0x1F) unit separator I've never seen these in the wild though.
- alex_suzuki 3y ago
  They‘re used a lot in barcodes, e.g. for delimiting the different fields of a driving license.
  - euroderf 3y ago
    That sounds like the premise for an utterly fascinating deep dive.
    - alex_suzuki 3y ago
      It‘s quite the rabbit hole, I can assure you.
- adammarples 3y ago
  Not seeing them in the wild is good, it means they will work when you use them. The more they get used the more often you'll find they crop up in the text fields you're trying to separate and the whole csv escaping nightmare will continue.
  - Repulsion9513 3y ago
    If you've got delimited-text embedded inside your delimited-text, you've got a nightmare that escaping can't save you from. (obligatory https://knowyourmeme.com/memes/xzibit-yo-dawg https://knowyourmeme.com/memes/xzibit-yo-dawg)
- dhagberg 3y ago
  They're definitely used in ACARS messages that go to every commercial airplane cockpit...
ykonstant 3y ago
The POSIX spec defines portable and non-portable ASCII characters, prudently placing the separators in the non-portable set. In order to nest tables, base64 (or whatever portable encoding) code the table into the field. This works much better, easier and more error-free than any escaping strategy. Regarding visibility in editors, if you are nesting tables I don't think you care too much about manual editing, but if you do, it is easy to setup vim/emacs/vscode to display the ASCII separators. I am being told even notepad++ can display those, there are no excuses.

lolive 3y ago

https://github.com/SixArm/usv/tree/main/doc/faq#why-use-control-picture-characters-rather-than-the-control-characters-themselves https://github.com/SixArm/usv/tree/main/doc/faq#why-use-cont...

deleted 3y ago
[deleted]

wombatpm 3y ago

No you don’t. It’s a holdover from when files were on tapes. The logic is all inverted too. Record separators were at the beginning of a record. Group and unit separators could then be nested. You really needed a spec for whatever document you were trying to makt

rightbyte 3y ago
It doesn't matter if the sentinel byte is after or before each record. Having it before is nice for syncing byte streams.
- wombatpm 3y ago
  It matters when you trying to roundtrip the data through a text editor because existing tools balk at a 300MB file with a single line.
Repulsion9513 3y ago
You need a "spec" just the same for a CSV: does it have headers, is there a consistent column ordering, etc. Control characters work the exact same as commas and newlines, but you don't have to worry about escaping them because (unless your data is binary or recursive) the control characters won't show up in the data!
jdougan 3y ago
Do you have a reference to how this worked?