5 ms·

Does it have nesting operators? I want to embedd ASCII within my ASCII fields. So I can have a table within my table.

by josefx 3y ago

lolive 3y ago
A manager: I wish I could have a CSV inside my CSV. Any sane person: NO!
- josefx 3y ago
  base64 encoded fields it is.
  - lolive 3y ago
    That was the right answer. You are hired!
- bruce511 3y ago
  Well you might be wrong, but EDI in general and HL7 specifically allow 3 levels of "fields in fields in field". As long as your parser copes, and as long as you have appropriate structures to import into, its no big deal.
  - iainmerrick 3y ago
    So now when I'm exporting data I need to know what nesting level it's going to live at so I can generate the correct separators? I really think that might be the worst idea I've heard for a while!
    - bruce511 3y ago
      It's not toooo bad :) But it's very much a "thing" in the real word. It's called EDI, and it's been around for a long time.
- wiredfool 3y ago
  Been there, done the Interop.
- icedchai 3y ago
  About 10 years ago, I worked at a place where we were embedding both XML and JSON in CSV fields. Then there are always the people who can't generate a valid CSV due to escaping issues... Nothing is ever simple.
vaylian 3y ago
It kind of does. See `man ascii` * FS (0x1C) file separator * GS (0x1D) group separator * RS (0x1E) record separator * US (0x1F) unit separator I've never seen these in the wild though.
- alex_suzuki 3y ago
  They‘re used a lot in barcodes, e.g. for delimiting the different fields of a driving license.
  - euroderf 3y ago
    That sounds like the premise for an utterly fascinating deep dive.
    - alex_suzuki 3y ago
      It‘s quite the rabbit hole, I can assure you.
- adammarples 3y ago
  Not seeing them in the wild is good, it means they will work when you use them. The more they get used the more often you'll find they crop up in the text fields you're trying to separate and the whole csv escaping nightmare will continue.
  - Repulsion9513 3y ago
    If you've got delimited-text embedded inside your delimited-text, you've got a nightmare that escaping can't save you from. (obligatory https://knowyourmeme.com/memes/xzibit-yo-dawg https://knowyourmeme.com/memes/xzibit-yo-dawg)
- dhagberg 3y ago
  They're definitely used in ACARS messages that go to every commercial airplane cockpit...
ykonstant 3y ago
The POSIX spec defines portable and non-portable ASCII characters, prudently placing the separators in the non-portable set. In order to nest tables, base64 (or whatever portable encoding) code the table into the field. This works much better, easier and more error-free than any escaping strategy. Regarding visibility in editors, if you are nesting tables I don't think you care too much about manual editing, but if you do, it is easy to setup vim/emacs/vscode to display the ASCII separators. I am being told even notepad++ can display those, there are no excuses.

Does it have nesting operators? I want to embedd ASCII within my ASCII fields. So I can have a table within my table.

by josefx 3y ago

Does it have nesting operators? I want to embedd ASCII within my ASCII fields. So I can have a table within my table.

lolive 3y ago

A manager: I wish I could have a CSV inside my CSV. Any sane person: NO!

josefx 3y ago
base64 encoded fields it is.
- lolive 3y ago
  That was the right answer. You are hired!
bruce511 3y ago
Well you might be wrong, but EDI in general and HL7 specifically allow 3 levels of "fields in fields in field". As long as your parser copes, and as long as you have appropriate structures to import into, its no big deal.
- iainmerrick 3y ago
  So now when I'm exporting data I need to know what nesting level it's going to live at so I can generate the correct separators? I really think that might be the worst idea I've heard for a while!
  - bruce511 3y ago
    It's not toooo bad :) But it's very much a "thing" in the real word. It's called EDI, and it's been around for a long time.
wiredfool 3y ago
Been there, done the Interop.
icedchai 3y ago
About 10 years ago, I worked at a place where we were embedding both XML and JSON in CSV fields. Then there are always the people who can't generate a valid CSV due to escaping issues... Nothing is ever simple.

vaylian 3y ago

It kind of does. See `man ascii` * FS (0x1C) file separator * GS (0x1D) group separator * RS (0x1E) record separator * US (0x1F) unit separator I've never seen these in the wild though.

alex_suzuki 3y ago
They‘re used a lot in barcodes, e.g. for delimiting the different fields of a driving license.
- euroderf 3y ago
  That sounds like the premise for an utterly fascinating deep dive.
  - alex_suzuki 3y ago
    It‘s quite the rabbit hole, I can assure you.
adammarples 3y ago
Not seeing them in the wild is good, it means they will work when you use them. The more they get used the more often you'll find they crop up in the text fields you're trying to separate and the whole csv escaping nightmare will continue.
- Repulsion9513 3y ago
  If you've got delimited-text embedded inside your delimited-text, you've got a nightmare that escaping can't save you from. (obligatory https://knowyourmeme.com/memes/xzibit-yo-dawg https://knowyourmeme.com/memes/xzibit-yo-dawg)
dhagberg 3y ago
They're definitely used in ACARS messages that go to every commercial airplane cockpit...

ykonstant 3y ago

The POSIX spec defines portable and non-portable ASCII characters, prudently placing the separators in the non-portable set. In order to nest tables, base64 (or whatever portable encoding) code the table into the field. This works much better, easier and more error-free than any escaping strategy. Regarding visibility in editors, if you are nesting tables I don't think you care too much about manual editing, but if you do, it is easy to setup vim/emacs/vscode to display the ASCII separators. I am being told even notepad++ can display those, there are no excuses.