5 ms·

Wait a second … he’s not proposing using unit/record/group/file separators as separators, he’s proposing using the symbols for those separators as separators!

Wait a second … he’s not proposing using unit/record/group/file separators as separators, he’s proposing using the symbols for those separators as separators! Why not just use the separators themselves‽ Yes, rather than using U+1F (the ASCII and Unicode unit separator), he proposes using U+241F (the Unicode symbol for the unit separator). I almost feel like this must be an early April Fool’s joke? Also, he writes ‘comprised of’ rather than ‘composed of’ or ‘comprises’ throughout his RFC.

ahofmann 3y ago
They explain in the FAQ that this approach works with most text editors and copy-paste situations.
- philipwhiuk 3y ago
  It doesn't "work" because I can't read the darn things at a sane zoom level.
bryanlarsen 3y ago
Using a visible character rather than an invisible one makes editing in an editor a lot easier.
- tgv 3y ago
  It won't wrap at the record separator, so you'll get a very long line.
  - bryanlarsen 3y ago
    The example seems to use `␞\n` as a separator rather than just `␞`. I assume their proposed standard is more definitive.
    - vidarh 3y ago
      Their ABNF uses RS, defined as U+241E, not U+241E + '\n' as the record separator. They seem to add an "USV escape" in front of the linefeeds. My bet is that this will lead to implementations that wrongly treats "␞␛\n" (RS ESC \m) as the real record separator, the same way lots of "CSV" implementations just split on comma and LF. Seems to me if you're going to add support for something like that you should just bite the bullet and declare an LF immediately following an RS as part of the record separator, or you're falling in the same trap as CSV of being "close enough" to naively splittable that people will do it because it works often enough.
      - shawnz 3y ago
        The escape symbol lets you ignore any non-special character, not just newlines: https://github.com/sixarm/usv?tab=readme-ov-file#escape-esc https://github.com/sixarm/usv?tab=readme-ov-file#escape-esc
        vidarh 3y ago
        I'm aware. I don't think that serves a useful purpose - I think the way they've done it is likely to make people more likely to get the parsing wrong for pretty much zero benefit. My guess is you'll end up seeing a lot of "pseudo-USV" parsers the same way we have a ton of "pseudo-CSV" parsers that breaks on escapes or quoted strings with commas, and so I think they fundamentally failed to learn the lessons of CSV.
    - theamk 3y ago
      that's a lie as far as I can see, the csv-to-usv tools does not add any newlines: [0] https://github.com/SixArm/csv-to-usv-rust-crate/blob/30a0324d8ebd1086bae52ab2fc8d3c9abda53c0a/src/lib.rs#L11 https://github.com/SixArm/csv-to-usv-rust-crate/blob/30a0324...
      - bryanlarsen 3y ago
        The examples here have them: https://github.com/SixArm/usv/tree/main/examples https://github.com/SixArm/usv/tree/main/examples
- NoMoreNicksLeft 3y ago
  If you're doing spreadsheets, then it should show in a spreadsheet and not in an editor. It's like complaining that he can't edit jpegs in Sublime or something... there's a reason that's working poorly. Speaking of which, last time I had a control code heavy file open in Sublime, it actually did show the control codes as special characters, and it was possible to copy/paste those. This proposal is so bad I suspect it will become a standard.
  - dimask 3y ago
    There are a lot of cases where I would rather inspect/quickfix a csv file in a text editor rather than open it as a spreadsheet. Especially cases where something is wrong in the format, and it will just not open as a spreadsheet at all. Adding unnecessary levels of obfuscation to your data should never be considered a good idea imo.
- eadmund 3y ago
  The ASCII separators are visible in my editor. If something doesn’t support ASCII text, that sounds like a bug which should be fixed, not a reason to misuse graphical characters for something other than their purpose.
SigmundurM 3y ago
They cover the reasoning for using the control picture characters instead of the control characters in the FAQ: "We tried using the control characters, and also tried configuring various editors to show the control characters by rendering the control picture characters. First, we encountered many difficulties with editor configurations, attempting to make each editor treat the invisible zero-width characters by rendering with the visible letter-width characters. Second, we encountered problems with copy/paste functionality, where it often didn't work because the editor implementations and terminal implementations copied visible letter-width characters, not the underlying invisible zero-width characters. Third, users were unable to distinguish between the rendered control picture characters (e.g. the editor saw ASCII 31 and rendered Unicode Unit Separator) versus the control picture characters being in the data content (e.g. someone actually typed Unicode Unit Separator into the data content)." - https://github.com/SixArm/usv/tree/main/doc/faq#why-use-control-picture-characters-rather-than-the-control-characters-themselves https://github.com/SixArm/usv/tree/main/doc/faq#why-use-cont...
- nostrademons 3y ago
  https://xkcd.com/927/ https://xkcd.com/927/
  - pie_flavor 3y ago
    'Too many competing standards' is not one of the quoted reasons.
  - ascorbic 3y ago
    https://github.com/SixArm/usv/blob/main/doc/criticisms/index.md#xkcd-one-universal-standard https://github.com/SixArm/usv/blob/main/doc/criticisms/index...
- vidarh 3y ago
  I can't read those characters at the size I can/prefer to read the text at, so I need the tooling to support and render these differently anyway... This feels like solving the wrong problem in a way that will still end up with the same amount of work.
ape4 3y ago
An issue with CSV is that commas need to be escaped. Are the U+241F characters escaped in this USV format?
hermitcrab 3y ago
I don't see any real advantage over using ASCII unit and record separators (.asv). Also I am not convinced about the need for an escape character. If you really need to use ASCII unit or record separators as data - tough use a different format. If only editors would display the ASCII unit separator (Notepad++ does) and treat the ASCII record as a carriage return (Notepad++ doesn't) then .asv format would be a huge improvement on CSV.
skirmish 3y ago
'comprised of' is standard verbiage in US patents, and I am guessing he is trying to sound formal and official. Also see https://en.wikipedia.org/wiki/Comprised_of https://en.wikipedia.org/wiki/Comprised_of .

ahofmann 3y ago

They explain in the FAQ that this approach works with most text editors and copy-paste situations.

philipwhiuk 3y ago
It doesn't "work" because I can't read the darn things at a sane zoom level.

bryanlarsen 3y ago

Using a visible character rather than an invisible one makes editing in an editor a lot easier.

tgv 3y ago
It won't wrap at the record separator, so you'll get a very long line.
- bryanlarsen 3y ago
  The example seems to use `␞\n` as a separator rather than just `␞`. I assume their proposed standard is more definitive.
  - vidarh 3y ago
    Their ABNF uses RS, defined as U+241E, not U+241E + '\n' as the record separator. They seem to add an "USV escape" in front of the linefeeds. My bet is that this will lead to implementations that wrongly treats "␞␛\n" (RS ESC \m) as the real record separator, the same way lots of "CSV" implementations just split on comma and LF. Seems to me if you're going to add support for something like that you should just bite the bullet and declare an LF immediately following an RS as part of the record separator, or you're falling in the same trap as CSV of being "close enough" to naively splittable that people will do it because it works often enough.
    - shawnz 3y ago
      The escape symbol lets you ignore any non-special character, not just newlines: https://github.com/sixarm/usv?tab=readme-ov-file#escape-esc https://github.com/sixarm/usv?tab=readme-ov-file#escape-esc
      - vidarh 3y ago
        I'm aware. I don't think that serves a useful purpose - I think the way they've done it is likely to make people more likely to get the parsing wrong for pretty much zero benefit. My guess is you'll end up seeing a lot of "pseudo-USV" parsers the same way we have a ton of "pseudo-CSV" parsers that breaks on escapes or quoted strings with commas, and so I think they fundamentally failed to learn the lessons of CSV.
  - theamk 3y ago
    that's a lie as far as I can see, the csv-to-usv tools does not add any newlines: [0] https://github.com/SixArm/csv-to-usv-rust-crate/blob/30a0324d8ebd1086bae52ab2fc8d3c9abda53c0a/src/lib.rs#L11 https://github.com/SixArm/csv-to-usv-rust-crate/blob/30a0324...
    - bryanlarsen 3y ago
      The examples here have them: https://github.com/SixArm/usv/tree/main/examples https://github.com/SixArm/usv/tree/main/examples
NoMoreNicksLeft 3y ago
If you're doing spreadsheets, then it should show in a spreadsheet and not in an editor. It's like complaining that he can't edit jpegs in Sublime or something... there's a reason that's working poorly. Speaking of which, last time I had a control code heavy file open in Sublime, it actually did show the control codes as special characters, and it was possible to copy/paste those. This proposal is so bad I suspect it will become a standard.
- dimask 3y ago
  There are a lot of cases where I would rather inspect/quickfix a csv file in a text editor rather than open it as a spreadsheet. Especially cases where something is wrong in the format, and it will just not open as a spreadsheet at all. Adding unnecessary levels of obfuscation to your data should never be considered a good idea imo.
eadmund 3y ago
The ASCII separators are visible in my editor. If something doesn’t support ASCII text, that sounds like a bug which should be fixed, not a reason to misuse graphical characters for something other than their purpose.

SigmundurM 3y ago

They cover the reasoning for using the control picture characters instead of the control characters in the FAQ: "We tried using the control characters, and also tried configuring various editors to show the control characters by rendering the control picture characters. First, we encountered many difficulties with editor configurations, attempting to make each editor treat the invisible zero-width characters by rendering with the visible letter-width characters. Second, we encountered problems with copy/paste functionality, where it often didn't work because the editor implementations and terminal implementations copied visible letter-width characters, not the underlying invisible zero-width characters. Third, users were unable to distinguish between the rendered control picture characters (e.g. the editor saw ASCII 31 and rendered Unicode Unit Separator) versus the control picture characters being in the data content (e.g. someone actually typed Unicode Unit Separator into the data content)." - https://github.com/SixArm/usv/tree/main/doc/faq#why-use-control-picture-characters-rather-than-the-control-characters-themselves https://github.com/SixArm/usv/tree/main/doc/faq#why-use-cont...

nostrademons 3y ago
https://xkcd.com/927/ https://xkcd.com/927/
- pie_flavor 3y ago
  'Too many competing standards' is not one of the quoted reasons.
- ascorbic 3y ago
  https://github.com/SixArm/usv/blob/main/doc/criticisms/index.md#xkcd-one-universal-standard https://github.com/SixArm/usv/blob/main/doc/criticisms/index...
vidarh 3y ago
I can't read those characters at the size I can/prefer to read the text at, so I need the tooling to support and render these differently anyway... This feels like solving the wrong problem in a way that will still end up with the same amount of work.

ape4 3y ago

An issue with CSV is that commas need to be escaped. Are the U+241F characters escaped in this USV format?

hermitcrab 3y ago

I don't see any real advantage over using ASCII unit and record separators (.asv). Also I am not convinced about the need for an escape character. If you really need to use ASCII unit or record separators as data - tough use a different format. If only editors would display the ASCII unit separator (Notepad++ does) and treat the ASCII record as a carriage return (Notepad++ doesn't) then .asv format would be a huge improvement on CSV.

skirmish 3y ago

'comprised of' is standard verbiage in US patents, and I am guessing he is trying to sound formal and official. Also see https://en.wikipedia.org/wiki/Comprised_of https://en.wikipedia.org/wiki/Comprised_of .