3 ms·

Not if you just say those characters are invalid data. I first heard about them decades ago, but I don't think I have ever once seen them in use. The real pro

Not if you just say those characters are invalid data. I first heard about them decades ago, but I don't think I have ever once seen them in use. The real problem is that there is no easy universal way to type them with a keyboard. So it would require software interfaces in the application, and at that point it’s basically binary.

Ukv 2y ago
> Not if you just say those characters are invalid data The author's claim is "with no restrictions on the text". It's easy if you can forbid certain characters, but then you can't store arbitrary text (e.g: filepaths, or scraped comments).
- dec0dedab0de 2y ago
  but these specific characters are not text. they exist solely to be delimiters. It would be like trying to escape a column in your spreadsheet.
  - Ukv 2y ago
    > but these specific characters are not text. they exist solely to be delimiters. Even if people used them only for their intended purpose, someone could use them as delimiters within the text you want to store (e.g: list of tags in filename) - unless I'm misunderstanding. > It would be like trying to escape a column in your spreadsheet. Other formats do allow escaping their delimiters, so that you can use that character literally or even nest a string of that format within an entry.
    - dec0dedab0de 2y ago
      Right, nesting wouldn’t be possible. But there is never any reason to use these characters literally, they are just delimiters.
      - deleted 2y ago
        [deleted]
      - Ukv 2y ago
        > But there is never any reason to use these characters literally, they are just delimiters. I can put []* in my comment (maybe because I'm demonstrating the format, referencing the characters, or just being capricious), and now someone scraping and storing comments has a need to use those characters literally. Sometimes fine to ignore certain content or store it lossily, but often not. *: (copy-paste between the brackets into https://bobpritchett.com/unicode-inspector https://bobpritchett.com/unicode-inspector)
        dec0dedab0de 2y ago
        yes, you would still need to clean your inputs before randomly adding it to your table. Your contrived example brings me back to my original assertion that as long as you’re ok with those characters not being valid data it works fine. So, sure if someone really wanted to store those two literal non-visible characters in a text file that would not work. Everyone else could just not do that.
        Ukv 2y ago
        > yes, you would still need to clean your inputs before randomly adding it to your table. Lossy is fine in some cases, but in many cases you do actually need the specific text you're trying to store - not just something similar to it. Hence my objection to "never any reason to use these characters literally". > Your contrived example [...] if someone really wanted to store those two literal non-visible characters in a text file Needing to store these specific characters is rare, but needing to store arbitrary text (possibly from adversarial/mischievous parties, or just a large enough dataset that encountering all edge-cases is inevitable) is common. For instance, for security reasons a log shouldn't break or have a blindspot for folders with those characters. > as long as you’re ok with those characters not being valid data it works fine Which is what I'm saying in my original comment with "or alternatively, a restriction for the stored text not to have them"

Ukv 2y ago

> Not if you just say those characters are invalid data The author's claim is "with no restrictions on the text". It's easy if you can forbid certain characters, but then you can't store arbitrary text (e.g: filepaths, or scraped comments).

dec0dedab0de 2y ago
but these specific characters are not text. they exist solely to be delimiters. It would be like trying to escape a column in your spreadsheet.
- Ukv 2y ago
  > but these specific characters are not text. they exist solely to be delimiters. Even if people used them only for their intended purpose, someone could use them as delimiters within the text you want to store (e.g: list of tags in filename) - unless I'm misunderstanding. > It would be like trying to escape a column in your spreadsheet. Other formats do allow escaping their delimiters, so that you can use that character literally or even nest a string of that format within an entry.
  - dec0dedab0de 2y ago
    Right, nesting wouldn’t be possible. But there is never any reason to use these characters literally, they are just delimiters.
    - deleted 2y ago
      [deleted]
    - Ukv 2y ago
      > But there is never any reason to use these characters literally, they are just delimiters. I can put []* in my comment (maybe because I'm demonstrating the format, referencing the characters, or just being capricious), and now someone scraping and storing comments has a need to use those characters literally. Sometimes fine to ignore certain content or store it lossily, but often not. *: (copy-paste between the brackets into https://bobpritchett.com/unicode-inspector https://bobpritchett.com/unicode-inspector)
      - dec0dedab0de 2y ago
        yes, you would still need to clean your inputs before randomly adding it to your table. Your contrived example brings me back to my original assertion that as long as you’re ok with those characters not being valid data it works fine. So, sure if someone really wanted to store those two literal non-visible characters in a text file that would not work. Everyone else could just not do that.
        Ukv 2y ago
        > yes, you would still need to clean your inputs before randomly adding it to your table. Lossy is fine in some cases, but in many cases you do actually need the specific text you're trying to store - not just something similar to it. Hence my objection to "never any reason to use these characters literally". > Your contrived example [...] if someone really wanted to store those two literal non-visible characters in a text file Needing to store these specific characters is rare, but needing to store arbitrary text (possibly from adversarial/mischievous parties, or just a large enough dataset that encountering all edge-cases is inevitable) is common. For instance, for security reasons a log shouldn't break or have a blindspot for folders with those characters. > as long as you’re ok with those characters not being valid data it works fine Which is what I'm saying in my original comment with "or alternatively, a restriction for the stored text not to have them"