3 ms·

> with no restrictions on the text in fields or the need to try and escape characters. Maybe I'm missing something, but wouldn't it still need escaping for tho

by Ukv 2y ago

> with no restrictions on the text in fields or the need to try and escape characters. Maybe I'm missing something, but wouldn't it still need escaping for those ASCII separator characters (or alternatively, a restriction for the stored text not to have them)? It's true that having to deal with escaping much less often (since the ASCII separator characters are rarer than commas/quotes) would be convenient for manual reading/writing, but I feel that's canceled out by the characters being hard to type/see (likely the reason why they're rare) - and it wouldn't necessarily save on writer/parser code complexity.

missblit 2y ago
See this is why I once used moon-viewing-ceremony-seperated-values (MVCSV). The Moon Viewing Ceremony emoji was unlikely to show up in my dataset, and not only is the emoji visible, it's quite visually pleasing.
- dazzaji 2y ago
  I’m now free-falling down a moon viewing ceremony rabbit hole of emoji history, and enjoying the ride!
- Macha 2y ago
  Wait until you expand into the Japanese market and all your users are talking about 月見
huem0n 2y ago
Thank you for writing that complaint out so I don't have to. It solves nothing.
chubot 2y ago
Yup exactly, it just pushes the problem around, without solving it. The delimiters can occur in binary data, or when there's nesting -- trying to store TSV in TSV, or JSON in JSON, etc. The latter definitely happens a lot, for better or worse
- chrishill89 2y ago
  You can disallow those metacharacters in the data proper. Then you have a format that can store any utf8 or whatever except the non-whitespace control codes without any escaping. That solves a problem in an opinionated way. Just like how json is opinionated (utf8 only). You can convert to another format if you need something crazier than rows and columns consisting of normal text.
  - chubot 2y ago
    That already exists -- TSV. You disallow the tab metacharacter.
    - thayne 2y ago
      And the newline character. Both of which commonly occur in normal text.
    - chrishill89 2y ago
      Then I don't understand. Like the sibling comment said the really "problematic" character in TSV is the line feed. But a tab can occur as well. The format that I described does not already exist in the form of TSV. And further based on your original comment I would have thought that both TSV and this format would be discarded as not-useful.
- smarx007 2y ago
  The title says ASCII Delimited Text not ASCII Delimited Binary Data. For the purposes of CSV, I consider text to be anything that satisfies the regex ^\P{Cc}+$ (https://www.compart.com/en/unicode/category/Cc https://www.compart.com/en/unicode/category/Cc) and I normally strip chars in that category before saving some text (for single-line text). ^[\p{Cc}&&[^\n]]+$ is a regex that can be used to strip all control chars except for the newline.
  - chrishill89 2y ago
    Thanks for that. Handy. I think I’ll have use for that myself.
dec0dedab0de 2y ago
Not if you just say those characters are invalid data. I first heard about them decades ago, but I don't think I have ever once seen them in use. The real problem is that there is no easy universal way to type them with a keyboard. So it would require software interfaces in the application, and at that point it’s basically binary.
- Ukv 2y ago
  > Not if you just say those characters are invalid data The author's claim is "with no restrictions on the text". It's easy if you can forbid certain characters, but then you can't store arbitrary text (e.g: filepaths, or scraped comments).
  - dec0dedab0de 2y ago
    but these specific characters are not text. they exist solely to be delimiters. It would be like trying to escape a column in your spreadsheet.
    - Ukv 2y ago
      > but these specific characters are not text. they exist solely to be delimiters. Even if people used them only for their intended purpose, someone could use them as delimiters within the text you want to store (e.g: list of tags in filename) - unless I'm misunderstanding. > It would be like trying to escape a column in your spreadsheet. Other formats do allow escaping their delimiters, so that you can use that character literally or even nest a string of that format within an entry.
      - dec0dedab0de 2y ago
        Right, nesting wouldn’t be possible. But there is never any reason to use these characters literally, they are just delimiters.
        deleted 2y ago
        [deleted]
jader201 2y ago
I feel like this (and some of the replies to this) is missing the point a bit. I don’t think the goal was to make a bullet-proof delimiter that fails at nothing. The goal was to solve the problem of not allowing things like commas, quotes, newlines, tabs, pipes, etc. in text files. I feel like using the proposed ASCII characters would eliminate these limitations, while also allowing machine creatable and readable format (emphasis on machine as opposed to human). Yes, it would still be tough for a human to type or read these delimiters, so in that case, go with traditional CSV or TSV (or MVCSV!). But if you only need to use a machine to create/read the text, this sounds like a great solution, allowing all of the normal characters you might see in text.
- Ukv 2y ago
  If you need a machine-readable format, why not go with escaping like most other formats, or length-before-text, to include all characters - instead of a format that fails on some (albeit rare) characters?
  - jader201 2y ago
    Both of those are fine, but they add additional complexities (even if small) where there is very little, if any, complexity added with using these two characters as delimiters.
    - Ukv 2y ago
      For machine-readable formats, I'd argue length-before-text is simpler to parse than splitting by separators - even before having to add extra application logic to handle this method being fallible (e.g: if you want to use it to store filenames, you now need a check and pop-up about some OS-valid filenames not being supported).
      - chrishill89 2y ago
        > For machine-readable formats, I'd argue length-before-text is simpler to parse than splitting by separators Yes. > even before having to add extra application logic to handle this method being fallible (e.g: if you want to use it to store filenames, you now need a check and pop-up about some OS-valid filenames not being supported). I don't see why you'd need that. CSV does not have anything like that. That's a higher-level concern. This ascii-delimited format (like CSV) is supposed to be a stupid row/column format. And also simpler to implement than CSV.

by Ukv 2y ago

missblit 2y ago

See this is why I once used moon-viewing-ceremony-seperated-values (MVCSV). The Moon Viewing Ceremony emoji was unlikely to show up in my dataset, and not only is the emoji visible, it's quite visually pleasing.

dazzaji 2y ago
I’m now free-falling down a moon viewing ceremony rabbit hole of emoji history, and enjoying the ride!
Macha 2y ago
Wait until you expand into the Japanese market and all your users are talking about 月見

huem0n 2y ago

Thank you for writing that complaint out so I don't have to. It solves nothing.

chubot 2y ago

Yup exactly, it just pushes the problem around, without solving it. The delimiters can occur in binary data, or when there's nesting -- trying to store TSV in TSV, or JSON in JSON, etc. The latter definitely happens a lot, for better or worse

chrishill89 2y ago
You can disallow those metacharacters in the data proper. Then you have a format that can store any utf8 or whatever except the non-whitespace control codes without any escaping. That solves a problem in an opinionated way. Just like how json is opinionated (utf8 only). You can convert to another format if you need something crazier than rows and columns consisting of normal text.
- chubot 2y ago
  That already exists -- TSV. You disallow the tab metacharacter.
  - thayne 2y ago
    And the newline character. Both of which commonly occur in normal text.
  - chrishill89 2y ago
    Then I don't understand. Like the sibling comment said the really "problematic" character in TSV is the line feed. But a tab can occur as well. The format that I described does not already exist in the form of TSV. And further based on your original comment I would have thought that both TSV and this format would be discarded as not-useful.
smarx007 2y ago
The title says ASCII Delimited Text not ASCII Delimited Binary Data. For the purposes of CSV, I consider text to be anything that satisfies the regex ^\P{Cc}+$ (https://www.compart.com/en/unicode/category/Cc https://www.compart.com/en/unicode/category/Cc) and I normally strip chars in that category before saving some text (for single-line text). ^[\p{Cc}&&[^\n]]+$ is a regex that can be used to strip all control chars except for the newline.
- chrishill89 2y ago
  Thanks for that. Handy. I think I’ll have use for that myself.

dec0dedab0de 2y ago

Not if you just say those characters are invalid data. I first heard about them decades ago, but I don't think I have ever once seen them in use. The real problem is that there is no easy universal way to type them with a keyboard. So it would require software interfaces in the application, and at that point it’s basically binary.

Ukv 2y ago
> Not if you just say those characters are invalid data The author's claim is "with no restrictions on the text". It's easy if you can forbid certain characters, but then you can't store arbitrary text (e.g: filepaths, or scraped comments).
- dec0dedab0de 2y ago
  but these specific characters are not text. they exist solely to be delimiters. It would be like trying to escape a column in your spreadsheet.
  - Ukv 2y ago
    > but these specific characters are not text. they exist solely to be delimiters. Even if people used them only for their intended purpose, someone could use them as delimiters within the text you want to store (e.g: list of tags in filename) - unless I'm misunderstanding. > It would be like trying to escape a column in your spreadsheet. Other formats do allow escaping their delimiters, so that you can use that character literally or even nest a string of that format within an entry.
    - dec0dedab0de 2y ago
      Right, nesting wouldn’t be possible. But there is never any reason to use these characters literally, they are just delimiters.
      - deleted 2y ago
        [deleted]

jader201 2y ago

I feel like this (and some of the replies to this) is missing the point a bit. I don’t think the goal was to make a bullet-proof delimiter that fails at nothing. The goal was to solve the problem of not allowing things like commas, quotes, newlines, tabs, pipes, etc. in text files. I feel like using the proposed ASCII characters would eliminate these limitations, while also allowing machine creatable and readable format (emphasis on machine as opposed to human). Yes, it would still be tough for a human to type or read these delimiters, so in that case, go with traditional CSV or TSV (or MVCSV!). But if you only need to use a machine to create/read the text, this sounds like a great solution, allowing all of the normal characters you might see in text.

Ukv 2y ago
If you need a machine-readable format, why not go with escaping like most other formats, or length-before-text, to include all characters - instead of a format that fails on some (albeit rare) characters?
- jader201 2y ago
  Both of those are fine, but they add additional complexities (even if small) where there is very little, if any, complexity added with using these two characters as delimiters.
  - Ukv 2y ago
    For machine-readable formats, I'd argue length-before-text is simpler to parse than splitting by separators - even before having to add extra application logic to handle this method being fallible (e.g: if you want to use it to store filenames, you now need a check and pop-up about some OS-valid filenames not being supported).
    - chrishill89 2y ago
      > For machine-readable formats, I'd argue length-before-text is simpler to parse than splitting by separators Yes. > even before having to add extra application logic to handle this method being fallible (e.g: if you want to use it to store filenames, you now need a check and pop-up about some OS-valid filenames not being supported). I don't see why you'd need that. CSV does not have anything like that. That's a higher-level concern. This ascii-delimited format (like CSV) is supposed to be a stupid row/column format. And also simpler to implement than CSV.