6 ms·

I don't really see the point... Using special characters makes it harder to manipulate by hand, and you still don't solve conflicts (where content is able to co

I don't really see the point... Using special characters makes it harder to manipulate by hand, and you still don't solve conflicts (where content is able to corrupt the container structure). Ultimately, the only safe way is to use Pascal string-style length prefixes. Escaping is too easy to ignore, until it comes back to bite you in the ass. Balanced delimiters break as soon as the content contains invalid data (or is delimited according to a different convention than the container).

jrochkind1 5y ago
It's much less likely that content is going to include ASCII 28 to 31 than that it will include a comma or a tab. But you're right, certainly not impossible. I think the human-editability you mention may be a major reason why comma-separated has been such a practical winner? Since ASCII 28 to 31 are invisible or indistinguishable in most editors, I suspect a format relying on them would be subject to routine corruption. (I have some experience working with a "legacy" "binary" format which DOES use those ASCII separator values... https://en.wikipedia.org/wiki/MARC_standards https://en.wikipedia.org/wiki/MARC_standards)
- Nullabillity 5y ago
  > It's much less likely that content is going to include ASCII 28 to 31 than that it will include a comma or a tab. But you're right, certainly not impossible. Yeah, that's the same problem as UTF-8's almost-compatibility with ASCII. Things work, until they break spectacularly. Ultimately, you either have to solve the issue properly, or keep it visible enough that people are forced to handle it.
- slver 5y ago
  In machine encoding formats, "much less likely" is the same as "very likely". You're either OK for something to happen or you categorically make it impossible to happen. Invisible characters also mean you can no longer call a data format "human readable". The special ASCII characters you're referring to are mostly a historical oddity today.
  - jrochkind1 5y ago
    How much have you worked with actually existing CSV data? If it was as easy as "making it categorically impossible to happen", working with CSV wouldn't be such a mess! You can say whatever you want in whatever standard you want, data bugs still happen. Any CSV format description anywhere, will of course you need to use various internal escaping mechanisms for a literal comma, or forbid them entirely. Does that "categorically make it impossible to happen"? Obviously not, words in a standard can't make something categorically impossible. How likely a protocol/standard violation is to happen of course matters to actual practical pain. Yes, of course the ascii separator values are a historical path not taken. This thread, not started by me, was musing about why, and what-if, by people who, believe it or not, have plenty of experience dealing with data, and bad data.
    - slver 5y ago
      You’ve misinterpreted my comment. And arguing about unrelated things.
Pet_Ant 5y ago
Manipulating by hand is easily solved with alt sequences and adding support to distinctly display those characters to text editors. It's easier than figuring out the corner cases of YAML.
simongr3dal 5y ago
If you're editing by hand then length prefixes are also broken easily and counting the length of strings is cumbersome unless you're using a "technical" text editor like VS Code in which case its just as easy to write a delimiter like \x1F (unit separator) as it is to highlight the text and look at character count. Unless "editing by hand" means using something like excel?
- Nullabillity 5y ago
  There's a difference between "take on this inconvenience and gain nothing" and "take on this inconvenience in exchange for solving this problem".
chriswarbo 5y ago
> Ultimately, the only safe way is to use Pascal string-style length prefixes. Note that length-prefixes aren't necessarily safer when nested. For example, `line(10 bytes, [cell(1000 bytes, "xxxxxxx...`. This can lead to vulnerabilities like buffer overflows (naively allocating 10 bytes for the line, then trying to put 1000 bytes into it)
- Nullabillity 5y ago
  Sure, without bounds-checking all hope is lost, but then you're in a hopeless place anyway.
slver 5y ago
> Escaping is too easy to ignore This is why we have editors that do syntax checking as we type. Editing CSV by hand is just not important enough for such editors to be commonplace. CSV files are exported and imported. And since we have JSON, whose spec is stable, known and at this point consistently implemented, CSV should really just be sun set rather than "fixed". Also, the simplest escaping format is wrap in quotes and represent quotes by typing quotes twice. That's easy to remember and relatively hard to screw up even when editing by hand.
throw0101a 5y ago
> Using special characters makes it harder to manipulate by hand I generally use CSV as an import/export format, and would like for software to at least support the option (even if CSV/TSV remains the default).

jrochkind1 5y ago

It's much less likely that content is going to include ASCII 28 to 31 than that it will include a comma or a tab. But you're right, certainly not impossible. I think the human-editability you mention may be a major reason why comma-separated has been such a practical winner? Since ASCII 28 to 31 are invisible or indistinguishable in most editors, I suspect a format relying on them would be subject to routine corruption. (I have some experience working with a "legacy" "binary" format which DOES use those ASCII separator values... https://en.wikipedia.org/wiki/MARC_standards https://en.wikipedia.org/wiki/MARC_standards)

Nullabillity 5y ago
> It's much less likely that content is going to include ASCII 28 to 31 than that it will include a comma or a tab. But you're right, certainly not impossible. Yeah, that's the same problem as UTF-8's almost-compatibility with ASCII. Things work, until they break spectacularly. Ultimately, you either have to solve the issue properly, or keep it visible enough that people are forced to handle it.
slver 5y ago
In machine encoding formats, "much less likely" is the same as "very likely". You're either OK for something to happen or you categorically make it impossible to happen. Invisible characters also mean you can no longer call a data format "human readable". The special ASCII characters you're referring to are mostly a historical oddity today.
- jrochkind1 5y ago
  How much have you worked with actually existing CSV data? If it was as easy as "making it categorically impossible to happen", working with CSV wouldn't be such a mess! You can say whatever you want in whatever standard you want, data bugs still happen. Any CSV format description anywhere, will of course you need to use various internal escaping mechanisms for a literal comma, or forbid them entirely. Does that "categorically make it impossible to happen"? Obviously not, words in a standard can't make something categorically impossible. How likely a protocol/standard violation is to happen of course matters to actual practical pain. Yes, of course the ascii separator values are a historical path not taken. This thread, not started by me, was musing about why, and what-if, by people who, believe it or not, have plenty of experience dealing with data, and bad data.
  - slver 5y ago
    You’ve misinterpreted my comment. And arguing about unrelated things.

Pet_Ant 5y ago

Manipulating by hand is easily solved with alt sequences and adding support to distinctly display those characters to text editors. It's easier than figuring out the corner cases of YAML.

simongr3dal 5y ago

If you're editing by hand then length prefixes are also broken easily and counting the length of strings is cumbersome unless you're using a "technical" text editor like VS Code in which case its just as easy to write a delimiter like \x1F (unit separator) as it is to highlight the text and look at character count. Unless "editing by hand" means using something like excel?

Nullabillity 5y ago
There's a difference between "take on this inconvenience and gain nothing" and "take on this inconvenience in exchange for solving this problem".

chriswarbo 5y ago

> Ultimately, the only safe way is to use Pascal string-style length prefixes. Note that length-prefixes aren't necessarily safer when nested. For example, `line(10 bytes, [cell(1000 bytes, "xxxxxxx...`. This can lead to vulnerabilities like buffer overflows (naively allocating 10 bytes for the line, then trying to put 1000 bytes into it)

Nullabillity 5y ago
Sure, without bounds-checking all hope is lost, but then you're in a hopeless place anyway.

slver 5y ago

> Escaping is too easy to ignore This is why we have editors that do syntax checking as we type. Editing CSV by hand is just not important enough for such editors to be commonplace. CSV files are exported and imported. And since we have JSON, whose spec is stable, known and at this point consistently implemented, CSV should really just be sun set rather than "fixed". Also, the simplest escaping format is wrap in quotes and represent quotes by typing quotes twice. That's easy to remember and relatively hard to screw up even when editing by hand.

throw0101a 5y ago

> Using special characters makes it harder to manipulate by hand I generally use CSV as an import/export format, and would like for software to at least support the option (even if CSV/TSV remains the default).