4 ms·

Note, this (wrongly) uses Control Picture characters rather than the control characters themselves. There can be perfectly valid reasons to want to include the

Note, this (wrongly) uses Control Picture characters rather than the control characters themselves. There can be perfectly valid reasons to want to include the Control Picture characters in the data, and the inability to makes this unacceptable IMO. (Just imagine using USV to export a set of blog articles, except you can’t because you talk about USV in the blog articles using the picture characters)

jph 4y ago
Yes you're right. For special character escaping like your blog post idea, have you seen Unicode Separated Values Extensions (USVX)? It's USV plus extras for backslash escape, whitespace trim, etc. These use a smarter parser.
- kazinator 4y ago
  The language being embedded should provide these features, so USV is kept simple. If the data is HTML, then you can encode the special characters using ampersand notation. And that can be kept out of the USV spec entirely as an external convention. So that is to say, use the native escaping of whatever data format you are storing. A similar principle is used in embedding JS or JSON in HTML. A sequence like "</script" could occur in a JS or JSON string literal. For that reason there is a way to escape the backslash \/. Or else the slash can be encoded numerically. This is all provided by the embedded language; the embedding language (HTML) knows nothing about it. The user, or software, doing the embedding just has to know about this and put in the escapes if they occur in the to-be-embedded data. USV is nice the way it is; don't make a complicated skullduggery out of it with escape mechanisms. The whole point of choosing those characters is to avoid it. If you're gonna have escapes, then might as well stick with commmas for field separation, newlines for records.
  - jph 4y ago
    I hear you. The feedback from this post and other testers is approximately evenly split. Some people like the simplicity, some people want/need extras especially whitespace trim, backspace escape, and final newline. What's your opinion of USV being simple (as you describe above) and USVX being a different standard that is USV + extras?
    - kazinator 4y ago
      USV is not simple if USVX exists. There is a potential for confusion. E.g. someone carelessly documents that some software accepts USV, when they mean USVX, or vice versa. You will forever have to be double-checking. Imagine if we had JSON and JSONX to contend with. White space trim means that if you don't expect trim, your significant white space is eaten, causing a bug. Most of the time, USV will be machine-generated. If the machine generates a field with leading or trailing space, then that's what the data is.
kazinator 4y ago
> There can be perfectly valid reasons to want to include the Control Picture characters in the data. What you want is null characters. Multiple null characters can be used for multiple separating roles/levels. Microsoft implemented this idea in the Registry; there are string tables where each string is null terminated, and then an extra null indicates the end of the table. e.g. foo<NUL>bar<NUL><NUL> This is an empty table containing no strings: <NUL> This is a table with one empty string: <NUL><NUL> This looks like it should extend to more dimensions with more NULs: record, table, group, file, ... The long-winded point here is that your fictitious blogger trying to write about the scheme in all likelihood will not have any luck embedding literal NULs into their article, so we are are safe.
deleted 4y ago
[deleted]

jph 4y ago

Yes you're right. For special character escaping like your blog post idea, have you seen Unicode Separated Values Extensions (USVX)? It's USV plus extras for backslash escape, whitespace trim, etc. These use a smarter parser.

kazinator 4y ago
The language being embedded should provide these features, so USV is kept simple. If the data is HTML, then you can encode the special characters using ampersand notation. And that can be kept out of the USV spec entirely as an external convention. So that is to say, use the native escaping of whatever data format you are storing. A similar principle is used in embedding JS or JSON in HTML. A sequence like "</script" could occur in a JS or JSON string literal. For that reason there is a way to escape the backslash \/. Or else the slash can be encoded numerically. This is all provided by the embedded language; the embedding language (HTML) knows nothing about it. The user, or software, doing the embedding just has to know about this and put in the escapes if they occur in the to-be-embedded data. USV is nice the way it is; don't make a complicated skullduggery out of it with escape mechanisms. The whole point of choosing those characters is to avoid it. If you're gonna have escapes, then might as well stick with commmas for field separation, newlines for records.
- jph 4y ago
  I hear you. The feedback from this post and other testers is approximately evenly split. Some people like the simplicity, some people want/need extras especially whitespace trim, backspace escape, and final newline. What's your opinion of USV being simple (as you describe above) and USVX being a different standard that is USV + extras?
  - kazinator 4y ago
    USV is not simple if USVX exists. There is a potential for confusion. E.g. someone carelessly documents that some software accepts USV, when they mean USVX, or vice versa. You will forever have to be double-checking. Imagine if we had JSON and JSONX to contend with. White space trim means that if you don't expect trim, your significant white space is eaten, causing a bug. Most of the time, USV will be machine-generated. If the machine generates a field with leading or trailing space, then that's what the data is.

kazinator 4y ago

> There can be perfectly valid reasons to want to include the Control Picture characters in the data. What you want is null characters. Multiple null characters can be used for multiple separating roles/levels. Microsoft implemented this idea in the Registry; there are string tables where each string is null terminated, and then an extra null indicates the end of the table. e.g. foo<NUL>bar<NUL><NUL> This is an empty table containing no strings: <NUL> This is a table with one empty string: <NUL><NUL> This looks like it should extend to more dimensions with more NULs: record, table, group, file, ... The long-winded point here is that your fictitious blogger trying to write about the scheme in all likelihood will not have any luck embedding literal NULs into their article, so we are are safe.

deleted 4y ago

[deleted]