6 ms·

Show HN: USV = Unicode Separated Values

32 points by jph 4y ago

robbiejs 4y ago
Cool! Great way to not have to deal with all those csv escape rules :-)
pestatije 4y ago
I don't get the no-escape rule...it should be possible to escape
- giraffe_lady 4y ago
  Why? Without escaping it's very simple to implement a parser or verifier for it. With escaping it has no real advantages over csv. The characters are pretty much just for this, when would you ever need to put them in the data? If you're working on tooling for unicode itself? Just use... any other format. Literally any.
- selfhoster11 4y ago
  In what world would you ever use a tabular format that contains Unicode escapes as a natural part of the content?
- jph 4y ago
  No-escape makes serialization and deserialization simpler, faster, and easier to test. For example, import/export is a breeze on the command line with typical Unix tools such as awk, sed, grep, mlr. No-escape makes international standardization clearer for Backus–Naur form (BNF), the same way other standards use no-escape BNF. For example, no-escape is the same choice as with the existing international standard for IANA tab separated values.
- 0x073 4y ago
  That's the problem of all CSV formats, wrong or custom implementations. Someone will generate wrong usv e.g. escape or encapsulate values. And if you get that wrong format no one cares if they gave you a wrong format. => Your parser grows and grows...
derbOac 4y ago
This needs to be in base R, Julia, python, and all spreadsheet programs yesterday.
0x073 4y ago
Group separator and file separator examples would be great.
- jph 4y ago
  Good feedback thank you. I'm adding these now for you. USV with 2 units by 2 records by 2 groups by 2 files: a␟b␞c␟d␝e␟f␞g␟h␜i␟j␞k␟l␝m␟n␞o␟p USV with typical shell commands and pretty output: $ echo "a␟b␞c␟d␝e␟f␞g␟h␜i␟j␞k␟l␝m␟n␞o␟p" | sed 's/␟/,/g; s/␞/\n/g; s/␝/\n---\n/g; s/␜/\n===\n/g;' a,b c,d --- e,f g,h === i,j k,l --- m,n o,p
lazypenguin 4y ago
Sometimes things are so obvious you think, “why didn’t we do this before?”. This makes a lot of sense I would definitely use it. Only barrier to adoption IMO is lack of implementation. A few libs for a few popular languages and the friction for using it is gone.
- jph 4y ago
  Great idea. I'll pledge $50 as a token of thanks and encouragement for any open source developer who can add USV to a popular language or tool such as R, Julia, OpenOffice, LibreOffice, TextQL, Q, JavaScript, Python, etc.
  - adgjlsfhk1 4y ago
    Julia's CSV package is pretty close to supporting it already. (it lets you pick the delimiter already.) It should be pretty easy to change the newline character.
  - cylinder714 4y ago
    Good news: the excellent VisiData (https://www.visidata.org/ https://www.visidata.org/) supports USV. Bad news: it looks like they got ␟ and ␞ mixed up, there are trailing ␞␟ characters after each record (should just be an ␞ between each record), and no characters at all after the last record: $ cat test.csv one,two,three, four,five,six seven,eight,nine $ vd test.csv -b -o test.usv opening test.csv as csv saving 1 sheets to test.usv as usv Think about what you're doing. test.usv save finished $ cat test.usv one␞two␞three␞␟four␞five␞six␞␟seven␞eight␞nine␞␟$ $
    - anjakefala 4y ago
      Would you be open to filing an issue with these details? https://github.com/saulpw/visidata/issues/new?assignees=&labels=bug&template=bug_report.md&title= https://github.com/saulpw/visidata/issues/new?assignees=&lab... We do not have anyone who is deeply familiar with USV spec details on the team, and could use your help to understand what needs to be fixed.
tester89 4y ago
Note, this (wrongly) uses Control Picture characters rather than the control characters themselves. There can be perfectly valid reasons to want to include the Control Picture characters in the data, and the inability to makes this unacceptable IMO. (Just imagine using USV to export a set of blog articles, except you can’t because you talk about USV in the blog articles using the picture characters)
- jph 4y ago
  Yes you're right. For special character escaping like your blog post idea, have you seen Unicode Separated Values Extensions (USVX)? It's USV plus extras for backslash escape, whitespace trim, etc. These use a smarter parser.
  - kazinator 4y ago
    The language being embedded should provide these features, so USV is kept simple. If the data is HTML, then you can encode the special characters using ampersand notation. And that can be kept out of the USV spec entirely as an external convention. So that is to say, use the native escaping of whatever data format you are storing. A similar principle is used in embedding JS or JSON in HTML. A sequence like "</script" could occur in a JS or JSON string literal. For that reason there is a way to escape the backslash \/. Or else the slash can be encoded numerically. This is all provided by the embedded language; the embedding language (HTML) knows nothing about it. The user, or software, doing the embedding just has to know about this and put in the escapes if they occur in the to-be-embedded data. USV is nice the way it is; don't make a complicated skullduggery out of it with escape mechanisms. The whole point of choosing those characters is to avoid it. If you're gonna have escapes, then might as well stick with commmas for field separation, newlines for records.
    - jph 4y ago
      I hear you. The feedback from this post and other testers is approximately evenly split. Some people like the simplicity, some people want/need extras especially whitespace trim, backspace escape, and final newline. What's your opinion of USV being simple (as you describe above) and USVX being a different standard that is USV + extras?
d--b 4y ago
See, you can already see the problem with this: 1. Editors don’t return carriage on RS 2. There is no US or RS key on my keyboard I am keeping my tsv. Tab is _never_ a data character. And if return carriage is a data character, you probably shouldn’t use a dsv file in the first place.
- justsomehnguy 4y ago
  > Tab is _never_ a data character. Until it is. Just like a semicolon for CSV. > And if return carriage is a data character, you probably shouldn’t use a dsv file in the first place. Agree. EDIT: > I am keeping my tsv As a solely import-export data format (ie no manual editing) it has some virtue. But at this point you can just use XML, CLIXML or even SQLite *shurg_emoji*
deleted 4y ago
[deleted]

32 points by jph 4y ago

robbiejs 4y ago

Cool! Great way to not have to deal with all those csv escape rules :-)

pestatije 4y ago

I don't get the no-escape rule...it should be possible to escape

giraffe_lady 4y ago
Why? Without escaping it's very simple to implement a parser or verifier for it. With escaping it has no real advantages over csv. The characters are pretty much just for this, when would you ever need to put them in the data? If you're working on tooling for unicode itself? Just use... any other format. Literally any.
selfhoster11 4y ago
In what world would you ever use a tabular format that contains Unicode escapes as a natural part of the content?
jph 4y ago
No-escape makes serialization and deserialization simpler, faster, and easier to test. For example, import/export is a breeze on the command line with typical Unix tools such as awk, sed, grep, mlr. No-escape makes international standardization clearer for Backus–Naur form (BNF), the same way other standards use no-escape BNF. For example, no-escape is the same choice as with the existing international standard for IANA tab separated values.
0x073 4y ago
That's the problem of all CSV formats, wrong or custom implementations. Someone will generate wrong usv e.g. escape or encapsulate values. And if you get that wrong format no one cares if they gave you a wrong format. => Your parser grows and grows...

derbOac 4y ago

This needs to be in base R, Julia, python, and all spreadsheet programs yesterday.

0x073 4y ago

Group separator and file separator examples would be great.

jph 4y ago
Good feedback thank you. I'm adding these now for you. USV with 2 units by 2 records by 2 groups by 2 files: a␟b␞c␟d␝e␟f␞g␟h␜i␟j␞k␟l␝m␟n␞o␟p USV with typical shell commands and pretty output: $ echo "a␟b␞c␟d␝e␟f␞g␟h␜i␟j␞k␟l␝m␟n␞o␟p" | sed 's/␟/,/g; s/␞/\n/g; s/␝/\n---\n/g; s/␜/\n===\n/g;' a,b c,d --- e,f g,h === i,j k,l --- m,n o,p

lazypenguin 4y ago

Sometimes things are so obvious you think, “why didn’t we do this before?”. This makes a lot of sense I would definitely use it. Only barrier to adoption IMO is lack of implementation. A few libs for a few popular languages and the friction for using it is gone.

jph 4y ago
Great idea. I'll pledge $50 as a token of thanks and encouragement for any open source developer who can add USV to a popular language or tool such as R, Julia, OpenOffice, LibreOffice, TextQL, Q, JavaScript, Python, etc.
- adgjlsfhk1 4y ago
  Julia's CSV package is pretty close to supporting it already. (it lets you pick the delimiter already.) It should be pretty easy to change the newline character.
- cylinder714 4y ago
  Good news: the excellent VisiData (https://www.visidata.org/ https://www.visidata.org/) supports USV. Bad news: it looks like they got ␟ and ␞ mixed up, there are trailing ␞␟ characters after each record (should just be an ␞ between each record), and no characters at all after the last record: $ cat test.csv one,two,three, four,five,six seven,eight,nine $ vd test.csv -b -o test.usv opening test.csv as csv saving 1 sheets to test.usv as usv Think about what you're doing. test.usv save finished $ cat test.usv one␞two␞three␞␟four␞five␞six␞␟seven␞eight␞nine␞␟$ $
  - anjakefala 4y ago
    Would you be open to filing an issue with these details? https://github.com/saulpw/visidata/issues/new?assignees=&labels=bug&template=bug_report.md&title= https://github.com/saulpw/visidata/issues/new?assignees=&lab... We do not have anyone who is deeply familiar with USV spec details on the team, and could use your help to understand what needs to be fixed.

tester89 4y ago

Note, this (wrongly) uses Control Picture characters rather than the control characters themselves. There can be perfectly valid reasons to want to include the Control Picture characters in the data, and the inability to makes this unacceptable IMO. (Just imagine using USV to export a set of blog articles, except you can’t because you talk about USV in the blog articles using the picture characters)

jph 4y ago
Yes you're right. For special character escaping like your blog post idea, have you seen Unicode Separated Values Extensions (USVX)? It's USV plus extras for backslash escape, whitespace trim, etc. These use a smarter parser.
- kazinator 4y ago
  The language being embedded should provide these features, so USV is kept simple. If the data is HTML, then you can encode the special characters using ampersand notation. And that can be kept out of the USV spec entirely as an external convention. So that is to say, use the native escaping of whatever data format you are storing. A similar principle is used in embedding JS or JSON in HTML. A sequence like "</script" could occur in a JS or JSON string literal. For that reason there is a way to escape the backslash \/. Or else the slash can be encoded numerically. This is all provided by the embedded language; the embedding language (HTML) knows nothing about it. The user, or software, doing the embedding just has to know about this and put in the escapes if they occur in the to-be-embedded data. USV is nice the way it is; don't make a complicated skullduggery out of it with escape mechanisms. The whole point of choosing those characters is to avoid it. If you're gonna have escapes, then might as well stick with commmas for field separation, newlines for records.
  - jph 4y ago
    I hear you. The feedback from this post and other testers is approximately evenly split. Some people like the simplicity, some people want/need extras especially whitespace trim, backspace escape, and final newline. What's your opinion of USV being simple (as you describe above) and USVX being a different standard that is USV + extras?

d--b 4y ago

See, you can already see the problem with this: 1. Editors don’t return carriage on RS 2. There is no US or RS key on my keyboard I am keeping my tsv. Tab is _never_ a data character. And if return carriage is a data character, you probably shouldn’t use a dsv file in the first place.

justsomehnguy 4y ago
> Tab is _never_ a data character. Until it is. Just like a semicolon for CSV. > And if return carriage is a data character, you probably shouldn’t use a dsv file in the first place. Agree. EDIT: > I am keeping my tsv As a solely import-export data format (ie no manual editing) it has some virtue. But at this point you can just use XML, CLIXML or even SQLite *shurg_emoji*

deleted 4y ago

[deleted]