9 ms·
For those wondering what USV is, like myself: > Unicode separated values (USV) is a data format that uses Unicode symbol characters between data parts. USV com
by jiehong 3y ago
For those wondering what USV is, like myself:
> Unicode separated values (USV) is a data format that uses Unicode symbol characters between data parts. USV competes with comma separated values (CSV), tab separated values (TSV), ASCII separated values (ASV), and similar systems. USV offers more capabilities and standards-track syntax.
> Separators:
>
> ␟ U+241F Symbol for Unit Separator (US)
>
> ␞ U+241E Symbol for Record Separator (RS)
>
> ␝ U+241D Symbol for Group Separator (GS)
>
> ␜ U+241C Symbol for File Separator (FS)
>
> Modifiers:
>
> ␛ U+241B Symbol for Escape (ESC)
>
> ␗ U+2417 Symbol for End of Transmission Block (ETB)
>
> ␖ U+2416 Symbol For Synchronous Idle (SYN)
- calvinmorrison 3y agoI always wonder why we don't use this.
- curtisblaine 3y agoNot easily readable / editable using a regular text editor.
- timmg 3y agoDo those characters map to something visually useful in (typical) unicode fonts? That would be neat :) Edit: Apparently, kinda (e.g. https://www.compart.com/en/unicode/U+241E https://www.compart.com/en/unicode/U+241E ) Not the most creative....
- bryanlarsen 3y agoAccording to their GitHub README: ```USV works with many kinds of editors. Any editor that can render the USV characters will work. We use vi, emacs, Coda, Notepad++, TextMate, Sublime, VS Code, etc.``` I loaded an example in my fairly generic Emacs and it worked out of the box. The separators were pretty small so I had to increase my font size to distinguish US from RS. And of course I have no idea how to enter those characters. I'm sure there is, but cut & paste worked.
- Solvency 3y agoWho here uses a "regular" text editor, let's be real
- ahofmann 3y agoI'm fascinated that a lot of posters in this thread are not understanding the ideas and experiences, that the inventors of this file format had or made. They invented this format because it works for machines as well as for humans. Text editors can handle the proposed UTF characters just fine. Humans can see them. The only challenge is that it is cumbersome to type the delimiters. And that the format is not used in any relevant software (like Excel). Both are reason enough, that USV will not be used anywhere. But I can see why they went this way on their file format.
- wtetzner 3y agoI don't really see what benefit it provides over CSV other than needing to escape less frequently. That hardly seems like it's worth it.
- vidarh 3y agoWe might be able to see them, but for me they're just a blur unless I zoom in significantly, so I'll need editor accommodations just as much for these characters as if they used the already existing RS/FS/US/GS characters. It feels like instead of fixing it properly, they went with an option that will still need tool improvements, will be controversial, and adds unnecessary details (e.g. the SYN they've added will be an active nuisance and I'd be willing to bet will get ignored by enough tools to become a hazard to data integrity). I quite like an initiative to make use of proper record and unit separators, but this feels poorly thought through in several respects (e.g. their quirky escape characters that adds differently depending on the class of the following character will be a 'fun' source of bugs; that splitting records on LF requires three characters almost certainly will mean a number of tools will incorrectly treat those three characters as a unit, etc. -- these assumptions are based on how slapdash a lot of CSV parsing and generation is; if you want to compete with CSV you ought to learn those lessons)
- dahart 3y agoCSV works for machines as well as humans, why do you assume or imply otherwise? Making the separator hard to type makes this ‘invention’ hard for humans to use. Using the glyphs instead of the semantic Unicode separators might also make this harder to use, even if you can understand why they did it, and to some degree it subverts the intent of the Unicode standard’s separator and glyph characters.
- deleted 3y ago[deleted]
- theamk 3y agoAll downsides, no upsides. You cannot edit it in regular editor, like csv/tsv/jsonlines. There is no schema or efficient storage, like binary formats. There is no wide library support. Not all data is representable.
- mavhc 3y ago1 Editors can be improved 2 Same as CSV etc then 3 Libraries can be improved 4 Escaping characters exists ASCII 1963 had 8 separators, 1965 reduced it to 4, and named them. See 6.3.12 of https://dl.acm.org/doi/pdf/10.1145/363831.363839 https://dl.acm.org/doi/pdf/10.1145/363831.363839
- cxr 3y agoThe task here is to explain why one should use this over CSV. By your own admissions, there is no reason to prefer this over CSV.
- mavhc 3y agoThere's no standard for CSV files, thus no one can parse them properly The only time you need to escape a character is if it's a control character that's rarely used, unlike the " and , characters
- cxr 3y ago> There's no standard for CSV files Literally untrue. (And were it true, it still wouldn't be a reason why one should use this over CSV—not sure what's so hard to grasp about the conversational/contextual premise here.)
- ixwt 3y ago> You cannot edit it in regular editor, like csv/tsv/jsonlines. If only there were shortcuts on modern operating systems to allow us to do things that aren't readily on our keyboards. Like upper case characters. Or copy and paste. Or close windows. Our lives would be so much better. If ASV had caught on, there could be common shared shortcuts to type them, and fonts would regularly display them (just like the unicode characters proposed). But CSV was simple enough and readily type-able. > There is no schema or efficient storage, like binary formats. I'm not quite certain where you're trying to go with this. Binary formats aren't really meant to be human readable in an average text editor. It doesn't know to differentiate 1, 2, 4, or 8 bytes as an integer or a float. Even current hex editors to make it easier to navigate these formats don't really know unless you are able to tell it somehow. > There is no wide library support. It's a critical mass problem. Not enough people are using them, so no libraries are being made. > Not all data is representable. I'm not quite certain what data couldn't be represented. f you can represent your data in CSV, you can represent it in ASV. It's all plain text that gets interpreted based on what you need. They're nearly a 1:1 replacement. Commas get replaced by unit separators, new lines get replaced by group separators. Then you have record and file separators to do with for further levels of abstraction if you need.
- edent 3y agoI've sketched out a replacement for JSON which would use these characters - https://shkspr.mobi/blog/2017/03/kyli-because-it-is-superior-to-json/ https://shkspr.mobi/blog/2017/03/kyli-because-it-is-superior...
- deleted 3y ago[deleted]
- michaelt 3y agoCSV is the javascript of the tabular data world. Everyone thinks they can do better, but nothing's more widely supported (for a sufficiently generous definition of 'supported')
- hermitcrab 3y agoUnfortunately CSVs vary a lot in the wild. Some people use commas as a delimiter, some use semi-colons. Escaping rules vary. And the text encoding is not specified. I randomly generated some CSVs and fed them into Excel and Numbers and they were differently interpreted.
- mst 3y agoThis is why I tend to use the Pg COPY version of TSV - works beautifully with 'cut' and friends, loads trivially into most databases, and the 'vary a lot' problem is (ish) avoided by specifying COPY escaping which is clearly documented and something people often already recognise. Generally my only interaction with CSV itself is to fling it through https://p3rl.org/Text::CSV https://p3rl.org/Text::CSV since that seems to be able to get a pretty decent parse of every sort of CSV I've yet had to deal with in the wild.
- Tijdreiziger 3y agoCountries that use . as the thousands separator (e.g. 1.000) use , as the CSV separator. Countries that use , as the thousands separator (e.g. 1,000) use ; as the CSV separator. Why? Because that’s how Excel does it.
- Tijdreiziger 3y agoErrata: as the decimal separator, not as the thousands separator.
- dimask 3y agoFunny thing, excel, which is the most common spreadsheet editor, does not practically support CSV files if you happen to live in countries where the default official convention is using commas for decimal points in numbers. Unless you go around and manually set stuff in how it imports it or you change your default settings. It has reached meme levels at my work. Tab separated files are much better imo in not getting confused with the delimiter for a sufficiently sane tsv file.
- chasil 3y agoIn a POSIX shell, I actually prefer to use the bell character for IFS. while IFS="$(printf \\a)" read -r field1 field2... do ... done This works just as well as anything outside the range of printing characters.
- quesera 3y ago> I actually prefer to use the bell character for IFS Heaven help you if you cat the source file in a shell, though!
- pavon 3y agoBack when rolling your own application level protocol on top of TCP was common (as opposed to using http, zeromq, etc) I frequently used file/record/group/unit separators for delimiters, and considered them an underrated gem, especially for plain-text data where they were prohibited to occur in the message body so you didn't have to escape them (still good to scan and reject messages containing them). As a modern example they (and most other ASCII control characters) are disallowed in json strings.
- gkbrk 3y agoYou can put control characters in JSON strings, you just need to escape them.
- pavon 3y agoThe way I read the json standard, the only way to include control characters is to encode them as hex. For example BEL can be encoded as "\u0007", but escaping it by using a backslash followed by a literal BEL character is not allowed. So literal control characters should never be in json text.
- lsaferite 3y agoThis makes me wonder if the Escape control character (\u001B) would work in a JSON string. Time to go test things out. :)
- paulddraper 3y agoMLLP (Minimal Lower Layer Protocol) -- used extensively to transmit HL7 in health systems -- uses file separators to delimit messages. OB vertical tab <content> 1C file separator 0D carriage return I wrote one of the most popular translators for MLLP, which converts it to HTTP [1]. --- P.S. Ironically, HL7 messages have something literally called a "field separator" but don't use the field separator character, usually they use vertical bar. [1] https://github.com/rivethealth/mllp-http https://github.com/rivethealth/mllp-http
- queuebert 3y agoCSV is honestly not that problematic. Figuring out if an field contains and comma and then properly quoting it is trivial. And fields without commas don't need quoting. Sometimes your application even guarantees no commas, especially if CSV is into it from the beginning.
- deleted 3y ago[deleted]
- hermitcrab 3y agoI'm guessing you haven't worked in custom support where people send you their "CSV" files. Even the field delimiter varies (many Europeans use semi-colons).
- queuebert 3y agoNo, I have. I don't consider abuse of the format a problem with the format. Though I can see how having to delimit with special characters will help the type of person who writes print(','.join(stuff)).
- hermitcrab 3y ago>I don't consider abuse of the format a problem with the format. That's a fair point. But you could argue that when the abuse is so widespread, it becomes a defacto part of the format (even if it isn't in the RFC).
- teddyh 3y agoUsing Unicode graphic characters as metasyntactic escape characters is fundamentally wrong. Those Unicode characters are for displaying the symbols for Unit Separator, Record Separator, etc. and not for actually being separators! ASCII already has those! Included in Unicode!
- layer8 3y agoTo be fair, I don’t quite get those graphic characters, because the original characters should already be displayed that way, shouldn’t they? Now when I see such a character, I have no idea if it’s the real character or just it’s graphic-character counterpart.
- teddyh 3y agoI like the LINE FEED character to be interpreted as a line feed, not displayed as “␊”.
- lsaferite 3y agoI mean, my assumption (yeah, I know) is that the 'display' variant is more for documentation talking _about_ the control character and not meant to _be_ the control character. Abusing the 'display' variant this way seems... misguided.
- eadmund 3y agoWait a second … he’s not proposing using unit/record/group/file separators as separators, he’s proposing using the symbols for those separators as separators! Why not just use the separators themselves‽ Yes, rather than using U+1F (the ASCII and Unicode unit separator), he proposes using U+241F (the Unicode symbol for the unit separator). I almost feel like this must be an early April Fool’s joke? Also, he writes ‘comprised of’ rather than ‘composed of’ or ‘comprises’ throughout his RFC.
- ahofmann 3y agoThey explain in the FAQ that this approach works with most text editors and copy-paste situations.
- philipwhiuk 3y agoIt doesn't "work" because I can't read the darn things at a sane zoom level.
- bryanlarsen 3y agoUsing a visible character rather than an invisible one makes editing in an editor a lot easier.
- tgv 3y agoIt won't wrap at the record separator, so you'll get a very long line.
- bryanlarsen 3y agoThe example seems to use `␞\n` as a separator rather than just `␞`. I assume their proposed standard is more definitive.
- vidarh 3y agoTheir ABNF uses RS, defined as U+241E, not U+241E + '\n' as the record separator. They seem to add an "USV escape" in front of the linefeeds. My bet is that this will lead to implementations that wrongly treats "␞␛\n" (RS ESC \m) as the real record separator, the same way lots of "CSV" implementations just split on comma and LF. Seems to me if you're going to add support for something like that you should just bite the bullet and declare an LF immediately following an RS as part of the record separator, or you're falling in the same trap as CSV of being "close enough" to naively splittable that people will do it because it works often enough.
- dr_kiszonka 3y agoI am very confused. The author provides "assistive accessibility software to people with vision/cognition/mobility impairments", but these character symbols are indistinguishable for folks with impaired vision.
- jph 3y agoGood connection to BoldContacts.org :-) Screen readers handle these well. If someone wants to create a font with big bold separators, that would be awesome.
- dr_kiszonka 3y agoI checked Windows Narrator, and two other utilities and they didn't utter anything. I think requiring users to install a custom font would further hold back the adoption of your format. I hope you succeed; just my 2c.