16 ms·
Show HN: Comma Separated Values (CSV) to Unicode Separated Values (USV)
- difer7 3y agoDoes USV supports nested fields? While reading the USV GitHub's README I did not clearly understand the purpose of the "group separator"
- philsnow 3y agoIn the same way that CSV supports fields that contain nested CSV documents: cumbersomely / painfully, with lots of escaping of the delimiter characters.
- jiehong 3y agoFor those wondering what USV is, like myself: > Unicode separated values (USV) is a data format that uses Unicode symbol characters between data parts. USV competes with comma separated values (CSV), tab separated values (TSV), ASCII separated values (ASV), and similar systems. USV offers more capabilities and standards-track syntax. > Separators: > > ␟ U+241F Symbol for Unit Separator (US) > > ␞ U+241E Symbol for Record Separator (RS) > > ␝ U+241D Symbol for Group Separator (GS) > > ␜ U+241C Symbol for File Separator (FS) > > Modifiers: > > ␛ U+241B Symbol for Escape (ESC) > > ␗ U+2417 Symbol for End of Transmission Block (ETB) > > ␖ U+2416 Symbol For Synchronous Idle (SYN)
- calvinmorrison 3y agoI always wonder why we don't use this.
- curtisblaine 3y agoNot easily readable / editable using a regular text editor.
- timmg 3y agoDo those characters map to something visually useful in (typical) unicode fonts? That would be neat :) Edit: Apparently, kinda (e.g. https://www.compart.com/en/unicode/U+241E https://www.compart.com/en/unicode/U+241E ) Not the most creative....
- bryanlarsen 3y agoAccording to their GitHub README: ```USV works with many kinds of editors. Any editor that can render the USV characters will work. We use vi, emacs, Coda, Notepad++, TextMate, Sublime, VS Code, etc.``` I loaded an example in my fairly generic Emacs and it worked out of the box. The separators were pretty small so I had to increase my font size to distinguish US from RS. And of course I have no idea how to enter those characters. I'm sure there is, but cut & paste worked.
- Solvency 3y agoWho here uses a "regular" text editor, let's be real
- ahofmann 3y agoI'm fascinated that a lot of posters in this thread are not understanding the ideas and experiences, that the inventors of this file format had or made. They invented this format because it works for machines as well as for humans. Text editors can handle the proposed UTF characters just fine. Humans can see them. The only challenge is that it is cumbersome to type the delimiters. And that the format is not used in any relevant software (like Excel). Both are reason enough, that USV will not be used anywhere. But I can see why they went this way on their file format.
- wtetzner 3y agoI don't really see what benefit it provides over CSV other than needing to escape less frequently. That hardly seems like it's worth it.
- vidarh 3y agoWe might be able to see them, but for me they're just a blur unless I zoom in significantly, so I'll need editor accommodations just as much for these characters as if they used the already existing RS/FS/US/GS characters. It feels like instead of fixing it properly, they went with an option that will still need tool improvements, will be controversial, and adds unnecessary details (e.g. the SYN they've added will be an active nuisance and I'd be willing to bet will get ignored by enough tools to become a hazard to data integrity). I quite like an initiative to make use of proper record and unit separators, but this feels poorly thought through in several respects (e.g. their quirky escape characters that adds differently depending on the class of the following character will be a 'fun' source of bugs; that splitting records on LF requires three characters almost certainly will mean a number of tools will incorrectly treat those three characters as a unit, etc. -- these assumptions are based on how slapdash a lot of CSV parsing and generation is; if you want to compete with CSV you ought to learn those lessons)
- theamk 3y agoAll downsides, no upsides. You cannot edit it in regular editor, like csv/tsv/jsonlines. There is no schema or efficient storage, like binary formats. There is no wide library support. Not all data is representable.
- mavhc 3y ago1 Editors can be improved 2 Same as CSV etc then 3 Libraries can be improved 4 Escaping characters exists ASCII 1963 had 8 separators, 1965 reduced it to 4, and named them. See 6.3.12 of https://dl.acm.org/doi/pdf/10.1145/363831.363839 https://dl.acm.org/doi/pdf/10.1145/363831.363839
- cxr 3y agoThe task here is to explain why one should use this over CSV. By your own admissions, there is no reason to prefer this over CSV.
- mavhc 3y agoThere's no standard for CSV files, thus no one can parse them properly The only time you need to escape a character is if it's a control character that's rarely used, unlike the " and , characters
- cxr 3y ago> There's no standard for CSV files Literally untrue. (And were it true, it still wouldn't be a reason why one should use this over CSV—not sure what's so hard to grasp about the conversational/contextual premise here.)
- ixwt 3y ago> You cannot edit it in regular editor, like csv/tsv/jsonlines. If only there were shortcuts on modern operating systems to allow us to do things that aren't readily on our keyboards. Like upper case characters. Or copy and paste. Or close windows. Our lives would be so much better. If ASV had caught on, there could be common shared shortcuts to type them, and fonts would regularly display them (just like the unicode characters proposed). But CSV was simple enough and readily type-able. > There is no schema or efficient storage, like binary formats. I'm not quite certain where you're trying to go with this. Binary formats aren't really meant to be human readable in an average text editor. It doesn't know to differentiate 1, 2, 4, or 8 bytes as an integer or a float. Even current hex editors to make it easier to navigate these formats don't really know unless you are able to tell it somehow. > There is no wide library support. It's a critical mass problem. Not enough people are using them, so no libraries are being made. > Not all data is representable. I'm not quite certain what data couldn't be represented. f you can represent your data in CSV, you can represent it in ASV. It's all plain text that gets interpreted based on what you need. They're nearly a 1:1 replacement. Commas get replaced by unit separators, new lines get replaced by group separators. Then you have record and file separators to do with for further levels of abstraction if you need.
- edent 3y agoI've sketched out a replacement for JSON which would use these characters - https://shkspr.mobi/blog/2017/03/kyli-because-it-is-superior-to-json/ https://shkspr.mobi/blog/2017/03/kyli-because-it-is-superior...
- deleted 3y ago[deleted]
- michaelt 3y agoCSV is the javascript of the tabular data world. Everyone thinks they can do better, but nothing's more widely supported (for a sufficiently generous definition of 'supported')
- hermitcrab 3y agoUnfortunately CSVs vary a lot in the wild. Some people use commas as a delimiter, some use semi-colons. Escaping rules vary. And the text encoding is not specified. I randomly generated some CSVs and fed them into Excel and Numbers and they were differently interpreted.
- mst 3y agoThis is why I tend to use the Pg COPY version of TSV - works beautifully with 'cut' and friends, loads trivially into most databases, and the 'vary a lot' problem is (ish) avoided by specifying COPY escaping which is clearly documented and something people often already recognise. Generally my only interaction with CSV itself is to fling it through https://p3rl.org/Text::CSV https://p3rl.org/Text::CSV since that seems to be able to get a pretty decent parse of every sort of CSV I've yet had to deal with in the wild.
- Tijdreiziger 3y agoCountries that use . as the thousands separator (e.g. 1.000) use , as the CSV separator. Countries that use , as the thousands separator (e.g. 1,000) use ; as the CSV separator. Why? Because that’s how Excel does it.
- Tijdreiziger 3y agoErrata: as the decimal separator, not as the thousands separator.
- dimask 3y agoFunny thing, excel, which is the most common spreadsheet editor, does not practically support CSV files if you happen to live in countries where the default official convention is using commas for decimal points in numbers. Unless you go around and manually set stuff in how it imports it or you change your default settings. It has reached meme levels at my work. Tab separated files are much better imo in not getting confused with the delimiter for a sufficiently sane tsv file.
- chasil 3y agoIn a POSIX shell, I actually prefer to use the bell character for IFS. while IFS="$(printf \\a)" read -r field1 field2... do ... done This works just as well as anything outside the range of printing characters.
- quesera 3y ago> I actually prefer to use the bell character for IFS Heaven help you if you cat the source file in a shell, though!
- pavon 3y agoBack when rolling your own application level protocol on top of TCP was common (as opposed to using http, zeromq, etc) I frequently used file/record/group/unit separators for delimiters, and considered them an underrated gem, especially for plain-text data where they were prohibited to occur in the message body so you didn't have to escape them (still good to scan and reject messages containing them). As a modern example they (and most other ASCII control characters) are disallowed in json strings.
- gkbrk 3y agoYou can put control characters in JSON strings, you just need to escape them.
- pavon 3y agoThe way I read the json standard, the only way to include control characters is to encode them as hex. For example BEL can be encoded as "\u0007", but escaping it by using a backslash followed by a literal BEL character is not allowed. So literal control characters should never be in json text.
- lsaferite 3y agoThis makes me wonder if the Escape control character (\u001B) would work in a JSON string. Time to go test things out. :)
- paulddraper 3y agoMLLP (Minimal Lower Layer Protocol) -- used extensively to transmit HL7 in health systems -- uses file separators to delimit messages. OB vertical tab <content> 1C file separator 0D carriage return I wrote one of the most popular translators for MLLP, which converts it to HTTP [1]. --- P.S. Ironically, HL7 messages have something literally called a "field separator" but don't use the field separator character, usually they use vertical bar. [1] https://github.com/rivethealth/mllp-http https://github.com/rivethealth/mllp-http
- queuebert 3y agoCSV is honestly not that problematic. Figuring out if an field contains and comma and then properly quoting it is trivial. And fields without commas don't need quoting. Sometimes your application even guarantees no commas, especially if CSV is into it from the beginning.
- deleted 3y ago[deleted]
- hermitcrab 3y agoI'm guessing you haven't worked in custom support where people send you their "CSV" files. Even the field delimiter varies (many Europeans use semi-colons).
- queuebert 3y agoNo, I have. I don't consider abuse of the format a problem with the format. Though I can see how having to delimit with special characters will help the type of person who writes print(','.join(stuff)).
- hermitcrab 3y ago>I don't consider abuse of the format a problem with the format. That's a fair point. But you could argue that when the abuse is so widespread, it becomes a defacto part of the format (even if it isn't in the RFC).
- teddyh 3y agoUsing Unicode graphic characters as metasyntactic escape characters is fundamentally wrong. Those Unicode characters are for displaying the symbols for Unit Separator, Record Separator, etc. and not for actually being separators! ASCII already has those! Included in Unicode!
- layer8 3y agoTo be fair, I don’t quite get those graphic characters, because the original characters should already be displayed that way, shouldn’t they? Now when I see such a character, I have no idea if it’s the real character or just it’s graphic-character counterpart.
- teddyh 3y agoI like the LINE FEED character to be interpreted as a line feed, not displayed as “␊”.
- lsaferite 3y agoI mean, my assumption (yeah, I know) is that the 'display' variant is more for documentation talking _about_ the control character and not meant to _be_ the control character. Abusing the 'display' variant this way seems... misguided.
- eadmund 3y agoWait a second … he’s not proposing using unit/record/group/file separators as separators, he’s proposing using the symbols for those separators as separators! Why not just use the separators themselves‽ Yes, rather than using U+1F (the ASCII and Unicode unit separator), he proposes using U+241F (the Unicode symbol for the unit separator). I almost feel like this must be an early April Fool’s joke? Also, he writes ‘comprised of’ rather than ‘composed of’ or ‘comprises’ throughout his RFC.
- ahofmann 3y agoThey explain in the FAQ that this approach works with most text editors and copy-paste situations.
- philipwhiuk 3y agoIt doesn't "work" because I can't read the darn things at a sane zoom level.
- bryanlarsen 3y agoUsing a visible character rather than an invisible one makes editing in an editor a lot easier.
- tgv 3y agoIt won't wrap at the record separator, so you'll get a very long line.
- bryanlarsen 3y agoThe example seems to use `␞\n` as a separator rather than just `␞`. I assume their proposed standard is more definitive.
- vidarh 3y agoTheir ABNF uses RS, defined as U+241E, not U+241E + '\n' as the record separator. They seem to add an "USV escape" in front of the linefeeds. My bet is that this will lead to implementations that wrongly treats "␞␛\n" (RS ESC \m) as the real record separator, the same way lots of "CSV" implementations just split on comma and LF. Seems to me if you're going to add support for something like that you should just bite the bullet and declare an LF immediately following an RS as part of the record separator, or you're falling in the same trap as CSV of being "close enough" to naively splittable that people will do it because it works often enough.
- dr_kiszonka 3y agoI am very confused. The author provides "assistive accessibility software to people with vision/cognition/mobility impairments", but these character symbols are indistinguishable for folks with impaired vision.
- jph 3y agoGood connection to BoldContacts.org :-) Screen readers handle these well. If someone wants to create a font with big bold separators, that would be awesome.
- dr_kiszonka 3y agoI checked Windows Narrator, and two other utilities and they didn't utter anything. I think requiring users to install a custom font would further hold back the adoption of your format. I hope you succeed; just my 2c.
- evrimoztamur 3y agoFirst time hearing about USV, nifty! However, I think the adoptability challenge remains here to be Excel support (very tough).
- croes 3y agoExcel can't even handle CSV correctly without using the import function.
- yewenjie 3y agoI'm still confused whether this is a joke or not.
- jefftk 3y agoI don't think it's a joke; at https://github.com/sixarm/usv https://github.com/sixarm/usv they discuss how they're working on IANA standardization.
- romeoblade 3y agoApparently it is not. They have submitted it to the ietf. I will have to watch closely to see if librecalc/excel and languages/libraries adopt support. Seems like it does solve some common problems with CSV. https://www.ietf.org/archive/id/draft-unicode-separated-values-00.html https://www.ietf.org/archive/id/draft-unicode-separated-valu... https://datatracker.ietf.org/doc/draft-unicode-separated-values/ https://datatracker.ietf.org/doc/draft-unicode-separated-val...
- usrusr 3y agoI certainly hope that anyone proposing a Unicode CSV variant as a joke would pick some raised hand emoji as the separator and the victory gesture (0xe011, also popular as an approximation of how an air quote emoji would look like) as the quote character. But we already keep stumbling over missing support for the on-demand quote character even with separators like comma and tab, using more exotic characters as the separator will only make it worse. The value of less escaping is negative.
- knallfrosch 3y agoCompletely unreadable. Then again, Germans know the pain of decimal points. We write 3.000,00 for exactly three thousand, instead of 3,000.00 Now imagine how often parsing breaks.
- alwyn 3y agoIn my head 3.000,00 is correct and I always get confused because it seems most(?) people use the other method.
- michaelmior 3y agoIf I understand the API correctly from my brief glance, the crate returns a triply-nested vector with the outermost vector being the equivalent of CSV rows, then CSV columns, then "units" which don't have a direct CSV equivalent. It would be helpful if there was an API method that returned results without this final level of nesting, perhaps panicking if there is more than one unit. This would make it easier to deal with the common case (in CSV at least) where each column only has a single value.
- hiccuphippo 3y agoI think the units are the csv fields, records are rows, groups would be multiple CSV files (or multiple sheets in an excel file) and file separator... a zip with multiple CSV files? (or multiple excel files).
- michaelmior 3y agoMy mistake then about the correspondence :)
- somishere 3y agoI'm interested in this too. IMO there's actually a huge benefit to being able to concisely represent 3D / 4D data (i.e. xarrays, slices, datasets) in an easily digestible text format. Mainly thinking about this approach over e.g. the netCDF format or deeply nested JSON.
- jph 3y agoYou're directionally correct. :-) USV terminology is units, records, groups, files. Spreadsheet equivalents are cells, lines, sheets, folios. Database equivalents are fields, rows, tables, schemas. The USV Rust crate provides iterators str.units(), str.records(), str.groups(), str.files(), so it's easy to get the parts you want.
- eli 3y agoNot sure I understand the advantage over ASCII Separated Values (ASV) which use ASCII control characters 0x1E and 0x1F
- p_l 3y agoSurprisingly, they actually did write a FAQ entry on it (I'm honestly surprised): https://github.com/SixArm/usv/tree/main/doc/comparisons#ascii-separated-values-asv-aka-del-delimited-ascii https://github.com/SixArm/usv/tree/main/doc/comparisons#asci...
- jdeisenberg 3y agoAddressed in the FAQ: https://github.com/SixArm/usv/tree/main/doc/faq#why-choose-usv-over-asv https://github.com/SixArm/usv/tree/main/doc/faq#why-choose-u... Main point: ”USV provides typically-visible letter-width characters (such as Unicode 241F), whereas ASV provides typically-invisible zero-width characters (such as ASCII 31).”
- AdamH12113 3y agoUSV would have the disadvantage of using multi-byte characters as delimiters, so you have to decode the file in order to separate records. And you still can’t type the characters directly or be guaranteed to display them without font support. This honestly seems like cleverness for cleverness’s sake.
- eli 3y agoAh fair enough. Of course you could configure your shell/editor/whatever to make control characters visible. Seems like if you were going to edit USV or ASV by hand you'd probably want a customized editor anyway.
- a-priori 3y agoThe way I would have gone would be to define the standard to support both, such that the two sets of codes MUST be considered semantically equivalent, but that generation tools SHOULD prefer to generate the control codes for new files. This way people can initially use the visible glyphs while editors don't support the format, and this will always be supported. But, as editors add support and start to generate the files via tools or manually in tabular interfaces where the codes themselves disappear, usage will automatically transition over to the control codes.
- jefftk 3y agoDescription of USV: https://github.com/sixarm/usv https://github.com/sixarm/usv
- tambourine_man 3y agoASCII has a field delimiter character. The fact that we chose comma and tabs because a field delimiter character is hard to type or see is one of those things that saddens me in computing. Imagine the amount of pain that could have been spared if we had done it right from the start some 50 years ago.
- g4zj 3y agoInteresting. Are you referring to the unit separator (1F)? https://www.ascii-code.com/31 https://www.ascii-code.com/31
- tambourine_man 3y agoYes, we have unit, record, group and file separators. And we chose never to use them.
- g4zj 3y agoIt seems as though one could easily build a file format far more useful than CSV simply by utilizing these separators, and I'm sure it's been done countless times. Perhaps this would make an interesting personal project. Are you aware of any hurdles, missing key features, etc. that previous attempts at creating such a format have run into (other than adoption, obviously)?
- hermitcrab 3y agoThe ASCII unit separator and record separator characters are not well supported by editors. That is why people stick to the (horrible and inconsistent) CSV format.
- tambourine_man 3y agoPeople don’t like invisible hard to type character. They prefer suffering quoting, escaping, escaping quotes and all that fun stuff
- remram 3y agoWhy not use parquet at this point? (or a row-oriented equivalent like Avro or SQLite) If you don't have a human-readable file, might as well be compressible, queriable, and metadata-enabled I think.
- codeulike 3y agoCSV is like an invasive plant species, or perhaps a curse; you're never going to be able to root it out even thought there are a billion better data formats.
- croes 3y agoFor its use case a good and simple format with just three simple rules and three special purpose characters.
- codeulike 3y agoTrue but there's so much scope for people to do naive implementations with join() or split() functions and then you end up with nothing escaped properly and a big mess
- HideousKojima 3y agoCSV can be manually read/edited by non-technical/non-developer humans using commonly available tools like Excel and Notepad. Not many of the better data formats match that criteria.
- otherme123 3y agoNotepad, I agree. Excel... not so much: it tends to change data silently unless you are very cautious with your environment (e.g. dates transformed to number of days since 1900, and some strings to dates)
- HideousKojima 3y agoActually Excel finally added a "stop &$*@ing up my data" option recently: https://mashable.com/article/microsoft-excel-disable-setting-auto-conversion-data-into-dates https://mashable.com/article/microsoft-excel-disable-setting...
- otherme123 3y ago
- teddyh 3y agoThis is needlessly adding yet another standard¹ to the mix. If you are in a position to choose what standard you use, just use: • Whatever is best for the data model and/or languages you use. JSON is a common modern choice, suitable for most things. • If you want something more tabular, closer to CSV (which is a valid choice for bulk data), use strict RFC 4180 compliant data. • If you want to specify your own binary super-compact data, use ASN.1. I am also given to understand that Protobuf is a popular modern choice. If you aren’t in a position to choose your standards, just do whatever you need to do to parse whatever junk you are given, and emit as standards-compliant data as possible as output; again, RFC 4180 is a great way to standardize your own CSV output, as long as you stick to a subset which the receiving party can parse. Nobody needs “USV”, and nobody should use it. 1. <https://xkcd.com/927/ https://xkcd.com/927/>
- jph 3y agoThank you for the feedback. Your perspective makes sense to me for most people. I've added this to the "criticisms" page.
- ochrist 3y agoIf you live in a place where comma is the decimal separator, your CSV files will often use semicolon as the separator instead of comma. Will this tool cater for that?
- wodenokoto 3y agoWhat do you mean cater to that? The point is you separate with a value that is not used within the fields. So decimal your numbers however you want.
- greenshackle2 3y agoThis is (nominally) a discussion about the csv-to-usv tool. They are asking if the csv-to-usv tool also accepts semi-colon delimited files as input. Have you maybe lost track of what post you're commenting under? (I believe the answer is no BTW, the tool only supports , as delimiter in its input.)
- ochrist 3y agoYes, this. Thank you. If I work with CSV files they are most often not comma-separated but semicolon-separated because of the numbers. An Excel installation localized for decimal comma would not read 'real' CSV files correct. If csv-to-usv cannot cater for this type of CSV files, it would not be usable in a large part of the world.
- greenshackle2 3y agoYeah they should add it. The tool is like 20 lines of Rust code. It's a thin wrapper around the csv Rust crate, which does support specifying alternative delimiters.
- jph 3y agoYes, the next release adds custom delimiters, e.t.a. one week.
- code-faster 3y agoCSV is great because excel can import it, but it can't import USV, so at that point, why use USV when you can use JSON? https://github.com/tyleradams/json-toolkit/ https://github.com/tyleradams/json-toolkit/
- hiccuphippo 3y agoMaybe their objective in submiting to the ietf is to get programs like Excel to start supporting it.
- extraduder_ire 3y agoCan you not customize the separators used when importing csv-likes into excel? Libreoffice has a neat little window for it that even shows a preview of what values go into which cells.
- da_chicken 3y agoSure if you want to stop and fiddle with Excel. If you want to just double click and get to work, no.
- forgetfulness 3y agoSeems complex enough that you'd only manipulate files in this format by serializing through a tool, and by then it's competing with established binary formats rather than CSV.
- jonathaneunice 3y agoFascinated this uses the Unicode glyphs / symbols for unit and record separator rather than the unit and record separators themselves (ASCII US and RS). Perfect deployment of David Wheeler's aphorism: > All problems in computer science can be solved by adding another level of indirection. https://en.wikipedia.org/wiki/David_Wheeler_(computer_scientist) https://en.wikipedia.org/wiki/David_Wheeler_(computer_scient...
- ale42 3y agoIndeed... I didn't read the standard in detail to check whether escaping is allowed/taken into account, but what if my data contains those symbols? I mean, they are perfectly legal Unicode printable characters, unlike the ASCII ones.
- BugsJustFindMe 3y agoThere's an escape.
- theamk 3y agoI thought the point is you don't need escapes? If you still need to implement escape mechanism, might as well do CSV/TSV.
- sumergomonstro 3y agoYou shouldn't need escapes for separator characters precisely because they are not designed for data. Their entire purpose is to separate hierarchical data. If it turns out that escaping is needed, it will still be far rarer than escaping commas and newlines.
- BugsJustFindMe 3y agoThe point is ASCII DSV, which gives innately better hierarchy than CSV, but with visible tokens and stream accommodation. You should read the github readme. It's not that long. https://github.com/SixArm/usv/tree/main/doc/faq#why-choose-usv-over-csv-or-tsv https://github.com/SixArm/usv/tree/main/doc/faq#why-choose-u... As for still needing escapes, using obscure symbols instead of ones that are extremely common in writing inherently means needing far far faaaaaaar fewer of them.
- _obviously 3y agoUnicode is Turing complete which makes it an attack vector.
- hermitcrab 3y agoIt is a set of glyphs and their encodings. How is that 'Turing complete'?
- zzo38computer 3y agoUnicode involves more than the set of glyphs and their encodings; it also involves properties, etc. However, it can be an attack vector even ignoring that stuff; it does not have to be Turing-complete to be an attack vector. But, the specific kind of attacks depends on the application. Different kind of character sets and character encodings will be good for different purposes. Unicode is "equally bad" for many uses.
- _obviously 3y agoYes turning complete wasn't the right term. , it would be better to say that it's surprisingly complex to parse
- zzo38computer 3y agoYes, Unicode is too complicated and too messy, whether or not it is Turing-complete (it is complicated enough that maybe it is Turing-complete; I don't know).
- otabdeveloper4 3y agoAbsolutely terrible documentation. The RFC doesn't even explain the purpose of the "End of Transmission Block" token.
- jph 3y agoGood catch, thank you. An accidental omission. Fixed now in the repo. Fixed in the next IETF Internet Draft.
- bombledmonk 3y agoI've actually been employing Emoji Separated Values (ESV), often , here and there when doing some of this kind of work. Granted, it's not standard, but it's been really useful when I've needed it. *edit Apparently emojis don't fly here, but it was an index finger pointing right.
- gausswho 3y agoMy delimiter over several projects over the years has been: Only a matter of time before something breaks catastrophically but it hasn't happened yet.
- philipwhiuk 3y agoThe benefit of this is that you can use different emojis to denote content type. e.g. if it's a frowny face you know it's an invoice.
- ubutler 3y agoFunnily enough, I published a Python library two days ago that uses emojis to indicate where certain non-msgpackable builtin types have been forced into msgpackable objects: https://github.com/umarbutler/persist-cache/blob/main/src/persist_cache/serialization.py https://github.com/umarbutler/persist-cache/blob/main/src/pe... is used for tuples, for sets, for frozen sets, for pickles, for bytes and for bytearrays. I thought it was pretty ingenious but clearly I’m not the only one to think of it.
- MrOxiMoron 3y ago/me looks at calendar, nope not April 1st yet.
- pquki4 3y agoThe usv github repository says it is "the standard for data markup of ...", has 66 stars, and is currently applying for "text/usv" MIME type. That's all about it. Maybe I'll consider it when it does not belong to a company, has two more zeros in the number of stars, and has RFC/ISO attached to it. Because right now it is not much more of a "standard" than a hobby project I create on a whim.
- netsharc 3y agoYeah, I can't imagine the ego one needs to basically go "Hey everyone, I've invented a new standard!"...
- renewiltord 3y agoAbout the most annoying thing about the modern Internet is this kind of chip-on-the-shoulder comment about "oh he has such a big ego" and nonsense like that. Man, I preferred it when people could just write up and propose things. The insufferable "is that professional?", "What about consensus?", "Wow the ego to propose something". Time to return to monke.
- twosdai 3y agoYeah I agree, the project seems like a decent enough idea to discuss, and by the amount of engagement it's getting here there is some data to back up that assertion. Additionally I'm sure that those ("they have such a big ego") types of comments and thoughts existed in the early internet as well since its fairly human reaction whenever anyone tries to build or propose something that disrupts the status quo.
- FrustratedMonky 3y agoWhat a Pedantic take on what constitutes a 'standard'.
- paulddraper 3y agoWe'll be waiting on baited breath
- vidarh 3y agoTheir examples if anything convinced me not to use this for a long time. I need to zoom to be able to tell these apart, so I'll need editor support for it to be convenient to work with these anyway. And then clicking through to the comparisons, it demonstrates the difference existing support for CSV "everywhere" makes - Github renders the CSV examples nicely as tables, while again I need to zoom in to see which separator is which for USV. Maybe once there is widespread editor support. But if you need editor support for it to be comfortable anyway, then the main benefit vs. using the old-school actual separator characters goes out the window.
- strunz 3y agocsvkit makes displaying CSV in a terminal trivial and has all the tools to manipulate/filter data I've ever needed - https://csvkit.readthedocs.io/en/latest/ https://csvkit.readthedocs.io/en/latest/ I don't really get this project at all.
- vidarh 3y agoThe thing is, while I'll probably just stick with CSV too, I'm sympathetic to the intent, but given I expect it'll need tooling anyway I'm less sympathetic to them not picking the existing separator. I also think there are failed lessons here that reduces the incentive for switching. E.g. If you're going to improve on CSV, a key improvement would be to aim to make the format trivially splittable, because the lesson from CSV is that when a format looks this trivial people will assume they can just split on a fixed string or trivial regex, and so the more you can reduce the harm of that the better. As such, I'd avoid most of the escaping they show, especially for line endings, and just make RS '\n' the record separator, or possibly RS '\n'*. Optionally do the same for US. Require escaping LF immediately after RS/US, and only allow escaping RS, so unescaping can be done with a trivial fixed replace per field if you have a reason to assume your data might have leading linefeeds in fields - a lot of apps will get away with just ignoring that. Then parsing is reduced to something like `data.split(RS).map{|row| row.split(US).map{|col| col.gsub(ESCAPE,"\n") } }` (assuming RS, US, and ESCAPE are regexps that include the optional trailing linefeeds and escapes leading linefeeds respectively). Being able to copy a correct one-liner from Stackoverflow ought to avoid most of the problems with broken CSV/TSV parsing. I'm also not convinced adding GS, FS, ETB is a good idea, partly for that reason, partly because a lot of the tools people will want to load data into will not handle more than one set of records, and so you'll end up splitting files anyway, in which case I'd just use a proper archive format... Those characters feels like they're trying to do too much given they're "competing" primarily with CSV/TSV. Their spec also needs to talk about encoding, because unless I've missed something, they only talk about codepoints, and they're likely to e.g. get people splitting on the UTF8 sequence etc. This to me is another reason for using the ASCII values - they encode the same in ASCII based characters sets and UTF8, and so it feels likely to be more robust against the horrors of people doing naive split-based parsing.
- nilslice 3y agoIf you would like to run csv-to-usv from 15+ languages (not only rust!) then check out this demo I made, converting the library to an Extism plugin function: https://github.com/extism/extism-csv-to-usv https://github.com/extism/extism-csv-to-usv Here's a snippet that runs it in your browser: // Simple example to run this in your browser! But will work in Go, PHP, Ruby, Java, Python, etc... const extism = await import("https://esm.sh/@extism/extism"); const plugin = await extism.createPlugin("https://cdn.modsurfer.dylibso.com/api/v1/module/a28e7322a6fde92cc27344584b5e86c211dbd5a345fe6ec95f1389733c325541.wasm", { useWasi: false } ); let out = await plugin.call("csv_to_usv", "a,b,c"); console.log(out.text());
- greenshackle2 3y agoI'm sorry but.. why? The library is a single function consisting of 10 lines of Rust code. And would be about 10 LOCs to re-implement in any language that has native csv libs. It seems a little bit unnecessary to load a WASM runtime for that.
- theamk 3y agoBut without WASM, how are you are going to get 500ms+ startup time and an 3rd party server dependency in your critical path?
- philipwhiuk 3y agoAnd two domains you're blindly trusting not to be hijacked.
- nilslice 3y agoSorry do you know what “demo” means?
- nilslice 3y agofor sure — do it!
- yixas40465 3y ago[flagged]
- SuperHeavy256 3y agoI've long wanted a successor to CSV, but this is kinda stupid. People like CSVs because they look good, feel natural even in plaintext. This is the same reason that Markdown in successful. As for including commas in your data, it could just have been managed with a simple escape character like a \, for when there's actually a comma in your data. That's it.
- hermitcrab 3y ago>As for including commas in your data, it could just have been managed with a simple escape character like a \, for when there's actually a comma in your data. That's it. Not quite. What if there is a \ in your data? Then you have to escape that.
- lelanthran 3y ago> Not quite. What if there is a \ in your data? Then you have to escape that. No problem, any character following a `\` is a literal character. `\\` => literal `\`. `\,` => literal comma. `\a` => literal `a`, etc. Parsing this is easy, generating it is easy, and there is only one rule to remember for humans reading or generating it. Each rule added for parsing is one more added complexity and point of failure.
- philipwhiuk 3y agoYou still have to solve this in USV.
- euroderf 3y agoOr two commas in a row can be the escape, without overloading backslash.
- crq-yml 3y agoIt's sensible in principle: * Editors will play nicely with the graphical representation. If you need better graphics, it's done with font customization, which everyone already supports. * It announces that the data is source text, vs transmitted bytes. The type/token distinction is not easy to overcome. * It sits way out in Unicode's space where a collision is unlikely. The whole reason why CSV-type formats create frustration is because the tooling is ad-hoc, never does the right thing and uses the lower byte spaces where people stuff all kinds of random junk. This is the "fuck it, you get the same treatment as a Youtube video id" kind of solution. That said, if used, someone will attack it by printing those characters as input.
- pimlottc 3y ago> Is USV aiming to become a standard? > > Yes and we've submitted the first draft of the USV standard to the IETF: link. This is a nice idea, and all, but seems unlikely to become a meaningful standard without some major backing behind that "we".
- evnix 3y agoThere are some well researched alternatives to CSV, From the top of my head, I can highly recommend SML https://dev.stenway.com/SML/SimpleML.html https://dev.stenway.com/SML/SimpleML.html Recommend watching the, 'stop using CSV video' too https://youtu.be/mGUlW6YgHjE?si=zDG_9Jv8LSy-ttP4 https://youtu.be/mGUlW6YgHjE?si=zDG_9Jv8LSy-ttP4
- jph 3y agoThank you, I'll add a todo to compare SML.
- justtinker 3y agoThis is the XKCD comic in action. https://xkcd.com/927/ https://xkcd.com/927/ Someone should write a family of filters of the form CSV2ASV, CSV2USV, CSV2JSON ,USV2XML , TOML2USV, USV2Cuneiform.......
- hermitcrab 3y agoAlternatives to CSV are also covered in length at: https://news.ycombinator.com/item?id=31220841 https://news.ycombinator.com/item?id=31220841
- isoprophlex 3y agoThis is just ESV files with extra complexity! ESV: eggplant-separated values. Because who is ever going to put AUBERGINE (U+1F346) into a dataset? It's the perfect record separator!
- mongol 3y agoI prefer tomato as separator. It is tastier
- tamimio 3y agoI am uncertain, but this is likely to reintroduce the issue of Unicode buffer overflow into the mainstream. What are your proposed solutions, considering it is expected to become standardized?
- jph 3y agoUSV defaults to UTF-8, which prevents the issue of Unicode buffer overflow-- presuming you mean the typical kind where Unicode flows into ASCII. The primary USV library implementation uses Rust, which is notably good at UTF-8 and conversions between operating system string encodings (such as ASCII) and UTF-8,
- nayuki 3y agoNope, this isn't a good approach. I prefer tab-separated values (TSV) and use it as much as possible.
- Fileformat 3y agoA similar concept that is (IMHO) much nicer: RSV It doesn't need any escaping or quoting: a field just has to be valid UTF-8. The trick is that the delimiters are bytes that are invalid UTF-8. The spec fits on a napkin, parsing is trivial, you can jump to the middle of a doc and find the nearest row, etc. Main downside is you need an editor/viewer that can handle it. https://github.com/Stenway/RSV-Specification https://github.com/Stenway/RSV-Specification
- jph 3y agoThank you. I added a comparison to RSV.
- zzo38computer 3y agoI have seen Unicode Separated Values. I don't like Unicode and I even more don't like USV. I like ASCII Separated Values, which can encode each separator as a single byte, and can be used with character encodings other than Unicode (and, even if you do use it with Unicode, does not prevent you from using the Unicode control pictures in your data; USV does prevent you from using those characters in your data even though the data is (allegedly) Unicode). What they say about display and input really depends on the specific editors and viewers that you are using (and perhaps on the fonts as well). When I use vi, I have no difficulty entering ASCII control characters in the text. However, there is also the problem with line breaking, with ASV and with USV, anyways; and they do mention this in the issues anyways. Fortunately, I can write a program to convert these formats without too much difficulty, even without implementing Unicode (since it is a fixed sequence of bytes that will need to be replaced; however, it does mean that it will need to read multiple bytes to figure out whether or not it is a record separator, which is not as simple as ASV).
- philsnow 3y ago> The Synchronous Idle (SYN) symbol is a heartbeat, and is especially useful for streaming data, such as to keep a connection alive. > > SYN tells the data reader that data streaming is still in progress. > > SYN has no effect on the output content. > > Example of a unit that contains a Synchronous Idle: > > a␖b␞ Why would this go in-band inside a document format? Just why? If you want keep-alives, use a kind of connection that supports out-of-band keepalives. If you download the same document twice, and the second time the server is heavily loaded (or it's waiting on some dependency, or whatever), presumably the server will helpfully generate some SYNs in the middle of the document to keep the connection alive (?), but now you've got the same document "spelled" two different ways, that won't checksum alike. SYN along with the weirdness of > Escape + [non-USV-special] character: the character is ignored means that you have arbitrarily many ways of writing semantically-same documents.
- paulddraper 3y agoThis entire thing is a solution in search of a problem, and this is the most obvious one. Why does a file format need a transport protocol? --- Existing transport protocols (TCP, QUIC) already provide this.
- jph 3y agoGreat point about SYN. I'll drop it.
- deleted 3y ago[deleted]
- hughw 3y agoUSV is doomed because Worse is Better[1] (edit: fix url) [1] https://dreamsongs.com/RiseOfWorseIsBetter.html https://dreamsongs.com/RiseOfWorseIsBetter.html
- 1vuio0pswjnm7 3y agoText-only, no Javascript: https://static.crates.io/readmes/csv-to-usv/csv-to-usv-1.1.2.html https://static.crates.io/readmes/csv-to-usv/csv-to-usv-1.1.2... https://github.com/sixarm/csv-to-usv-rust-crate/ https://github.com/sixarm/csv-to-usv-rust-crate/
- samatman 3y agoY'know, I greatly dislike this. It's an actual emotional reaction. This should not be standardized. No one should use this. This is a bad idea and deserves to die in obscurity. I'll tell you why, it's pretty simple. The characters this... thing is stealing, exist to represent invisible control sequences. That is their use. The fact that they can be mentioned by direct input is inevitable, but not to be encouraged. I will be greatly disappointed if this is accepted as a standard. The fact that a USV file looks like a rendered ASV file is a show stopping bug, an anti-feature, an insult to life itself. Kill it with fire.
- quesera 3y agoI can't disagree. I know we're not supposed to do low-brow dismissal here, but everything about this idea is wrong. Rationale, solution, representation, everything.
- rgmerk 3y agoI think I'd prefer to wear out my keyboard typing XML tags than deal with this.
- ggm 3y agoconversion of file encoding from simple ASCII to UTF-8 has consequences beyond the field/record problem. Some tools will randomly convert " to 'LEFT DOUBLE QUOTATION MARK' and 'RIGHT DOUBLE QUOTATION MARK' if they see UTF-8 flagging. Thus, the file is converted without your voluntary participation.
- sukmaagung 3y agoInstead of using separator-character: a <comma> b <comma> c <enter> d <comma> e <comma> f why not using header-character: <row><cell> a <cell> b <cell> c <row><cell> d <cell> e <cell> f
- jbaber 3y agoI've been using an emoji separated values format for a personal project where fields contain lots of special characters including whitespace. I'd previously given up using ASV because of the printability and copy/paste problems described. Replacing the control characters by their printable glyphs solves all my previous problems and is as genius as it is naughty. I sympathize with the arguments people here present against and agree the SYN character and Group Separator are weird -- but cause no harm. I'm not bothered by the same data having multiple representations since I'm insisting on human readability rather than byte-by-byte perfection in the first place. It took 20 minutes to convert my project and I'm very happy. Only tooling change I had to make was adding digraphs to vim digraph rs 9246 us 9247 etc. Easy to type directly in my .usv file. Easy to type and read in some Python consuming it. Regardless of it becoming a standard and my lingering grouchiness about multi-byte characters, needing to use non-xterm, etc. this works very well for me.
- jph 3y agoGood points. I dropped SYN now because of feedback here. Thanks for the digraph vi info. Thanks to you, I added a vi-specific page in the repo for your tip.
- two_handfuls 3y agoNice! Is there a Python library?