6 ms·
Totally agree. His arguments are basically "performance!" (which is honestly not important to 99% of CSV export users) and "It's underderspecified!" And while I
by hn_throwaway_99 3y ago
Totally agree. His arguments are basically "performance!" (which is honestly not important to 99% of CSV export users) and "It's underderspecified!" And while I can agree with the second, at least partly, in the real world the spec is essentially "Can you import it to Excel?". I'm amazed at how much programmers can discount "It already works pretty much everywhere" for the sake of more esoteric improvements.
All that said (and perhaps counter to what I said), I do hope "Unicode Separated Values" takes off. It's essentially just a slight tweak to CSV where the delimiters are special unicode characters, so you don't have to have complicated quoting/escaping logic, and it also supports multiple sheets (i.e. a workbook) in a single file.
- ryncewynd 3y agoAre there unicode characters specifically for delimiters? If Excel had a standardised "Save as USV" option it would solve so many issues for me. I get so many broken CSVs from third-parties
- bhaney 3y agoHell, there's ASCII characters specifically for delimiters. 0x1C to 0x1F are respectively defined as file, group, record, and unit separators. Unicode naturally inherits them all.
- smcin 3y agoExcept nobody uses them. Another previous discussion: https://news.ycombinator.com/item?id=33935140 https://news.ycombinator.com/item?id=33935140
- hiccuphippo 3y agoASCII has characters for unit, record, group and file separator. And a some days ago there was a story here about using the unicode printable representation of these for an editor friendly format. https://news.ycombinator.com/item?id=39679378 https://news.ycombinator.com/item?id=39679378
- crq-yml 3y agoThere are characters from ASCII for delimiting records, which are underused because they cause confusion about whether they should be represented as a state change like a backspace character, or as a glyph. See also: "nobody can agree on the line ending character sequence". The USV proposal uses additional codepoints introduced in Unicode for the representation of the record delimiters, so they will always look and edit like character glyphs, and nobody is using them for some other purpose. The standardized look of these glyphs is unappealing, and they aren't easy to type, but it's fixable with a font and some editing functions. Most of the issue hinges on Excel support.
- samatman 3y ago> nobody is using them for some other purpose. There's a lot of tooling which uses them for their intended purpose, which is to represent the C0 control characters in a font, so they can be printed when they appear in a document. Your editor is probably one of those. Which is why I consider USV a terrible idea. If I see ␇ in a file, I don't want to be constantly wondering if it's "mention ␇" or "use ␇ to represent \x07". That's why the control pictures block exists: to provide pictographs of invisible control characters. Not to make a cutesy hack "look! it's a picture of a control character pretending to be what it isn't!!" format.
- rightbyte 3y agoI don't underatand why this is a question up for debate. You need eye tracking, so that there is a beep when you read the relevant part.
- ykonstant 3y agoI agree about USV, it creates confusion where none needs to exist. For personal use, though, it is not that bad to receive a USV: it should be postmarked ".usv" and in any case if you suspect shenanigans you can `grep` for the offending (literally!) unicode characters and `tr` them into proper ASCII separators. Now, if there is nesting in the USV, I give up. I share the lament: the whole table issue was solved before it became a problem. POSIX divides ASCII into portable and non-portable characters; only portable characters are allowed in the fields and separators are non-portable. If you need nesting, use a portable encoding of the inner table. This scheme repeats indefinitely without escaping hell or exceptions, preventing tons of errors and headache. Visibility is such a bizarre complaint. Text editors already handle control characters: they handle tabs, they handle newlines, it is not a tremendous, earth-shattering feature request to make them handle separators gracefully.
- hn_throwaway_99 3y agoYes, this recent discussion has lots of good info and links in the comments: https://news.ycombinator.com/item?id=39679378 https://news.ycombinator.com/item?id=39679378
- lostlogin 3y ago> Are there unicode characters specifically for delimiters? We could use the HL7 pipe ‘|’ and all enjoy that hell.
- chaps 3y agoMy worst nightmare was a semicolon-delimited file. Where one of the columns had hand-typed street names - without quotes.. so "WELLS" was often "WE;;S". Since it was the only column like that, the # of columns to the left of the annoying column and the # on the right would always stay the same. So it was pretty easy to clean.
- doctor_eval 3y agoIt’s been years since I last worked with HL7. Isn’t there also ^ and ~ to deal with? Hell indeed.
- deathanatos 3y agoYes, there is. Multi-dimensional CSV?
- lostlogin 3y agoThere is the white space bs too. Sometimes it matters, sometimes it doesn’t. What type of white space is it? Seriously rough.
- alexvoda 3y agoIsn't that what HL7 stands for? Hell Layer 7 as in the seventh circle of hell.
- deathanatos 3y agoGod, please no. For those unfamiliar with the atrocity that is HL7v2, the format is essentially CSV, but with the record separator set to a lone CR, and the field separator usually set to |. Usually, because the format lets the consumer of the format redefine it, for whatever reasons. (The first use of the field separator it determines whatever character it will be. Thankfully, the first use is in a fixed spot, so it's determinable, but still. Oh, but we don't know the character encoding until like the 18th field in … and it doesn't necessarily have to be an ASCII superset. So I have no idea what an HL7v2 message in a non-ASCII superset even looks like, or how a parser is even supposed to reasonably parse such a thing. I presume attempt a decoding in all possible decodings, and then see which one matches the embedded character set, and pray nobody can create a polyglot?) There's also further separators, delimiting within a field. It also has its own escape sequences, to deal with the above. … and it is what carries an unfortunate amount of medical data, and is generally how providers interoperate, despite the existence of more civilized standards like FHIR
- fbdab103 3y agoMy significantly bigger beef would be all of the auto-formatting Excel does to mangle data. Excel loves to turn entries into dates. Human genes had to be renamed so as to avoid this Excel features.
- potatoicecoffee 3y agoexcel now has a prompt so you can tell it to not convert stuff automatically
- fbdab103 3y agoGasp. Big news. I do not recall ever seeing this, so I wonder if $JOB is running some hilariously outdated version for compatibility with a load bearing VBA script.
- sgerenser 3y agoAutomatic Data Conversion toggle was only added in the past ~year: https://insider.microsoft365.com/en-us/blog/control-data-conversions-in-excel-for-windows-and-mac https://insider.microsoft365.com/en-us/blog/control-data-con...
- diogocp 3y ago> in the real world the spec is essentially "Can you import it to Excel?" And the answer to that is always no. You will it think it's yes because it works for you, but when you send it to someone who has a different Excel version or simply different regional settings, it won't work. The recipient will first have to figure out what dialect you used to export.
- chaps 3y ago"Always no"? What. Really not sure what you mean here when you agree that the possibility of it working exists. I get the sentiment -- when I request data through FOIA, I will almost always request it as "an excel format" because I know that I'll at least be able to import it. CSV is much less of a guarantee and will have issues -- missing quotes, wrong delimiters, inconsistent column counts, things like that. So requesting "an excel format" implies "make the best effort to give me something that will load in my excel, but without asking what version of excel I have". Removes a fair amount of hassle, especially when it took months to get the data. It also means that if they fuck up the columns by doing a conversion, you have some means of telling them that the data is simply wrong, rather than the data is hard to work with. It does mean dealing with [0-9]+GB sized excel files sometimes, though. That all said, I prefer to share CSV files. Haven't had much of a problem with it and I can guarantee some consistency from my side. Like, the people I share files with aren't going to know what the hell a parquet file is. A CSV though? Just double click it if you're less technical, or open it up in less if you can use a terminal. It usually compresses well, despite what the author wrote.
- throwaway2037 3y ago> when I request data through FOIA Fascinating. Can you share any details? Did you ever think to share some of your interesting finds here on HN as a submission?
- chaps 3y agoEvery now and then, yep :) https://mchap.io/that-time-the-city-of-seattle-accidentally-gave-me-32m-emails-for-40-dollars4997.html https://mchap.io/that-time-the-city-of-seattle-accidentally-...
- Dalewyn 3y ago>I do hope "Unicode Separated Values" takes off. It's essentially just a slight tweak to CSV where the delimiters are special unicode characters Commas can be typed by anyone on any keyboard and readable by anyone. Special Unicode Characters(tm) can't be typed by anyone on any keyboard and readable by noone. Convenience is a virtue.
- hn_throwaway_99 3y agoI can't remember the last time I, or anyone I've ever worked with for that matter, ever typed up a CSV from scratch. The whole point of USV is that the delimiters can't normally be typed so you don't have to worry about escaping. USV supports displayable delimiters (see https://github.com/SixArm/usv https://github.com/SixArm/usv), so for the much more common case of editing an existing CSV in a text editor, you can just copy and paste.
- Dalewyn 3y agoI've valued the virtue of CSVs being readable by any text editor known to man, and I've occasionally edited them by hand. The pure simplicity of reading and typing commas trumps any value provided by more esoteric configurations. As for escaping, that's for the subsequent programmers (which could also be me) to figure out. If it is me, I'll deal with it because it keeps things simple.
- groestl 3y ago> I've occasionally edited them by hand. Yeah, usually when the quoting was f'up.
- albert_e 3y agoEveryone of us was a beginner at some point. The first time we came across CSV format we likely typed it in notepad by hand. A lot of issues with CSVs are also sometimes troubleshooted by hand-- by manually fixing a quote or a comma. There is value is the ability to do this level of editing and troubleshooting.
- palmfacehn 3y agoJSON objects as a CSV field has been mostly agreeable for my usage. It would be nice if some of the spreadsheet apps displayed the object tree.
- djbusby 3y agoASCII has had field and record separators since like, forever. Wish we had kept using those.
- josefx 3y agoDoes it have nesting operators? I want to embedd ASCII within my ASCII fields. So I can have a table within my table.
- lolive 3y agoA manager: I wish I could have a CSV inside my CSV. Any sane person: NO!
- josefx 3y agobase64 encoded fields it is.
- lolive 3y agoThat was the right answer. You are hired!
- bruce511 3y agoWell you might be wrong, but EDI in general and HL7 specifically allow 3 levels of "fields in fields in field". As long as your parser copes, and as long as you have appropriate structures to import into, its no big deal.
- iainmerrick 3y agoSo now when I'm exporting data I need to know what nesting level it's going to live at so I can generate the correct separators? I really think that might be the worst idea I've heard for a while!
- 3y ago
- paulddraper 3y agoUSV has a mountain of problems. And really is in search of a problem to solve.
- chrisandchris 3y agoSo true. Working with imports/exports in CSV from ERP software. One can't imagine how often "Oh, this import doesn't work. I'll just fix the CSV file" occurs. Try that with some compressed, "esoteric" file, or even CML and users will break it. Besides all the downsides CSV has, as soon as it's not only machine-machine communication and a human is involved, CSV is just simole enough.
- likis 3y agoIt seems like you missed the conclusion in the article. If users want CSV exports, let them have it. If you have important data being shuffled around systems, pick something with a specification instead.
- xeromal 3y agoCheck out Polars in python if you want some CSV performance lol. I recently got a 5 million row CSV from a 3rd party and I could manipulate columns (filtering, sorting, grouping) in actions that took less than a second. It's an incredible tool.
- medstrom 3y agoThe IANA standard for TSV already disallows tabs inside fields, so you can skip writing any quoting logic (in principle). The MIME type is `text/tab-separated-values`. https://www.iana.org/assignments/media-types/text/tab-separated-values https://www.iana.org/assignments/media-types/text/tab-separa...
- puppymaster 3y agoditto. if it fits in the ram, file types don't matter.
- deleted 3y ago[deleted]