10 ms·
One thing that has always puzzled me is why CSV exists, given the ASCII unit/record seperators appear to do exactly what the comma and newline characters are at
by cesaref 6y ago
One thing that has always puzzled me is why CSV exists, given the ASCII unit/record seperators appear to do exactly what the comma and newline characters are attempting to do with codes which could appear within records, and have been well defined since some point in the 60s.
And here we are, 60 years later still struggling to work out where a record ends...
- skrebbel 6y agoBacuse csv is human readable and human writeable
- nojito 6y agoSo is tsv, but for some reason csv is more popular despite being the worst character to use.
- hrktb 6y agoFor a lot of people, meaningful space is a hard concept to understand. It's made worse by the number of applications where they will see spaces reformatted willy nilly (for instance web forms eating their line breaks. From there explaining there are spaces and tabs, and that both can look the same on screen but they are different is just asking for trouble outside of our circles.
- mgkimsal 6y ago> is just asking for trouble outside of our circles it's asking for trouble in our circles too. Imagine getting a tab-sep file, opening it up in an editor and having it automatically convert to 4 spaces "because", then sending it back without checking something that doesn't look wrong.
- SAI_Peregrinus 6y agoOr an input/editor where the `tab` key moves your focus from the text field to the next button. Like Slack. You can't just hit `tab` in Slack when typing a code snippet for something that needs them like a Makefile.
- JamesSwift 6y agoWhat is human readable about a tab?
- nojito 6y agoOpen up a csv in notepad vs opening up a tsv. One actually looks like a table the other is a garbled mess.
- JamesSwift 6y agoI think we have different definitions of human readable then. To me, human readable means if I print it in a printer, it loses no fidelity. What you describe is what I would call machine-readable. It is able to be imported into a program. The same that a binary separator would also let it be imported.
- nojito 6y agoEven in your example TSV wins.
- luckylion 6y agoYeah, we've never had similar issues with line-endings ;)
- barbegal 6y agoI think that stems from the fact that no software ever properly supported them, they are non-printable characters so you can't see them easily. Whereas with CSV you can fire up notepad.exe and explore the data. Given the inability to standardise on line endings "\n" "\r\n" or "\r" I don't think we can standardise on using unit or record separators
- tinus_hn 6y agoNo, we certainly can’t expect people to use the ASCII ‘Record separator’ character as a record separator, that would be madness!
- mark-r 6y agoHave you ever seen the ASCII separator characters used as they were intended? I don't think I have. It's obvious the problem they were trying to solve, but it was too little too late. It doesn't help that they're control characters that aren't meant to be displayed so they're practically invisible.
- billme 6y agoSimple Google explains why: - non printable character - no single keyboard key - intended for systems, not people - etc
- atoav 6y agoEasy: CSVs are human readable and writeable and humans tend to have a comma sign on their keyboard, while they don't have the ASCII record seperator visible (teaching them to use eg Alt + 30 will not work). And if they had that key on their keyboard then you'd have the comma problems all over again: What if a ASCII record seperator shows up in the field?
- throwaway_pdp09 6y ago> What if a ASCII record seperator shows up point is, being not human readable and having no keyboard key, you can reasonably expect those special separators not to. I suppose I'd consider such special-separated files (26 to 29 from memory?) to be machine generated and machine readable only, not intended as human readable without a bit of extra software or eg. a special emacs mode
- sokoloff 6y agoWhen considering standard A that can only work with machine generated and machine readable files and standard B that can work with all those files plus human-readable/editable files, I don’t find it surprising at all that standard B wins in the market.
- throwaway_pdp09 6y agoThere is no standard but one for CSV files (the RFC 4180 mentioned). The whole article pretty well rams home the fact there is no standard B because there is no standard which is adhered to.
- sokoloff 6y agoHumans are not bound by the computer science definition of “standard”. But you’re correct that I was using the word “standard” imprecisely to mean something more like “file extension” rather than an IETF RFC.
- ken 6y ago
- snazz 6y agoThe TSV (tab-separated values) format is supported by a lot of software (Google Sheets, for instance) and solves the "comma in the field" problem quite nicely. Tabs can easily be typed by a human, but they don't usually occur within a one-line string, so they don't cause the same problems as CSVs do.
- hulahoof 6y agoTSV, CSV, PSV etc. all share the same class of problem. The important thing is that you are quoting your fields and escaping your quotes, regardless of the symbols used.
- JoBrad 6y agoImportantly, tabs aren’t usually inserted into fields by Excel users. With Power BI, exporting TSV is a little easier for non-dev Excel power users, too.
- IggleSniggle 6y agoNailed it. When Excel users press tab, they mean "go to the next cell." Conveniently, this is also what the format means to the computer when it encounters the tab separator. It's not foolproof, but it's much better. The human interpretation of the character matches the machine interpretation.
- DataWorker 6y ago“Don’t usually occur” They do. People write code to create all sorts of bastardized abominations of “csv” or tab delim or whatever. It’s why the featured article gets reposted every few years. You can define a standard for csv files, but then Excel does it’s own thing and here we are.
- steerablesafe 6y agoIf those characters had a visual representation and were easily typeable and we settled on using them as separators then we would struggle escaping those special characters within records. The problem is lack of standardization, not lack of special characters for record separators.
- rb808 6y agoWow never heard of it before. The biggest problem is Excel and text editors rule the small data world.
- Spooky23 6y agoI think CSV is more popular than one would think because Excel was awful back in the old days. A lot of the crufty edge cases are artifacts of whatever people had to deal with in 1995 to get data into spreadsheets. We all got used to pushing data around with CSV because the people consuming the data needed it.
- eyelidlessness 6y agoBased on my understanding of CSV, unit separators are valid column delimiters. I doubt you could use record separators between as row delimiters though.
- atrettel 6y agoCSV is human readable and writable, but that is only part of the advantage of CSV. Its simple format is great as an archival format. It's the plain text of data formats. Decades pass and you can still read it without much of an issue, though people are rightly pointing out that the format is ambiguous is nature and subject to a lot of interpretation. I've recently been compiling a lot of decades old experimental datasets, and it's honestly great when I find plain text files from the 1980s, since we can still read them! It's the more exotic data formats that came about later that are often more difficult to read properly. The Library of Congress lists CSV files as one of its preferred formats for archival datasets: https://www.loc.gov/preservation/resources/rfs/data.html https://www.loc.gov/preservation/resources/rfs/data.html
- carapace 6y agoThis, a thousand times. As a fun exercise write a CSV-like spec but using those ASCII chars. table = [row.split(unit_sep) for row in data.split(rec_sep)] There's your ASCIISV decoder.
- hexmiles 6y agoThat's funny, when i was twelve i need to store some data and i didn't really know database or csv, so i actually used the ascii control character to implement tables and record. I later discovered sqlite, soooo much simpler...
- cryptonector 6y agoASCII FS/RS/GS/US do not have a visual representation, and users do not know how to enter them. That means they're essentially for binary-like encodings -- they're not really text if users can't see or enter them easily. But users can see the commas in CSV, and they can trivially enter them. Yeah, the result is messy. The lesson here is that the separator control characters should have been visual and had a visual indicator on keyboard keycaps to indicate how to enter them. Because they aren't and didn't, they are essentially useless for text. EDIT: I do happen to know how to enter these on *nix systems. The ascii(7) man page tells you how: 034 28 1C FS (file separator) 134 92 5C \ '\\' 035 29 1D GS (group separator) 135 93 5D ] 036 30 1E RS (record separator) 136 94 5E ^ 037 31 1F US (unit separator) 137 95 5F _ so FS is ^\ (which you have to be careful does not cause the tty to generate SIGQUIT -- you have to ^V^\), ^], ^^, and ^_. That is, <control>\, <control>], <control>^, and <control><shift>-. On U.S. keyboards anyways.