5 ms·
Easy: CSVs are human readable and writeable and humans tend to have a comma sign on their keyboard, while they don't have the ASCII record seperator visible (te
by atoav 7y ago
Easy: CSVs are human readable and writeable and humans tend to have a comma sign on their keyboard, while they don't have the ASCII record seperator visible (teaching them to use eg Alt + 30 will not work).
And if they had that key on their keyboard then you'd have the comma problems all over again: What if a ASCII record seperator shows up in the field?
- throwaway_pdp09 7y ago> What if a ASCII record seperator shows up point is, being not human readable and having no keyboard key, you can reasonably expect those special separators not to. I suppose I'd consider such special-separated files (26 to 29 from memory?) to be machine generated and machine readable only, not intended as human readable without a bit of extra software or eg. a special emacs mode
- sokoloff 7y agoWhen considering standard A that can only work with machine generated and machine readable files and standard B that can work with all those files plus human-readable/editable files, I don’t find it surprising at all that standard B wins in the market.
- throwaway_pdp09 7y agoThere is no standard but one for CSV files (the RFC 4180 mentioned). The whole article pretty well rams home the fact there is no standard B because there is no standard which is adhered to.
- sokoloff 7y agoHumans are not bound by the computer science definition of “standard”. But you’re correct that I was using the word “standard” imprecisely to mean something more like “file extension” rather than an IETF RFC.
- ken 7y agoIt happens. XPM/PBM/XBM lost to TIFF (and later PNG). Postscript lost to PDF. OpenOffice.org XML couldn't displace Excel's Binary File Format. As much as I wish that beauty or usability was the primary indicator of market success, the simpler explanation that explains all of these (including CSV) is: Microsoft Office supported them.
- throwaway_pdp09 7y agoI don't know about XPM/PBM/XBM lost but AIUI PDF is just a subset of postscript with compression. If you think MS's binary excel file format is all that secret, I suggest you try unzipping it - it is no more than compressed XML. I have rescued parts of corrupt excel and word files by doing just that.
- mark-r 7y agoMS completely redesigned their binary formats a few years ago. You're right about the current format, but the earlier one was completely undecipherable. Even the current XML has quirks in it to be compatible with the older format.
- mike22 7y agoIt was reversible enough to write the Apache POI library. I maintain code that uses the HSSF streaming API portion of the library, which mainly gives you low-level access to the structures in XLS files. The OO.org folks made some very nice documentation: http://www.openoffice.org/sc/excelfileformat.pdf http://www.openoffice.org/sc/excelfileformat.pdf
- ken 7y agoPDF diverged from Postscript years ago. Microsoft’s binary Excel format is not XML -- you’re thinking of its replacement. It was COM/OLE or some such.
- greggman3 7y agoBinary formats win I think when the ASCII format is significantly greater in storage size and parsing speed. XPM/PBM/XBM are enormous compared to any binary image format. And remember these images used to get put on 1.4meg floppies and sent over 2400baud modems. It's basically use the right tool for the job. For pixels that's a binary format. For mostly text, text itself works pretty well.
- roenxi 7y agoYou're almost certainly describing a file format that already exists and that nobody uses because it is not human-interactable. Anyone can edit a csv. I often have. An important feature. The only real stumper in the CSV format is why they didn't use \ as the escape in strings like everyone else. Probably some good reason.
- throwaway_pdp09 7y agoInteresting. I don't think I have ever hand edited a CSV (unless you mean created from scratch, well then, perhaps). OTOH the ability to just look at it... that I've found very valuable and agree there.
- eyelidlessness 7y agoWhile backslash escaping is very common, double-quote (or double-special-anything) is fairly common as well. I use it all the time in SQL.
- ISL 7y agoIf someone copy-pastes a line from one CSV to a string within a field of another, the second will have record separators from the first within a single field.
- louis_pasteur 7y agoBut humans rarely use notepad these days. They use code editors like Coda or VSCode at the very least which have all kinds of advanced features. Surely, those can include support for ASCII separators?
- beaconstudios 7y agoDevelopers rarely use notepad. Non technical users still have to edit csvs by hand, all over the world.
- JauntyHatAngle 7y agoI don't think your first line is at all true. Modifying Csvs with notepad is rife. I'd wager more than using any one particular code editor.
- arethuza 7y agoOr they use Excel, which can introduce its own entertaining errors. A few years back while working on something that unavoidably used large quantities of CSV data we would sternly warn people not to use Excel, but people still did it ("But I just looked at it!").
- roto 7y agoDevelopers are by no means the only humans that use and/or edit CSV files.
- the_angry_angel 7y ago> But humans rarely use notepad these days. They use code editors like Coda or VSCode at the very least which have all kinds of advanced features. Surely, those can include support for ASCII separators? I do a fair amount of work with companies that do "EDI" over CSV (or worse CSV-like - think 2 CSVs jammed together with different formats, no headers, no support for escaping or quoting) and fixed width documents. I can absolutely assure you that humans do open these files in notepad far more often than I'd like. Often one of the main reasons they don't use things like X12, ASCII separators, etc. is because a "human needs to open it at some point" was a prevailing business decision some number of years ago (think "what happens if the IT system fails? how can we still ship stuff even in a complete emergency") and now it's baked into their documented process so deeply its like shouting into the wind to alter things. Third party warehouses are the worst at this.
- u801e 7y ago> CSVs are human readable and writeable and humans tend to have a comma sign on their keyboard Do people usually compose CSV data by hand? I thought they would use a program like Excel, enter or generate their data in separate cells and then save the file as a CSV. There's no reason why a program like Excel (or any other program for that matter) couldn't use the record separator instead of a comma as a delimiter when generating or consuming such files.
- ganstyles 7y agoI do quite a bit for GitHub markdown tables, for making small annotations for ML, for python parsing, etc. Very common and would be annoying too open up excel every time I needed some csv.
- u801e 7y agoI thought markdown tables used the pipe character as the delimiter between table columns. Also, is not necessary to use excel, and a lot of the issues brought up in the article wouldn't be the case with a different delimiter character. Editors could easily be updated to make entering that delimiter character easier to enter by mapping it to a key like tab whenever a file like that is opened. It just seems like a problem that could be solved, but can't be due to inertia.
- yiyus 7y ago> I thought markdown tables used the pipe character as the delimiter between table columns. Markdown (the original one by John Gruber) does not include any syntax for tables. Other implementations have included it, but there is no standard. As far as I know, using the pipe character to separate columns is indeed quite common, but it is not the only way.
- BiteCode_dev 7y agoCompose, rarely. Read and edit with plain text editor, it's common.
- r00fus 7y agoIf the ASCII separators were supported regularly in text editors since the early days then it would remove 90% of the problems. I prefer the | pipe character as a delimiter - easy to see, not part of common speech or titles, and enterable via keyboard. Yes, it can exist in the field but less likely.
- chapium 7y agoIt looks a LOT like l and I.
- r00fus 7y agoLuckily code sees it as ASCII 124. If you're saying that for compliance, | can be converted to l/I then it's a feature, not a downside.
- niccl 7y agoBut if it _might_ turn up, you have to assume it will and deal with it, and then you're into the exact same problem as before, but with something even less 'standard' than before. I think it's like cryptography. Why bother to roll your own when there are people who are cleverer (certainly than me. I don't know about you) who've already put a lot of effort in to this, so just use one of the well tested standard libraries and don't mess with it
- r00fus 7y agoI'm not recommending "rolling your own". You still use CSV libraries, but with a pipe delimiter instead.