8 ms·
"In favour of what?", that is the matter. CSV is a format more for humans and less for machines, but that is the use case: a format that is good enough to be c
by izietto 5y ago
"In favour of what?", that is the matter.
CSV is a format more for humans and less for machines, but that is the use case: a format that is good enough to be compiled by humans and read by machines. At the moment there aren't many alternatives.
- habitue 5y agoObjectively, CSV is terrible for humans despite being a plaintext format. No one reads CSVs: they're incomprehensible since the columns are not aligned with the headings. (You might be drawing an analogy with JSON, which is often human readable because it puts the keys right there next to the values). The best that can be said for its simplicity is that it's easy to write code that can dump data out in CSV format (and to a lesser extent, it's easy to parse it, though watch out for those variants). This is not a really strong argument, most everyone is going to use a library for serialization, there's no reason to write your own unless it's for learning.
- fighterpilot 5y agoI'm a human and I love CSVs. What other format can I open in a spreadsheet, access through cat/grep/awk, and easily load into any programming language? Any other format has to sacrifice one of these three things, and that's bad. It's this trifecta that makes it so versatile and human friendly.
- madaxe_again 5y agoDeveloper friendly. Not human friendly. I used to receive CSVs of product data from clients, which they often handcrafted or manipulated by hand, and inevitably, and I mean inevitably, broke. Of course, excel will make just as much of a mess - when the client hands back their sheet with all of their UPCs expressed as exponents and æ€ ligatures jammed next to every apostrophe, there’s no guessing as to what happened. In theory, it’s a great and simple format. In practice, it causes much pain.
- asdff 5y agoHonestly though, if people are doing that with CSVs there is no system where they won't also go off the reservation and do something insane with an excel file or sqlite or whatever. At least with CSVs there are common lines of code all around the internet to wrangle all sorts of issues.
- bobthepanda 5y agoAnything written by hand will be screwed up. Chances are if they're writing it by hand and it's badly formatted, they're not using something that has input or output validation anyways, and they're probably not going to budge from whatever workflow they're using.
- xtracto 5y ago> access through cat/grep/awk, and easily load into any programming language Until the CSV fields contain commas themselves. Even if fields are surrounded by "".
- megous 5y agoThen you just switch to a 3-line php script (fgetcsv() in a loop) and continue on... :)
- breck 5y agoSome people like me never generate CSVs like that. If someone sends me a CSV with quotes, I reject it and say "clean up your data". It's okay to say, "we're not going to allow escape characters in our CSV grammar. We're going to have a higher standard"
- danellis 5y agoNot everyone has the option of rejecting it, though. Sometimes you need that data and have no control over how it's being generated.
- the_only_law 5y agoI recall one of my old Phones could dump SMS messages as CSV. I wanted to be build and interface that would let me search through my messages. Turns out when you have data that contains both commas and quotes things get screwey real quick. You could have quoted data like: 123,ABC,”,””,456 Where ,” is column data. I think that standard method is to double quote the field, but the dump sure wasn’t doing that for me, plus what happens when the data is something like abc””,?
- deleted 5y ago[deleted]
- rendall 5y ago> 123,ABC,”,””,456 That is badly formatted CSV, assuming ” is a double-quote. Double-quote (") is how you escape commas, so the 3rd entry will make a parser barf, since there is no close quote. It should be 123,ABC,"","",456 > Turns out when you have data that contains both commas and quotes things get screwey real quick. Not really. It's pretty logical. If the entry has even one comma, put quotes around it. If it also has quotes, double each of them. Otherwise don't worry about it. > what happens when the data is something like abc””,? abc"", becomes "abc"""","
- wutwutwutwut 5y agoWhich format would you propose if I am writing a .NET tool to produce a file to be consumed by Excel?
- habitue 5y agoI would expect .NET to have excellent .xlsx capabilities? Most libraries have an easy way to write those from tabular data.
- wutwutwutwut 5y agoWhy would you expect that?
- izietto 5y agoI like CSV instead, it's pretty intuitive to me to provide a list of records separating each column by comma and each record by newline.
- rutthenut 5y agoSurprised you would think that JSON is more human-readable, as the layout of that makes it visually quite appalling - not so bad when through a pretty-printer utility, admittedly, but may as well go for XML as another use case
- anigbrowl 5y agoFor ages I would just dump json files in a browser and let it auto-pretty print it for a first look. A while back I discovered https://dadroit.com/ https://dadroit.com/ which is small (in feature set) but perfectly formed (nicely designed and extremely fast, even on huge files).
- godshatter 5y agoI try to use tab-delimited files when writing out columnar text to be loaded into something else (assuming it can parse them). It's even easier to read since you don't have extraneous commas and double quotes everywhere and most viewers will line things up to some degree via the tabs. Tabs are less common in data fields (at least when compared to double quotes and commas), at least in my experience.
- asdff 5y agoThe columns are aligned with the headings if you use the tool for the job, like R or python or awk or insert favorite cli parsing method here.
- grenoire 5y agoI think this issue can be bypassed if a better format such as Arrow or Parquet can be used in Excel.
- masklinn 5y agoIdeally excel would just speak sqlite. Not import/export or connect via odbc, straight be an sqlite client.
- cryptonym 5y agoThat doesn't qualify as "good enough to be compiled by humans". We need something dumb and easy you can author from scratch with any text editor, and read/understand with just `cat`.
- pdonis 5y agoCSV isn't supported in Excel because Microsoft prefers it. MS prefers its own Excel format. Excel supports CSV because a large enough majority of Excel users want it, which means MS includes support for it. For MS to remove CSV support in favor of Arrow or Parquet support would not benefit MS at all, and would make many of its customers push back, so I don't see it happening.
- mr_toad 5y agoEven if they did support Parquet they’d use the same embrace and extend mechanisms they do with CSV files.
- xtracto 5y agoInteresting that the two main "alternatives" mentioned here are columnar data storages. The problem I see with both Arrow and Parquet are that they are binary-based files. Sure, binary files can be VERY efficient. But the "magic" behind files like CSV, JSON or YAML is that they are both "sufficiently" both human and machine readable. Maybe a CSV killer would be a human readable columnar based file format. Nevertheless, the article basically discusses the issues encountered with the manual "editability" of CSV files, not so much with its performance. It also mentions parquet or arrow and concedes that they require specialized format to read/write. If we are looking to that, then there are a lot of options such as sqlite format, BerkleDB (used by some cryptocurrency projects) among plenty of others.
- masklinn 5y agoAnd CSV can be consumed and produced by spreadsheet software, which is useful to critical in many contexts where CSV is part of the pipeline. There the alternative would be to use xlsx files which… isn't necessarily any better. Until someone gets excel to ingest and produce something in a better format, we're pretty much stuck.
- Sanzig 5y agoAs I mentioned down-thread, I can generate a CSV with a couple of fprintf statements and a loop. I definitely can't do that with .xlsx. There is almost zero friction to bolting CSV export capability to an existing system, which is part of why it's so popular.
- masklinn 5y ago> As I mentioned down-thread, I can generate a CSV with a couple of fprintf statements and a loop. And usually generate garbage for anything but the most trivial case, which really nobody gives a shit about. That's the main reason why CSV absolutely sucks too, you have to waste month diagnosing the broken shit you're given to implement the workarounds necessary to deal with it. > I definitely can't do that with .xlsx. You probably can though. An xlsx file is just a bunch of XML files in a zip.
- pdonis 5y ago> An xlsx file is just a bunch of XML files in a zip. A bunch of XML files with opaque formats that MS constantly makes changes to to make its competitors have to keep chasing the format.
- masklinn 5y agoEven if it were true it wouldn't matter a whit to the production side of the format, which is what "produce CSVs using fprintf" is: excel can consume them all.
- acwan93 5y agoAlso, "will the other side adopt it?" After working with so many retailers and online sales channels, things that are considered "legacy" or "outdated" by the HN crowd doesn't seem like it will go away unless both sides make a change. There are numerous articles posted on HN about how "FTP is dead" or no one uses it anymore, when it's far from the case. Even Amazon's marketplace and vendor files are still using SFTP and EDI files. They've recently made changes, but it's been slow and hasn't had widespread adoption. There's also the universality and "simplicity" CSV provides to the non-computer literate, and convincing them to make a change to a new standard provides itself some non-technical challenges. CSV is a bad standard, but it's the best one given what it does and its flexibility.
- danellis 5y agoWhat are all these hypothetical humans doing with CSV files? I've worked with them a lot, and the only time I read one manually is to find out why it isn't parsing properly. I'm not writing them by hand. I'm not extracting data from them by hand. What are other people doing?
- crazygringo 5y agoMainly Ctrl+F and diffs, in my case. Sometimes you need to look up a single code in a 50 MB file. And sometimes you need a quick check to see if one line or a million lines changes. It's "exception not the rule" type stuff... but it sure comes in handy to be able to check this stuff quick with basic text tools than have to run it through some binary parser. Same as JSON. But unlike protobufs for example.
- danellis 5y agoInteresting. I would reach for `grep` and `diff` in those cases, so it's not much of a leap to imagine similarly using a tool for a binary format. After all, I only need the textual representation of the thing I'm searching for, not the entire dataset.
- crazygringo 5y agoSure, but those specific tools need to be written for each specific binary format. And honestly they usually simply don't exist. In the vast majority of cases, in my experience.