6 ms·

The post should at least mention in passing the major problem with CSV: it is a "no spec" family of de-facto formats, not a single thing (it is an example of "h

• y42 2y ago
  why you hate csv, not the program that is not able to properly create csv?
• golly_ned 2y ago
  It does mention this. Point 2.
• jimbokun 2y ago
  The post does mention it, as a positive: https://github.com/medialab/xan/blob/master/docs/LOVE_LETTER.md#2-csv-is-a-collective-idea https://github.com/medialab/xan/blob/master/docs/LOVE_LETTER...
• sramsay64 2y ago
  In fairness there are also several ambiguities with JSON. How do you handle multiple copies of the same key? Does the order of keys have semantic meaning? jq supports several pseudo-JSON formats that are quite useful like record separator separated JSON, newline separated JSON. These are obviously out of spec, but useful enough that I've used them and sometimes piped them into a .json file for storage. Also, encoding things like IEEE NaN/Infinity, and raw byte arrays has to be in proprietary ways.
  • diekhans 2y ago
    Plus the 64-bit integer problem, really 52-bit integers, due to JS not having integers.
    • 0cf8612b2e1e 2y ago
      That’s a JavaScript problem, not JSON.
    • d0mine 2y ago
      JSON itself is not limited to neither 52 nor 64-bit integers. integer = -? (digit | onenine digit+) https://json.org/ https://json.org/
    • dtech 2y ago
      Most good parsers have an option to parse to integers or arbitrary precision decimals.
      • VMG 2y ago
        Agreed. Which means that Javascript does not have a good parser.
        • exogen 2y ago
          `JSON.parse` actually does give you that option via the `reviver` parameter, which gives you access to the original string of digits (to pass to `BigInt` or the number type of your choosing) – so per this conversation fits the "good parser" criteria.
• d0mine 2y ago
  I lived through SOAP/WSDL horror with their numerous standards and the lack of compatibility between stacks in different programming languages. Having seen abused XML, CSV formats. CSV is preferable over XML. Human-readability matters. Relative simplicity matters. Despite JSON may also be interpreted differently by different tools, it is a good default choice for communicating between programs
  • ElectricalUnion 2y ago
    > lack of compatibility between stacks in different programming languages Well, that sure beats OpenAPI lack of compatibility between stacks in the same programming language. I think the fact one can't randomly concatenate strings and call it "valid XML" a huge bonus over the very common "join strings with comma and \r\n", non-rfc4180 compliant (therefore mostly unparseable without human/LLM interaction) garbage people often pretend is CSV.
• deleted 2y ago
  [deleted]
• pcwalton 2y ago
  > Unlike XML or JSON, there isn't a document defining the grammar of well-formed or valid CSV files There is such a document: RFC 4180. It may not be a good document, but it does exist.
• rkagerer 2y ago
  Waiting for someone to write a love letter to the infamous Windows INI file format I actually miss that. It was nice when settings were stored right alongside your software, instead of being left behind all over a bloated registry. And the format was elegant, if crude. I wrote my own library for encoding/writing/reading various datatypes and structure into ini's, in a couple different languages, and it served me well for years.
  • xp84 2y ago
    I feel like YAML is a spiritual successor to the .ini, since it shares a notable ideal of simple human readability/writability.
    • estebank 2y ago
      Whenever I ask myself "should I use YAML?" I answer myself "Norway".
      • Ygg2 2y ago
        To be fair to YAML that's been solved in 1.2. https://yaml.org/spec/1.2.2/#10212-boolean https://yaml.org/spec/1.2.2/#10212-boolean
    • ElectricalUnion 2y ago
      Given how YAML does magic and sometimes accidental type conversions of potentially nested objects, I think TOML is the well-defined sucessor to .ini
    • lelanthran 2y ago
      > I feel like YAML is a spiritual successor to the .ini, since it shares a notable ideal of simple human readability/writability. It doesn't feel that way to me: it's neither simple to read nor to write. I suppose that that's a builtin problem due to tree representation, which is something that INI files were never expected to represent. TBH, I actually prefer the various tree representation workarounds used by INI files: using whitespace to indicate child nodes stops being readable once you have more than a screenful of children in a node.
• lelanthran 2y ago
  To be honest, I'm wondering why you are rating JSON higher than CSV. > Unlike XML or JSON, there isn't a document defining the grammar of well-formed or valid CSV files, There is, actually, RFC 4180 IIRC. > there are many flavours that are incompatible with each other in the sense that a reader for one flavour would not be suitable for reading the other and vice versa. "There are many flavours that deviate from the spec" is a JSON problem too. > you cannot tell programmatically whether line 1 contains column header names or already data (you will have to make an educated guess but there ambiguities in it that cannot be resolved by machine). Also a problem in JSON > Quoting, escaping, UTF-8 support are particular problem areas, Sure, but they are no more nor no less a problem in JSON as well.
  • IanCal 2y ago
    Have you had to work with csv files from the wild much? I'm not being snarky but what you're talking about is night and day to what I've experienced over the years. There aren't vast numbers of different JSON formats. There's practically one and realistically maybe two. Headers are in each line, utf8 has never been an issue for me and quoting and escaping are well defined and obeyed. This is because for datasets, almost exclusively, the file is machine written and rarely messed with. Csv files have all kinds of separators, quote characters, some parsers don't accept multi lines and some do, people sort files which mostly works until there's a multi line. All kinds of line endings, encodings and mixed encodings where people have combined files. I tried using ASCII record separators after dealing with so many issues with commas, semicolons, pipes, tabs etc and still data in the wild had these jammed into random fields. Lots of these things don't break when you hit the issue either, the parsers happily churn on with garbage data, leading to further broken datasets. Also they're broken for clients if the first character is a capital I.
    • lelanthran 2y ago
      Sure, I get your arguments and we're probably mostly in agreement, but in practice I see very few problems arising with using CSV. I mean, right now, the data interchange format between multiple working systems is CSV; think payment systems, inter-bank data interchange, ERP systems, CRM systems, billing systems ... the list goes on. I just recently had a coffee with a buddy who's a salesman for some enterprise system: of the most common enterprise systems we recently worked with (SAP type things, but on smaller scales), every single one of them had CSV as the standard way to get data between themselves and other systems. And yet, they work. The number of people uploading excel files to be processed or downloading excel files for local visualistation and processing would floor you. It's done multiple times a day, on multiple systems, in multiple companies. And yet, they work. I get your argument though - a JSON array of arrays can represent everything that CSV can, and is preferable to CSV, and is what I would choose when given the choice, but the issues with using that are not going to be fewer than issues with CSV using RFC1480.
• otabdeveloper4 2y ago
  People who say that CSV is "simpler" are talking about whatever format Excel exports. Also these people have only ever had to deal with the American Excel localization. So yeah, with the caveat of "only ever use Excel and only ever the American edition" CSV is pretty nice.
  • cgio 2y ago
    Also keeping in mind all the locales where comma is the decimal point…tsv for the world.
    • matwood 2y ago
      And all the 'simple' formats start failing when dealing with blocks of text.
  • mbnielsen 2y ago
    As someone living in a country where , is used as the decimal separator, I cannot begin to describe the number of times CSV data has caused me grief. This becomes especially common in an office environment where Excel is the de facto only data handling tool that most people can and will use. Here the behavior of loading data becomes specific to the individual machine and changes over time (e.g. when IT suddenly forces a reset of MS Office application languages to the local one). That said, I don't really know of any alternative that won't be handled even worse by my colleagues...
• cess11 2y ago
  While CSV isn't exactly grammared or standardised like XML I think if it as more schema:d than JSON. There might be data corruption or consistency issues, but there is implicitly a schema: every line is exactly n fields, and the first line might contain field names. When a JSON API turns out to have optional fields it usually shows through trial and error, and unlike CSV it's typically not considered a bug you can expect the API owner to fix. In CSV 'missing data' is an empty string rather than nulls or their cousins because missing fields aren't allowed, which is nice. I also like that I can write my own ad hoc CSV encoder in most programming languages that can do string concatenation, and probably also a suitable decoder. It helps a lot in some ETL tasks and debugging. Decent CSV also maps straight to RDBMS tables, if the database for some reason fails at immediate import (e.g. too strict expectations) into a newly created table it's almost trivial to write an importer that does it.
  • cgio 2y ago
    JSON is not schema’d per se and intentionally so. There’s jsonschema which has better expressiveness than inference of a tabular schema, as it can reflect relationships.
    • cess11 2y ago
      Sure. I have yet to come across a data source with JSON Schema, I'll develop an opinion of it when I do.
• jajko 2y ago
  There is no file format that works out of box under all extreme corner cases. You would think that ie XML-defined WSDL with XSD schema is well battle proven. I've encountered 2 years ago (and still dealing with that) WSDL from a major banking vendor that is technically valid, but no open source library in Java (from all languages) was able to parse it successfully or generate binding classes out of box. Heck, flat files can end up with extreme cases, just work enough with legacy banking or regulatory systems and you will see some proper shit. The thing is, any sort of critical integration needs to be battle tested and continuously maintained, otherwise it will eventually go bad, even a decade after implementation and regular use without issues.
  • hajile 2y ago
    What about S-expressions? Where do they break?
• immibis 2y ago
  XML is a pretty good markup language. Using XML to store structured data is an abuse of it. All the features that are useful for markup are not useful for structured data and only add to the confusion.
• no_wizard 2y ago
  INI was for a long time a seemingly preferable format in the Python community for configuration for a long time, as I recall it. Haven’t been a full time Python dev in sometime though, it seems TOML has supplanted that, but I remember thinking how interesting it was that Python had a built in INI parser and serializer
• realitysballs 2y ago
  They do specifically mention this: “No one owns CSV. It has no real specification (yes, I know about the controversial ex-post RFC 4180), just a set of rules everyone kinda agrees to respect implicitly. It is, and will forever remain, an open and free collective idea.”
  • Gormo 2y ago
    That's true of the vast majority of protocols that people use in real life to exchange date. We're using one of them right now, in fact.
  • thayne 2y ago
    They even seem to think it is a good thing. But I don't see how not having a bunch of implementations that can't agree on the specifics of a file/interchange format is a good thing. And being free and open is completely orthogonal. There are many proprietary formats that don't have a spec, and many open formats that do have a spec (like, say, json).
• sgarland 2y ago
  > Waiting for someone to write a love letter to the infamous Windows INI file format... Honestly, it’s fine. TOML is better if you can use it, but otherwise for simple applications, it’s fine. PgBouncer still uses INI, though that in particular makes me twitch a bit, due to discovering that if it fails to parse its config, it logs the failed line (reasonable), which can include passwords if it’s a DSN string.
  • flkenosad 2y ago
    I should write a love letter to JSON.
    • recursive 2y ago
      For as good as JSON is or is not, it's definitely not under-rated.
  • marcosdumay 2y ago
    Well, once you get over the fact that information on a TOML file can be out of order in any place, denominated by any mix of 3 different key encodings, and broken down in any random way... then yes, the rest of TOML is good.