5 ms·

I think it’s because csv is good enough. All the standards I’ve seen haven’t been worth the effort to implement. So since csv, with all its flaws, is good enou

• quesera 5y ago
  I think CSV is crappy because commas are so common in real data. For almost all scenarios I've had to work with, I'd have been perfectly happy with TSV where literal Tab was a disallowed character. No escaping histrionics required.
  • hermitcrab 5y ago
    Not being able to include Tabs and Carriage Returns in your data can be a problem though.
    • deleted 5y ago
      [deleted]
    • mjevans 5y ago
      Depending on the library, (double) quoted fields _may_ be able to contain carriage returns (0x13) and/or line feeds (0x10). Sometimes it works, sometimes it doesn't, can be a real pain in the butt.
    • justin_oaks 5y ago
      The ideal would be a format that avoids characters that are common in data, but still allows a way to encode rare characters that are part of data. I think that was the motivation for multi-character delimiters like <% %> or {{ }} in templating languages.
    • pishpash 5y ago
      Unicode just needs a single special delimiter character that is only used as a delimiter.
      • GekkePrutser 5y ago
        Ideally a separate row and a column delimiter
        • BenjiWiebe 5y ago
          As I understand it, ASCII already has this. And isn't every ASCII codepoint valid UTF-8?
  • prepend 5y ago
    I prefer commas because I can see them over tabs. I spend zero time escaping commas because the libraries and read and write with (usually pandas but pretty much everything) escape for me. So unless I’m manually building csvs myself it’s a non-issue and certainly not histrionics.
    • thayne 5y ago
      Newlines are pretty common in text too, depending on your data. What do you do about those?
    • mbreese 5y ago
      I prefer tabs because my data can often have commas in it. Seeing tabs isn’t an issue as I also have invisible characters visible in all of my editors. But having your delimiter not be allowed in the record (as in \t is disallowed), makes parsing so much easier. CSV is a bear to parse because you have to read each value from a buffer to handle the quoting. line.strip().split(‘\t’) Is so handy. Batteries included are one thing, but I don’t want to pull in a library just to parse CSV files.
      • tech2 5y ago
        Given your example looks very much like python you could use the CSV module which is built into the standard library and handles all of this for you in a standards compliant manner. You really do want to use a library to parse CSV since there are a number of corner cases. For example, your example code may not read a whole row since rows can have newlines in them.
        • mbreese 5y ago
          > You really do want to use a library to parse CSV Right… you absolutely need to use a library. I’ve written the parsers for CSV which handle the edge cases (at least RFC edge cases). But I don’t want to include a library to be able to do it. Most of my scripts are small, and adding libraries makes them more difficult to move around. So, I shy away from CSV as a format as a result. I’m okay with not allowing new lines and tabs in my fields. It’s a valid trade off for my data. Yes, many of my scripts are Python, but not all, so adding new libraries for different languages is more difficult for me to remember as opposed to just splitting a line by a record separator. Also, many of my files require having comments in the file’s header. Another reason why CSV can’t be easily used, as this isn’t part of the RFC.
  • nerdponx 5y ago
    Pipes are usually pretty safe too.
    • qayxc 5y ago
      Depends entirely on context. I did consulting in wholesale and supplier data very often contained pipes as content separators (e.g. for product categories, sizes, etc.).
• hedora 5y ago
  CSV is fine. If you care about edge cases, implement RFC 4180: https://www.rfc-archive.org/getrfc.php?rfc=4180 https://www.rfc-archive.org/getrfc.php?rfc=4180 If you don't, then split each line on ",". Problem solved. If you find tab delimited easier to read (as I do), then check out the IANA spec for TSV files: https://www.iana.org/assignments/media-types/text/tab-separated-values https://www.iana.org/assignments/media-types/text/tab-separa... It's easier to parse than CSV. Unfortunately, you have to decide how to handle newlines and tabs yourself (probably as \n, \t, with \\ for backslashes).
  • makeworld 5y ago
    > If you don't, then split each line on ",". Problem solved. And the millionth bad CSV parser is born.
    • naikrovek 5y ago
      yeah the "CSV is fine" bit scared me. people think they know what everyone is doing and why; in reality (where these problems actually need to be solved) no one knows.
    • hedora 5y ago
      More than the millionth, I'm sure. If you know your input doesn't have commas (for example, because you generated it) and the code is throwaway, then splitting on comma is fine. But, yeah, I prefer tab delimited, since "no commas" is painful, and data that contains quotes leads to scenarios where the crappy CSV I generated can't be parsed by RFC 4180 parsers.
  • gnulinux 5y ago
    > If you don't, then split each line on ",". Problem solved. What? Real data contains '"' quote character too. That doesn't work.
    • hedora 5y ago
      Read the RFC I linked. It handles all the characters.
• kall 5y ago
  Yeah I agree. Just the combination of opening in excel on double click and being dead simple to spit out from any software makes it a winner. It helps that it's usable in plain text too. If it would have used ANYTHING other than half the world's decimal separator as a value separator, it would be a no brainer. If it had been .psv (pipe separated values) the edge cases would be so few that we could just ignore them.
  • hedora 5y ago
    Gnumeric (an open source spreadsheet) has a great dialog box that pops up and lets you tell it exactly how to parse the tab delimited / csv file you just handed it. It even has a preview of what your parsing options will do to the file you're opening. Also, "gnumeric foo.csv" does the right thing from the command line.
• Diggsey 5y ago
  CSV has more problems than just quoting: one problem is that excel is so commonly used to open CSV files that various weirdnesses in excel have essentially become part of the CSV format. For example, if a value looks like a formula, then the only way to get excel to treat it as text is to put a single quote in front of it, "'=not a formula". This in turn means that its common for values in a CSV to begin with a single quote, but for that single quote to not be intended as part of the value. I think a decent format could be like CSV, but tab separated, where the header row is mandatory, and with explicit support for data types. Tabs and newlines (along with any other unicode character) can be included via backslash escaping. eg. (imagine the spaces are tabs) int:Id str:Name json:Attributes dyn:Other 1 John\tSmith {"age": 30} int:42 2 Bill {"age": 40} str:Hello Applications can use their own type specifiers (eg. "formula:") to avoid needing to introduce special rules. The value of a field (before interpretation) is the exact string written in the file. The type specifier indicates how that string should be interpreted as a typed value by the application. For example, the `str` type specifier says that backslash escape sequences should be converted to the corresponding characters when interpreting the value. This has the following properties: - Machine and human readable and writable. - Does not place artificial restrictions on values. - Can be extended as needed with new type specifiers. - Can be processed losslessly even if an application does not understand all type specifiers in use. - Is not ambiguous about whether data is part of a header or not. - Can store common data patterns (eg. all values in a column have the same type) efficiently without being restrictive (values can still have different types even in the same column). - Can easily jump to a line or field (newlines and tabs not allowed except as separators - they must be escaped when in a value). - Easy to remember.
  • prepend 5y ago
    There’s lots of protocols like what you describe and I’ve written a few myself. But they are a pain to teach to other people. Csv isn’t perfect, but I’d rather live with its weaknesses than use anything else.
  • gnulinux 5y ago
    I like the format description, thanks. Sounds like something I might implement some day for fun! The reason I still don't like this is because most of the time I'm sharing csv is to a jira ticket or an email, or to customer and customer insists it needs to be csv. When customer wants csv there is nothing I can do. But in other cases I see this CSV file. So it's visual clarity is important. CSV has all the problems you listed, but TSV (and similar, such as your format) has the short coming of relying on an invisible separator. And worse, the width of tab is undefined, undecided. I don't know what the best trade-of... Maybe we need better tooling so frontends show CSV/TSV et al as separated by cells.
• GuB-42 5y ago
  I live in France and let me tell you: CSV is not good enough. What about France you say? The decimal point is a coma, and to avoid confusion, the list separator is a semicolon. So "1.5, 1.6" becomes "1,5; 1,6" when localized. And if you think it is terrible for CSVs, well, it is worse than that. If we all used RFC4180 consistently, it would work, but of course, that would be too simple. Some software insist on localization. So unless you need to know the software and locale that goes with that CSV. In the most simple case: you can treat both comas and semicolons as separators and it will work for reading, for writing, you have to pick one that the target software hopefully understands. Hell starts when you have decimal numbers, because sometimes, you get exported CSVs where the coma is used both as a decimal point and as a separator. It makes the number ambiguous and your file becomes useless. It can result in data loss (personal experience). For example "1,2,3" may be [1.2, 3] or [1, 2.3]. And even without that, you still have to guess if you are using points and comas or comas and semicolons. And of course, there the quoting/escaping issues every country has. Good thing the RFC has clear rules, I heard you can even find software where they are properly implemented. Of course, in France we also have the added bonus of questioning whether or not comas should be quoted since they are not the localized separator. And character encoding? Of course it is a mess, as with every text file, but CVS makes no effort to help, is it UTF-8? Latin1? I have seen both, and I have seen files mangled because they were opened with the wrong encoding. Also, line endings should be CRLF, for those who care. CSV as defined by the RFC is good enough, the problem is how it is used in practice.
• WorldMaker 5y ago
  I don't think it's necessarily that CSV is good enough but that its the only ASCII tabular format that "double clicks into Excel" by default. Excel supports a lot more tabular formats these days, but you have to search the ribbon for their importers. It's JSON support is actually surprisingly good, for instance. Sometimes I think the best thing that the Excel team could do to make the state of the world of tabular formats for us better as developers is just to add a bunch of new file extensions that you can rename files to. For instance, if you could just rename a JSON file to end with something like .XLJSON and a Business Analyst or Accountant can just double click to open in Excel, that would open up a lot more nicer tabular format options beyond CSV with better developer ergonomics.