7 ms·

Stop this horrible genre now! As discussed in the recent video programming post (https://news.ycombinator.com/item?id=13259686 https://news.ycombinator.com/item

by vortico 10y ago

Stop this horrible genre now! As discussed in the recent video programming post (https://news.ycombinator.com/item?id=13259686 https://news.ycombinator.com/item?id=13259686), this neither helps the beginner because it is unnecessarily overwhelming, nor the expert because he has already read https://tools.ietf.org/html/rfc4180 https://tools.ietf.org/html/rfc4180 and is aware of these problems with bad CSV parsers/emitters. In the intermediate case, it could be useful for someone maintaining a CSV parser or emitter who is not aware of a few of these points and wants 100% compatibility with all CSV parsers/emitters in existence, but the list style is confusing grammatically and offers no counterexamples or suggestions for improvement. With that said, if you are interested in using CSV for encoding and decoding in your own environment, I suggest a new file standard called μTSV. 1. All μTSV files are UTF-8. 2. All values are delimited by "\t", and all lines must end with "\n" (including the last one). 3. If you want to use a "\t" or "\n" character in your value, tough luck---use JSON.

deleted 10y ago
[deleted]
joshuata 10y ago
Many of these problems could be avoided if we used the character codes for unit (31) and record (30) separator. This would avoid any problems with tabs or line feeds, as well as better matching the intended semantics of ASCII/UTF-8
- kedean 10y ago
  That's suggested often, and the obvious rebuttal is that CSV is intended to be easily edited and relatively human-friendly. The unit/record characters are not on most(all?) keyboards, so editing the file in a text editor becomes pretty much impossible.
  - stouset 10y ago
    As a counterpoint, nobody seems to be editing CSV files by hand any more. It appears at this point to be an interchange format between different spreadsheets, row-oriented databases, etc.
    - kedean 10y ago
      I promise they're still used by hand all the time, usually as a form of supplying data to a program in cases where a database is just too much. CSV is still a very common base for data-driven unit tests.
donatj 10y ago
I mostly just thought it was fun to write. Put my pain down on a page, get a little bit of catharsis. I actually had written out some "counterexamples or suggestions" but decided to keep it terse and with the general style of the other similar posts. I was not aware of the scorn for the style, I just knew I enjoy reading them. As opposed to your μTSV may I suggest DSV. The Art of Unix Programming makes a strong case for how it's superior to CSV [1] 1. http://www.catb.org/esr/writings/taoup/html/ch05s02.html#id2901882 http://www.catb.org/esr/writings/taoup/html/ch05s02.html#id2...
- vram22 10y ago
  Heh, by coincidence, I had recently written two posts about DSV, and had mentioned the same TAOUP DSV link you gave above: Processing DSV data (Delimiter-Separated Values) with Python: https://jugad2.blogspot.in/2016/11/processing-dsv-data-delimiter-separated.html https://jugad2.blogspot.in/2016/11/processing-dsv-data-delim... The example program in the post lets you use either a "-c delimiter_char" or a "-n delimiter_ASCII_code" to specify the delimiter for your DSV data being processed by it. and [xtopdf] Publish DSV data (Delimiter-Separated Values) to PDF: https://jugad2.blogspot.in/2016/12/xtopdf-publish-dsv-data-delimiter.html https://jugad2.blogspot.in/2016/12/xtopdf-publish-dsv-data-d...
  - groovy2shoes 10y ago
    You're both wrong. EDN is the best. https://github.com/edn-format/edn https://github.com/edn-format/edn
    - vram22 10y ago
      >You're both wrong. Bah. How am I wrong when I haven't even claimed DSV to be the best? Just quoting that section in ESR's book does not imply I endorse or agree with it. Logic ...
      - vram22 10y ago
        Plus, just providing that EDN link is hardly proof that it is the best ... And further, "best" in many cases is subjective. You want to claim EDN is the best, you lay out an objective proof.
mnarayan01 10y ago
Using the "μ" in a header (e.g. Content-Type) is at least historically compliant as it exists in ISO-8859-1...but doing so needlessly would be insane.
- pjc50 10y ago
  Rather like the space in "Program Files", the "μ" would serve the function of making sure that the caller has at least some working UTF-8.
  - mnarayan01 10y ago
    AFAIK using the UTF-8 encoding of "μ" would not be standard compliant.
lisivka 10y ago
Much easier solution is to develop a standard for meta-field, which will describe CSV features used. Example: #!CSV: escape=:\\', quote='"', quotealg='c-like', separator='\t', eof='\n\r', comment='#', encoding='utf-8' foo bar baz Such standard (Lets call it mCSV) can be incorporated into existing solutions. Moreover, existing files can be converted from/to a proprietary flavor of CSV to/from metainized CSV.
- seagreen 10y ago
  Self-describing data is generally a bad idea. If you can't get programs to agree on the original document format, what makes you think you'll be able to get them to agree on this meta format? EDIT: The right solution is the simple one. Follow RFC4180 exactly. Reject everything that doesn't match it.
  - lisivka 10y ago
    CSV is format for information interchange. General rule for II formats is: be strict in what you emit, be liberal in what you accept. So, yes, follow RFC4180 strictly in your own code, but be able to read CSV files produced by others too. Single line of metainformation can greatly reduce confusion and is easy to throw out, when it is unnecessary.
    - x1798DE 10y ago
      This rule sounds nice, but you end up with a bunch of malformed documents out there because people emitting it never get feedback about what they are doing wrong. I think I've said this in another thread, but I think rejecting malformed documents even though you could "figure out what they meant" is the equivalent of not telling people when they have food on their face. I think the right pattern for something like this is to complain loudly when you are given a malformed document, then say, "If you really want me to parse this malformed document, use --force and I'll try to make some guesses, but really you should just ask for a well-formed document."
      - 10y ago
azernik 10y ago
Yeah, the genre was good for non-standardized things like names and (to a lesser extent) time, but file formats have specs and reference implementations. Refer to those.
- duskwuff 10y ago
  Good file formats have specifications and reference implementations. By this standard, CSV is a terrible format.
  - webmaven 10y ago
    ObNitPick: "Standard" is a bit confusing in this context. I suggest using "measure", "criterion", or something similar.
Tloewald 10y ago
If you're going to replace CSV then why not use the ASCII record and field delimiter characters and avoid using characters like tab and new line which are in band and actually useful? And if you're using tabs as field delimiters then why do you need quotation marks in the mix?
amyjess 10y ago
In practice, I made heavy use of this format at my last company. I was dealing with lots and lots of Google Sheets where I knew there wouldn't be any tabs or returns in the columns, and my code was all ad-hoc stuff that would never see production, so I'd just read in files, line-by-line, split on "\t", and bam instant parser.
julg 10y ago
PSV (https://github.com/jgis/psv-spec https://github.com/jgis/psv-spec) allows "\t" and "\n" in values.

by vortico 10y ago

deleted 10y ago

[deleted]

joshuata 10y ago

Many of these problems could be avoided if we used the character codes for unit (31) and record (30) separator. This would avoid any problems with tabs or line feeds, as well as better matching the intended semantics of ASCII/UTF-8

kedean 10y ago
That's suggested often, and the obvious rebuttal is that CSV is intended to be easily edited and relatively human-friendly. The unit/record characters are not on most(all?) keyboards, so editing the file in a text editor becomes pretty much impossible.
- stouset 10y ago
  As a counterpoint, nobody seems to be editing CSV files by hand any more. It appears at this point to be an interchange format between different spreadsheets, row-oriented databases, etc.
  - kedean 10y ago
    I promise they're still used by hand all the time, usually as a form of supplying data to a program in cases where a database is just too much. CSV is still a very common base for data-driven unit tests.

donatj 10y ago

I mostly just thought it was fun to write. Put my pain down on a page, get a little bit of catharsis. I actually had written out some "counterexamples or suggestions" but decided to keep it terse and with the general style of the other similar posts. I was not aware of the scorn for the style, I just knew I enjoy reading them. As opposed to your μTSV may I suggest DSV. The Art of Unix Programming makes a strong case for how it's superior to CSV [1] 1. http://www.catb.org/esr/writings/taoup/html/ch05s02.html#id2901882 http://www.catb.org/esr/writings/taoup/html/ch05s02.html#id2...

vram22 10y ago
Heh, by coincidence, I had recently written two posts about DSV, and had mentioned the same TAOUP DSV link you gave above: Processing DSV data (Delimiter-Separated Values) with Python: https://jugad2.blogspot.in/2016/11/processing-dsv-data-delimiter-separated.html https://jugad2.blogspot.in/2016/11/processing-dsv-data-delim... The example program in the post lets you use either a "-c delimiter_char" or a "-n delimiter_ASCII_code" to specify the delimiter for your DSV data being processed by it. and [xtopdf] Publish DSV data (Delimiter-Separated Values) to PDF: https://jugad2.blogspot.in/2016/12/xtopdf-publish-dsv-data-delimiter.html https://jugad2.blogspot.in/2016/12/xtopdf-publish-dsv-data-d...
- groovy2shoes 10y ago
  You're both wrong. EDN is the best. https://github.com/edn-format/edn https://github.com/edn-format/edn
  - vram22 10y ago
    >You're both wrong. Bah. How am I wrong when I haven't even claimed DSV to be the best? Just quoting that section in ESR's book does not imply I endorse or agree with it. Logic ...
    - vram22 10y ago
      Plus, just providing that EDN link is hardly proof that it is the best ... And further, "best" in many cases is subjective. You want to claim EDN is the best, you lay out an objective proof.

mnarayan01 10y ago

Using the "μ" in a header (e.g. Content-Type) is at least historically compliant as it exists in ISO-8859-1...but doing so needlessly would be insane.

pjc50 10y ago
Rather like the space in "Program Files", the "μ" would serve the function of making sure that the caller has at least some working UTF-8.
- mnarayan01 10y ago
  AFAIK using the UTF-8 encoding of "μ" would not be standard compliant.

lisivka 10y ago

Much easier solution is to develop a standard for meta-field, which will describe CSV features used. Example: #!CSV: escape=:\\', quote='"', quotealg='c-like', separator='\t', eof='\n\r', comment='#', encoding='utf-8' foo bar baz Such standard (Lets call it mCSV) can be incorporated into existing solutions. Moreover, existing files can be converted from/to a proprietary flavor of CSV to/from metainized CSV.

seagreen 10y ago
Self-describing data is generally a bad idea. If you can't get programs to agree on the original document format, what makes you think you'll be able to get them to agree on this meta format? EDIT: The right solution is the simple one. Follow RFC4180 exactly. Reject everything that doesn't match it.
- lisivka 10y ago
  CSV is format for information interchange. General rule for II formats is: be strict in what you emit, be liberal in what you accept. So, yes, follow RFC4180 strictly in your own code, but be able to read CSV files produced by others too. Single line of metainformation can greatly reduce confusion and is easy to throw out, when it is unnecessary.
  - x1798DE 10y ago
    This rule sounds nice, but you end up with a bunch of malformed documents out there because people emitting it never get feedback about what they are doing wrong. I think I've said this in another thread, but I think rejecting malformed documents even though you could "figure out what they meant" is the equivalent of not telling people when they have food on their face. I think the right pattern for something like this is to complain loudly when you are given a malformed document, then say, "If you really want me to parse this malformed document, use --force and I'll try to make some guesses, but really you should just ask for a well-formed document."
    - 10y ago

azernik 10y ago

Yeah, the genre was good for non-standardized things like names and (to a lesser extent) time, but file formats have specs and reference implementations. Refer to those.

duskwuff 10y ago
Good file formats have specifications and reference implementations. By this standard, CSV is a terrible format.
- webmaven 10y ago
  ObNitPick: "Standard" is a bit confusing in this context. I suggest using "measure", "criterion", or something similar.

Tloewald 10y ago

If you're going to replace CSV then why not use the ASCII record and field delimiter characters and avoid using characters like tab and new line which are in band and actually useful? And if you're using tabs as field delimiters then why do you need quotation marks in the mix?

amyjess 10y ago

In practice, I made heavy use of this format at my last company. I was dealing with lots and lots of Google Sheets where I knew there wouldn't be any tabs or returns in the columns, and my code was all ad-hoc stuff that would never see production, so I'd just read in files, line-by-line, split on "\t", and bam instant parser.

julg 10y ago

PSV (https://github.com/jgis/psv-spec https://github.com/jgis/psv-spec) allows "\t" and "\n" in values.