3 ms·

> what's wrong with [CSV]? The encoding, line endings, and the used separators are random. Encoding/decoding behavior may be locale-dependent. E.g. if you use

by ahoge 12y ago

> what's wrong with [CSV]? The encoding, line endings, and the used separators are random. Encoding/decoding behavior may be locale-dependent. E.g. if you use Excel with a de-DE locale to create a CSV file, Excel with a en-US locale won't open it. The former uses ';' as separators and the latter expects ','. Since the format is so (seemingly) simple, there are millions of ad hoc implementations out there. Naturally, most of them are horribly broken. Fun fact: ASCII actually reserved control characters for this stuff. 1F is the "unit separator" and 1E is the "record separator". There is even a "group separator" (1D) and a "file separator" (1C). http://en.wikipedia.org/wiki/ASCII#ASCII_control_code_chart http://en.wikipedia.org/wiki/ASCII#ASCII_control_code_chart

alexvoda 12y ago
We need the equivalent of CommonMark. CommonCSV anyone?
- burntsushi 12y ago
  We have that already, RFC 4180: https://tools.ietf.org/html/rfc4180 https://tools.ietf.org/html/rfc4180 The problem is adoption. Most CSV parsers accept a superset of RFC 4180, because the only thing worse than not adhering to a good spec is tripping over CSV data in the wild.
  - ahoge 12y ago
    RFC 4180 does not specify the character encoding. This isn't a problem with JSON, XML, ASN.1, Protobufs, or Thrift. Anything other than CSV, really.
    - shared4you 12y ago
      Indeed it does, although not directly. From the RFC: > TEXTDATA = %x20-21 / %x23-2B / %x2D-7E So, it is a subset of ASCII whose hex codes must lie in the ranges given above. Basically, any printable ASCII character with the exception of 0x22 (= double quote) and 0x2C (=comma) which serve other purposes.
      - ahoge 12y ago
        From "3. MIME Type Registration of text/csv": Common usage of CSV is US-ASCII, but other character sets defined by IANA for the "text" tree may be used in conjunction with the "charset" parameter. Basically, you can specify that it's UTF-8 or whatever via the Content-Type response header. Setting this response header isn't something an encoder could do. A CSV parsing library also shouldn't be responsible for loading the CSV file via HTTP. In most cases, CSV files aren't directly fetched via HTTP anyways. Anyhow, ASCII is utter nonsense. We aren't living in an ASCII world. Even the Americans don't do that. An ASCII-only format is completely useless.
deafbybeheading 12y ago
>Fun fact: ASCII actually reserved control characters for this stuff. 1F is the "unit separator" and 1E is the "record separator". There is even a "group separator" (1D) and a "file separator" (1C). Which means you can't safely use it for arbitrary data (since your records themselves could contain these separators). Most of the time that doesn't matter; sometimes it does.
- deleted 12y ago
  [deleted]
- ahoge 12y ago
  You can't use these control characters in CSV either. The grammar from RFC 4180 excludes the 0x00 to 0x1F range.

by ahoge 12y ago

alexvoda 12y ago

We need the equivalent of CommonMark. CommonCSV anyone?

burntsushi 12y ago
We have that already, RFC 4180: https://tools.ietf.org/html/rfc4180 https://tools.ietf.org/html/rfc4180 The problem is adoption. Most CSV parsers accept a superset of RFC 4180, because the only thing worse than not adhering to a good spec is tripping over CSV data in the wild.
- ahoge 12y ago
  RFC 4180 does not specify the character encoding. This isn't a problem with JSON, XML, ASN.1, Protobufs, or Thrift. Anything other than CSV, really.
  - shared4you 12y ago
    Indeed it does, although not directly. From the RFC: > TEXTDATA = %x20-21 / %x23-2B / %x2D-7E So, it is a subset of ASCII whose hex codes must lie in the ranges given above. Basically, any printable ASCII character with the exception of 0x22 (= double quote) and 0x2C (=comma) which serve other purposes.
    - ahoge 12y ago
      From "3. MIME Type Registration of text/csv": Common usage of CSV is US-ASCII, but other character sets defined by IANA for the "text" tree may be used in conjunction with the "charset" parameter. Basically, you can specify that it's UTF-8 or whatever via the Content-Type response header. Setting this response header isn't something an encoder could do. A CSV parsing library also shouldn't be responsible for loading the CSV file via HTTP. In most cases, CSV files aren't directly fetched via HTTP anyways. Anyhow, ASCII is utter nonsense. We aren't living in an ASCII world. Even the Americans don't do that. An ASCII-only format is completely useless.

deafbybeheading 12y ago

>Fun fact: ASCII actually reserved control characters for this stuff. 1F is the "unit separator" and 1E is the "record separator". There is even a "group separator" (1D) and a "file separator" (1C). Which means you can't safely use it for arbitrary data (since your records themselves could contain these separators). Most of the time that doesn't matter; sometimes it does.

deleted 12y ago
[deleted]
ahoge 12y ago
You can't use these control characters in CSV either. The grammar from RFC 4180 excludes the 0x00 to 0x1F range.