4 ms·

None of this stuff works. Whenever CSV comes up I always feel a bit sad. ASCII includes a set of four out-of-band delimiters[0] that can be used instead of sil

None of this stuff works. Whenever CSV comes up I always feel a bit sad. ASCII includes a set of four out-of-band delimiters[0] that can be used instead of silly formats like CSV that use in-band delimiters which necessitate complicated quoting rules. You can't just treat CSV as text. It's not and woe betide you if you ever use this stuff in a script instead of using a proper CSV parsing tool. If we used the ASCII delimiters instead it would be possibly to treat it as text and stuff like this would work. [0] https://en.wikipedia.org/wiki/Delimiter#ASCII_delimited_text https://en.wikipedia.org/wiki/Delimiter#ASCII_delimited_text

chrishill89 3y ago
I agree. And some tools support it, e.g. Miller.[1] There are a lot of power tools for CSV. All they need to do to support this format is to support it as input and output. It should be possible for example to add support to Csvkit[2] as input/output ("format"). Just based on HN comments alone (these things always come up) there is a decent demand for this kind of format. And there are a few implementations out there. But the implementations often look more like standards or proofs of concepts rather than full-fledged implementations. (A full-fledged implementation should probably provide functions/programs to convert to and from CSV as described in RFC-4180.) We could make this happen. [1] https://news.ycombinator.com/item?id=31258316 https://news.ycombinator.com/item?id=31258316 [2] https://csvkit.readthedocs.io/en/latest/scripts/in2csv.html https://csvkit.readthedocs.io/en/latest/scripts/in2csv.html
ykonstant 3y ago
Preach; the POSIX standard also understands this, and partitions control characters into portable and non-portable ones; the ASCII field/record/group delimiters are designated as non-portable, and therefore can be safely used to delimit portable character data. The fact that the whole file is then non-portable is a feature, not a bug; if you need to pack an entire table into a field, encode it in some portable encoding (eg base64) and add a decoder field. The "arguments" against this are absurd; mainly that the resulting file is not plain text and cannot be read/modified immediately by plain text editors. This ignores the fact that plain text editors already deal with control characters because they have to decide what to do with <tab>, and makes a mountain out of a molehill when the in-band horrors of escaping delimiters create trouble every single day.
- chrishill89 3y ago
  > The fact that the whole file is then non-portable is a feature, not a bug; if you need to pack an entire table into a field, encode it in some portable encoding (eg base64) and add a decoder field. I guess you could: 1. Use unit-separator and record-separator for text 2. Use group-separator for (1) 3. Use file-separator for (2) > The "arguments" against this are absurd; mainly that the resulting file is not plain text and cannot be read/modified immediately by plain text editors. This ignores the fact that plain text editors already deal with control characters because they have to decide what to do with <tab>, and makes a mountain out of a molehill when the in-band horrors of escaping delimiters create trouble every single day. Could not agree more.
hermitcrab 3y ago
Yes, has been discussed before. https://news.ycombinator.com/item?id=31220841 https://news.ycombinator.com/item?id=31220841 The only thing needed for this to work really well for text editors to treat the end of record character like a carriage return (I'm not aware of any that do).

chrishill89 3y ago

I agree. And some tools support it, e.g. Miller.[1] There are a lot of power tools for CSV. All they need to do to support this format is to support it as input and output. It should be possible for example to add support to Csvkit[2] as input/output ("format"). Just based on HN comments alone (these things always come up) there is a decent demand for this kind of format. And there are a few implementations out there. But the implementations often look more like standards or proofs of concepts rather than full-fledged implementations. (A full-fledged implementation should probably provide functions/programs to convert to and from CSV as described in RFC-4180.) We could make this happen. [1] https://news.ycombinator.com/item?id=31258316 https://news.ycombinator.com/item?id=31258316 [2] https://csvkit.readthedocs.io/en/latest/scripts/in2csv.html https://csvkit.readthedocs.io/en/latest/scripts/in2csv.html

ykonstant 3y ago

Preach; the POSIX standard also understands this, and partitions control characters into portable and non-portable ones; the ASCII field/record/group delimiters are designated as non-portable, and therefore can be safely used to delimit portable character data. The fact that the whole file is then non-portable is a feature, not a bug; if you need to pack an entire table into a field, encode it in some portable encoding (eg base64) and add a decoder field. The "arguments" against this are absurd; mainly that the resulting file is not plain text and cannot be read/modified immediately by plain text editors. This ignores the fact that plain text editors already deal with control characters because they have to decide what to do with <tab>, and makes a mountain out of a molehill when the in-band horrors of escaping delimiters create trouble every single day.

chrishill89 3y ago
> The fact that the whole file is then non-portable is a feature, not a bug; if you need to pack an entire table into a field, encode it in some portable encoding (eg base64) and add a decoder field. I guess you could: 1. Use unit-separator and record-separator for text 2. Use group-separator for (1) 3. Use file-separator for (2) > The "arguments" against this are absurd; mainly that the resulting file is not plain text and cannot be read/modified immediately by plain text editors. This ignores the fact that plain text editors already deal with control characters because they have to decide what to do with <tab>, and makes a mountain out of a molehill when the in-band horrors of escaping delimiters create trouble every single day. Could not agree more.

hermitcrab 3y ago

Yes, has been discussed before. https://news.ycombinator.com/item?id=31220841 https://news.ycombinator.com/item?id=31220841 The only thing needed for this to work really well for text editors to treat the end of record character like a carriage return (I'm not aware of any that do).