6 ms·

The thing that puzzles me the most is, that people use _C_SV at all. Separation by comma, or any other member of the printable subset of ASCII in the first plac

The thing that puzzles me the most is, that people use _C_SV at all. Separation by comma, or any other member of the printable subset of ASCII in the first place. What this essentially boils down to is ambiguous in-band-signalling and a contextual grammar. ASCII had addressed the problem of separating entries ever since its creation: Separator control codes. There are: x01 SOH "Start of Heading" x02 STX "Start of Text" x03 ETX "End of Text" x04 EOT "End of Transmission" x1C FS "File Separator" x1D GS "Group Separator" x1E RS "Record Separator" x1F US "Unit Separator" You can use those just fine for exchanging data as you would using CSV, but without the ambiguities of separation characters and the need to quote strings. Heck if payload data is limited to the subset ASCII/UTF-8 without control codes you can just dump anything without the need for escaping or quoting. So my suggestion is simple. Don't use CSV or "P"SV (printable separated values). Use ASV (ASCII separated values).

davedx 9y ago
The article kind of addresses this. There are millions of spreadsheets and applications out in the wild that use CSV to communicate. Sure, if you're building some kind of system where you need to ingest data from one application from another application you control, then using a different interchange format like ASV is an option. But then people tend to use more powerful formats like JSON or XML.
- dspillett 9y ago
  > There are millions of spreadsheets and applications out in the wild that use CSV to communicate. That, and data in CSV format is human readable in any old text editor or even work processor which many use as a quick sanity check to make sure their data looks sane. A lot of editors will not display the ASCII control characters at all so the fields on the line get mashed together, or may even reject the file as containing what it considers to be unexpected characters.
  - steventhedev 9y ago
    More to the point, it's easy to export from Excel, which is a massive pain to load in any non-.NET language. While it's great to hope to use a well defined transport for machine-to-machine communication, it's exhausting to explain anything beyond CSV to Bob from sales.
baldfat 9y ago
I am happy when I see I can get data via CSV over the other delivery methods people use. My local school board prints out all their data and then scans them into a PDF, ugh. I had one vendor that on purpose made the data only available in forms that would take me 600+ lines of code to clean up in mangled ASCII format. I use CSV all the time when I am working with R. My data can come in the form of CSV, XLS, or PDF. Which would you want to work with? I can easily look at the data. I never touch my incoming data and my output is in reports, but CSV can be the easiest way to get data into a computer.
- mnx 9y ago
  Actually, XLS is not bad to work with, if you have a library for it. And it's well defined, unlike CSV. Insofar as I know, there's no way to make a CSV file that will open and show nicely in all popular versions of excel / google docs/ open office, especially across language settings. And a well formed XLS file will just work.
ajdlinux 9y ago
Give me a version of every standard text editor that can let me display and edit these ASV files when I just need to quickly hack something, and sure, I'll use it. CSV is directly editable in any text editor and manipulable by standard text processing tools, that's one of its key advantages.
- datenwolf 9y ago
  How about Vim? :help digraph :help digraph-table Feel free to implement mappings for quickly accessing these digraphs. Those pesky F<n> keys are perfect for this. Easy to reach, gets the job done.
  - emidln 9y ago
    For vim/evil, the following works. ASCII Name - Vim Insert - Visual Repr -------------------------------------------------- Start of Heading - Ctrl-v Ctrl-a - ^A Start of Text - Ctrl-v Ctrl-b - ^B End of Text - Ctrl-v Ctrl-c - ^C End of Transmission - Ctrl-v Ctrl-d - ^D File Separator - Ctrl-v Ctrl-\ - ^\ Group Separator - Ctrl-v Ctrl-] - ^] For insertion you can also always just Ctrl-v DDD or Ctrl-v xHH where DDD is the three digit decimal value or HH is two hex values of the ascii code.
- emidln 9y ago
  Vim and Emacs can. If your editor can't, maybe it should get with the (54 year old) program.
  - somecontext 9y ago
    For some context in case anyone was curious, Wikipedia believes that both vi and Emacs were (originally) written in 1976, which is 41 years ago. Unix dates from ~1969--1973 depending on your definition, which is 44--48 years ago. Accordingly, the reference to "54 year old" appears to be to the first standard as well as first commercial use of ASCII, in 1963. That first ASCII standard from 1963 specified eight "separators" simply named S0 through S7 at codes 0x18--0x1F. The 1965 update reused the first four for other purposes (eg cancel and escape) and labeled 0x1C--0x1F with the more descriptive names we now know.
burntsushi 9y ago
This comes up every single time someone mentions CSV. Without fail. The bottom line is that CSV is human readable and writable in plain text. If you start using fancy ASCII characters, then it becomes neither because our text editors don't support it.
- paulie_a 9y ago
  Honestly I dispute that it is "human readable". It is sort of legible but incredibly inconvenient to read or manually write. They might be slightly more convenient than tabular files such as ACH or DDF
  - paulddraper 9y ago
    More convenient than JSON, more consistent than YAML.
    - Piskvorrr 9y ago
      Consistent? What do you mean, "consistent"? Sometimes it's comma separated, sometimes it's semicolon separated (depending on the user's locale), sometimes it's separated by tabs (because it's a _C_SV file, yeah, no biggie), no content encoding hint (Unicode? Latin-1251? Win-1252? Nobody knows), not to mention you've written this comment under an article that shows just about the least consistent behavior ever. (Line breaks? Ahahahaha!) The only consistent thing about CSV is its ubiquity; other than that, it's a hairy, inconsistent mess that appears simple. (Source: having parsed millions of blobs that all identified themselves as CSV, despite being almost completely different in structure.)
      - paulddraper 9y ago
        > Sometimes it's comma separated, sometimes it's semicolon separated (depending on the user's locale), sometimes it's separated by tabs CSV is comma separated. [1] Valid YAML foo: bar baz Invalid YAML foo: "bar" baz Valid YAML foo: "bar baz" Invalid YAML foo: "bar baz Valid YAML foo: bar baz" [1] https://tools.ietf.org/html/rfc4180 https://tools.ietf.org/html/rfc4180
eli 9y ago
I don't think this necessarily addresses the security vulnerabilities in the article, which involve abusing the application reading the CSV, not the file format itself. If Excel decides that text between Start of Text and End of Text that begins with a "=" is a formula, then you're in the same spot.
sbierwagen 9y ago
If a dev is going to use a weirdo non-CSV data interchange format, they would just use XSLX or JSON or etc etc etc. "ASV" is only a viable option if you then also use your time machine to go back 40 years and make everyone start using it then.
thepompano 9y ago
This might create some integration-related hiccups with XML, as most ASCII control characters are forbidden per the XML 1.0/1.1 specs.

davedx 9y ago

The article kind of addresses this. There are millions of spreadsheets and applications out in the wild that use CSV to communicate. Sure, if you're building some kind of system where you need to ingest data from one application from another application you control, then using a different interchange format like ASV is an option. But then people tend to use more powerful formats like JSON or XML.

dspillett 9y ago
> There are millions of spreadsheets and applications out in the wild that use CSV to communicate. That, and data in CSV format is human readable in any old text editor or even work processor which many use as a quick sanity check to make sure their data looks sane. A lot of editors will not display the ASCII control characters at all so the fields on the line get mashed together, or may even reject the file as containing what it considers to be unexpected characters.
- steventhedev 9y ago
  More to the point, it's easy to export from Excel, which is a massive pain to load in any non-.NET language. While it's great to hope to use a well defined transport for machine-to-machine communication, it's exhausting to explain anything beyond CSV to Bob from sales.

baldfat 9y ago

I am happy when I see I can get data via CSV over the other delivery methods people use. My local school board prints out all their data and then scans them into a PDF, ugh. I had one vendor that on purpose made the data only available in forms that would take me 600+ lines of code to clean up in mangled ASCII format. I use CSV all the time when I am working with R. My data can come in the form of CSV, XLS, or PDF. Which would you want to work with? I can easily look at the data. I never touch my incoming data and my output is in reports, but CSV can be the easiest way to get data into a computer.

mnx 9y ago
Actually, XLS is not bad to work with, if you have a library for it. And it's well defined, unlike CSV. Insofar as I know, there's no way to make a CSV file that will open and show nicely in all popular versions of excel / google docs/ open office, especially across language settings. And a well formed XLS file will just work.

ajdlinux 9y ago

Give me a version of every standard text editor that can let me display and edit these ASV files when I just need to quickly hack something, and sure, I'll use it. CSV is directly editable in any text editor and manipulable by standard text processing tools, that's one of its key advantages.

datenwolf 9y ago
How about Vim? :help digraph :help digraph-table Feel free to implement mappings for quickly accessing these digraphs. Those pesky F<n> keys are perfect for this. Easy to reach, gets the job done.
- emidln 9y ago
  For vim/evil, the following works. ASCII Name - Vim Insert - Visual Repr -------------------------------------------------- Start of Heading - Ctrl-v Ctrl-a - ^A Start of Text - Ctrl-v Ctrl-b - ^B End of Text - Ctrl-v Ctrl-c - ^C End of Transmission - Ctrl-v Ctrl-d - ^D File Separator - Ctrl-v Ctrl-\ - ^\ Group Separator - Ctrl-v Ctrl-] - ^] For insertion you can also always just Ctrl-v DDD or Ctrl-v xHH where DDD is the three digit decimal value or HH is two hex values of the ascii code.
emidln 9y ago
Vim and Emacs can. If your editor can't, maybe it should get with the (54 year old) program.
- somecontext 9y ago
  For some context in case anyone was curious, Wikipedia believes that both vi and Emacs were (originally) written in 1976, which is 41 years ago. Unix dates from ~1969--1973 depending on your definition, which is 44--48 years ago. Accordingly, the reference to "54 year old" appears to be to the first standard as well as first commercial use of ASCII, in 1963. That first ASCII standard from 1963 specified eight "separators" simply named S0 through S7 at codes 0x18--0x1F. The 1965 update reused the first four for other purposes (eg cancel and escape) and labeled 0x1C--0x1F with the more descriptive names we now know.

burntsushi 9y ago

This comes up every single time someone mentions CSV. Without fail. The bottom line is that CSV is human readable and writable in plain text. If you start using fancy ASCII characters, then it becomes neither because our text editors don't support it.

paulie_a 9y ago
Honestly I dispute that it is "human readable". It is sort of legible but incredibly inconvenient to read or manually write. They might be slightly more convenient than tabular files such as ACH or DDF
- paulddraper 9y ago
  More convenient than JSON, more consistent than YAML.
  - Piskvorrr 9y ago
    Consistent? What do you mean, "consistent"? Sometimes it's comma separated, sometimes it's semicolon separated (depending on the user's locale), sometimes it's separated by tabs (because it's a _C_SV file, yeah, no biggie), no content encoding hint (Unicode? Latin-1251? Win-1252? Nobody knows), not to mention you've written this comment under an article that shows just about the least consistent behavior ever. (Line breaks? Ahahahaha!) The only consistent thing about CSV is its ubiquity; other than that, it's a hairy, inconsistent mess that appears simple. (Source: having parsed millions of blobs that all identified themselves as CSV, despite being almost completely different in structure.)
    - paulddraper 9y ago
      > Sometimes it's comma separated, sometimes it's semicolon separated (depending on the user's locale), sometimes it's separated by tabs CSV is comma separated. [1] Valid YAML foo: bar baz Invalid YAML foo: "bar" baz Valid YAML foo: "bar baz" Invalid YAML foo: "bar baz Valid YAML foo: bar baz" [1] https://tools.ietf.org/html/rfc4180 https://tools.ietf.org/html/rfc4180

eli 9y ago

I don't think this necessarily addresses the security vulnerabilities in the article, which involve abusing the application reading the CSV, not the file format itself. If Excel decides that text between Start of Text and End of Text that begins with a "=" is a formula, then you're in the same spot.

sbierwagen 9y ago

If a dev is going to use a weirdo non-CSV data interchange format, they would just use XSLX or JSON or etc etc etc. "ASV" is only a viable option if you then also use your time machine to go back 40 years and make everyone start using it then.

thepompano 9y ago

This might create some integration-related hiccups with XML, as most ASCII control characters are forbidden per the XML 1.0/1.1 specs.