5 ms·

There are similar problems with csv. A deceptively simple format with many variants to handle various edge cases.

by slaymaker1907 6y ago

lstamour 6y ago
Yes, except it's easy to debug a csv problem because it's generally text in some encoding you can hopefully identify without looking at specific byte offsets in a hex editor: http://fileformats.archiveteam.org/wiki/Tape_Archive#Identification http://fileformats.archiveteam.org/wiki/Tape_Archive#Identif... In fact, one of the biggest questions I've still yet to answer is whether Docker or OCI have ever clarified just which TAR format must be followed? As far as I can tell, it's a mix of "whatever library X supports"? https://www.cyphar.com/blog/post/20190121-ociv2-images-i-tar https://www.cyphar.com/blog/post/20190121-ociv2-images-i-tar for example. I'm not sure I want a more complicated format, but I do wish somebody would pick a new file extension for their next use of tar so we could clearly differentiate one tar format from another the next time someone builds on top of simple tar files for their binary distribution, etc.
brazzy 6y ago
How many, though? Off the top of my head, you need to specify text encoding, line endings, a quoting mechanism in case a field contains the separator, and an escaping mechanism in case a field contains multiple lines or the quote character. If you have those specified and everyone follows the specification, there shouldn't be any problems, right? Woe be unto you if you have to process input data where those are not used consistently, of course.
- roberto 6y ago
  First, there's no specification. Second, this: http://georgemauer.net/2017/10/07/csv-injection.html http://georgemauer.net/2017/10/07/csv-injection.html
  - brazzy 6y ago
    Quote from the site you linked to: "So who’s fault is all of this anyways? Well it’s not the CSV format’s." And what I wrote before is basically all you need for a specification. Yes, it sucks that there is no fixed one that everyone uses.
- probably_wrong 6y ago
  > If you have those specified and everyone follows the specification, there shouldn't be any problems, right? Well, yes, but I feel that that's the point at which CSV definitely earns its "ill-specified format" hat. Given that there is no way to extract the file specification from the file itself, you can only follow the specification if someone tells you the specification beforehand. You can even use the "wrong" specification and don't notice it because your specification overlaps with the "true" specification in the types of data you're typically seeing (looking at you, optional quotes).
- tinus_hn 6y ago
  All you need to do is specify all these unspecified things and it’ll be fine. You forgot the localized decimal separator symbol.
  - VMG 6y ago
    if you start going down that route you will have to specify the format for dates, network addresses, postal addresses, phone numbers... it never ends
    - tinus_hn 6y ago
      Either you accept that CSV can’t reliably contain fractional numbers or you specify the decimal separator.
      - brazzy 6y ago
        CSV doesn't contain numbers at all; it contains strings. How those should be parsed into numbers is not part of the format, unless by CSV you additionally mean "to be imported by Excel". Which is certainly a common use case but not always the case.
        tinus_hn 6y ago
        By this line of thinking, any file type is specified, because the specification of a file is a sequence of bytes. Arguably right, practically useless.
        brazzy 6y ago
        The distinction between file format and payload format is perfectly normal and not at all useless. All file formats have limits for how much they define types for the payload. Most have very limited types. JSON for example knows objects, arrays, strings, numbers, booleans and null. It's not that big a difference to have a format that has only strings.
- Denvercoder9 6y ago
  > Off the top of my head, you need to specify text encoding, line endings, a quoting mechanism in case a field contains the separator, and an escaping mechanism in case a field contains multiple lines or the quote character. You also need to specify the separator itself, which despite the name isn't always a comma, but can also be a semicolon or (less commonly) a tab.
  - medstrom 6y ago
    The IANA standard for TSV files [1] bans tab characters inside of fields. It's a convenient assumption, so if I'm writing a quick way to sync some data to disk, I choose TSV because the functions can be simpler. https://www.iana.org/assignments/media-types/text/tab-separated-values https://www.iana.org/assignments/media-types/text/tab-separa...
    - mFixman 6y ago
      To this day I don't understand why standard table formats don't use the standard ASCII unit and record separators. They are there, they are convenient, and they don't require any visible character sacrifices.
      - marcthe12 6y ago
        I gues lack of program support. But yeas. That would have been great
- dathinab 6y ago
  Besides inconsistencies, that a non defined quoting and escaping mechanism is already quite painful. Let's look at following single column single line csv: '\0' Depending on the source this might be: - quote back-slash zero quote - zero - back-slash zero - a byte with the numeric value of 0 - quote zero quote - quote back-slash zero quote - quote byte-with-value-0 quote - malformed and should be ignored Worse the same application might mix different of this quoting mechanisms for different columns or even in the same column. Normally it shouldn't but it's defintely a thing you can find in programs which don't use a proper marschalling library/code, and you know. CSV is easy so you surly don't need to add a library dependency just to use it. (sorry sarcasm). Or with other words "because CSV is easy" it's somteimes not properly specified and sometimes no "proper" marschalling/serialization library/module is used resulting in ad-hoc fixes for quoting where needed which potentially don't follow any specification either.
  - disgruntledphd2 6y ago
    The worst part about CSV is the quoting. Oh the quoting. As soon as a human writes something in a plain text field, your nice CSV parsing gets much, much harder. Salespeople enter things like Company Foo, the One (which used to be Bar). The worst part is that often these are legal names, which means you do need to store them. When I worked for a FAANG this was one of my major annoyances as there was no canonical number for a customer, so everyone did string matching which broke whenever a company changed their name (like for instance, if they'd just gone public). Just say no to CSV's with commas in quotes.
- happymellon 6y ago
  Always makes me sad that ASCII delimited was never a thing. It's almost like they really wanted to avoid this issue back in the 70s so it is built right into ASCII and unicode.
  - yrro 6y ago
    What I like about this is that ending your data with FS means that you can detect file truncation, yet another thing that CSV makes impossible. I wonder how you embed US/RS/GS/FS within a field though. Prefix with ESC?
- stinos 6y ago
  How many, though? As many as there are localized datetime formats :) there shouldn't be any problems, right? That would be reallys awesome, yes. With the aid of the other sibling comments you already figured by now it's not as simple, unfortunately.
- dkarl 6y ago
  I suspect that most of the CSV issues I've encountered were produced by ad-hoc writers and parsers embedded in applications rather than serious attempts to implement a CSV library. Behaviors I've encountered during various integrations: - Values in most columns are escaped, but a few are not. (You have to count commas from both ends of the line and then log a warning and make a guess if there are too many commas.) - ASCII whitespace is url-encoded (why?) but no other characters are. - Columns without double quotes can only contain digits. - Double commas are used as the field separator because they didn't want to implement quoting. (Some of the fields came from user input on a web page, and I prayed for them to hit a double comma and learn the error of their ways, but it never happened. I think their front end developers were sanitizing for them.) - And my favorite, no quoting was supported at all, double quotes were left unquoted, commas were stripped from values when writing, and when reading, fields were compared against a list of mappings like "Acme Inc." -> "Acme, Inc." to restore the commas.

There are similar problems with csv. A deceptively simple format with many variants to handle various edge cases.

Yes, except it's easy to debug a csv problem because it's generally text in some encoding you can hopefully identify without looking at specific byte offsets in a hex editor: http://fileformats.archiveteam.org/wiki/Tape_Archive#Identification http://fileformats.archiveteam.org/wiki/Tape_Archive#Identif... In fact, one of the biggest questions I've still yet to answer is whether Docker or OCI have ever clarified just which TAR format must be followed? As far as I can tell, it's a mix of "whatever library X supports"? https://www.cyphar.com/blog/post/20190121-ociv2-images-i-tar https://www.cyphar.com/blog/post/20190121-ociv2-images-i-tar for example. I'm not sure I want a more complicated format, but I do wish somebody would pick a new file extension for their next use of tar so we could clearly differentiate one tar format from another the next time someone builds on top of simple tar files for their binary distribution, etc.

brazzy 6y ago

How many, though? Off the top of my head, you need to specify text encoding, line endings, a quoting mechanism in case a field contains the separator, and an escaping mechanism in case a field contains multiple lines or the quote character. If you have those specified and everyone follows the specification, there shouldn't be any problems, right? Woe be unto you if you have to process input data where those are not used consistently, of course.

roberto 6y ago
First, there's no specification. Second, this: http://georgemauer.net/2017/10/07/csv-injection.html http://georgemauer.net/2017/10/07/csv-injection.html
- brazzy 6y ago
  Quote from the site you linked to: "So who’s fault is all of this anyways? Well it’s not the CSV format’s." And what I wrote before is basically all you need for a specification. Yes, it sucks that there is no fixed one that everyone uses.
probably_wrong 6y ago
> If you have those specified and everyone follows the specification, there shouldn't be any problems, right? Well, yes, but I feel that that's the point at which CSV definitely earns its "ill-specified format" hat. Given that there is no way to extract the file specification from the file itself, you can only follow the specification if someone tells you the specification beforehand. You can even use the "wrong" specification and don't notice it because your specification overlaps with the "true" specification in the types of data you're typically seeing (looking at you, optional quotes).
tinus_hn 6y ago
All you need to do is specify all these unspecified things and it’ll be fine. You forgot the localized decimal separator symbol.
- VMG 6y ago
  if you start going down that route you will have to specify the format for dates, network addresses, postal addresses, phone numbers... it never ends
  - tinus_hn 6y ago
    Either you accept that CSV can’t reliably contain fractional numbers or you specify the decimal separator.
    - brazzy 6y ago
      CSV doesn't contain numbers at all; it contains strings. How those should be parsed into numbers is not part of the format, unless by CSV you additionally mean "to be imported by Excel". Which is certainly a common use case but not always the case.
      - tinus_hn 6y ago
        By this line of thinking, any file type is specified, because the specification of a file is a sequence of bytes. Arguably right, practically useless.
        brazzy 6y ago
        The distinction between file format and payload format is perfectly normal and not at all useless. All file formats have limits for how much they define types for the payload. Most have very limited types. JSON for example knows objects, arrays, strings, numbers, booleans and null. It's not that big a difference to have a format that has only strings.
Denvercoder9 6y ago
> Off the top of my head, you need to specify text encoding, line endings, a quoting mechanism in case a field contains the separator, and an escaping mechanism in case a field contains multiple lines or the quote character. You also need to specify the separator itself, which despite the name isn't always a comma, but can also be a semicolon or (less commonly) a tab.
- medstrom 6y ago
  The IANA standard for TSV files [1] bans tab characters inside of fields. It's a convenient assumption, so if I'm writing a quick way to sync some data to disk, I choose TSV because the functions can be simpler. https://www.iana.org/assignments/media-types/text/tab-separated-values https://www.iana.org/assignments/media-types/text/tab-separa...
  - mFixman 6y ago
    To this day I don't understand why standard table formats don't use the standard ASCII unit and record separators. They are there, they are convenient, and they don't require any visible character sacrifices.
    - marcthe12 6y ago
      I gues lack of program support. But yeas. That would have been great
dathinab 6y ago
Besides inconsistencies, that a non defined quoting and escaping mechanism is already quite painful. Let's look at following single column single line csv: '\0' Depending on the source this might be: - quote back-slash zero quote - zero - back-slash zero - a byte with the numeric value of 0 - quote zero quote - quote back-slash zero quote - quote byte-with-value-0 quote - malformed and should be ignored Worse the same application might mix different of this quoting mechanisms for different columns or even in the same column. Normally it shouldn't but it's defintely a thing you can find in programs which don't use a proper marschalling library/code, and you know. CSV is easy so you surly don't need to add a library dependency just to use it. (sorry sarcasm). Or with other words "because CSV is easy" it's somteimes not properly specified and sometimes no "proper" marschalling/serialization library/module is used resulting in ad-hoc fixes for quoting where needed which potentially don't follow any specification either.
- disgruntledphd2 6y ago
  The worst part about CSV is the quoting. Oh the quoting. As soon as a human writes something in a plain text field, your nice CSV parsing gets much, much harder. Salespeople enter things like Company Foo, the One (which used to be Bar). The worst part is that often these are legal names, which means you do need to store them. When I worked for a FAANG this was one of my major annoyances as there was no canonical number for a customer, so everyone did string matching which broke whenever a company changed their name (like for instance, if they'd just gone public). Just say no to CSV's with commas in quotes.
happymellon 6y ago
Always makes me sad that ASCII delimited was never a thing. It's almost like they really wanted to avoid this issue back in the 70s so it is built right into ASCII and unicode.
- yrro 6y ago
  What I like about this is that ending your data with FS means that you can detect file truncation, yet another thing that CSV makes impossible. I wonder how you embed US/RS/GS/FS within a field though. Prefix with ESC?
stinos 6y ago
How many, though? As many as there are localized datetime formats :) there shouldn't be any problems, right? That would be reallys awesome, yes. With the aid of the other sibling comments you already figured by now it's not as simple, unfortunately.
dkarl 6y ago
I suspect that most of the CSV issues I've encountered were produced by ad-hoc writers and parsers embedded in applications rather than serious attempts to implement a CSV library. Behaviors I've encountered during various integrations: - Values in most columns are escaped, but a few are not. (You have to count commas from both ends of the line and then log a warning and make a guess if there are too many commas.) - ASCII whitespace is url-encoded (why?) but no other characters are. - Columns without double quotes can only contain digits. - Double commas are used as the field separator because they didn't want to implement quoting. (Some of the fields came from user input on a web page, and I prayed for them to hit a double comma and learn the error of their ways, but it never happened. I think their front end developers were sanitizing for them.) - And my favorite, no quoting was supported at all, double quotes were left unquoted, commas were stripped from values when writing, and when reading, fields were compared against a list of mappings like "Acme Inc." -> "Acme, Inc." to restore the commas.