4 ms·

Dedicated separator characters don't solve the problem--you'd still need to escape them. Or validate that the data (which may come from untrusted web forms etc.

Dedicated separator characters don't solve the problem--you'd still need to escape them. Or validate that the data (which may come from untrusted web forms etc.) does not contain them, which means you have another error condition to handle.

hermitcrab 3y ago
Or specify that the data can't contain this data. If it does, you have to use a different format. This keeps everything super simple. And how often are ASCII US and RS characters used in data? I don't think I have ever seem one in the wild, apart from in a .asv file.
- g4zj 3y ago
  I'm no expert on character encodings or Unicode itself, but would this be as simple as checking for the byte 1F in the data? Assuming the file is ASCII or UTF-8 encoded (or attempting to confirm this as much as possible as well), it seems like that check would suffice to validate the absence of the code point in the data, but I imagine it's not quite so simple.
  - rhelz 3y ago
    For text data, it would work fine, but you'd have to do some finagling with binary data; $1F is a perfectly valid byte to have in, say, a 4-byte integer.
    - hermitcrab 3y ago
      Without escaping, it wouldn't be suitable for arbitary binary data.
    - runlaszlorun 3y ago
      My going assumption is that arbitrary binary data should be in a binary format. Feel free to correct me, but I figure that as long as data can be from 0x00 to 0xFF per byte, no format that uses characters in that range will ever be safe. I’m not a big C developer but I figure the null terminated strings have the same limitation. But if its something entered by keyboard you should be ok to use control codes. Personally, I find tab and return to be fine for text driven stuff. Shows up in an editor just like intented.
tambourine_man 3y ago
The “problem” I’m referring to is that we chose a widely used character as a field separator. Of course you still have to write a parser, etc, it’s just a lot easier if you choose a dedicated character.
AdamH12113 3y ago
There's an ASCII character for escaping, too, if you need it. The advantage of ASV is not that you can't have invalid or insecure data, it's that valid data will almost never contain ASCII control characters in the record fields themselves. Commas, quotation marks, and backslashes, meanwhile, are everywhere.

hermitcrab 3y ago

Or specify that the data can't contain this data. If it does, you have to use a different format. This keeps everything super simple. And how often are ASCII US and RS characters used in data? I don't think I have ever seem one in the wild, apart from in a .asv file.

g4zj 3y ago
I'm no expert on character encodings or Unicode itself, but would this be as simple as checking for the byte 1F in the data? Assuming the file is ASCII or UTF-8 encoded (or attempting to confirm this as much as possible as well), it seems like that check would suffice to validate the absence of the code point in the data, but I imagine it's not quite so simple.
- rhelz 3y ago
  For text data, it would work fine, but you'd have to do some finagling with binary data; $1F is a perfectly valid byte to have in, say, a 4-byte integer.
  - hermitcrab 3y ago
    Without escaping, it wouldn't be suitable for arbitary binary data.
  - runlaszlorun 3y ago
    My going assumption is that arbitrary binary data should be in a binary format. Feel free to correct me, but I figure that as long as data can be from 0x00 to 0xFF per byte, no format that uses characters in that range will ever be safe. I’m not a big C developer but I figure the null terminated strings have the same limitation. But if its something entered by keyboard you should be ok to use control codes. Personally, I find tab and return to be fine for text driven stuff. Shows up in an editor just like intented.

tambourine_man 3y ago

The “problem” I’m referring to is that we chose a widely used character as a field separator. Of course you still have to write a parser, etc, it’s just a lot easier if you choose a dedicated character.

AdamH12113 3y ago

There's an ASCII character for escaping, too, if you need it. The advantage of ASV is not that you can't have invalid or insecure data, it's that valid data will almost never contain ASCII control characters in the record fields themselves. Commas, quotation marks, and backslashes, meanwhile, are everywhere.