3 ms·

USV has no escaping on purpose because it's simpler and faster. USV is for the 99.999% of cases that don't embed USV Control Picture characters in the content

by jph 4y ago

USV has no escaping on purpose because it's simpler and faster. USV is for the 99.999% of cases that don't embed USV Control Picture characters in the content. If you need escaping, then I can suggest you consider USVX which is USV + extensions for escaping and other conveniences, or you can use your content's own escaping such as ampersand escaping for HTML text, or you can use any other format such as JSON, XML, SQL, etc.

layer8 4y ago
This just creates another “CSV” were you don’t know which format variation was actually used.
tomsmeding 4y ago
> USV is for the 99.999% of cases that don't embed USV Control Picture characters in the content. I agree that wanting to put one of these characters in a csv is super rare. The vast majority of use cases would not notice that certain inout characters are prohibited. But certain inout characters _are_ prohibited, which means that encoders must check this case, and tools using those encoders neex to handle its failure. In a way, the fact that failure will be super rare will make it more dangerous, because people will omit these checks because "it won't happen for them" -- and then at some point it will wreak havoc. And with USVX all we've gained is less backslashes but more bytes per separator, so for usual data that doesn't contain a comma in every field, the USVX encoding will even increase file size, without requiring less code anywhere. I admit there is no ideal solution, though; while out-of-band signalling (i.e. length-prefixing) avoids all of these issues (which is why binary formats are universally length-prefixing formats), escaped formats are work much better for humans. And if a human wouldn't need to read it, one wouldn't use csv anyway, usually.
- specialist 4y ago
  You just reminded me of the argument against seat belts based on the assumption that they'll encourage riskier driving thereby leading to more deaths. Not doing the obviously correct superior easy strategy because there's some infrequent weird corner case prevents any kind of progress. Perfection is the enemy of good enough, engineering is balancing constraints, ad nauseum...

by jph 4y ago

layer8 4y ago

This just creates another “CSV” were you don’t know which format variation was actually used.

tomsmeding 4y ago

> USV is for the 99.999% of cases that don't embed USV Control Picture characters in the content. I agree that wanting to put one of these characters in a csv is super rare. The vast majority of use cases would not notice that certain inout characters are prohibited. But certain inout characters _are_ prohibited, which means that encoders must check this case, and tools using those encoders neex to handle its failure. In a way, the fact that failure will be super rare will make it more dangerous, because people will omit these checks because "it won't happen for them" -- and then at some point it will wreak havoc. And with USVX all we've gained is less backslashes but more bytes per separator, so for usual data that doesn't contain a comma in every field, the USVX encoding will even increase file size, without requiring less code anywhere. I admit there is no ideal solution, though; while out-of-band signalling (i.e. length-prefixing) avoids all of these issues (which is why binary formats are universally length-prefixing formats), escaped formats are work much better for humans. And if a human wouldn't need to read it, one wouldn't use csv anyway, usually.

specialist 4y ago
You just reminded me of the argument against seat belts based on the assumption that they'll encourage riskier driving thereby leading to more deaths. Not doing the obviously correct superior easy strategy because there's some infrequent weird corner case prevents any kind of progress. Perfection is the enemy of good enough, engineering is balancing constraints, ad nauseum...