4 ms·

Ben this is great, thank you. Would you consider adding Unicode Separated Values (USV)? https://github.com/sixarm/usv https://github.com/sixarm/usv USV is lik

by jph 4y ago

Ben this is great, thank you. Would you consider adding Unicode Separated Values (USV)? https://github.com/sixarm/usv https://github.com/sixarm/usv USV is like CSV and simpler because of no escaping and no quoting. I can donate $50 to you or your charity of choice as a token of thanks and encouragement.

tomsmeding 4y ago
> no escaping That makes no sense. Sure, they've chosen significantly less common separator characters than something like ','. but they are still characters that may appear in the data. How do you represent a value containing ␟ Unit Separator in USV? In-band signalling is ever going to remain in-band signalling. And in-band signalling will need escaping.
- jph 4y ago
  USV has no escaping on purpose because it's simpler and faster. USV is for the 99.999% of cases that don't embed USV Control Picture characters in the content. If you need escaping, then I can suggest you consider USVX which is USV + extensions for escaping and other conveniences, or you can use your content's own escaping such as ampersand escaping for HTML text, or you can use any other format such as JSON, XML, SQL, etc.
  - layer8 4y ago
    This just creates another “CSV” were you don’t know which format variation was actually used.
  - tomsmeding 4y ago
    > USV is for the 99.999% of cases that don't embed USV Control Picture characters in the content. I agree that wanting to put one of these characters in a csv is super rare. The vast majority of use cases would not notice that certain inout characters are prohibited. But certain inout characters _are_ prohibited, which means that encoders must check this case, and tools using those encoders neex to handle its failure. In a way, the fact that failure will be super rare will make it more dangerous, because people will omit these checks because "it won't happen for them" -- and then at some point it will wreak havoc. And with USVX all we've gained is less backslashes but more bytes per separator, so for usual data that doesn't contain a comma in every field, the USVX encoding will even increase file size, without requiring less code anywhere. I admit there is no ideal solution, though; while out-of-band signalling (i.e. length-prefixing) avoids all of these issues (which is why binary formats are universally length-prefixing formats), escaped formats are work much better for humans. And if a human wouldn't need to read it, one wouldn't use csv anyway, usually.
    - specialist 4y ago
      You just reminded me of the argument against seat belts based on the assumption that they'll encourage riskier driving thereby leading to more deaths. Not doing the obviously correct superior easy strategy because there's some infrequent weird corner case prevents any kind of progress. Perfection is the enemy of good enough, engineering is balancing constraints, ad nauseum...
- imtringued 4y ago
  Just reject nested USV documents in the spec. Use something else if you want nested data.
  - jerf 4y ago
    That has the same in-band problems as existing specs, which is that things generating the data need to have that callout. There are many cases where the "solution" to the CSV inband signalling problem was to just reject values with commas, because they should never come in and if they ever do they should be investigated because they weren't valid data for whatever the CSV was storing. The whole problem is that programmers don't think to do that. The siren call of the string append function is just too strong, especially when programmers don't even realize they should be resisting.
  - hedora 4y ago
    That means it does not support binary data, which TSV does.
- deckard1 4y ago
  > In-band signalling is [n]ever going to remain in-band signalling This is the root of all spreadsheet evil, right here. TSV. Replace/strip/ban tab + newline from fields when writing. Done. If you need those characters, encode them using backslashes if you absolutely must (i.e. you're an Excel weenie doing Excel weenie things that really aren't "spreadsheet" things but since you only know of one hammer you then use that hammer to do everything)
benhoyt 4y ago
Thanks. I haven't tried this, but it should actually work already in standard AWK input mode with FS and RS set to those Unicode separators. I'll test it tomorrow when back at my laptop.
- benhoyt 4y ago
  Yep, this works fine (in GoAWK, gawk, and mawk, though not in original awk): $ cat t.usv && echo id␟name␟age␞1␟Bob "Billy" Smith␟42␞2␟Jane Brown␟37 $ goawk -F␟ -vRS=␞ -vOFS=, '{ print $1, $2, $3 }' t.usv id,name,age 1,Bob "Billy" Smith,42 2,Jane Brown,37 I've also added explicit tests for ASCII and Unicode unit and record separators, just to ensure I don't regress: https://github.com/benhoyt/goawk/commit/215652de58f33630edb0c4463def18419e886ea4 https://github.com/benhoyt/goawk/commit/215652de58f33630edb0...
cylinder714 4y ago
More expressions of the same idea: https://www.ccsv.io/ https://www.ccsv.io/ And this perennial favorite here at HN: https://ronaldduncan.wordpress.com/2009/10/31/text-file-formats-ascii-delimited-text-not-csv-or-tab-delimited-text/ https://ronaldduncan.wordpress.com/2009/10/31/text-file-form... I've been toying with the idea of starting a change.org campaign, asking Microsoft to add support to Excel for importing and exporting such files. I don't see any way to make any progress if Microsoft won't push the industry forward.

by jph 4y ago

tomsmeding 4y ago

> no escaping That makes no sense. Sure, they've chosen significantly less common separator characters than something like ','. but they are still characters that may appear in the data. How do you represent a value containing ␟ Unit Separator in USV? In-band signalling is ever going to remain in-band signalling. And in-band signalling will need escaping.

jph 4y ago
USV has no escaping on purpose because it's simpler and faster. USV is for the 99.999% of cases that don't embed USV Control Picture characters in the content. If you need escaping, then I can suggest you consider USVX which is USV + extensions for escaping and other conveniences, or you can use your content's own escaping such as ampersand escaping for HTML text, or you can use any other format such as JSON, XML, SQL, etc.
- layer8 4y ago
  This just creates another “CSV” were you don’t know which format variation was actually used.
- tomsmeding 4y ago
  > USV is for the 99.999% of cases that don't embed USV Control Picture characters in the content. I agree that wanting to put one of these characters in a csv is super rare. The vast majority of use cases would not notice that certain inout characters are prohibited. But certain inout characters _are_ prohibited, which means that encoders must check this case, and tools using those encoders neex to handle its failure. In a way, the fact that failure will be super rare will make it more dangerous, because people will omit these checks because "it won't happen for them" -- and then at some point it will wreak havoc. And with USVX all we've gained is less backslashes but more bytes per separator, so for usual data that doesn't contain a comma in every field, the USVX encoding will even increase file size, without requiring less code anywhere. I admit there is no ideal solution, though; while out-of-band signalling (i.e. length-prefixing) avoids all of these issues (which is why binary formats are universally length-prefixing formats), escaped formats are work much better for humans. And if a human wouldn't need to read it, one wouldn't use csv anyway, usually.
  - specialist 4y ago
    You just reminded me of the argument against seat belts based on the assumption that they'll encourage riskier driving thereby leading to more deaths. Not doing the obviously correct superior easy strategy because there's some infrequent weird corner case prevents any kind of progress. Perfection is the enemy of good enough, engineering is balancing constraints, ad nauseum...
imtringued 4y ago
Just reject nested USV documents in the spec. Use something else if you want nested data.
- jerf 4y ago
  That has the same in-band problems as existing specs, which is that things generating the data need to have that callout. There are many cases where the "solution" to the CSV inband signalling problem was to just reject values with commas, because they should never come in and if they ever do they should be investigated because they weren't valid data for whatever the CSV was storing. The whole problem is that programmers don't think to do that. The siren call of the string append function is just too strong, especially when programmers don't even realize they should be resisting.
- hedora 4y ago
  That means it does not support binary data, which TSV does.
deckard1 4y ago
> In-band signalling is [n]ever going to remain in-band signalling This is the root of all spreadsheet evil, right here. TSV. Replace/strip/ban tab + newline from fields when writing. Done. If you need those characters, encode them using backslashes if you absolutely must (i.e. you're an Excel weenie doing Excel weenie things that really aren't "spreadsheet" things but since you only know of one hammer you then use that hammer to do everything)

benhoyt 4y ago

Thanks. I haven't tried this, but it should actually work already in standard AWK input mode with FS and RS set to those Unicode separators. I'll test it tomorrow when back at my laptop.

benhoyt 4y ago
Yep, this works fine (in GoAWK, gawk, and mawk, though not in original awk): $ cat t.usv && echo id␟name␟age␞1␟Bob "Billy" Smith␟42␞2␟Jane Brown␟37 $ goawk -F␟ -vRS=␞ -vOFS=, '{ print $1, $2, $3 }' t.usv id,name,age 1,Bob "Billy" Smith,42 2,Jane Brown,37 I've also added explicit tests for ASCII and Unicode unit and record separators, just to ensure I don't regress: https://github.com/benhoyt/goawk/commit/215652de58f33630edb0c4463def18419e886ea4 https://github.com/benhoyt/goawk/commit/215652de58f33630edb0...

cylinder714 4y ago

More expressions of the same idea: https://www.ccsv.io/ https://www.ccsv.io/ And this perennial favorite here at HN: https://ronaldduncan.wordpress.com/2009/10/31/text-file-formats-ascii-delimited-text-not-csv-or-tab-delimited-text/ https://ronaldduncan.wordpress.com/2009/10/31/text-file-form... I've been toying with the idea of starting a change.org campaign, asking Microsoft to add support to Excel for importing and exporting such files. I don't see any way to make any progress if Microsoft won't push the industry forward.