6 ms·

Fascinated this uses the Unicode glyphs / symbols for unit and record separator rather than the unit and record separators themselves (ASCII US and RS). Perfec

Fascinated this uses the Unicode glyphs / symbols for unit and record separator rather than the unit and record separators themselves (ASCII US and RS). Perfect deployment of David Wheeler's aphorism: > All problems in computer science can be solved by adding another level of indirection. https://en.wikipedia.org/wiki/David_Wheeler_(computer_scientist) https://en.wikipedia.org/wiki/David_Wheeler_(computer_scient...

ale42 3y ago
Indeed... I didn't read the standard in detail to check whether escaping is allowed/taken into account, but what if my data contains those symbols? I mean, they are perfectly legal Unicode printable characters, unlike the ASCII ones.
- BugsJustFindMe 3y ago
  There's an escape.
  - theamk 3y ago
    I thought the point is you don't need escapes? If you still need to implement escape mechanism, might as well do CSV/TSV.
    - sumergomonstro 3y ago
      You shouldn't need escapes for separator characters precisely because they are not designed for data. Their entire purpose is to separate hierarchical data. If it turns out that escaping is needed, it will still be far rarer than escaping commas and newlines.
    - BugsJustFindMe 3y ago
      The point is ASCII DSV, which gives innately better hierarchy than CSV, but with visible tokens and stream accommodation. You should read the github readme. It's not that long. https://github.com/SixArm/usv/tree/main/doc/faq#why-choose-usv-over-csv-or-tsv https://github.com/SixArm/usv/tree/main/doc/faq#why-choose-u... As for still needing escapes, using obscure symbols instead of ones that are extremely common in writing inherently means needing far far faaaaaaar fewer of them.
      - theamk 3y ago
        What's the point of visible tokens if it's all squished in one line? You are not going to be editing this in regular editor once you have non-trivial amount of data. And yes, I read README and source code, so I know that newlines are optional, existing tools don't generate them, and multi-line examples are basically fake.
        BugsJustFindMe 3y ago
        > What's the point of visible tokens if it's all squished in one line? It doesn't have to be all squished in one line, it just doesn't hurt anything. Visually splitting squished lines for presentation or perusal is trivial because of the record separator. > You are not going to be editing this in regular editor I know (or at least I think) that you meant this in relation to squished lines getting very long, but maybe we can talk about it in a broader context, since record splitting is trivial... One could easily say these same words about documents written in right-to-left languages. But people in Israel manage to create files too somehow, so that's clearly not an insurmountable barrier.
        couchand 3y ago
        Editors generally support composing right-to-left languages that way? So I suppose the metaphor suggests that all editors should directly support the visible glyphs semantically? And yet, that's explicitly not the semantic purpose of those glyphs. The actual delimiters already exist at a lower code point. If we're asking editors to semantically support delimiters we should be asking them to support the semantic delimiters.
        jph 3y ago
        Good point. I'm adding automatic record separator newlines to the crate now.
- 6510 3y ago
  I one time attempted to write a blog post about escaping stuff in rss feeds, while technically correct nothing could parse the rss feed for the blog.
deleted 3y ago
[deleted]
red_admiral 3y ago
Indeed, if the result is to be encoded with UTF-8, using 1-byte separators vs the multi-byte encoding of (241F) would make sense to me. I'd also prefer if escapes were done in the "traditional" manner of, for example, "\t" for a tab because you can then read in stuff with something like input.split("\t").map(unescape); you know any actual tab character in the input is a field separator, and then you can go through the fields to put back the escaped ones.
- eadmund 3y ago
  > you can then read in stuff with something like input.split("\t").map(unescape) What about input lines like 'asdf\\thjkl\tzxcvb'? That should be two fields, one the string ‘asdf\thjkl’ and the other the string ‘zxcvb.’ I think that your way is a bit like trying to match context-free grammars with a regular expression. The right way is to parse the input character by character.
  - cxr 3y ago
    Although matching up nested pairs of brackets requires something at least as powerful as a pushdown automaton (CFG matcher), discriminating between an arbitrary number of escaped backslashes followed by an unescaped 't' versus an arbitrary number of escaped backslashes followed by the '\t' escape sequence doesn't require anything more powerful than a finite state machine.
  - qzzi 3y ago
    > you know any actual tab character in the input is a field separator, and then you can go through the fields to put back the escaped ones The "\t" in "split" is not a "slash-tee" but an actual tab character and then escape sequences in fields are handled by the "unescape" function.
  - fiddlerwoaroof 3y ago
    I think the suggestion is that the field separator is an actual tab character (ascii code 9) but tabs inside the field are `\t`. So, splitting on the tab character always works because fields cannot contain ascii code 9 but must use the two character escape instead.
ajdude 3y ago
This makes me sad; such a missed opportunity.
default-kramer 3y ago
Two links away is the answer: https://github.com/SixArm/usv/tree/main/doc/faq#why-use-control-picture-characters-rather-than-the-control-characters-themselves https://github.com/SixArm/usv/tree/main/doc/faq#why-use-cont...
- divbzero 3y ago
  The answer makes sense to me, but I wish we could fix editors to properly handle the ASCII separators (1C, 1D, 1E, 1F) instead of resorting to Unicode control picture characters (241C, 241D, 241E, 241F). Maybe if editors are fixed up we could adopt ASCII Separated Values (ASV) as the new standard.
  - throwanem 3y ago
    Emacs has handled literal ASCII control characters correctly I believe since around the time I was born - probably somewhat earlier, if we count back further than GNU. Unicode works fine there too, so it makes no nevermind to me which flavor people use. I just think it's funny how "everything old is new again".
  - jph 3y ago
    Yes you're right. That's a long term goal.
- marwis 3y ago
  Why not combine zero width character with visible character, i.e. use 2 characters for separators? ,<FS> for fields \n<RS> for records This removes ambiguity in parsing and remains user readable. It's also relatively easy to auto-fix files edited by users in normal editors. It also mostly removes need for escaping. It's also smaller or same size as unicode multibyte characters (haven't checked).
  - Renaud 3y ago
    Wouldn't this easily break your file in subtle ways when someone tries to edit it in their editor and the zero width character is not visible? How could you make the difference with a standard CSV file if it looks like a standard CSV file? They explain why they don't use control characters. Editors are not consistent in how they show control/zero-length characters: https://github.com/SixArm/usv/tree/main/doc/faq#why-use-control-picture-characters-rather-than-the-control-characters-themselves https://github.com/SixArm/usv/tree/main/doc/faq#why-use-cont...
HL33tibCe7 3y ago
Perfect deployment of HL33tibCe7’s aphorism: > For every interesting HN post, there’s at least one smug commenter who thinks he knows better, but actually doesn’t https://github.com/SixArm/usv/tree/main/doc/faq#why-use-control-picture-characters-rather-than-the-control-characters-themselves https://github.com/SixArm/usv/tree/main/doc/faq#why-use-cont...
- ok_dad 3y ago
  The OP was probably assuming no human would want to actually read a CSV raw, and so was probably correct from their POV. Your POV is probably from someone who reads CSVs raw. You don't have to be so rude about it, you're being even more smug than the OP, probably.
  - groby_b 3y ago
    One of the two likely works with CSVs for a living, and it's definitely not the person suggesting "What if it just was hard to eyeball/edit". If you don't understand why something is the way it is, it might be better to start with a question than with a statement implying the tech misses existing tech. Chesterton's fence still applies, and ignoring it means you're outsourcing your work to others. RTFM is a perfectly valid answer at that point.
    - ok_dad 3y ago
      I use CSVs for a living but I rarely read them manually. I’d rather have ASCII than Unicode in my CSVs. My point above, though, is that everyone has opinions and you don’t have to be a dickhead about “correcting” them.
1vuio0pswjnm7 3y ago
(For text processing, I use octal \034 all the time.) Perhaps there is a software developer version of "Needs more cowbell" called "Needs more complexity" Computer languages generally use the Latin alphabet. And even in a case like APL, which some HN commenters call "hieroglyphics", the number of symbols is limited and each is precisely defined (cf. potentially up to 1.1 million Unicode symbols and "emojis" that are open to interpretation).
- Tijdreiziger 3y ago
  Well, yeah, not every language uses the Latin alphabet.
pmarreck 3y ago
I tried to create a "printable binary" format for better visual inspection of raw binary data (and a to/from format) https://github.com/pmarreck/elixir-snippets/blob/master/printable_binary.exs https://github.com/pmarreck/elixir-snippets/blob/master/prin...

ale42 3y ago

Indeed... I didn't read the standard in detail to check whether escaping is allowed/taken into account, but what if my data contains those symbols? I mean, they are perfectly legal Unicode printable characters, unlike the ASCII ones.

BugsJustFindMe 3y ago
There's an escape.
- theamk 3y ago
  I thought the point is you don't need escapes? If you still need to implement escape mechanism, might as well do CSV/TSV.
  - sumergomonstro 3y ago
    You shouldn't need escapes for separator characters precisely because they are not designed for data. Their entire purpose is to separate hierarchical data. If it turns out that escaping is needed, it will still be far rarer than escaping commas and newlines.
  - BugsJustFindMe 3y ago
    The point is ASCII DSV, which gives innately better hierarchy than CSV, but with visible tokens and stream accommodation. You should read the github readme. It's not that long. https://github.com/SixArm/usv/tree/main/doc/faq#why-choose-usv-over-csv-or-tsv https://github.com/SixArm/usv/tree/main/doc/faq#why-choose-u... As for still needing escapes, using obscure symbols instead of ones that are extremely common in writing inherently means needing far far faaaaaaar fewer of them.
    - theamk 3y ago
      What's the point of visible tokens if it's all squished in one line? You are not going to be editing this in regular editor once you have non-trivial amount of data. And yes, I read README and source code, so I know that newlines are optional, existing tools don't generate them, and multi-line examples are basically fake.
      - BugsJustFindMe 3y ago
        > What's the point of visible tokens if it's all squished in one line? It doesn't have to be all squished in one line, it just doesn't hurt anything. Visually splitting squished lines for presentation or perusal is trivial because of the record separator. > You are not going to be editing this in regular editor I know (or at least I think) that you meant this in relation to squished lines getting very long, but maybe we can talk about it in a broader context, since record splitting is trivial... One could easily say these same words about documents written in right-to-left languages. But people in Israel manage to create files too somehow, so that's clearly not an insurmountable barrier.
        couchand 3y ago
        Editors generally support composing right-to-left languages that way? So I suppose the metaphor suggests that all editors should directly support the visible glyphs semantically? And yet, that's explicitly not the semantic purpose of those glyphs. The actual delimiters already exist at a lower code point. If we're asking editors to semantically support delimiters we should be asking them to support the semantic delimiters.
      - jph 3y ago
        Good point. I'm adding automatic record separator newlines to the crate now.
6510 3y ago
I one time attempted to write a blog post about escaping stuff in rss feeds, while technically correct nothing could parse the rss feed for the blog.

deleted 3y ago

[deleted]

red_admiral 3y ago

Indeed, if the result is to be encoded with UTF-8, using 1-byte separators vs the multi-byte encoding of (241F) would make sense to me. I'd also prefer if escapes were done in the "traditional" manner of, for example, "\t" for a tab because you can then read in stuff with something like input.split("\t").map(unescape); you know any actual tab character in the input is a field separator, and then you can go through the fields to put back the escaped ones.

eadmund 3y ago
> you can then read in stuff with something like input.split("\t").map(unescape) What about input lines like 'asdf\\thjkl\tzxcvb'? That should be two fields, one the string ‘asdf\thjkl’ and the other the string ‘zxcvb.’ I think that your way is a bit like trying to match context-free grammars with a regular expression. The right way is to parse the input character by character.
- cxr 3y ago
  Although matching up nested pairs of brackets requires something at least as powerful as a pushdown automaton (CFG matcher), discriminating between an arbitrary number of escaped backslashes followed by an unescaped 't' versus an arbitrary number of escaped backslashes followed by the '\t' escape sequence doesn't require anything more powerful than a finite state machine.
- qzzi 3y ago
  > you know any actual tab character in the input is a field separator, and then you can go through the fields to put back the escaped ones The "\t" in "split" is not a "slash-tee" but an actual tab character and then escape sequences in fields are handled by the "unescape" function.
- fiddlerwoaroof 3y ago
  I think the suggestion is that the field separator is an actual tab character (ascii code 9) but tabs inside the field are `\t`. So, splitting on the tab character always works because fields cannot contain ascii code 9 but must use the two character escape instead.

ajdude 3y ago

This makes me sad; such a missed opportunity.

default-kramer 3y ago

Two links away is the answer: https://github.com/SixArm/usv/tree/main/doc/faq#why-use-control-picture-characters-rather-than-the-control-characters-themselves https://github.com/SixArm/usv/tree/main/doc/faq#why-use-cont...

divbzero 3y ago
The answer makes sense to me, but I wish we could fix editors to properly handle the ASCII separators (1C, 1D, 1E, 1F) instead of resorting to Unicode control picture characters (241C, 241D, 241E, 241F). Maybe if editors are fixed up we could adopt ASCII Separated Values (ASV) as the new standard.
- throwanem 3y ago
  Emacs has handled literal ASCII control characters correctly I believe since around the time I was born - probably somewhat earlier, if we count back further than GNU. Unicode works fine there too, so it makes no nevermind to me which flavor people use. I just think it's funny how "everything old is new again".
- jph 3y ago
  Yes you're right. That's a long term goal.
marwis 3y ago
Why not combine zero width character with visible character, i.e. use 2 characters for separators? ,<FS> for fields \n<RS> for records This removes ambiguity in parsing and remains user readable. It's also relatively easy to auto-fix files edited by users in normal editors. It also mostly removes need for escaping. It's also smaller or same size as unicode multibyte characters (haven't checked).
- Renaud 3y ago
  Wouldn't this easily break your file in subtle ways when someone tries to edit it in their editor and the zero width character is not visible? How could you make the difference with a standard CSV file if it looks like a standard CSV file? They explain why they don't use control characters. Editors are not consistent in how they show control/zero-length characters: https://github.com/SixArm/usv/tree/main/doc/faq#why-use-control-picture-characters-rather-than-the-control-characters-themselves https://github.com/SixArm/usv/tree/main/doc/faq#why-use-cont...

HL33tibCe7 3y ago

Perfect deployment of HL33tibCe7’s aphorism: > For every interesting HN post, there’s at least one smug commenter who thinks he knows better, but actually doesn’t https://github.com/SixArm/usv/tree/main/doc/faq#why-use-control-picture-characters-rather-than-the-control-characters-themselves https://github.com/SixArm/usv/tree/main/doc/faq#why-use-cont...

ok_dad 3y ago
The OP was probably assuming no human would want to actually read a CSV raw, and so was probably correct from their POV. Your POV is probably from someone who reads CSVs raw. You don't have to be so rude about it, you're being even more smug than the OP, probably.
- groby_b 3y ago
  One of the two likely works with CSVs for a living, and it's definitely not the person suggesting "What if it just was hard to eyeball/edit". If you don't understand why something is the way it is, it might be better to start with a question than with a statement implying the tech misses existing tech. Chesterton's fence still applies, and ignoring it means you're outsourcing your work to others. RTFM is a perfectly valid answer at that point.
  - ok_dad 3y ago
    I use CSVs for a living but I rarely read them manually. I’d rather have ASCII than Unicode in my CSVs. My point above, though, is that everyone has opinions and you don’t have to be a dickhead about “correcting” them.

1vuio0pswjnm7 3y ago

(For text processing, I use octal \034 all the time.) Perhaps there is a software developer version of "Needs more cowbell" called "Needs more complexity" Computer languages generally use the Latin alphabet. And even in a case like APL, which some HN commenters call "hieroglyphics", the number of symbols is limited and each is precisely defined (cf. potentially up to 1.1 million Unicode symbols and "emojis" that are open to interpretation).

Tijdreiziger 3y ago
Well, yeah, not every language uses the Latin alphabet.

pmarreck 3y ago

I tried to create a "printable binary" format for better visual inspection of raw binary data (and a to/from format) https://github.com/pmarreck/elixir-snippets/blob/master/printable_binary.exs https://github.com/pmarreck/elixir-snippets/blob/master/prin...