4 ms·

That would be solved by using the ASCII control chars Record Separator / Unit Separator! I don't get how this is not widely used as standard.

mjevans 2y ago
The _entire_ point of a CSV file is that it's fully human readable and write-able. The characters you mention could be used in a custom delimiter variant of the format, but at that point it's back to a binary machine format.
- kevmo314 2y ago
  What do you mean? I just push the Record Separator key on my keyboard. /s in case :)
  - mbreese 2y ago
    I’m pretty sure those used to exist. But when looking for a picture to back up my (likely flawed) memory, Google helpfully told me that you can get a record separator character by hitting Ctrl-^ (caret). Who knew?
  - corysama 2y ago
    The entire argument against ASCII Delimited Text boils down to "No one bothered to support it in popular editors back in 1984. Because I grew up without it, it is impossible to imagine supporting it today." You need 4 new keyboard shortcuts. Use ctrl+, ctrl+. ctrl+[ ctrl+] You need 4 new character symbols. You need a bit of new formatting rules. Pretty much page breaks decorated with the new symbols. It's really not that hard. But, like many problems in tech, the popular advice is "Everyone recognizes the problem and the solution. But, the problematic way is already widely used and the solution is not. Therefore everyone doing anything new should invest in continuing to support the problem forever."
    - zzo38computer 2y ago
      The keys would be something other than those, though. They would be: CTRL+\ for file separator, CTRL+] for group separator, CTRL+^ for record separator, CTRL+_ for unit separator. Other than that, it would work like you described, I think. > But, like many problems in tech, the popular advice is "Everyone recognizes the problem and the solution. But, the problematic way is already widely used and the solution is not This is unfortunately common. However, what else happens too, is disagreement about what is the problem and the solution.
    - WorldMaker 2y ago
      In Windows (and DOS EDIT.COM and a few other similarly ancient tools) there have existed Alt+028, Alt+029, Alt+030, and Alt+031 for a long time. I vaguely recall some file format I was working with in QBASIC used some or all of them and I was editing those files for some reason. That was not quite as far back as 1984, but sometime in the early 1990s for sure. I believe EDIT.COM had basic glyphs for them too, but I don't recall what they were, might have been random Wingdings like the playing card suits. Having keyboard shortcuts doesn't necessarily solve why people don't want to use that format, either.
      - zzo38computer 2y ago
        > I believe EDIT.COM had basic glyphs for them too, but I don't recall what they were, might have been random Wingdings like the playing card suits. That is not specific to EDIT.COM; they are the PC characters with the same codes as the corresponding control characters, so they appear as graphic characters. (They can be used in any program that can use PC character set.) However, in EDIT.COM and QBASIC you can also prefix a control character with CTRL+P in order to enter it directly into the file (and they appear as graphic characters, since I think the only control characters they will handle as control characters are tabs and line breaks). Suits are PC characters 3 to 6; these are PC characters 28 to 31 which are other shapes.
    - LegionMammal978 2y ago
      > The entire argument against ASCII Delimited Text boils down to "No one bothered to support it in popular editors back in 1984. Because I grew up without it, it is impossible to imagine supporting it today." There's also the argument of "Now you have two byte values that cannot be allowed to appear in a record under any circumstances. (E.g., incoming data from uncontrolled sources MUST be sanitized to reject or replace those bytes.)" Unless you add an escaping mechanism, in which case the argument shifts to "Why switch from CSV/TSV if the alternative still needs an escaping mechanism?"
- EGreg 2y ago
  Can't there be some magic sequence (like two unescaped newlines) to start a new record?
  - macintux 2y ago
    You’ll still find that sequence in data; it’ll just be rare enough that it won’t rear its ugly head until your solution has been in production for a while.
- mikepurvis 2y ago
  As soon as you give those characters magic meanings then suddenly people will have reason to want to use them— it'll be a CSV containing localization strings for tooltips that contain that character and bam, we'll be back to escaping. Except the usages of that character will be rare and so potentially way more scary. At least with quotes and commas, the breakages are everywhere so you confront them sooner rather than later.
  - BeFlatXIII 2y ago
    Best to add deliberate breakage to the spec, then.
  - immibis 2y ago
    Graphical representations of the control characters begin at U+2400 in the "Control Pictures" Unicode block. Instead of the actual U+001E Record Separator, you put the U+241E Symbol for Record Separator in the help text.
    - mikepurvis 2y ago
      .... with a note underneath urging readers not to copy and paste the character because it's only the graphical representation of it, not the thing itself. Perhaps a more salient example might be CSV nested in CSV. This happens all the time with XML (hello junit) and even JSON— when you plug a USB drive into my LG TV, it creates a metadata file on it that contains {"INFO":"{ \"thing\": true, <etc> }"}
      - immibis 2y ago
        You wouldn't use this format for that. It's not a universal format, but an application-specific one. It works in some applications, not in others.
- mbreese 2y ago
  And that’s why I tend to use tab delimited files more… when viewed with invisible characters shown, it’s pretty clear to read/write separate fields and have an easier to parse format. This, of course, assumes that your input doesn’t include tabs or newlines… because then you’re still stuck with the same problem, just with a different delimiter.
zoover2020 2y ago
Perhaps the popularity,or lack thereof? More often than not, the bad standard wins the long term market
andrewflnr 2y ago
Can't type them on a keyboard I guess, or generally work with them in the usual text-oriented tools? Part of the appeal of CSV is you can just open it up in Notepad or something if you need to. Maybe that's more a critique of text tools than it is of ASCII record separator characters.
jandrese 2y ago
If there were visible well known characters that could be printed for those and keys on a keyboard for inputting them we would probably have RSV files. Because they are buried down in the nonprintable section of the ASCII chart they are a pain for people to deal with. All it would have taken is one more key on the keyboard, maybe splitting the tab key in half.
- thesuitonym 2y ago
  > If there were visible well known characters that could be printed... ...There would be datasets that include those characters, and so they wouldn't be as useful for record separators. Look into your heart and know it to be true.
  - jandrese 2y ago
    I wouldn't feel too bad about blindly scrubbing those characters out of inputs unlike commas, tabs, and quotes.
sundarurfriend 2y ago
I remembered seeing a comment like this before, and... comment: https://news.ycombinator.com/item?id=26305052 https://news.ycombinator.com/item?id=26305052 comment: https://news.ycombinator.com/item?id=39679662 https://news.ycombinator.com/item?id=39679662 "ASCII Delimited Text – Not CSV or Tab Delimited Text" post [2014]: https://news.ycombinator.com/item?id=7474600 https://news.ycombinator.com/item?id=7474600 same post [2024]: https://news.ycombinator.com/item?id=42100499 https://news.ycombinator.com/item?id=42100499 comment: https://news.ycombinator.com/item?id=15440801 https://news.ycombinator.com/item?id=15440801 (...and many more.) "This comes up every single time someone mentions CSV. Without fail." - top reply from burntsushi in that last link, and it remains as true today as in 2017 :D You're not wrong though, we just need some major text editor to get the ball rolling and start making some attempts to understand these characters, and the rest will follow suit. We're kinda stuck at a local optimum which is clearly not ideal but also not troublesome enough to easily drum up wide support for ADSV (ASCII Delimiter Separated Values).
- ryandrake 2y ago
  Hahah, I came here to make the comment about ASCII's control characters, so I'm glad someone else beat me to it, and also that someone further pointed out that this topic comes up every time someone mentions CSV!
- scythe 2y ago
  >we just need some major text editor to get the ball rolling and start making some attempts to understand these characters Many text editors offer extensions APIs, including Vim, Emacs, Notepad++. But the ideal behavior would be to auto-align record separators and treat unit separators as a special kind of newline. That would allow the file to actually look like a table within the text editor. Input record separator as shift+space and unit separator as shift+enter.
  - hermitcrab 2y ago
    I think it would be enough for: 1. the field separator to be shown as a special character 2. the row separator to be (optionally) be interpreted as a linefeed IIRC 1) is true for Notepad++, but not 2).
orthoxerox 2y ago
But what if one of your columns contains arbitrary binary data?
- ndsipa_pomu 2y ago
  You'd likely need to uuencode it or similar as CSV isn't designed for binary data.
masklinn 2y ago
> I don't get how this is not widely used as standard. It requires bespoke tools for edition, and while CSV is absolute garbage it can be ingested and produced by most spreadsheet software, as well as databases.
marxisttemp 2y ago
I was really excited when I learned of these characters, but ultimately if it’s in ASCII then it’s in-band and will eventually require escaping leading to the same problem.

mjevans 2y ago

The _entire_ point of a CSV file is that it's fully human readable and write-able. The characters you mention could be used in a custom delimiter variant of the format, but at that point it's back to a binary machine format.

kevmo314 2y ago
What do you mean? I just push the Record Separator key on my keyboard. /s in case :)
- mbreese 2y ago
  I’m pretty sure those used to exist. But when looking for a picture to back up my (likely flawed) memory, Google helpfully told me that you can get a record separator character by hitting Ctrl-^ (caret). Who knew?
- corysama 2y ago
  The entire argument against ASCII Delimited Text boils down to "No one bothered to support it in popular editors back in 1984. Because I grew up without it, it is impossible to imagine supporting it today." You need 4 new keyboard shortcuts. Use ctrl+, ctrl+. ctrl+[ ctrl+] You need 4 new character symbols. You need a bit of new formatting rules. Pretty much page breaks decorated with the new symbols. It's really not that hard. But, like many problems in tech, the popular advice is "Everyone recognizes the problem and the solution. But, the problematic way is already widely used and the solution is not. Therefore everyone doing anything new should invest in continuing to support the problem forever."
  - zzo38computer 2y ago
    The keys would be something other than those, though. They would be: CTRL+\ for file separator, CTRL+] for group separator, CTRL+^ for record separator, CTRL+_ for unit separator. Other than that, it would work like you described, I think. > But, like many problems in tech, the popular advice is "Everyone recognizes the problem and the solution. But, the problematic way is already widely used and the solution is not This is unfortunately common. However, what else happens too, is disagreement about what is the problem and the solution.
  - WorldMaker 2y ago
    In Windows (and DOS EDIT.COM and a few other similarly ancient tools) there have existed Alt+028, Alt+029, Alt+030, and Alt+031 for a long time. I vaguely recall some file format I was working with in QBASIC used some or all of them and I was editing those files for some reason. That was not quite as far back as 1984, but sometime in the early 1990s for sure. I believe EDIT.COM had basic glyphs for them too, but I don't recall what they were, might have been random Wingdings like the playing card suits. Having keyboard shortcuts doesn't necessarily solve why people don't want to use that format, either.
    - zzo38computer 2y ago
      > I believe EDIT.COM had basic glyphs for them too, but I don't recall what they were, might have been random Wingdings like the playing card suits. That is not specific to EDIT.COM; they are the PC characters with the same codes as the corresponding control characters, so they appear as graphic characters. (They can be used in any program that can use PC character set.) However, in EDIT.COM and QBASIC you can also prefix a control character with CTRL+P in order to enter it directly into the file (and they appear as graphic characters, since I think the only control characters they will handle as control characters are tabs and line breaks). Suits are PC characters 3 to 6; these are PC characters 28 to 31 which are other shapes.
  - LegionMammal978 2y ago
    > The entire argument against ASCII Delimited Text boils down to "No one bothered to support it in popular editors back in 1984. Because I grew up without it, it is impossible to imagine supporting it today." There's also the argument of "Now you have two byte values that cannot be allowed to appear in a record under any circumstances. (E.g., incoming data from uncontrolled sources MUST be sanitized to reject or replace those bytes.)" Unless you add an escaping mechanism, in which case the argument shifts to "Why switch from CSV/TSV if the alternative still needs an escaping mechanism?"
EGreg 2y ago
Can't there be some magic sequence (like two unescaped newlines) to start a new record?
- macintux 2y ago
  You’ll still find that sequence in data; it’ll just be rare enough that it won’t rear its ugly head until your solution has been in production for a while.
mikepurvis 2y ago
As soon as you give those characters magic meanings then suddenly people will have reason to want to use them— it'll be a CSV containing localization strings for tooltips that contain that character and bam, we'll be back to escaping. Except the usages of that character will be rare and so potentially way more scary. At least with quotes and commas, the breakages are everywhere so you confront them sooner rather than later.
- BeFlatXIII 2y ago
  Best to add deliberate breakage to the spec, then.
- immibis 2y ago
  Graphical representations of the control characters begin at U+2400 in the "Control Pictures" Unicode block. Instead of the actual U+001E Record Separator, you put the U+241E Symbol for Record Separator in the help text.
  - mikepurvis 2y ago
    .... with a note underneath urging readers not to copy and paste the character because it's only the graphical representation of it, not the thing itself. Perhaps a more salient example might be CSV nested in CSV. This happens all the time with XML (hello junit) and even JSON— when you plug a USB drive into my LG TV, it creates a metadata file on it that contains {"INFO":"{ \"thing\": true, <etc> }"}
    - immibis 2y ago
      You wouldn't use this format for that. It's not a universal format, but an application-specific one. It works in some applications, not in others.
mbreese 2y ago
And that’s why I tend to use tab delimited files more… when viewed with invisible characters shown, it’s pretty clear to read/write separate fields and have an easier to parse format. This, of course, assumes that your input doesn’t include tabs or newlines… because then you’re still stuck with the same problem, just with a different delimiter.

zoover2020 2y ago

Perhaps the popularity,or lack thereof? More often than not, the bad standard wins the long term market

andrewflnr 2y ago

Can't type them on a keyboard I guess, or generally work with them in the usual text-oriented tools? Part of the appeal of CSV is you can just open it up in Notepad or something if you need to. Maybe that's more a critique of text tools than it is of ASCII record separator characters.

jandrese 2y ago

If there were visible well known characters that could be printed for those and keys on a keyboard for inputting them we would probably have RSV files. Because they are buried down in the nonprintable section of the ASCII chart they are a pain for people to deal with. All it would have taken is one more key on the keyboard, maybe splitting the tab key in half.

thesuitonym 2y ago
> If there were visible well known characters that could be printed... ...There would be datasets that include those characters, and so they wouldn't be as useful for record separators. Look into your heart and know it to be true.
- jandrese 2y ago
  I wouldn't feel too bad about blindly scrubbing those characters out of inputs unlike commas, tabs, and quotes.

sundarurfriend 2y ago

I remembered seeing a comment like this before, and... comment: https://news.ycombinator.com/item?id=26305052 https://news.ycombinator.com/item?id=26305052 comment: https://news.ycombinator.com/item?id=39679662 https://news.ycombinator.com/item?id=39679662 "ASCII Delimited Text – Not CSV or Tab Delimited Text" post [2014]: https://news.ycombinator.com/item?id=7474600 https://news.ycombinator.com/item?id=7474600 same post [2024]: https://news.ycombinator.com/item?id=42100499 https://news.ycombinator.com/item?id=42100499 comment: https://news.ycombinator.com/item?id=15440801 https://news.ycombinator.com/item?id=15440801 (...and many more.) "This comes up every single time someone mentions CSV. Without fail." - top reply from burntsushi in that last link, and it remains as true today as in 2017 :D You're not wrong though, we just need some major text editor to get the ball rolling and start making some attempts to understand these characters, and the rest will follow suit. We're kinda stuck at a local optimum which is clearly not ideal but also not troublesome enough to easily drum up wide support for ADSV (ASCII Delimiter Separated Values).

ryandrake 2y ago
Hahah, I came here to make the comment about ASCII's control characters, so I'm glad someone else beat me to it, and also that someone further pointed out that this topic comes up every time someone mentions CSV!
scythe 2y ago
>we just need some major text editor to get the ball rolling and start making some attempts to understand these characters Many text editors offer extensions APIs, including Vim, Emacs, Notepad++. But the ideal behavior would be to auto-align record separators and treat unit separators as a special kind of newline. That would allow the file to actually look like a table within the text editor. Input record separator as shift+space and unit separator as shift+enter.
- hermitcrab 2y ago
  I think it would be enough for: 1. the field separator to be shown as a special character 2. the row separator to be (optionally) be interpreted as a linefeed IIRC 1) is true for Notepad++, but not 2).

orthoxerox 2y ago

But what if one of your columns contains arbitrary binary data?

ndsipa_pomu 2y ago
You'd likely need to uuencode it or similar as CSV isn't designed for binary data.

masklinn 2y ago

> I don't get how this is not widely used as standard. It requires bespoke tools for edition, and while CSV is absolute garbage it can be ingested and produced by most spreadsheet software, as well as databases.

marxisttemp 2y ago

I was really excited when I learned of these characters, but ultimately if it’s in ASCII then it’s in-band and will eventually require escaping leading to the same problem.