5 ms·

The shortcoming of using the control characters is that there is no easy way to type them on a keyboard. You can trivially edit csv in a text editor.

edlebert 2y ago
And equally as important, you can cat a CSV file and easily understand it.
- bediger4000 2y ago
  Or concatenate them, or diff or grep.
zabzonk 2y ago
> no easy way to type them on a keyboard or probably to see them in the screen in any sensible form. Also, what to do if you want to embed those ctrl characters in a field? you are likely back to the way that CSV does it with quotes, commas and CRLFs.
- velcrovan 2y ago
  That’s the whole point of having field and record separators as distinct values in ASCII. There is no other valid use for them, so no escaping is necessary. Have you ever used ASCII value 30 for anything, anywhere, in your life?
  - evoke4908 2y ago
    No, no one has ever used that character because it can't be typed or displayed which is the entire point of the thread you're replying to
    - velcrovan 2y ago
      But that’s my point. The person I responded to was objecting that they might want to use control characters like ASCII 30 as values in a field. As you and I agree, that will never happen.
      - croes 2y ago
        What about the cases where people don’t want to use it but still do? People C&P values from a source that includes those characters and they‘ll break your export file
    - cornel_io 2y ago
      Sometimes the content of a cell in e.g. a CSV file will be another CSV file. This inevitably happens with pretty much every format, whether CSV, TSV, JSON, etc, so you'll need some sort of quoting rule no matter what you do.
  - zabzonk 2y ago
    I have seen no end of CSV data that embeds CSV data in a field. I'm sure this will happen for whatever character format you pick.
    - LegionMammal978 2y ago
      Out of curiosity, in which general circumstances do you tend to see this sort of data? A lot of the support I see for ASV/USV/whatever comes from the idea that this is negligibly rare, which has always sounded to me like a dangerous assumption for a general-purpose format.
      - zabzonk 2y ago
        Some (many?) years ago i wrote a FOSS tool for messing with CSV called CSVfix. People used to mail me CSV files and ask how to parse them. Also, in my general consulting/contracting work I came across all sorts of weird stuff, a lot of it machine-generated.
    - throw0101d 2y ago
      > I have seen no end of CSV data that embeds CSV data in a field. So in CSV [1] a record is separated by a CRLF (0x0D 0xA[2]) and a field value is separated by a comma (0x2c). In ADT (ASC?), the record separator is 0x1E and the unit/field separator is 0x1F. But there are two more separators defined: file (0x1C) and group (0x1D). I'm not sure if it's defined anywhere, but if if you wish to embed ADT data with-in a ADT file, and have it as part of the CSV-equivalent field (unit), you could say that: after the 0x1E record separator, put a group separator (0x1D) which will denote the beginning of ADT sequence which will be treated as a unit value. The end of the value shall ("MUST"?) be denoted by another group separator, after which a unit separator will indicate the next field. The fact that there are four separation characters would allow for some to be used for embedding applications to tell parsers that a new 'level' of parsing is being done. [1] https://datatracker.ietf.org/doc/html/rfc4180 https://datatracker.ietf.org/doc/html/rfc4180 [2] https://www.man7.org/linux/man-pages/man7/ascii.7.html https://www.man7.org/linux/man-pages/man7/ascii.7.html
  - crazygringo 2y ago
    > There is no other valid use for them, so no escaping is necessary. Famous last words... > Have you ever used ASCII value 30 for anything, anywhere, in your life? No but if this format took off I would be. And then because text editors start displaying them so the files make sense, people start using them for other purposes, like separating values on a line or copying spreadsheet cells to the clipboard, because it's more "semantic", and those lines get pasted into a field in a tool that exports its data as CSV. What then?
jiehong 2y ago
Maybe you with a hex editor that shows dual pane: hex and text.
- NBJack 2y ago
  I asser that is still not convenient or scalable. You will need to mentally parse each number (two characters) to ensure it is correct. Compare this with a simple glyph (i.e. a single comma) which is easy to eyeball.
- deleted 2y ago
  [deleted]
mattpallissard 2y ago
It looks like it would just be be ctrl+^, which seems pretty straightforward.
- smarx007 2y ago
  ctrl+_ to separate cells, ctrl+^ to separate rows - works perfectly in notepad++. I think the proposal can be improved by using ctrl+^ followed by a newline as a row separator, it looks much more readable plus will allow various line-based CLI tools to be used unless there are newlines in the cells.
- dahart 2y ago
  Straightforward is completely subjective. But a comma is relatively much simpler in an absolute sense. Ctrl+shift+6 is a 3-key chord, it’s potentially hard to discover (I can’t say I’ve actually ever seen it), it seems likely to be overridden by applications, and caret isn’t a natural separator and is more commonly used for other things, like exponents. A comma is 1 key on the keyboard, and it’s already a natural separator; the very meaning of comma is separator. Note how many commas are used in this thread compared to the number of record separators. :P Having to type both ctrl+shift+6 and ctrl+shift+minus a lot seems like a small physical and mental friction compared to using commas and returns each time a character is typed, that adds up to a lot of physical and mental friction over time. Enough that the eventual implication is that you need better tooling than a text editor provides in order to author delimited files, enough that it sort of undermines the idea of having a text file. It’s a mistake to think that because a key chord exists that it’s a solved problem, and a mistake to underestimate the value of making commonly used items as simple as possible, especially if it’s going to affect a lot of different people.
  - mattpallissard 2y ago
    While I agree with the sentiment of that, it still seems a lot easier than escaping characters. I would probably opt for a mix. Control separation with new lines
    - 2y ago
velcrovan 2y ago
This is a bit silly. Any modern text editor (whether vim or VSCode or BBEdit or Notepad++ or whatever) is capable of displaying control characters and of copy/pasting them. Keyboard shortcuts for inserting any characters whatsoever are easy to add. And even with CSV files, if you’re editing them by hand rather than manipulating them with code, you’re probably doing it wrong.
- pvg 2y ago
  All of this is easy (use the proper editor, configure it for this particular weirdass situation, do something other than the thing you want to do, etc) in a way that’s exactly analogous to ‘you can spin up your own dropbox over the weekend with ftp and rsync’.
Asooka 2y ago
Technically there is (not sure about easy, most of these require ctrl+shift, but they are on the keyboard): CTRL DEC HEX CHR NAME Ctrl-\ 28 1C FS File Separator (Right Arrow) Ctrl-] 29 1D GS Group Separator (Left Arrow) Ctrl-^ 30 1E RS Record Separator (Up Arrow) Ctrl-_ 31 1F US Unit Separator (Down Arrow) From https://www3.rocketsoftware.com/bluezone/help/v42/en/bzadmin/APPENDIX_C/ASCII_Character_Set.htm https://www3.rocketsoftware.com/bluezone/help/v42/en/bzadmin... You can type them in the terminal by prefixing with Ctrl-V, so you can enter a record separator by pressing Ctrl-V, Ctrl-Shift-6. Typing Ctrl-\ is tricky because some programs interpret it to mean end of input, e.g. it exits the Python repl, but I don't think that one in particular is super important to type manually. In hindsight if these were assigned to Ctrl-<letter>, they would have been a lot easier to type and use.
- zabzonk 2y ago
  ctrl-letters are used for other things

edlebert 2y ago

And equally as important, you can cat a CSV file and easily understand it.

bediger4000 2y ago
Or concatenate them, or diff or grep.

zabzonk 2y ago

> no easy way to type them on a keyboard or probably to see them in the screen in any sensible form. Also, what to do if you want to embed those ctrl characters in a field? you are likely back to the way that CSV does it with quotes, commas and CRLFs.

velcrovan 2y ago
That’s the whole point of having field and record separators as distinct values in ASCII. There is no other valid use for them, so no escaping is necessary. Have you ever used ASCII value 30 for anything, anywhere, in your life?
- evoke4908 2y ago
  No, no one has ever used that character because it can't be typed or displayed which is the entire point of the thread you're replying to
  - velcrovan 2y ago
    But that’s my point. The person I responded to was objecting that they might want to use control characters like ASCII 30 as values in a field. As you and I agree, that will never happen.
    - croes 2y ago
      What about the cases where people don’t want to use it but still do? People C&P values from a source that includes those characters and they‘ll break your export file
  - cornel_io 2y ago
    Sometimes the content of a cell in e.g. a CSV file will be another CSV file. This inevitably happens with pretty much every format, whether CSV, TSV, JSON, etc, so you'll need some sort of quoting rule no matter what you do.
- zabzonk 2y ago
  I have seen no end of CSV data that embeds CSV data in a field. I'm sure this will happen for whatever character format you pick.
  - LegionMammal978 2y ago
    Out of curiosity, in which general circumstances do you tend to see this sort of data? A lot of the support I see for ASV/USV/whatever comes from the idea that this is negligibly rare, which has always sounded to me like a dangerous assumption for a general-purpose format.
    - zabzonk 2y ago
      Some (many?) years ago i wrote a FOSS tool for messing with CSV called CSVfix. People used to mail me CSV files and ask how to parse them. Also, in my general consulting/contracting work I came across all sorts of weird stuff, a lot of it machine-generated.
  - throw0101d 2y ago
    > I have seen no end of CSV data that embeds CSV data in a field. So in CSV [1] a record is separated by a CRLF (0x0D 0xA[2]) and a field value is separated by a comma (0x2c). In ADT (ASC?), the record separator is 0x1E and the unit/field separator is 0x1F. But there are two more separators defined: file (0x1C) and group (0x1D). I'm not sure if it's defined anywhere, but if if you wish to embed ADT data with-in a ADT file, and have it as part of the CSV-equivalent field (unit), you could say that: after the 0x1E record separator, put a group separator (0x1D) which will denote the beginning of ADT sequence which will be treated as a unit value. The end of the value shall ("MUST"?) be denoted by another group separator, after which a unit separator will indicate the next field. The fact that there are four separation characters would allow for some to be used for embedding applications to tell parsers that a new 'level' of parsing is being done. [1] https://datatracker.ietf.org/doc/html/rfc4180 https://datatracker.ietf.org/doc/html/rfc4180 [2] https://www.man7.org/linux/man-pages/man7/ascii.7.html https://www.man7.org/linux/man-pages/man7/ascii.7.html
- crazygringo 2y ago
  > There is no other valid use for them, so no escaping is necessary. Famous last words... > Have you ever used ASCII value 30 for anything, anywhere, in your life? No but if this format took off I would be. And then because text editors start displaying them so the files make sense, people start using them for other purposes, like separating values on a line or copying spreadsheet cells to the clipboard, because it's more "semantic", and those lines get pasted into a field in a tool that exports its data as CSV. What then?

jiehong 2y ago

Maybe you with a hex editor that shows dual pane: hex and text.

NBJack 2y ago
I asser that is still not convenient or scalable. You will need to mentally parse each number (two characters) to ensure it is correct. Compare this with a simple glyph (i.e. a single comma) which is easy to eyeball.
deleted 2y ago
[deleted]

mattpallissard 2y ago

It looks like it would just be be ctrl+^, which seems pretty straightforward.

smarx007 2y ago
ctrl+_ to separate cells, ctrl+^ to separate rows - works perfectly in notepad++. I think the proposal can be improved by using ctrl+^ followed by a newline as a row separator, it looks much more readable plus will allow various line-based CLI tools to be used unless there are newlines in the cells.
dahart 2y ago
Straightforward is completely subjective. But a comma is relatively much simpler in an absolute sense. Ctrl+shift+6 is a 3-key chord, it’s potentially hard to discover (I can’t say I’ve actually ever seen it), it seems likely to be overridden by applications, and caret isn’t a natural separator and is more commonly used for other things, like exponents. A comma is 1 key on the keyboard, and it’s already a natural separator; the very meaning of comma is separator. Note how many commas are used in this thread compared to the number of record separators. :P Having to type both ctrl+shift+6 and ctrl+shift+minus a lot seems like a small physical and mental friction compared to using commas and returns each time a character is typed, that adds up to a lot of physical and mental friction over time. Enough that the eventual implication is that you need better tooling than a text editor provides in order to author delimited files, enough that it sort of undermines the idea of having a text file. It’s a mistake to think that because a key chord exists that it’s a solved problem, and a mistake to underestimate the value of making commonly used items as simple as possible, especially if it’s going to affect a lot of different people.
- mattpallissard 2y ago
  While I agree with the sentiment of that, it still seems a lot easier than escaping characters. I would probably opt for a mix. Control separation with new lines
  - 2y ago

velcrovan 2y ago

This is a bit silly. Any modern text editor (whether vim or VSCode or BBEdit or Notepad++ or whatever) is capable of displaying control characters and of copy/pasting them. Keyboard shortcuts for inserting any characters whatsoever are easy to add. And even with CSV files, if you’re editing them by hand rather than manipulating them with code, you’re probably doing it wrong.

pvg 2y ago
All of this is easy (use the proper editor, configure it for this particular weirdass situation, do something other than the thing you want to do, etc) in a way that’s exactly analogous to ‘you can spin up your own dropbox over the weekend with ftp and rsync’.

Asooka 2y ago

Technically there is (not sure about easy, most of these require ctrl+shift, but they are on the keyboard): CTRL DEC HEX CHR NAME Ctrl-\ 28 1C FS File Separator (Right Arrow) Ctrl-] 29 1D GS Group Separator (Left Arrow) Ctrl-^ 30 1E RS Record Separator (Up Arrow) Ctrl-_ 31 1F US Unit Separator (Down Arrow) From https://www3.rocketsoftware.com/bluezone/help/v42/en/bzadmin/APPENDIX_C/ASCII_Character_Set.htm https://www3.rocketsoftware.com/bluezone/help/v42/en/bzadmin... You can type them in the terminal by prefixing with Ctrl-V, so you can enter a record separator by pressing Ctrl-V, Ctrl-Shift-6. Typing Ctrl-\ is tricky because some programs interpret it to mean end of input, e.g. it exits the Python repl, but I don't think that one in particular is super important to type manually. In hindsight if these were assigned to Ctrl-<letter>, they would have been a lot easier to type and use.

zabzonk 2y ago
ctrl-letters are used for other things