5 ms·

I've always been curious, whether any popular systems made good use of US/RS/GS/FS. E.g. I'm surprised never to have seen any support for CSV files using US/RS

I've always been curious, whether any popular systems made good use of US/RS/GS/FS. E.g. I'm surprised never to have seen any support for CSV files using US/RS instead of TAB/(CR)LF or comma/(CR)LF. Or any support in Unix to use RS instead of LF or NUL to delineate records.

wglb 5y ago
I've written something that uses RS. The mind boggles at how much trouble (including security issues) if the inventors of CSV understood all of ASCII.
function_seven 5y ago
A few years ago I was consuming a SOAP webservice from a Remedy ticket handling system. While developing my end of it, I saw all sorts of weird 3-character clusters in some notes fields. I don't remember what they were exactly, but something like Ã¢ï¿½Å¾ (I just did that in CyberChef, I don't think that's what it really was at the time. It was only 3 glyphs I'm sure.) I started digging into it, a eventually worked out that some system at the start of the chain was using the ASCII US and RS characters to separate the notes! Each note would have the format of timestamp␟username␟note␞timestamp␟username␟note and so on. Then, through some chain of black boxes I never got to see, those ASCII values were converted into the visual ones I used above (U+241E and U+241F), then those were misinterpreted into Latin1 or some other non-Unicode encoding. What came out the other end were the jumbled messes I saw. I ended up just searching and splitting on those messy characters to recover the structured format of the notes. (They were all dumped into a single field in the SOAP record. I hate SOAP enough as it is, but was amazed how our systems managed to ruin it even more :)
thaumasiotes 5y ago
> I've always been curious, whether any popular systems made good use of US/RS/GS/FS. That can't be done; the concept of a "record separator" character already assumes that there is a value that can't occur within the data. If you play along with that, you're not making good use of anything. You're better off using commas and being forced to constantly confront the fact that tons of data contains commas than using a byte value that hides the problem slightly better than commas do.
- pacaro 5y ago
  I worked on a database (in a desktop GIS) that used PICK style record, field, value, sub-value separators. This worked for us because all data were text based. But it didn't work for us because we used characters 0xFC to OxFF as the separators which burned us with international customers. We'd have been better off using control characters for control. But as you correctly point out, we had to introduce escaping.
jlokier 5y ago
I've seen RS and LF used together to delineate records in log files. It's useful when there's a chance of power loss or program crash during logging and it's important to recover all the written records. With conventional logging that terminates each line with LF, if the write is interrupted, after the program restarts the next logged line is merged with the previous part-line. The result is the first line of each log file after each restart is junk and fails to match whatever patterns are scanned for later. When each line begins with RS and ends with LF, lines that are written whole are always found later.
mjevans 5y ago
CSV files have to survive human transcription and editing; it's part of what makes them useful.
- colanderman 5y ago
  I am imagining an alternate reality where text editors have basic support for displaying tabular data, formatted using US/RS/GS/FS, the same way they support TAB, CR/LF, and FF. But, modern text editors almost universally can't even manage to provide a coherent means of editing boldface text (cursors always get stuck on the wrong side of a hidden tag). So I doubt such a feature would have been well supported, even had it existed at all.
  - mjevans 5y ago
    I forgot the program, but older, probably many older, programs used to allow editing Rich Text formatted files in both a visual mode and also in a behind the scenes mode. The format codes could be both edited and copied. However that requires a smarter program to display the data the intended way. CSV files, and their cousin of sorts TSV files, can often be sufficiently read both raw as well as on line printers.

wglb 5y ago

I've written something that uses RS. The mind boggles at how much trouble (including security issues) if the inventors of CSV understood all of ASCII.

function_seven 5y ago

A few years ago I was consuming a SOAP webservice from a Remedy ticket handling system. While developing my end of it, I saw all sorts of weird 3-character clusters in some notes fields. I don't remember what they were exactly, but something like Ã¢ï¿½Å¾ (I just did that in CyberChef, I don't think that's what it really was at the time. It was only 3 glyphs I'm sure.) I started digging into it, a eventually worked out that some system at the start of the chain was using the ASCII US and RS characters to separate the notes! Each note would have the format of timestamp␟username␟note␞timestamp␟username␟note and so on. Then, through some chain of black boxes I never got to see, those ASCII values were converted into the visual ones I used above (U+241E and U+241F), then those were misinterpreted into Latin1 or some other non-Unicode encoding. What came out the other end were the jumbled messes I saw. I ended up just searching and splitting on those messy characters to recover the structured format of the notes. (They were all dumped into a single field in the SOAP record. I hate SOAP enough as it is, but was amazed how our systems managed to ruin it even more :)

thaumasiotes 5y ago

> I've always been curious, whether any popular systems made good use of US/RS/GS/FS. That can't be done; the concept of a "record separator" character already assumes that there is a value that can't occur within the data. If you play along with that, you're not making good use of anything. You're better off using commas and being forced to constantly confront the fact that tons of data contains commas than using a byte value that hides the problem slightly better than commas do.

pacaro 5y ago
I worked on a database (in a desktop GIS) that used PICK style record, field, value, sub-value separators. This worked for us because all data were text based. But it didn't work for us because we used characters 0xFC to OxFF as the separators which burned us with international customers. We'd have been better off using control characters for control. But as you correctly point out, we had to introduce escaping.

jlokier 5y ago

I've seen RS and LF used together to delineate records in log files. It's useful when there's a chance of power loss or program crash during logging and it's important to recover all the written records. With conventional logging that terminates each line with LF, if the write is interrupted, after the program restarts the next logged line is merged with the previous part-line. The result is the first line of each log file after each restart is junk and fails to match whatever patterns are scanned for later. When each line begins with RS and ends with LF, lines that are written whole are always found later.

mjevans 5y ago

CSV files have to survive human transcription and editing; it's part of what makes them useful.

colanderman 5y ago
I am imagining an alternate reality where text editors have basic support for displaying tabular data, formatted using US/RS/GS/FS, the same way they support TAB, CR/LF, and FF. But, modern text editors almost universally can't even manage to provide a coherent means of editing boldface text (cursors always get stuck on the wrong side of a hidden tag). So I doubt such a feature would have been well supported, even had it existed at all.
- mjevans 5y ago
  I forgot the program, but older, probably many older, programs used to allow editing Rich Text formatted files in both a visual mode and also in a behind the scenes mode. The format codes could be both edited and copied. However that requires a smarter program to display the data the intended way. CSV files, and their cousin of sorts TSV files, can often be sufficiently read both raw as well as on line printers.