3 ms·

I have seen no end of CSV data that embeds CSV data in a field. I'm sure this will happen for whatever character format you pick.

LegionMammal978 2y ago
Out of curiosity, in which general circumstances do you tend to see this sort of data? A lot of the support I see for ASV/USV/whatever comes from the idea that this is negligibly rare, which has always sounded to me like a dangerous assumption for a general-purpose format.
- zabzonk 2y ago
  Some (many?) years ago i wrote a FOSS tool for messing with CSV called CSVfix. People used to mail me CSV files and ask how to parse them. Also, in my general consulting/contracting work I came across all sorts of weird stuff, a lot of it machine-generated.
throw0101d 2y ago
> I have seen no end of CSV data that embeds CSV data in a field. So in CSV [1] a record is separated by a CRLF (0x0D 0xA[2]) and a field value is separated by a comma (0x2c). In ADT (ASC?), the record separator is 0x1E and the unit/field separator is 0x1F. But there are two more separators defined: file (0x1C) and group (0x1D). I'm not sure if it's defined anywhere, but if if you wish to embed ADT data with-in a ADT file, and have it as part of the CSV-equivalent field (unit), you could say that: after the 0x1E record separator, put a group separator (0x1D) which will denote the beginning of ADT sequence which will be treated as a unit value. The end of the value shall ("MUST"?) be denoted by another group separator, after which a unit separator will indicate the next field. The fact that there are four separation characters would allow for some to be used for embedding applications to tell parsers that a new 'level' of parsing is being done. [1] https://datatracker.ietf.org/doc/html/rfc4180 https://datatracker.ietf.org/doc/html/rfc4180 [2] https://www.man7.org/linux/man-pages/man7/ascii.7.html https://www.man7.org/linux/man-pages/man7/ascii.7.html

LegionMammal978 2y ago

Out of curiosity, in which general circumstances do you tend to see this sort of data? A lot of the support I see for ASV/USV/whatever comes from the idea that this is negligibly rare, which has always sounded to me like a dangerous assumption for a general-purpose format.

zabzonk 2y ago
Some (many?) years ago i wrote a FOSS tool for messing with CSV called CSVfix. People used to mail me CSV files and ask how to parse them. Also, in my general consulting/contracting work I came across all sorts of weird stuff, a lot of it machine-generated.

throw0101d 2y ago

> I have seen no end of CSV data that embeds CSV data in a field. So in CSV [1] a record is separated by a CRLF (0x0D 0xA[2]) and a field value is separated by a comma (0x2c). In ADT (ASC?), the record separator is 0x1E and the unit/field separator is 0x1F. But there are two more separators defined: file (0x1C) and group (0x1D). I'm not sure if it's defined anywhere, but if if you wish to embed ADT data with-in a ADT file, and have it as part of the CSV-equivalent field (unit), you could say that: after the 0x1E record separator, put a group separator (0x1D) which will denote the beginning of ADT sequence which will be treated as a unit value. The end of the value shall ("MUST"?) be denoted by another group separator, after which a unit separator will indicate the next field. The fact that there are four separation characters would allow for some to be used for embedding applications to tell parsers that a new 'level' of parsing is being done. [1] https://datatracker.ietf.org/doc/html/rfc4180 https://datatracker.ietf.org/doc/html/rfc4180 [2] https://www.man7.org/linux/man-pages/man7/ascii.7.html https://www.man7.org/linux/man-pages/man7/ascii.7.html