7 ms·
Anyone that's ever had to parse arbitrary data knows of the approximately 14 jiggityzillion corner cases involved when sucking in or outputting CSV/TAB delimite
by mikestew 13y ago
Anyone that's ever had to parse arbitrary data knows of the approximately 14 jiggityzillion corner cases involved when sucking in or outputting CSV/TAB delimited formats. Yet much like virtual memory and virtual machines, we find that a solution has existed since the 60s. For those wondering about the history and use of all those strange characters in your ASCII table: http://www.lammertbies.nl/comm/info/ascii-characters.html http://www.lammertbies.nl/comm/info/ascii-characters.html
- joosters 13y agoInteresting web page! Despite many years of using ASCII and knowing some of the more common control codes, I had never even thought about what the other mysterious 0-31 codes were defined as. Something that the page doesn't mention is that CR+LF were originally two separate control codes because the action of returning the print head to the left hand side would take too long with a standard line printer. Therefore, separating the actions into two codes meant that the printer would not miss out any printable characters. (At least, I read that somewhere on the internet and assumed it was true!)
- iclelland 13y agoIt's more likely that it's because they are two separate physical actions (returning the head to the left, and advancing the paper one line). They could be used independently: You could print a line in bold, for instance, by issuing a CR without an LF and then printing the same line again. A carriage-return operation takes much longer than a single character, or even two or three. It doesn't make sense to issue two characters just to take up time. The printers always had to have some internal buffer memory (and handshaking over the communication lines to say when the buffer is full) in order not to lose any characters.
- derefr 13y ago> You could print a line in bold, for instance, by issuing a CR without an LF and then printing the same line again. True, but mildly redundant: "overprinting" was explicitly the purpose of 0x08 backspace (which had nothing, originally, to do with 0x7F deletion.)
- jff 13y agoTo overprint a whole line using 0x08, you'd need one 0x08 for each character in the line. So an N-character line overprinted that way would take N3 characters in memory. Using CR, you'd need N2 + 1 characters.
- Someone 13y agoEven ignoring additional time needed to send those extra characters, it also was a lot faster than those control-H's, and (I guess) caused way less wear on your printer.
- mzs 13y agoI had a daisey wheel printer in the late '80s that had a few characters of buffer. I had to know that it took quite some time for every CR. It would do the niave bold of the full line with CR, but if it got X^HX it would hit the X and then slide the head over a bit to the right to smear and get the bold effect, which looked much better. It was not uncommon and that's another reason a lot code did it the BS way for OS capable HC devices.
- jff 13y agoUgh, wish I had checked this after sending it, now it's too late to edit. I think the gist of it is clear, but to make sure: Overprinted with 0x08: requires 3N characters Overprinted with CR: requires 2N+1 characters
- drivers99 13y ago"You could print a line in bold, for instance, by issuing a CR without an LF and then printing the same line again." Last I checked, this still works even on laser printers (at least on a LaserJet), when sending data to it as plain text. It's not actually printing over itself, but it knows to make the repeated characters bold.
- mzs 13y agoless (among other unix tools) does this too (but you have to do one character, bs and the character again). There are more, like _, bs, character underlines (like cat there is ul that handles this specifically). If your terminal supports os (overstrike) in it's terminal description it handles that natively.
- eli 13y agoDepends on your platform. Even today a Unix/Linux line ending is almost always just LF. See: http://en.wikipedia.org/wiki/Newline http://en.wikipedia.org/wiki/Newline
- chinpokomon 13y agoUnix = LF Mac = CR DOS = CRLF The DOS line endings are inherited from previous systems, and while they precisely convey carriage and paper movement of a printer it can be a pain to deal with today.
- ygra 13y agoSomewhat ironic that this detail of how teletypewriters work was not carried over into Unix whereas every intricacy of how video terminals worked still is with us today.
- gioele 13y agoUnix = LF Mac = CR DOS = CRLF Please add to that list also HTTP = CRLF http://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html http://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html However, I still have to see I single webserver that does not accept LF alone instead of CRLF.
- eli 13y agoOnly old Macs. They switched to POSIX-style LF with OS X.
- chinpokomon 13y agoTrue, but since Mac OSX is already known by most people to be *nix based, at least this audience, I felt like that might just add to confusion. There are certainly other OSs like BeOS that I didn't mention, in part because I didn't think they added to the discussion, and also while having fond memories of BeOS, I honestly don't remember what it used. I want to say it was LF, but I no longer remember.
- russell 13y agoIn the days of mechanical line printers CR by itself allowed overprinting for special effects like underlining and bold.
- csixty4 13y agoYes. Separate Carriage Return & Line Feed date back to Murray's 1901 variant of Baudot encoding, and ASCII was created to standardize the various teletype encodings out there so it inherited this way of doing things.
- olalonde 13y agoAgreed. A few days ago, I learned that spaces are illegal between separators in CSV. For example, `"val1", "val2"` is illegal (it should be `"val1","val2"`). Kind of unintuitive given that in most languages, non-delimited spaces are insignificant.
- Pxtl 13y agoI don't think CSV is a rigidly-defined format - I'm sure some implementations will happily accept spaces between the comma and the opening quote.
- mbreese 13y agoThis is one reason why I prefer tab delimited files... The format is pretty simple with few edge cases. There really only one caveat to worry about - fields with tab characters. And that's extremely rare in my field. CSV on the other hand has a few different variations.
- wiredfool 13y agoThe thing that gets to me is that the CSV RFC is over 4000, and is less than 10 years old.
- mjn 13y agoThat's one reason I prefer TSV... the other one is that you can't use any of the regular Unix tools reliably on CSVs.
- MichaelGG 13y agoCan you explain how tab is any easier than comma? If you have to deal with escaping a character, then certainly it doesn't matter which character it is? For the general case, that is.
- mbreese 13y agoWho said anything about escaping a character? For most datasets, having actual tab characters is rare, especially if you can just replace them with spaces. Same with newlines - the just aren't needed in a lot of data. If you're dealing with user-derived text content, tab delimited files might not be the best choice. However, it's great for tabular data. With CSV, you have to escape quotes, commas, and newlines. With tab delimited, you only have to escape tabs and newlines - and that's if they can't be sanitized out to begin with.
- slowmover 13y ago> One might question why all control codes in the ASCII character set have low values, but the DEL control code has value 127. This is, because this specific character was defined for deleting data on paper tapes. Most paper tapes in that time used 7 holes to code the data. The value 127 represents a binary pattern were all seven bits are high, so when using the DEL character on an existing paper tape, all holes are punched and existing data is erased. I love this, it shows just how old the roots of ASCII are.
- sesqu 13y agoThere's a problem with that explanation: the binary representation of "deleted data" has a different meaning than the control code "delete data".
- vram22 13y agoOr just google for "ASCII table" and open the first hit: http://www.asciitable.com/ http://www.asciitable.com/ But your link does have more in-depth explanations (including historical info) for some of the control characters.
- jpttsn 13y ago> man ascii Just sayin'
- cpwright 13y agoParsing and outputting CSV can definitely be a pain, and there are a lot of corner cases. I think that this is certainly one of those times where rolling your own should be avoided if at all possible. I am a big fan of Text::CSV or the Python csv module, but even using Text::CSV have had different behavior from different versions.
- chronomex 13y agoA much more complete history is detailed here: http://web.archive.org/web/20120213005708/http://www.transbay.net/~enf/ascii/ascii.pdf http://web.archive.org/web/20120213005708/http://www.transba...