17 ms·

CR+LF Has a Long History

102 points by rhabarba 5y ago

ck2 5y ago
My first "computer" was a thermal paper terminal, with modem-phone coupler to dial-in to Compuserve (like $6 an hour after 6pm with minimums) The separate actions of CR and LF were definitely important. PDP-11 at school also was a paper terminal with same importance. It's really amazing how far we've come considering what everyone has in their pocket.
TillE 5y ago
It's a dreadfully common story: some ugly hack which was necessary at the time persists for decades (or centuries, in the case of legal/social systems), because nobody can really be bothered to change it. The extra file size is not much of an issue, but it does mean we still require stupid workarounds with Git and anything else that needs to handle cross-platform plaintext.
- chrismorgan 5y ago
  But it’s still worse when the ugly hack was only useful for a very short time, probably wasn’t even necessary, and ruins other things by existing. UTF-16 is my favourite to complain of: created because a few large companies had put a couple of years of heavy investment into UCS-2—which was a massive breaking change itself—and didn’t want to let that go to waste (with all the bother they’d been putting users through), even though by the time they actually made UTF-16, UTF-8 with its ASCII compatibility had been invented. And so they ruined Unicode for everyone for ever after with that accursed abomination surrogates and the distinction between code points and scalar values.
  - colejohnson66 5y ago
    And now Unicode have declared themselves kneecapped to ~20 bits (1,111,998) because of it. I doubt we'll run out of space like we did with 16 bits, but never say never. The whole reason UCS-2 was used was companies saying "16 bits is enough" and then it not actually being enough. Almost every issue with Unicode is because developers made assumptions, despite Unicode themselves providing guidance saying otherwise.
    - layer8 5y ago
      Almost every issue with software in general is because developers made assumptions.
- wrs 5y ago
  How is this a "hack"? If anything, not using CRLF line endings is the hack, because it means you can't just send the file straight to a terminal: the line endings have to be translated by something on the way to the terminal for it to display properly.
  - moring 5y ago
    The "hack" is to wrongly define "move down, move left" to be the same as "line ending". It isn't. The first are commands to a machine, the second are elements of encoding text in binary storage. Or more general, the "hack" is to act as if ASCII was a character encoding, when it's actually the encoding for a set of commands to a machine.
    - hnlmorg 5y ago
      It's both, and more. It's also document encoding (there's characters to encode structured data too). Given the era ASCII was invented, I wouldn't really call it a hack. Character encodings were one of the lowest level abstractions. The size of a byte wasn't always fixed to 8 bits and it was the number of bits in your character set that often defined how big your byte was (yes, I know ASCII is technically 7-bit but one of the original considerations was that the 8th bit would be a parity bit: https://en.wikipedia.org/wiki/Parity_bit https://en.wikipedia.org/wiki/Parity_bit ). It's also the reason the lowest size of addressable data is a byte / char. Bare in mind that we are talking about an era before operating systems as we recognize them. Back then each machine would have it's own operating system and some institutes would even have different operating systems despite having the same machines. Really these operating systems were more like firmware than what we have today. So there wasn't any distinction between character encodings and machine commands but there were differences between machines. So it made sense that ASCII was cross purpose because that's exactly what characters were: raw addressable data containing instructions. Those instructions were sometimes printable data, it was sometimes formatting data, and it was sometimes control data. I doubt any designers of ASCII could have even imagined the way we use computers these days when writing that specification. Even just the fact that character encodings have become an interchangeable abstraction for rendering text would have blown their minds. Never mind everything else.
- wruza 5y ago
  because nobody can really be bothered to change it Python 2 to 3 all the things, I guess? You’d then need another workaround choosing between CR (now obsolete), CRLF, LF and RS (renamed to LE), the latter being a true newline.
- hnlmorg 5y ago
  It wasn't a hack though. Characters were low level addressable data. It's no coincidence that on machines that had a 6-bit wide character set the smallest addressable data (a byte) was also 6-bits (I know ASCII is 7-bits but that was to reduce transmission overhead or allow for a parity bit when required, eg on punch tape or cards that often supported 8 columns of bits). The way machines worked back then would be you'd address a byte of data, that byte contained instructions. Those instructions were sometimes printable data, it was sometimes formatting data, and it was sometimes control data. It wasn't until operating systems became more sophisticated, hardware become more sophisticated and the two became decoupled, that character encodings became the abstraction for rendering text that we think of them as today.
  - kps 5y ago
    Character codes predate computers. CR and LF codes date to 1901.
    - hnlmorg 5y ago
      Even earlier than that. CR and LF go back to Baudot code (which is where term 'baud' also originates) and that was created in the 1870s. https://en.wikipedia.org/wiki/Baudot_code https://en.wikipedia.org/wiki/Baudot_code
      - kps 5y ago
        Baudot's (ITA1) was linear, with no concept of lines. Murray's (which evolved into ITA2 with minor changes) added CR LF, since he developed a typewriter-like system.
        hnlmorg 5y ago
        Ahhh I see where you've gotten that 1901 date from now: > In 1901, Baudot's code was modified by Donald Murray > The Murray code also introduced what became known as "format effectors" or "control characters" – the CR (Carriage Return) and LF (Line Feed) codes. (source: https://en.wikipedia.org/wiki/Baudot_code#Murray_code https://en.wikipedia.org/wiki/Baudot_code#Murray_code) Thanks for the correction there. Learned something new today :)
drewcoo 5y ago
> CR and LF are used for each new line. The author's insistence on calling CR+LF "new line" only serves to confuse. LF is line feed, thus new line. The CR is to readjust the write head back to the starting position.
- bottled_poe 5y ago
  More intuitively, CR is the horizontal dimension and LF is the vertical dimension in the printer head position on the page, which makes it more obvious which is the new line char.
  - justsomehnguy 5y ago
    > More intuitively Not for me. I get the dimensions but hell if I can remember it. But mnemonic 'ReturN' (for the Windows style \r\n) helps (well, a bit) here: \r: carrier Return \n: liNe feed
    - mananaysiempre 5y ago
      A physical printer (like a dot-matrix one) couldn’t care less whether you told it to CR LF or to LF CR, you’ll end up at the start of the next line either way; neither does a terminal emulator, nornally (but mind the generic kernel terminal driver in between, which normally has the onlcr knob turned on). Apparently some systems even used LF CR as the line terminator. (Bare CR is more well-known, having been used by classic Mac OS, and also makes a bit of sense if you remember the parallel port had an AutoLF line that instructed the attached printer to translate CR to CR LF. I don’t know if that is the actual reason for the convention.) The caveat is that advancing the paper vertically is relatively fast, but backing up the carriage[1] horizontally can take some time, so on a dumb and slow printer LF CR abc might end up printing abc in the wrong place while the carriage is still moving backwards, while on a smart but slow printer it will just be slower than CR LF abc. I suspect the dumb part is the origin of the CR NUL convention in Net-ASCII[2]; at least both termcap (dC capability) and terminfo (padding machinery) can describe, and curses handles, the possibility that the terminal requires a delay between issuing a CR and printing new characters at the beginning of the line. [1] It’s a carriage, as in a moving thing that contains useful stuff (cf 3D printers), not a carrier, as in a bare unmodulated signal (cf modems). [2] https://tools.ietf.org/html/rfc20 https://tools.ietf.org/html/rfc20, which probably takes the prize as the lowest-numbered RFC you could still encounter as an up-to-date reference.
j1elo 5y ago
What a coincidence to see CRLF discussed in HN front page, this bit me just yesterday! While using the aws cli tool on Linux (the Docker containerized version) with the "text" output format, it turns out each output line ends with CRLF ("\r\n") instead of simply LF ("\n") as is common for any Linux cli tool, so I had to add a post-process step in my commands to remove CRs (a chained call to tr -d '\r'). Otherwise, my bash variables would end up containing those pesky carriage returns, and debug logs would show up unexpectedly bizarre. In retrospect it's obvious what happened, but it got me confused for a good while. Why would it output CRLF at all? Also of course, not a single mention of this in any of the docs I was reading.
- bee_rider 5y ago
  Vague memory, so take it with a grain of salt, but I think this might be a way of getting the break to show up in an editor that expects \n or one that expects \r. Maybe most editors that expect one simply ignore the other?
- CornCobs 5y ago
  The dos2unix tool is purpose-built for this!
  - Josteniok 5y ago
    Or flip (https://ccrma.stanford.edu/~craig/utility/flip/ https://ccrma.stanford.edu/~craig/utility/flip/)! I've been carrying this thing around with me for years.
  - mananaysiempre 5y ago
    Note that dos2unix and unix2dos do what you expect when used as filters, but overwrite the specified file in place when given a file name. This is a footgun second only to one in pdftotext, arguably, which (unsurprisingly) refuses to work as a filter and (surprisingly) will silently create a matching .txt file when passed a single .pdf as an argument.
    - spc476 5y ago
timonoko 5y ago
Remarkably uninteresting. Next: Let us discuss what the Unicode Character 'REVERSE LINE FEED' (U+008D) is good for.
- tdeck 5y ago
  VERTICAL TAB is much more puzzling.
  - mananaysiempre 5y ago
    Devices that needed to produce output on printed forms would use a physical reference to ensure the text went to the right place. One of the commands a tabulator could give the typewriter it drove was to move the form to the next marked position—a literal tab, or maybe a functional equivalent like a hope in a punched tape. Thus, vertical tab.
  - asddubs 5y ago
    interesting, hadn't heard of it before. Apparently it was primarily for printing purposes https://stackoverflow.com/questions/3380538/what-is-a-vertical-tab https://stackoverflow.com/questions/3380538/what-is-a-vertic...
- mark-r 5y ago
  Uninteresting depends on your perspective. For anyone old enough to have used a Teletype certainly it is no mystery, but how many of us are left? Speaking of, a Teletype didn't understand anything outside of the ASCII range. So what was reverse line feed good for? Absolutely nothing! The author remarks that his Teletype was old because it didn't have lower case. But in my experience that was almost universal - I think I only saw a single Teletype that had lower case.
  - timonoko 5y ago
    Olivetti Teletype had lower case. I had an interesting episode, where the University of Nitpicking Nazis discussed if my thesis, printed entirely on Teletype, was acceptable. -- It was not as good-looking as with IBM Selectric, but quite comparable to regular manual typewriters.
wrs 5y ago
There's no need for conspiracy theories about adding delays. CR and LF were just separate functions and it's useful for them to be separate. CRT terminals (or emulations thereof) still work the same way. You have to send CR and LF to a VT100 to get the cursor to both move to the left margin and then go to the next line down. The Unix stty command lets you set cr0/1/2/3 modes for inserting NUL characters after a CR for exactly this reason, to give the carriage time to return on your printing terminal.
- blihp 5y ago
  Not just terminals: most printers until the mid-80's or so also cared about the distinction for the same reason. They started adding DIP switches allowing you change the default behavior and by the end of the 80's the need for CR+LF was pretty much a thing of the past. While the original reason for this distinction probably did date back to teletypes, some practical use cases evolved out of the capability. IIRC, some word processors used the ability to provide an overstrike mode before bold was a thing most printers could do. ASCII art photos were also a thing both for printers that didn't support a graphics mode as well as providing a way of printing a larger 'image' on the paper faster while also using less (ribbon) ink.
  - romwell 5y ago
    The distinction goes back to typewriters. The author mentions that the typewriters had a return lever that could also do a line feed. What they missed is that: * You could still freely move the carriage without advancing the paper if you wanted: there was a carriage release button; * You could also advance paper manually at any time, while keeping the carriage at its current position; * You could often adjust line spacing, including setting it to 0, which means that the big lever would only return the carriage, but not advance the paper. In short, CR and LF were separate useful functions on a typewriter. The backspace key was useful for the same reason; obviously, it wouldn't erase text - it would move the carriage just like the space bar would, but in opposite direction. These were necessary for: * Underlining text (by overtyping the underscore character over lines of text); * Adding accents, or crossing zeros with overtyping; * Bold text by retyping the same line; Etc. Tab stops in your favorite document editor come from the same era: the "Tab" key would release the carriage, and the spring would pull it forward until the next physical stop (whose positions you would manually set). The feature was made to make typing tabular data easier, hence the name. That said, the author has a solid point about why we have CR and LF in that order. To wit, on typewriters, the LF+CR lever would advance the paper first, because it made sense both mechanically and from the UX perspective (you would nudge it to advance paper by whatever line spacing you set, as opposed to manual advance which could be a fraction of that). Returning the carriage, and then advancing paper indeed makes more sense on the terminal, where all operations but CR take a fixed amount of time. That was the value of the article, really : to give motivation for CR+LF rather than LF+CR for line ending. The former works well with the terminals, the latter - with typewriters. Computers therefore use CR+LF.
jfoutz 5y ago
I'm a little skeptical. Equipment from this era was designed by slide rule and sent to draftsmen to draw up and send down to the factory floor. The solution to problems was more power. I would imagine some wily old programmer knew commands would be latched, and maybe as maintenance fell behind, the CR would lag a little, but could be trusted to complete. Better to do CR LF than LF CR. but they both have to happen. I dunno. Maybe this model sucked. But every teletype I've ever used was kinda scary, those things will take your fingers off. They're power tools capable of violence. I'd guess, if you pulled out the maintenance guide, and applied proper lubrication, and replaced wear parts, it wouldn't miss those deadlines. I guess you might need to pull out the schematic and verify the electronics are still in spec. I mean, yeah, the author has a point, but I'd believe it's a wear issue that this teletype doesn't reliably return in time.
- Animats 5y ago
  The carriage return time spec for the 5-level teletypes is that CR LF should be sufficient. I've restored two Teletype Model 15 machines, and both work properly with plain CR LF. They're 45 baud, so two characters is 400ms. 8-level, 110 baud devices often needed more time. The early Model 15 teletypes would just overtype at the right margin if you didn't send a CR, and would type on top of the previous line if you didn't send an LF. Mechanical options requiring extra parts were later offered so that overrunning the end of line would force a CR LF. One letter would be typed about halfway across the page, but it was better than losing an entire line because one character was garbled. This was useful for links over shortwave radio, where losing characters was not uncommon. With that feature, CR implied LF. I have two Model 15 machines; one has that feature, the other does not. Too many years ago, I had to overhaul a Teletype driver to handle all the serial devices that appeared just before memory got cheap and everything got buffering. I put in a delay function of the form K1 + linelength*K2. Some serial printers worked with wide paper and needed long delays for their long lines. There was at least one device [1] that needed a negative K2 term. It had a minimum time to print a line, so short lines needed an extra delay. (The early Teletype machines are the dream of the right-to-repair people. Everything is fully documented, all parts are removeable, and they are very repairable. They're also high-maintenance. They need regular oiling, and there are several hundred oiling points and three different lubricants. In heavy use, they need an annual cleaning which involves partial disassembly, soaking in a cleaning bath, and lubrication. They have several hundred adjustments, and a whole "adjustments" manual. Nobody would put up with that today. That's the downside of "right to repair" - the hardware is repairable, but clunky.) [1] http://archive.computerhistory.org/resources/text/DataInterface/Data_Interface.DI-240.1973.102646310.pdf http://archive.computerhistory.org/resources/text/DataInterf...
ggm 5y ago
You couldn't do overtype without them being distinct. I think It's that simple.
- tonyedgecombe 5y ago
  That has always been my understanding. If you want bold, underline or strikethrough then you need to be able to send a carriage return on its own.
  - romwell 5y ago
    You could just backspace repeatedly, but it'd be a waste of bits.
- justinmchase 5y ago
  Well you could if you have a line-unfeed. So by default it would do both but then in the uncommon case where you want overtype you could go back up. A little late for that insight, if its even worthwhile, but typically I would optimize the common case by default and add features for doing advanced / uncommon things.
bullen 5y ago
HTTP also uses CRLF but the spec is flexible I think. Still most if not all implementations use it. That is alot of bytes when you sum them up. Also CRLFCRLF is used to demarkate end of headers and chunks.
- p_l 5y ago
  All textual protocols with legacy of IETF style design use CRLF, and that includes HTTP which heavily drunk from that legacy. Personally I suspect it's because ARPA was a bit cheap on some things and thus they used typewriters instead of protocol analyzers ;)
ochrist 5y ago
Back in the eighties I worked in the military with teletypes. Some examples are: https://www.radiomuseum.org/r/siemens_fernschreiber_t37.html https://www.radiomuseum.org/r/siemens_fernschreiber_t37.html https://www.cryptomuseum.com/telex/siemens/t100/index.htm https://www.cryptomuseum.com/telex/siemens/t100/index.htm https://www.camion-militar.com/spa/item/ART02979.html https://www.camion-militar.com/spa/item/ART02979.html There were also other makes, like Olivetti. They were connected to various networks, e.g. military, the civil Telex network or the ICAO AFTN network. https://en.wikipedia.org/wiki/Telex https://en.wikipedia.org/wiki/Telex https://en.wikipedia.org/wiki/Aeronautical_Fixed_Telecommunication_Network https://en.wikipedia.org/wiki/Aeronautical_Fixed_Telecommuni... They all used a 5 bit character set Baudot/ITA2: https://en.wikipedia.org/wiki/Baudot_code https://en.wikipedia.org/wiki/Baudot_code (I can still read a paper tape message directly from the reel) As these teletypes were quite slow (typically 50 baud) and mechanical, the procedure was for line shifts to always be CR CR LF. The reason for this is to allow for the carriage (printing head) to have time enough to move to the start of the line before the line feed. This was enforced in the procedures, so you might end up with a rejected message if you did not do it properly. When computers were introduced in the various networks, ITA2 was converted to ASCII and some of these restrictions were relaxed a bit.
justinmchase 5y ago
And hopefully a short future!

102 points by rhabarba 5y ago

ck2 5y ago

My first "computer" was a thermal paper terminal, with modem-phone coupler to dial-in to Compuserve (like $6 an hour after 6pm with minimums) The separate actions of CR and LF were definitely important. PDP-11 at school also was a paper terminal with same importance. It's really amazing how far we've come considering what everyone has in their pocket.

TillE 5y ago

It's a dreadfully common story: some ugly hack which was necessary at the time persists for decades (or centuries, in the case of legal/social systems), because nobody can really be bothered to change it. The extra file size is not much of an issue, but it does mean we still require stupid workarounds with Git and anything else that needs to handle cross-platform plaintext.

chrismorgan 5y ago
But it’s still worse when the ugly hack was only useful for a very short time, probably wasn’t even necessary, and ruins other things by existing. UTF-16 is my favourite to complain of: created because a few large companies had put a couple of years of heavy investment into UCS-2—which was a massive breaking change itself—and didn’t want to let that go to waste (with all the bother they’d been putting users through), even though by the time they actually made UTF-16, UTF-8 with its ASCII compatibility had been invented. And so they ruined Unicode for everyone for ever after with that accursed abomination surrogates and the distinction between code points and scalar values.
- colejohnson66 5y ago
  And now Unicode have declared themselves kneecapped to ~20 bits (1,111,998) because of it. I doubt we'll run out of space like we did with 16 bits, but never say never. The whole reason UCS-2 was used was companies saying "16 bits is enough" and then it not actually being enough. Almost every issue with Unicode is because developers made assumptions, despite Unicode themselves providing guidance saying otherwise.
  - layer8 5y ago
    Almost every issue with software in general is because developers made assumptions.
wrs 5y ago
How is this a "hack"? If anything, not using CRLF line endings is the hack, because it means you can't just send the file straight to a terminal: the line endings have to be translated by something on the way to the terminal for it to display properly.
- moring 5y ago
  The "hack" is to wrongly define "move down, move left" to be the same as "line ending". It isn't. The first are commands to a machine, the second are elements of encoding text in binary storage. Or more general, the "hack" is to act as if ASCII was a character encoding, when it's actually the encoding for a set of commands to a machine.
  - hnlmorg 5y ago
    It's both, and more. It's also document encoding (there's characters to encode structured data too). Given the era ASCII was invented, I wouldn't really call it a hack. Character encodings were one of the lowest level abstractions. The size of a byte wasn't always fixed to 8 bits and it was the number of bits in your character set that often defined how big your byte was (yes, I know ASCII is technically 7-bit but one of the original considerations was that the 8th bit would be a parity bit: https://en.wikipedia.org/wiki/Parity_bit https://en.wikipedia.org/wiki/Parity_bit ). It's also the reason the lowest size of addressable data is a byte / char. Bare in mind that we are talking about an era before operating systems as we recognize them. Back then each machine would have it's own operating system and some institutes would even have different operating systems despite having the same machines. Really these operating systems were more like firmware than what we have today. So there wasn't any distinction between character encodings and machine commands but there were differences between machines. So it made sense that ASCII was cross purpose because that's exactly what characters were: raw addressable data containing instructions. Those instructions were sometimes printable data, it was sometimes formatting data, and it was sometimes control data. I doubt any designers of ASCII could have even imagined the way we use computers these days when writing that specification. Even just the fact that character encodings have become an interchangeable abstraction for rendering text would have blown their minds. Never mind everything else.
wruza 5y ago
because nobody can really be bothered to change it Python 2 to 3 all the things, I guess? You’d then need another workaround choosing between CR (now obsolete), CRLF, LF and RS (renamed to LE), the latter being a true newline.
hnlmorg 5y ago
It wasn't a hack though. Characters were low level addressable data. It's no coincidence that on machines that had a 6-bit wide character set the smallest addressable data (a byte) was also 6-bits (I know ASCII is 7-bits but that was to reduce transmission overhead or allow for a parity bit when required, eg on punch tape or cards that often supported 8 columns of bits). The way machines worked back then would be you'd address a byte of data, that byte contained instructions. Those instructions were sometimes printable data, it was sometimes formatting data, and it was sometimes control data. It wasn't until operating systems became more sophisticated, hardware become more sophisticated and the two became decoupled, that character encodings became the abstraction for rendering text that we think of them as today.
- kps 5y ago
  Character codes predate computers. CR and LF codes date to 1901.
  - hnlmorg 5y ago
    Even earlier than that. CR and LF go back to Baudot code (which is where term 'baud' also originates) and that was created in the 1870s. https://en.wikipedia.org/wiki/Baudot_code https://en.wikipedia.org/wiki/Baudot_code
    - kps 5y ago
      Baudot's (ITA1) was linear, with no concept of lines. Murray's (which evolved into ITA2 with minor changes) added CR LF, since he developed a typewriter-like system.
      - hnlmorg 5y ago
        Ahhh I see where you've gotten that 1901 date from now: > In 1901, Baudot's code was modified by Donald Murray > The Murray code also introduced what became known as "format effectors" or "control characters" – the CR (Carriage Return) and LF (Line Feed) codes. (source: https://en.wikipedia.org/wiki/Baudot_code#Murray_code https://en.wikipedia.org/wiki/Baudot_code#Murray_code) Thanks for the correction there. Learned something new today :)

drewcoo 5y ago

> CR and LF are used for each new line. The author's insistence on calling CR+LF "new line" only serves to confuse. LF is line feed, thus new line. The CR is to readjust the write head back to the starting position.

bottled_poe 5y ago
More intuitively, CR is the horizontal dimension and LF is the vertical dimension in the printer head position on the page, which makes it more obvious which is the new line char.
- justsomehnguy 5y ago
  > More intuitively Not for me. I get the dimensions but hell if I can remember it. But mnemonic 'ReturN' (for the Windows style \r\n) helps (well, a bit) here: \r: carrier Return \n: liNe feed
  - mananaysiempre 5y ago
    A physical printer (like a dot-matrix one) couldn’t care less whether you told it to CR LF or to LF CR, you’ll end up at the start of the next line either way; neither does a terminal emulator, nornally (but mind the generic kernel terminal driver in between, which normally has the onlcr knob turned on). Apparently some systems even used LF CR as the line terminator. (Bare CR is more well-known, having been used by classic Mac OS, and also makes a bit of sense if you remember the parallel port had an AutoLF line that instructed the attached printer to translate CR to CR LF. I don’t know if that is the actual reason for the convention.) The caveat is that advancing the paper vertically is relatively fast, but backing up the carriage[1] horizontally can take some time, so on a dumb and slow printer LF CR abc might end up printing abc in the wrong place while the carriage is still moving backwards, while on a smart but slow printer it will just be slower than CR LF abc. I suspect the dumb part is the origin of the CR NUL convention in Net-ASCII[2]; at least both termcap (dC capability) and terminfo (padding machinery) can describe, and curses handles, the possibility that the terminal requires a delay between issuing a CR and printing new characters at the beginning of the line. [1] It’s a carriage, as in a moving thing that contains useful stuff (cf 3D printers), not a carrier, as in a bare unmodulated signal (cf modems). [2] https://tools.ietf.org/html/rfc20 https://tools.ietf.org/html/rfc20, which probably takes the prize as the lowest-numbered RFC you could still encounter as an up-to-date reference.

j1elo 5y ago

What a coincidence to see CRLF discussed in HN front page, this bit me just yesterday! While using the aws cli tool on Linux (the Docker containerized version) with the "text" output format, it turns out each output line ends with CRLF ("\r\n") instead of simply LF ("\n") as is common for any Linux cli tool, so I had to add a post-process step in my commands to remove CRs (a chained call to tr -d '\r'). Otherwise, my bash variables would end up containing those pesky carriage returns, and debug logs would show up unexpectedly bizarre. In retrospect it's obvious what happened, but it got me confused for a good while. Why would it output CRLF at all? Also of course, not a single mention of this in any of the docs I was reading.

bee_rider 5y ago
Vague memory, so take it with a grain of salt, but I think this might be a way of getting the break to show up in an editor that expects \n or one that expects \r. Maybe most editors that expect one simply ignore the other?
CornCobs 5y ago
The dos2unix tool is purpose-built for this!
- Josteniok 5y ago
  Or flip (https://ccrma.stanford.edu/~craig/utility/flip/ https://ccrma.stanford.edu/~craig/utility/flip/)! I've been carrying this thing around with me for years.
- mananaysiempre 5y ago
  Note that dos2unix and unix2dos do what you expect when used as filters, but overwrite the specified file in place when given a file name. This is a footgun second only to one in pdftotext, arguably, which (unsurprisingly) refuses to work as a filter and (surprisingly) will silently create a matching .txt file when passed a single .pdf as an argument.
  - spc476 5y ago

timonoko 5y ago

Remarkably uninteresting. Next: Let us discuss what the Unicode Character 'REVERSE LINE FEED' (U+008D) is good for.

tdeck 5y ago
VERTICAL TAB is much more puzzling.
- mananaysiempre 5y ago
  Devices that needed to produce output on printed forms would use a physical reference to ensure the text went to the right place. One of the commands a tabulator could give the typewriter it drove was to move the form to the next marked position—a literal tab, or maybe a functional equivalent like a hope in a punched tape. Thus, vertical tab.
- asddubs 5y ago
  interesting, hadn't heard of it before. Apparently it was primarily for printing purposes https://stackoverflow.com/questions/3380538/what-is-a-vertical-tab https://stackoverflow.com/questions/3380538/what-is-a-vertic...
mark-r 5y ago
Uninteresting depends on your perspective. For anyone old enough to have used a Teletype certainly it is no mystery, but how many of us are left? Speaking of, a Teletype didn't understand anything outside of the ASCII range. So what was reverse line feed good for? Absolutely nothing! The author remarks that his Teletype was old because it didn't have lower case. But in my experience that was almost universal - I think I only saw a single Teletype that had lower case.
- timonoko 5y ago
  Olivetti Teletype had lower case. I had an interesting episode, where the University of Nitpicking Nazis discussed if my thesis, printed entirely on Teletype, was acceptable. -- It was not as good-looking as with IBM Selectric, but quite comparable to regular manual typewriters.

wrs 5y ago

There's no need for conspiracy theories about adding delays. CR and LF were just separate functions and it's useful for them to be separate. CRT terminals (or emulations thereof) still work the same way. You have to send CR and LF to a VT100 to get the cursor to both move to the left margin and then go to the next line down. The Unix stty command lets you set cr0/1/2/3 modes for inserting NUL characters after a CR for exactly this reason, to give the carriage time to return on your printing terminal.

blihp 5y ago
Not just terminals: most printers until the mid-80's or so also cared about the distinction for the same reason. They started adding DIP switches allowing you change the default behavior and by the end of the 80's the need for CR+LF was pretty much a thing of the past. While the original reason for this distinction probably did date back to teletypes, some practical use cases evolved out of the capability. IIRC, some word processors used the ability to provide an overstrike mode before bold was a thing most printers could do. ASCII art photos were also a thing both for printers that didn't support a graphics mode as well as providing a way of printing a larger 'image' on the paper faster while also using less (ribbon) ink.
- romwell 5y ago
  The distinction goes back to typewriters. The author mentions that the typewriters had a return lever that could also do a line feed. What they missed is that: * You could still freely move the carriage without advancing the paper if you wanted: there was a carriage release button; * You could also advance paper manually at any time, while keeping the carriage at its current position; * You could often adjust line spacing, including setting it to 0, which means that the big lever would only return the carriage, but not advance the paper. In short, CR and LF were separate useful functions on a typewriter. The backspace key was useful for the same reason; obviously, it wouldn't erase text - it would move the carriage just like the space bar would, but in opposite direction. These were necessary for: * Underlining text (by overtyping the underscore character over lines of text); * Adding accents, or crossing zeros with overtyping; * Bold text by retyping the same line; Etc. Tab stops in your favorite document editor come from the same era: the "Tab" key would release the carriage, and the spring would pull it forward until the next physical stop (whose positions you would manually set). The feature was made to make typing tabular data easier, hence the name. That said, the author has a solid point about why we have CR and LF in that order. To wit, on typewriters, the LF+CR lever would advance the paper first, because it made sense both mechanically and from the UX perspective (you would nudge it to advance paper by whatever line spacing you set, as opposed to manual advance which could be a fraction of that). Returning the carriage, and then advancing paper indeed makes more sense on the terminal, where all operations but CR take a fixed amount of time. That was the value of the article, really : to give motivation for CR+LF rather than LF+CR for line ending. The former works well with the terminals, the latter - with typewriters. Computers therefore use CR+LF.

jfoutz 5y ago

I'm a little skeptical. Equipment from this era was designed by slide rule and sent to draftsmen to draw up and send down to the factory floor. The solution to problems was more power. I would imagine some wily old programmer knew commands would be latched, and maybe as maintenance fell behind, the CR would lag a little, but could be trusted to complete. Better to do CR LF than LF CR. but they both have to happen. I dunno. Maybe this model sucked. But every teletype I've ever used was kinda scary, those things will take your fingers off. They're power tools capable of violence. I'd guess, if you pulled out the maintenance guide, and applied proper lubrication, and replaced wear parts, it wouldn't miss those deadlines. I guess you might need to pull out the schematic and verify the electronics are still in spec. I mean, yeah, the author has a point, but I'd believe it's a wear issue that this teletype doesn't reliably return in time.

Animats 5y ago
The carriage return time spec for the 5-level teletypes is that CR LF should be sufficient. I've restored two Teletype Model 15 machines, and both work properly with plain CR LF. They're 45 baud, so two characters is 400ms. 8-level, 110 baud devices often needed more time. The early Model 15 teletypes would just overtype at the right margin if you didn't send a CR, and would type on top of the previous line if you didn't send an LF. Mechanical options requiring extra parts were later offered so that overrunning the end of line would force a CR LF. One letter would be typed about halfway across the page, but it was better than losing an entire line because one character was garbled. This was useful for links over shortwave radio, where losing characters was not uncommon. With that feature, CR implied LF. I have two Model 15 machines; one has that feature, the other does not. Too many years ago, I had to overhaul a Teletype driver to handle all the serial devices that appeared just before memory got cheap and everything got buffering. I put in a delay function of the form K1 + linelength*K2. Some serial printers worked with wide paper and needed long delays for their long lines. There was at least one device [1] that needed a negative K2 term. It had a minimum time to print a line, so short lines needed an extra delay. (The early Teletype machines are the dream of the right-to-repair people. Everything is fully documented, all parts are removeable, and they are very repairable. They're also high-maintenance. They need regular oiling, and there are several hundred oiling points and three different lubricants. In heavy use, they need an annual cleaning which involves partial disassembly, soaking in a cleaning bath, and lubrication. They have several hundred adjustments, and a whole "adjustments" manual. Nobody would put up with that today. That's the downside of "right to repair" - the hardware is repairable, but clunky.) [1] http://archive.computerhistory.org/resources/text/DataInterface/Data_Interface.DI-240.1973.102646310.pdf http://archive.computerhistory.org/resources/text/DataInterf...

ggm 5y ago

You couldn't do overtype without them being distinct. I think It's that simple.

tonyedgecombe 5y ago
That has always been my understanding. If you want bold, underline or strikethrough then you need to be able to send a carriage return on its own.
- romwell 5y ago
  You could just backspace repeatedly, but it'd be a waste of bits.
justinmchase 5y ago
Well you could if you have a line-unfeed. So by default it would do both but then in the uncommon case where you want overtype you could go back up. A little late for that insight, if its even worthwhile, but typically I would optimize the common case by default and add features for doing advanced / uncommon things.

bullen 5y ago

HTTP also uses CRLF but the spec is flexible I think. Still most if not all implementations use it. That is alot of bytes when you sum them up. Also CRLFCRLF is used to demarkate end of headers and chunks.

p_l 5y ago
All textual protocols with legacy of IETF style design use CRLF, and that includes HTTP which heavily drunk from that legacy. Personally I suspect it's because ARPA was a bit cheap on some things and thus they used typewriters instead of protocol analyzers ;)

ochrist 5y ago

Back in the eighties I worked in the military with teletypes. Some examples are: https://www.radiomuseum.org/r/siemens_fernschreiber_t37.html https://www.radiomuseum.org/r/siemens_fernschreiber_t37.html https://www.cryptomuseum.com/telex/siemens/t100/index.htm https://www.cryptomuseum.com/telex/siemens/t100/index.htm https://www.camion-militar.com/spa/item/ART02979.html https://www.camion-militar.com/spa/item/ART02979.html There were also other makes, like Olivetti. They were connected to various networks, e.g. military, the civil Telex network or the ICAO AFTN network. https://en.wikipedia.org/wiki/Telex https://en.wikipedia.org/wiki/Telex https://en.wikipedia.org/wiki/Aeronautical_Fixed_Telecommunication_Network https://en.wikipedia.org/wiki/Aeronautical_Fixed_Telecommuni... They all used a 5 bit character set Baudot/ITA2: https://en.wikipedia.org/wiki/Baudot_code https://en.wikipedia.org/wiki/Baudot_code (I can still read a paper tape message directly from the reel) As these teletypes were quite slow (typically 50 baud) and mechanical, the procedure was for line shifts to always be CR CR LF. The reason for this is to allow for the carriage (printing head) to have time enough to move to the start of the line before the line feed. This was enforced in the procedures, so you might end up with a rejected message if you did not do it properly. When computers were introduced in the various networks, ITA2 was converted to ASCII and some of these restrictions were relaxed a bit.

justinmchase 5y ago

And hopefully a short future!