15 ms·

Whence '\n'?

• amelius 2y ago
  A more interesting question: what would our code look like if ASCII (or strings in general) didn't have escape codes?
  • wongarsu 2y ago
    In PHP you see a lot of print('Hello World' . PHP_EOL); where the dot is string concatenation (underrated choice imho) and PHP_EOL is a predefined constant that maps to \n or \r\n depending on platform. You could easily extend that to have global constants for all non-printable ascii characters.
  • jiehong 2y ago
    The font you use could choose to display something for control characters, so they would have a visible shape on top of having a meaning. Perhaps like [0] (Unicode notation). [0]: https://rootr.net/im/ASCII/ASCII.gif https://rootr.net/im/ASCII/ASCII.gif
  • zokier 2y ago
    It would depend more on what we are intending to do, are we controlling a terminal, or are we writing to a file (with specific format). Terminal control is fairly easy answer, there would be some other API to control cursor position, so the code would need to call some function to move the cursor to next line. For files, it would depend on what the format is. So we might be writing just `<p>hello world</p>` instead of `hello world\n`. In fact I find it bit weird that we are using teletype (and telegraph etc) control protocol (what ASCII mostly is) as our "general purpose text" format; it doesn't make much sense to me.
    • rswail 2y ago
      Except that terminals didn't exist when ASCII was created, printers existed, either with a "carriage" that carried the actual print head (like IBM Selectric typewriters, or the original ASR33 teletypes) or "chain printers" that printed an entire line at a time with a set of rods for each character position that had the entire character set on them.
  • ivanjermakov 2y ago
    This is actually a valid problem when writing quines[1]. You need to escape string delimiters in a way without using its literal. This is what `chr(39)` is for in the following Python quine: a = 'a = {}{}{}; print(a.format(chr(39), a, chr(39)))'; print(a.format(chr(39), a, chr(39))) [1]: https://en.wikipedia.org/wiki/Quine_(computing) https://en.wikipedia.org/wiki/Quine_(computing)
    • eru 2y ago
      You don't need these gymnastics of knowing ASCII in Python quines: a = '''a = {} print(a.format(repr(a)))''' print(a.format(repr(a))) (OK, it's not a quine yet, because the output has slightly different formatting. But the formatting very quickly reaches a fixed point, if you keep feeding it into a Python interpreter.) The actual quine is the slightly less readable: a = 'a = {}\nprint(a.format(repr(a)))' print(a.format(repr(a))) https://github.com/matthiasgoergens/Quine/blob/c53f187a4b4030437005ddd12bd5a8594269b437/quine.c https://github.com/matthiasgoergens/Quine/blob/c53f187a4b403... has a C quine that doesn't rely on ASCII values either.
  • fsckboy 2y ago
    >what would our code look like if ASCII didn't have escape codes? ASCII is the layer that doesn't have escape codes (although it does have a single code for ESC), ascii is just a set of mappings from 7-bit numbers to/from mostly-printable characters
    • ezequiel-garzon 2y ago
      At least a full fourth of the ASCII code points, 0x00-0x1F, are not printable. A bit more, as we should add del (0x7F) and, according to some though more controversially, space (0x20).
      • fsckboy 2y ago
        the full forth may not be "printable if you define printable as using toner and ignoring positioning". But they aren't escape codes.
    • amelius 2y ago
      What would you call the lower 32 or so codes then? Control codes?
      • fsckboy 2y ago
        they are explicity defined as escape codes in terms of inputting them, but not in terms of receiving them. They are not by any stretch escape codes. See ANSI (which ASCII is a part of) if you want to learn about ESCAPE codes.
      • zzo38computer 2y ago
        Codes 0x00 to 0x1F are called control codes (and in some contexts (such as use of ASCII control codes with TRON character set) 0x20 is also considered to be a control code). The code 0x7F is also a control code. (The PC character set (which is a superset of ASCII) has graphics for those codes as well, although in plain ASCII there are no graphics for the control codes.)
  • rswail 2y ago
    If ASCII didn't have control sequences then teletypes wouldn't have worked. How do you make the carriage (a literal thing on a chain being driven back and forth) "return" to the start of the line without a "carriage return" code? How would you make the paper feed go up one line without a "line feed" code? Same for ringing the bell, tabs, backspace etc. A "new line" on a teletype was actually two characters, a CR and a LF. Unfortunately, using one of the other sequences (eg RS, for "Record Separator") would have saved billions of CPU cycles and misinterpreted text files dealing with the CRLF, CR, LF, LFCR sequences to mean "new line".
• cpach 2y ago
  Previous discussion: https://news.ycombinator.com/item?id=41564527 https://news.ycombinator.com/item?id=41564527
• nasso_dev 2y ago
  > This post was inspired by another post about exactly the same thing. I couldn't find it when I looked for it, so I wrote this. All credit to the original author for noticing how interesting this rabbit hole is. I think the author may be thinking of Ken Thompson's Turing Award lecture "Reflections on Trusting Trust".
  • Karellen 2y ago
    Although that presentation does point out that the technique is more generally used in quines. Given that there is a fair amount of research, papers and commentary on quines, it's possible that the author may have read something along those lines. https://en.wikipedia.org/wiki/Quine_(computing) https://en.wikipedia.org/wiki/Quine_(computing)
  • ktm5j 2y ago
    I totally missed that bit when the OP, but it definitely made me think of that paper so maybe.
  • yen223 2y ago
    I don't think so. I too recall seeing a post about this exact piece of trivia ('\n' in rust) years ago, but I couldn't find the source anymore.
    • tylerhou 2y ago
      It might have been https://research.swtch.com/nih https://research.swtch.com/nih ?
      • yen223 2y ago
        There's nothing in that article about Rust?
  • yuchi 2y ago
    Also have a read of this fabulous short web from 2009: https://www.teamten.com/lawrence/writings/coding-machines/ https://www.teamten.com/lawrence/writings/coding-machines/
• kijin 2y ago
  The incorrect capitalization made me think that, perhaps, there's a scarcely known escape sequence \N that is different from \n. Maybe it matches any character that isn't a newline? Nope, just small caps in the original article.
  • paulddraper 2y ago
    There is actually. Many systems use \N in CSVs or similar as NULL, to distinguish from an empty string. I figured this is what the article was about?
  • cpach 2y ago
    If you do view source, it’s actually \n, but it’s not displayed as such because of this CSS rule: .title { font-variant: small-caps; }
    • sedatk 2y ago
      So, the HN title is wrong.
      • isatty 2y ago
        The original title is.
        • deathanatos 2y ago
          In addition to what others have said about smallcaps being a stylistic rendering, if you copy & paste the original title, you'll get Whence '\n'?
        • niederman 2y ago
          No, the original title is correct, small caps are just an alternate way of setting lowercase letters.
          • neuroelectron 2y ago
• archmaster 2y ago
  if only this went into where the ocaml escape came from :)
  • diath 2y ago
    It does, it links to this: https://github.com/ocaml/ocaml/blob/4d6ecfb5cf4a5da814784dee7363a15ea278f324/lex/lexer.mll#L37 https://github.com/ocaml/ocaml/blob/4d6ecfb5cf4a5da814784dee...
    • fiddlerwoaroof 2y ago
      But this doesn’t really explain anything: ‘\010’ isn’t really any more primitive than ‘\x0a’: they’re just different representations of the same bit sequence
      • fluoridation 2y ago
        But it is more primitive than '\n', and can be rendered into binary without any further arbitrary conversion steps (arbitrary in that there's nothing in '\n' that says it should mean 10). It's just "transform the number after the backslash into the byte with that value".
        • xixixao 2y ago
          Yeah, but what maps the characters 1 and 0 to the numerical value 10? I bet it’s the same chain of compilation, at some point leading to an assembly compiler, and that itself leading to some proto binary implementation, maybe even going into bios/hardware implementation?
          • fluoridation 2y ago
            The logic can be implemented entirely in a parser written in a high level language. For example, //const int ZERO = '0'; const int ZERO = 0x30; int convert(const char *s){ int ret = 0; for (; *s; s++){ ret *= 10; ret += *s - ZERO; } return ret; } After that you just dump the value to the output verbatim. Any high level language can handle this, it's just a change of representation like any other. There's no need to do any further delegation.
• dist-epoch 2y ago
  I remember a similar article for some C compiler, and it turned out the only place the value 0x10 appeared was in the compiler binary, because in the source code it had something like "\\n" -> "\n"
• atoav 2y ago
  One rule of programming I figured out pretty quick is: if there are two ways of doing it and there is a 50/50 chance of one being correct and the other one isn't, chances are you will get it wrong the first time.
  • chgs 2y ago
    The USB rule. First time is the wrong way up Second time is also the wrong way up Third time works
    • fader 2y ago
      It's because of the quantum properties of USB connectors. They have spin 1/2.
      • SAI_Peregrinus 2y ago
        I thought it was because USB connectors occupy 4 spatial dimensions.
        • PaulDavisThe1st 2y ago
          That's good, because otherwise we'd never be able to find them when we need them.
          • inopinatus 2y ago
            Instead we always find a USB type mini B when needing a micro B, a micro B when needing a type C, and a type C when needing an extended micro B. If you reveal a spare extended micro B whilst rummaging around then it will in additional transpire that the next cable needed will be a mini B, irrespective of any prior expectation you may have held about the device in question. A randomly occurring old-school full-size type B may be encountered during any cable search, approximately 1% of the time, usually at the same moment your printer jams. What I really don't understand, however, is why I keep finding DB13W3s in my closet
• ncruces 2y ago
  I'm guessing the “other post” that inspired this might be: https://research.swtch.com/nih https://research.swtch.com/nih
  • dang 2y ago
    Discussed here: Running the "Reflections on Trusting Trust" Compiler - https://news.ycombinator.com/item?id=38020792 https://news.ycombinator.com/item?id=38020792 - Oct 2023 (67 comments)
• tzot 2y ago
  I always thought, maybe because of C, that \0??? is an octal escape; so in my mind \012 is \x0a or 0x0a, and \010 is 0x08. So I find this quite confusing; maybe OCaml does not have octal escapes but decimal ones, and \09 is the Tab character. I haven't checked.
  • dpassens 2y ago
    It is indeed a decimal escape: https://ocaml.org/manual/5.2/lex.html#char-literal https://ocaml.org/manual/5.2/lex.html#char-literal
  • fanf2 2y ago
    Yeah backslash-decimal character escapes are really rare, the only string syntaxes I know of that have them are in O’Caml, Lua, and DNS
    • binary132 2y ago
      Is O’Caml an Irish fork of OCaml? :)
      • fanf2 2y ago
        It was the spelling when I first learned the language, eg https://arxiv.org/pdf/2006.05862 https://arxiv.org/pdf/2006.05862 https://caml.inria.fr/pub/old_caml_site/FAQ/stephan.html https://caml.inria.fr/pub/old_caml_site/FAQ/stephan.html
        • binary132 2y ago
          Oh, interesting. Thanks for sharing
  • syncsynchalt 2y ago
    There's some truth in that direction, but it's not related to backslash escapes (which are symbolic/mnemonic, \n is "[Ne]wline", \r is "carriage [R]eturn", \t is "[T]ab", and so on). Instead, consider the convention of control characters, such as ^C (interrupt), ^G (bell), or ^M (carriage return). Those are characters in the C0 control set, where ^C is \0x3, ^G is \0x7, or ^M is \0xD. You're seeing a bit of cleverness that goes back to pre-Unix days: to represent the invisible C0 characters in ASCII, a terminal prepends the "^" character and prints the character AND-0x40, shifting it into a visible range. You may want to pull up an ASCII table such as https://www.asciitable.com https://www.asciitable.com to follow along. Each control character (first column) is mapped to the ^character two columns over, on that table. That's why \0 is represented with the odd choice of ^@, the escape key becomes ^[, and other hard-to-remember equivalents. These weren't choices made by Unix authors, they're artifacts of ASCII numbering.
• gjvc 2y ago
  this is a nothingburger of an article
• coolio1232 2y ago
  I thought this was going to be about '\N' but there's only '\n' here.
  • dang 2y ago
    It's in the html doc title but the article doesn't deliver.
• ynfnehf 2y ago
  First place I read about this idea (specifically newlines, not in general trusting trust) was day 42 in https://www.sigbus.info/how-i-wrote-a-self-hosting-c-compiler-in-40-days https://www.sigbus.info/how-i-wrote-a-self-hosting-c-compile... "For example, my compiler interprets "\n" (a sequence of backslash and character "n") in a string literal as "\n" (a newline character in this case). If you think about this, you would find this a little bit weird, because it does not have information as to the actual ASCII character code for "\n". The information about the character code is not present in the source code but passed on from a compiler compiling the compiler. Newline characters of my compiler can be traced back to GCC which compiled mine."
  • moefh 2y ago
    I was hoping GCC would do the same, leaving the decision about the value of '\n' to GCC's compiler, but apparently it hardcodes the numeric values for escapes[1], with options for ASCII or EBCDIC systems. [1] https://github.com/gcc-mirror/gcc/blob/8a4a967a77cb937a2df45f4d1d5032085a4080c6/libcpp/charset.cc#L2286 https://github.com/gcc-mirror/gcc/blob/8a4a967a77cb937a2df45...
    • codetrotter 2y ago
      But these numeric values are also ASCII representation of numbers, rather than being the actual byte that is written to the output. Maybe there is hope still. Where do the byte values for those numbers come from when the compiler writes output?
      • wahern 2y ago
        The C standard (see C23 5.2.1p3) requires the values of '0' through '9' to be contiguous, so it doesn't matter if you only care about round-tripping. '7' - '0' == 7 no matter the character set. Though, for round-tripping I suppose this isn't strictly necessary, but it certainly makes parsing and printing decimal notation very convenient. Notably, for both ASCII and EBCDIC 'A'..'F' and 'a'..'f' are also contiguous, so parsing and printing hexadecimal can be done much the same as decimal.
      • 2y ago
• happytoexplain 2y ago
  This is over my head. Why did we need to take a trip to discover why \n is encoded as a byte with the value 10? Isn't that expected? The author and HN comments don't say, so I feel stupid.
  • kibwen 2y ago
    The point is to ask "who" encoded that byte as the value of 10. If you're writing a parser and you parse a newline as the escape sequence `\n`, then where did the value 10 come from? If you instead parse a newline as the integer literal `10`, then where does the actual binary value 1010 come from? The ultimate point of this exercise is to alter your perception of what a compiler is (in the same way as the famous Reflections On Trusting Trust presentation). Which is to say: your compiler is not something that outputs your program; your compiler is also input to your program. And as a program itself, your compiler's compiler was an input to your compiler, which makes it transitively an input to your program, and the same is true of your compiler's compiler's compiler, and your compiler's compiler's compiler's compiler, and your compiler's compiler's compiler's compiler's compiler, and...
    • dotancohen 2y ago
      All right Stan, don't belabour the point.
  • mikl 2y ago
    The interesting point is how the value of 10 is not defined in Rust’s source code, but passed down as “word of mouth” from compiler to compiler.
    • analog31 2y ago
      The earliest reference on my shelf is Kernighan and Ritchie, 1978. Like you say, it's probably just been passed on from generation to generation. And it's easier to say "just like C" than to make up a new convention. Python uses the same convention.
• amelius 2y ago
  Why backslash?
  • o11c 2y ago
    Because backslash is a modern invention with no prior meaning in text. It was invented to allow writing the mathematical "and" and "or" symbols as /\ and \/.
    • dTal 2y ago
      Hm. According to Wiki, "As of November 2022, efforts to identify either the origin of this character or its purpose before the 1960s have not been successful." While your rationale was used to argue for its inclusion in ASCII, as an origin story however it is very unlikely, as (according to wiki again): "The earliest known reference found to date is a 1937 maintenance manual from the Teletype Corporation with a photograph showing the keyboard of its Kleinschmidt keyboard perforator WPE-3 using the Wheatstone system." The Kleinschmidt keyboard perforator was used for sending telegraphs, and is not well equipped with mathematical symbols, or indeed any symbols at all besides forward slash, backslash, question mark, and equals sign. Not even period!
  • sedatk 2y ago
    "Why character X for Y?" has usually a universal answer: character X wasn't used commonly for the given scope at the time, was available widely, and felt sensible enough for Y.
    • amelius 2y ago
      Since you mentioned Y, why was the Y combinator named Y? Were A...X already taken?
• binary132 2y ago
  Cool, but actually it was just 0x0A all along! The symbolic representation was always just an alias. It didn’t actually go through 81 generations of rustc to get back to “where it really came from”, as you’d be able to see if you could see the real binary behind the symbols. Yes I am just being a bit silly for fun, but at the same time, I know I personally often commit the error of imagining that my representations and abstractions are the real thing, and forgetting that they’re really just headstuff.
  • Etherlord87 2y ago
    The point of the article, that a lot of people in the comments seem to be missing, is that this alias is not defined in the SOURCE CODE of the foo. And not in the rust compiler. And Rust compiler is compiled... Using Rust compiler! So if you follow the source code, there's no definition, anywhere - that is, unless you have a knowledge (that isn't there in the source code, not in the latest revision at least), that the Rust compiler wasn't always compiled in Rust compiler (duh!) and you know where it was compiled. So the definition of this alias is an information that is somewhat "parasitic", secondary, embedded implicitly, like non-DNA information inherited from your parents, known as "Epigenetic mechanisms" (not memes).
    • binary132 2y ago
      Yeah, but my point is, it’s 0A in the compiler’s binary. ‘\n’ is just its name. If you could somehow see inline where the name maps to in the binary, you’d be able to go look at it.
• i4k 2y ago
  This is fascinating and terrifying.
• phibz 2y ago
  Backslash escape codes are a convention. They're so pervasive that we sometimes forget this. It could just as easily be some other character that is special and treated as an escape token.
• gnulinux 2y ago
  This is a fascinating post. It reads to me like some kind of cross between literate-programming and poetry. It's really trying to explain the idea that when you run `just foo` the very 0x0A byte comes from possibly hundreds of cycles of code generation. Back in the day, someone encoded this information into OCaml compiler -- somehow -- and years later here in my computer 0x0A information is stored due to this history. But the way in which this phenomena is explained is via actual code. The code itself is besides the point of course, it's not like anyone will ever run or compile this specific code, but it's put there for humans to follow the discussion.
• pmarreck 2y ago
  wait, Rust was bootstrapped via OCaml?
  • odo1242 2y ago
    Yep Also, the original Rust developers were OCaml devs, Rust borrowed design features (like matching and additive types) from OCaml, the syntax for lifetimes in Rust is the syntax for polymorphism in OCaml, and they borrowed semantics such as latent typing.
    • pmarreck 2y ago
      That is very interesting!
• defrost 2y ago
  Interesting that 10 hours in there's no thread hits for EBCDIC. All theories being bandied about should account for the fact that early C compilers appeared on non-ASCII systems that did not map \n "line feed" to decimal 10. https://en.wikipedia.org/wiki/EBCDIC https://en.wikipedia.org/wiki/EBCDIC As an added wrinkle EBCDIC had both an explicit NextLine and an explicit LineFeed character. For added fun: The gaps between letters made simple code that worked in ASCII fail on EBCDIC. For example for (c = 'A'; c <= 'Z'; ++c) putchar(c); would print the alphabet from A to Z if ASCII is used, but print 41 characters (including a number of unassigned ones) in EBCDIC. Sorting EBCDIC put lowercase letters before uppercase letters and letters before numbers, exactly the opposite of ASCII. The only guarentee in the C standard re: chracter encoding was that the digits '0'-'9' mapped in contiguous ascending order. In theory* simple C programs (that printed 10 lines of "Hello World") should have tha same source that compiled on either ASCII or EBCDIC systems and produced the same output. * many pitfalls aside
  • skissane 2y ago
    > As an added wrinkle EBCDIC had both an explicit NextLine and an explicit LineFeed character. Despite EBCDIC having a newline/next line character (NEL), it is rarely encountered in many EBCDIC systems. Early on, most EBCDIC systems (e.g. MVS, VM/CMS, OS/400, DOS/VSE) did not store text as byte stream files, but instead record-oriented files – storing lines as fixed-length or variable-length records. With fixed-length records, you'd declare a record length when creating the file (80 or 132 were the most common choices); every line in the file had to be of that length, shorter lines would be padded (normally with the EBCDIC space character, which is 0x40 not 0x20), longer lines would either be truncated or a continuation character would be used. With variable-length records, each record was prefixed with a record descriptor word (RDW) which gave its length (and a couple of spare bytes that theoretically could be used for additional metadata). However, in practice the use of variable-length records for text files (including program source code) was rather rare, fixed-length records were the norm. So even though NEL exists, it wasn't normally used in files on disk. Essentially, newline characters such as NEL are "in-band signalling "for line/record boundaries, but record-oriented filesystems used "out-of-band signalling" instead. I'm not sure exactly how stdio was implemented in the runtime libraries of EBCDIC C compilers – I assume \n did map to NEL internally, but then the stdio layer treated it as a record separator, and then wrote each record using a separate system call, padding as necessary. Later on, most of these operating systems gained POSIX compatibility subsystems, at which point they gained byte stream files as exist on mainstream systems. IBM systems generally support tagging files with a code page, so the files can be a mix of EBCDIC and ASCII, and the OS will perform translation between them in the IO layer (so an application which uses EBCDIC at runtime can read an ASCII file as EBCDIC, without having to manually call any character encoding conversion APIs, or be explicitly told whether the file to be read is EBCDIC or ASCII). Newer applications make increasing use of the POSIX-based filesystems, but older applications still mostly store data (even text files and program source code) in the classic record-oriented file systems. From what I understand, the most common place EBCDIC NEL would be encountered in the wild was EBCDIC line mode terminal connections (hard copy terminals such as IBM 2741 and IBM 3767).
• titwank 2y ago
  I wondered if clang has the same property, but it's explicitly coded as 10 (in lib/Lex/LiteralSupport.cpp): /// ProcessCharEscape - Parse a standard C escape sequence, which can occur in /// either a character or a string literal. static unsigned ProcessCharEscape(const char *ThisTokBegin, const char *&ThisTokBuf, const char *ThisTokEnd, bool &HadError, FullSourceLoc Loc, unsigned CharWidth, DiagnosticsEngine *Diags, const LangOptions &Features) { const char *EscapeBegin = ThisTokBuf; // Skip the '\' char. ++ThisTokBuf; // We know that this character can't be off the end of the buffer, because // that would have been \", which would not have been the end of string. unsigned ResultChar = *ThisTokBuf++; switch (ResultChar) { ... case 'n': ResultChar = 10; break; ...
  • titwank 2y ago
    Similarly in GCC it is hardcoded, with a choice of ASCII or EBCDIC (in gcc/libcpp/charset.cc): /* Convert an escape sequence (pointed to by FROM) to its value on the target, and to the execution character set. Do not scan past LIMIT. Write the converted value into TBUF, if TBUF is non-NULL. Returns an advanced pointer. Handles all relevant diagnostics. If LOC_READER is non-NULL, then RANGES must be non-NULL: location information is read from *LOC_READER, and *RANGES is updated accordingly. */ static const uchar * convert_escape (cpp_reader *pfile, const uchar *from, const uchar *limit, struct _cpp_strbuf *tbuf, struct cset_converter cvt, cpp_string_location_reader *loc_reader, cpp_substring_ranges *ranges, bool uneval) { /\* Values of \a \b \e \f \n \r \t \v respectively. */ #if HOST_CHARSET == HOST_CHARSET_ASCII static const uchar charconsts[] = { 7, 8, 27, 12, 10, 13, 9, 11 }; #elif HOST_CHARSET == HOST_CHARSET_EBCDIC static const uchar charconsts[] = { 47, 22, 39, 12, 21, 13, 5, 11 }; #else #error "unknown host character set" #endif uchar c; /* Record the location of the backslash. */ source_range char_range; if (loc_reader) char_range = loc_reader->get_next (); c = *from; switch (c) { ... case 'n': c = charconsts[4]; break; ...
• ProofHouse 2y ago
  A lot LOT better content than his stupid wanko manko perv post
• adamrezich 2y ago
  This is very interesting. I don't know if it was the intent, but Jon Blow doesn't plan to self-host the Jai compiler anytime soon, and I wonder if the rationale is at least in part due to things like this. It's interesting how he doesn't see self-hosting as any sort of immediate goal, quite unlike Rust and Zig.