18 ms·

This is over my head. Why did we need to take a trip to discover why \n is encoded as a byte with the value 10? Isn't that expected? The author and HN comments

This is over my head. Why did we need to take a trip to discover why \n is encoded as a byte with the value 10? Isn't that expected? The author and HN comments don't say, so I feel stupid.

kibwen 2y ago
The point is to ask "who" encoded that byte as the value of 10. If you're writing a parser and you parse a newline as the escape sequence `\n`, then where did the value 10 come from? If you instead parse a newline as the integer literal `10`, then where does the actual binary value 1010 come from? The ultimate point of this exercise is to alter your perception of what a compiler is (in the same way as the famous Reflections On Trusting Trust presentation). Which is to say: your compiler is not something that outputs your program; your compiler is also input to your program. And as a program itself, your compiler's compiler was an input to your compiler, which makes it transitively an input to your program, and the same is true of your compiler's compiler's compiler, and your compiler's compiler's compiler's compiler, and your compiler's compiler's compiler's compiler's compiler, and...
- dotancohen 2y ago
  All right Stan, don't belabour the point.
mikl 2y ago
The interesting point is how the value of 10 is not defined in Rust’s source code, but passed down as “word of mouth” from compiler to compiler.
- analog31 2y ago
  The earliest reference on my shelf is Kernighan and Ritchie, 1978. Like you say, it's probably just been passed on from generation to generation. And it's easier to say "just like C" than to make up a new convention. Python uses the same convention.
  - eru 2y ago
    The article is not about the convention, but about the mechanics of how the compiler came to 'learn' about the convention.
- happytoexplain 2y ago
  Ohh, a lot of spoken and unspoken things in the article made me think this was supposed to be a bug/quirk investigation. This makes much more sense.
  - antonvs 2y ago
    It is a kind of bug/quirk: it implies that the compiler is not buildable from scratch in its current form. It depends on a binary that knows how to translate \n.
    - happytoexplain 2y ago
      Right, I just mistook which thing was the bug/quirk.
yen223 2y ago
If you had to rebuild the rust compiler from scratch, and all you had was rustc's source code, there's nothing in the source code to tell you what '\n' actually maps to. It's an interesting real-world example of the Ken Thompson hack.
deleted 2y ago
[deleted]
crote 2y ago
The thing is, why 10? Why not 9 or 11? The code says "if you see 'string of newline character', output 'newline character'". How does the compiler know what a newline character is? Its code in turn just says "if you see 'string of newline character', treat it as 'newline character'"... As a human I can just Google "C string escape codes", but that table is nowhere to be found inside the compiler. If C 2025 is going to define Start of Heading as \h, is `'h' => cooked.push('\h')` going to magically start working? How could it possibly know? Clearly at some point someone must've manually programmed a `'n' => 10` mapping, but where is it!?
- TacticalCoder 2y ago
  > The thing is, why 10? Why not 9 or 11? Check the first Unicode codepoints, 10 is defined there: 000A is LINE FEED (LF) = new line (NL), end of line (EOL) It was already 10 in ASCII too (and the first 128 codepoints of Unicode are mostly the same as ASCII [I think there are a few tiny differences in some control characters]). So to answer your question: it's neither 9 nor 11 because '\n' stands for "new line" and not for "character tabulation" nor for "line tabulation" (which 9 and 11 respectively stands for). > Clearly at some point someone must've manually programmed a `'n' => 10` mapping I don't disagree with that.
  - antonvs 2y ago
    > Check the first Unicode codepoints, 10 is defined there The point is how the compiler knows that. Read the article.
    - johnisgood 2y ago
      Does https://news.ycombinator.com/item?id=41764555 https://news.ycombinator.com/item?id=41764555 answer the question?
- kreco 2y ago
  > The thing is, why 10? Why not 9 or 11? The code says "if you see 'string of newline character', output 'newline character'". How does the compiler know what a newline character is? Its code in turn just says "if you see 'string of newline character', treat it as 'newline character'"... You can look into the ascii table then. 10 is the line feed. '\n' is there to help human because it makes more sense than '10'. Nobody is asking why 'a' is equal to '61'. From the codebase, you know that '\n' is a char. A char is a value between 0 and 255, if you explicitly convert '\n' to int then you happen to find the ascii value and you are good to go and there is no need to pretend there is any poetry in this. Requoting you: > "if you see 'string of newline character', output 'newline character'" It simply becomes "if a you see 'the arbitrary symbol for the new line', output 'the corresponding ascii value'". I read the quote as "if you see 'a', output 'a' in ascii code." which is not mysterious in any kind of way. IMO, the article does not make sense at all because it pretends to wonder where an hexadecimal value is coming from, but with any other kind of symbol it would be the exact same article, you enter 'a' and find some weird hexadecimal value and you won't be able to correctly trace it from the source code. It would make sense it 'a' would display 'a' but '\n' would display some hexadecimal garbage in a text editor.

This is over my head. Why did we need to take a trip to discover why \n is encoded as a byte with the value 10? Isn't that expected? The author and HN comments don't say, so I feel stupid.

kibwen 2y ago

The point is to ask "who" encoded that byte as the value of 10. If you're writing a parser and you parse a newline as the escape sequence `\n`, then where did the value 10 come from? If you instead parse a newline as the integer literal `10`, then where does the actual binary value 1010 come from? The ultimate point of this exercise is to alter your perception of what a compiler is (in the same way as the famous Reflections On Trusting Trust presentation). Which is to say: your compiler is not something that outputs your program; your compiler is also input to your program. And as a program itself, your compiler's compiler was an input to your compiler, which makes it transitively an input to your program, and the same is true of your compiler's compiler's compiler, and your compiler's compiler's compiler's compiler, and your compiler's compiler's compiler's compiler's compiler, and...

dotancohen 2y ago
All right Stan, don't belabour the point.

mikl 2y ago

The interesting point is how the value of 10 is not defined in Rust’s source code, but passed down as “word of mouth” from compiler to compiler.

analog31 2y ago
The earliest reference on my shelf is Kernighan and Ritchie, 1978. Like you say, it's probably just been passed on from generation to generation. And it's easier to say "just like C" than to make up a new convention. Python uses the same convention.
- eru 2y ago
  The article is not about the convention, but about the mechanics of how the compiler came to 'learn' about the convention.
happytoexplain 2y ago
Ohh, a lot of spoken and unspoken things in the article made me think this was supposed to be a bug/quirk investigation. This makes much more sense.
- antonvs 2y ago
  It is a kind of bug/quirk: it implies that the compiler is not buildable from scratch in its current form. It depends on a binary that knows how to translate \n.
  - happytoexplain 2y ago
    Right, I just mistook which thing was the bug/quirk.

yen223 2y ago

If you had to rebuild the rust compiler from scratch, and all you had was rustc's source code, there's nothing in the source code to tell you what '\n' actually maps to. It's an interesting real-world example of the Ken Thompson hack.

deleted 2y ago

[deleted]

crote 2y ago

The thing is, why 10? Why not 9 or 11? The code says "if you see 'string of newline character', output 'newline character'". How does the compiler know what a newline character is? Its code in turn just says "if you see 'string of newline character', treat it as 'newline character'"... As a human I can just Google "C string escape codes", but that table is nowhere to be found inside the compiler. If C 2025 is going to define Start of Heading as \h, is `'h' => cooked.push('\h')` going to magically start working? How could it possibly know? Clearly at some point someone must've manually programmed a `'n' => 10` mapping, but where is it!?

TacticalCoder 2y ago
> The thing is, why 10? Why not 9 or 11? Check the first Unicode codepoints, 10 is defined there: 000A is LINE FEED (LF) = new line (NL), end of line (EOL) It was already 10 in ASCII too (and the first 128 codepoints of Unicode are mostly the same as ASCII [I think there are a few tiny differences in some control characters]). So to answer your question: it's neither 9 nor 11 because '\n' stands for "new line" and not for "character tabulation" nor for "line tabulation" (which 9 and 11 respectively stands for). > Clearly at some point someone must've manually programmed a `'n' => 10` mapping I don't disagree with that.
- antonvs 2y ago
  > Check the first Unicode codepoints, 10 is defined there The point is how the compiler knows that. Read the article.
  - johnisgood 2y ago
    Does https://news.ycombinator.com/item?id=41764555 https://news.ycombinator.com/item?id=41764555 answer the question?
kreco 2y ago
> The thing is, why 10? Why not 9 or 11? The code says "if you see 'string of newline character', output 'newline character'". How does the compiler know what a newline character is? Its code in turn just says "if you see 'string of newline character', treat it as 'newline character'"... You can look into the ascii table then. 10 is the line feed. '\n' is there to help human because it makes more sense than '10'. Nobody is asking why 'a' is equal to '61'. From the codebase, you know that '\n' is a char. A char is a value between 0 and 255, if you explicitly convert '\n' to int then you happen to find the ascii value and you are good to go and there is no need to pretend there is any poetry in this. Requoting you: > "if you see 'string of newline character', output 'newline character'" It simply becomes "if a you see 'the arbitrary symbol for the new line', output 'the corresponding ascii value'". I read the quote as "if you see 'a', output 'a' in ascii code." which is not mysterious in any kind of way. IMO, the article does not make sense at all because it pretends to wonder where an hexadecimal value is coming from, but with any other kind of symbol it would be the exact same article, you enter 'a' and find some weird hexadecimal value and you won't be able to correctly trace it from the source code. It would make sense it 'a' would display 'a' but '\n' would display some hexadecimal garbage in a text editor.