5 ms·

A little while ago, I discovered that... $"A{new List<string>{$"B{"{C}"}D"}.First()}E" ... was valid C#. This means that a C# compiler can't start wi

by billpg 5y ago

A little while ago, I discovered that... $"A{new List<string>{$"B{"{C}"}D"}.First()}E" ... was valid C#. This means that a C# compiler can't start with a simple tokenizing loop. That compiler phase would have to keep track of state in a stack, recording what each } character means while its still looping through code character-by-character. Now we're adding {{ and }} into the equation. Yay.

torginus 5y ago
>This means that a C# compiler can't start with a simple tokenizing loop. Not true, it just means that the parts between double quotes aren't bunched into a single token. To figure out how the compiler makes sense of this code, try https://roslynquoter.azurewebsites.net/ https://roslynquoter.azurewebsites.net/ You'll see how it tokenizes the string.
- billpg 5y ago
  That's not a simple tokenizer. A simple loop that would have worked with the 70s era of programming languages, would go through each character and once the boundary between two tokens has been identified, write out a token to a one-dimensional list. This would be a mostly stateless loop, tracking enough state for the current token in hand only. The next phase would go through the tokens and pair up brackets, etc. A C# tokenizer can't do that. It needs to keep a stack of state. When it sees a '}', it needs to know if that's a "normal" brace or the } that resumes a interpolated string literal. I was writing a tokenizer myself and I wanted to have something similar to string interpolation. I very quickly realized my simple loop that I would have written for my CS degree isn't going to cut it and I had to start over.
  - Metasyntactic 5y ago
    Hi, I'm one of teh C# language designers, and I work on the compiler implementation as well. C# has never had a "simple tokenizer". Indeed, even the first language has complex lexical constructs that are part and parcel of the language. For example, our comments can store structured data in them (like xml). > A simple loop that would have worked with the 70s era of programming languages Yes. But 70s era compilers had to deal with things like not having enough memory to even store basic amounts of data. It also had to work in spaces where things like a 'stack' was just not tenable. We're literally 50 years from that point, and having a compiler do stuff like keeping a stack is not an issue anymore :)
    - billpg 5y ago
      Then my compliments on your efforts. When I stumbled upon my own trouble writing a tokenizer, I wondered if the C# compiler had the same issue and maybe you couldn't have {} inside interpolated strings. I wrote my earlier expression with many nested {} expressions to see if your compiler would handle it and I was very happy to see it did. Although you did deny me an excuse to not have to start over. If the C# team didn't bother, why should I? (Except you did bother.)
Metasyntactic 5y ago
>Now we're adding {{ and }} into the equation. Yay. Hi, i'm the lang designer and feature implementor here :) The complexity of lexing/parsing did not get worse here. We actually just lex/parse this stuff the same way that interpolated strings have always been lexed/parsed. This has been supported in the language for almost 10 years at this point :)
GoblinSlayer 5y ago
And how do you tell if >> is a shift operator or two angle brackets?

by billpg 5y ago

torginus 5y ago

>This means that a C# compiler can't start with a simple tokenizing loop. Not true, it just means that the parts between double quotes aren't bunched into a single token. To figure out how the compiler makes sense of this code, try https://roslynquoter.azurewebsites.net/ https://roslynquoter.azurewebsites.net/ You'll see how it tokenizes the string.

billpg 5y ago
That's not a simple tokenizer. A simple loop that would have worked with the 70s era of programming languages, would go through each character and once the boundary between two tokens has been identified, write out a token to a one-dimensional list. This would be a mostly stateless loop, tracking enough state for the current token in hand only. The next phase would go through the tokens and pair up brackets, etc. A C# tokenizer can't do that. It needs to keep a stack of state. When it sees a '}', it needs to know if that's a "normal" brace or the } that resumes a interpolated string literal. I was writing a tokenizer myself and I wanted to have something similar to string interpolation. I very quickly realized my simple loop that I would have written for my CS degree isn't going to cut it and I had to start over.
- Metasyntactic 5y ago
  Hi, I'm one of teh C# language designers, and I work on the compiler implementation as well. C# has never had a "simple tokenizer". Indeed, even the first language has complex lexical constructs that are part and parcel of the language. For example, our comments can store structured data in them (like xml). > A simple loop that would have worked with the 70s era of programming languages Yes. But 70s era compilers had to deal with things like not having enough memory to even store basic amounts of data. It also had to work in spaces where things like a 'stack' was just not tenable. We're literally 50 years from that point, and having a compiler do stuff like keeping a stack is not an issue anymore :)
  - billpg 5y ago
    Then my compliments on your efforts. When I stumbled upon my own trouble writing a tokenizer, I wondered if the C# compiler had the same issue and maybe you couldn't have {} inside interpolated strings. I wrote my earlier expression with many nested {} expressions to see if your compiler would handle it and I was very happy to see it did. Although you did deny me an excuse to not have to start over. If the C# team didn't bother, why should I? (Except you did bother.)

Metasyntactic 5y ago

>Now we're adding {{ and }} into the equation. Yay. Hi, i'm the lang designer and feature implementor here :) The complexity of lexing/parsing did not get worse here. We actually just lex/parse this stuff the same way that interpolated strings have always been lexed/parsed. This has been supported in the language for almost 10 years at this point :)

GoblinSlayer 5y ago

And how do you tell if >> is a shift operator or two angle brackets?