I've put this into Codex fn hello(message: String) -> Result<String> { this is not part of code } Codex does a pretty code job at tokenizing the dif

I've put this into Codex fn hello(message: String) -> Result<String> { this is not part of code } Codex does a pretty code job at tokenizing the different parts of the first line (fn, open parenthesis, -> is considered too, its own token) but fails miserably on the second line. The second line should be a single token of invalid code. It should be tokenized into text, if that line was preceded by "//" or a start comment indicator. Interestingly, GPT-3/4 can probably explain the concept of commenting and specifically for Rust too. However, it can't apply it in this particular context.