For lines it is easy, you lex bare `\n` as a "token" and increment a counter, for columns you have to have a counter as well. You already have to account for Un

For lines it is easy, you lex bare `\n` as a "token" and increment a counter, for columns you have to have a counter as well. You already have to account for Unicode codepoints to tokenize correctly. It is also true that there's a mismatch between displayed columns and the byte position for multibyte Unicode codepoints, but you also need to account for `\t` when doing so, which is plain old ASCII. Sadly this has to be a "best effort" action or you need to escape the output, otherwise you might get things like an underline shifted by the the difference between the anticipated visual width and the byte length, and this might change from under you when the underlying platform is updated. With all of that said, the approach taken in `rustc` is to have an arena of all `Span`s for everytokens and embed "pointers" (not actual pointers) to them. This is done because we embed `Span`s in everything in order to be able to give reasonable diagnostic messages.