5 ms·

> Computing the column/line pair would indeed be cheap on the Lexer Calculating line and column may be complicated if you're doing it for display. Perhaps the

> Computing the column/line pair would indeed be cheap on the Lexer Calculating line and column may be complicated if you're doing it for display. Perhaps the input has a flag emoji, with a variation selector, followed by a tab... So it makes sense to store the minimum you can and defer line/column calculation, especially if your parse is perf sensitive. One place this can go wrong is when you have to show a lot of source locations, e.g. you're printing a backtrace for a stack overflow in an interpreter. Here you can easily get into an N^2 situation. So you want to defer but also memoize.

estebank 6y ago
For lines it is easy, you lex bare `\n` as a "token" and increment a counter, for columns you have to have a counter as well. You already have to account for Unicode codepoints to tokenize correctly. It is also true that there's a mismatch between displayed columns and the byte position for multibyte Unicode codepoints, but you also need to account for `\t` when doing so, which is plain old ASCII. Sadly this has to be a "best effort" action or you need to escape the output, otherwise you might get things like an underline shifted by the the difference between the anticipated visual width and the byte length, and this might change from under you when the underlying platform is updated. With all of that said, the approach taken in `rustc` is to have an arena of all `Span`s for everytokens and embed "pointers" (not actual pointers) to them. This is done because we embed `Span`s in everything in order to be able to give reasonable diagnostic messages.
- remram 6y ago
  Codepoints do not match characters. You only need to deal with codepoints to tokenize, but to find the column you need to count characters, which require a unicode database. I think this is what the parent was referring to, though I suspect a lot of compilers simply ignore this issue.

estebank 6y ago

For lines it is easy, you lex bare `\n` as a "token" and increment a counter, for columns you have to have a counter as well. You already have to account for Unicode codepoints to tokenize correctly. It is also true that there's a mismatch between displayed columns and the byte position for multibyte Unicode codepoints, but you also need to account for `\t` when doing so, which is plain old ASCII. Sadly this has to be a "best effort" action or you need to escape the output, otherwise you might get things like an underline shifted by the the difference between the anticipated visual width and the byte length, and this might change from under you when the underlying platform is updated. With all of that said, the approach taken in `rustc` is to have an arena of all `Span`s for everytokens and embed "pointers" (not actual pointers) to them. This is done because we embed `Span`s in everything in order to be able to give reasonable diagnostic messages.

remram 6y ago
Codepoints do not match characters. You only need to deal with codepoints to tokenize, but to find the column you need to count characters, which require a unicode database. I think this is what the parent was referring to, though I suspect a lot of compilers simply ignore this issue.