4 ms·
You could do that, with the caveat that a lot of tokens won't be &strs, like numeric literals, so you will need an alternative way of keeping positions/Spans in
by estebank 6y ago
You could do that, with the caveat that a lot of tokens won't be &strs, like numeric literals, so you will need an alternative way of keeping positions/Spans into the source code for those.
- couchand 6y agoWell, if your tokenizer returns `impl Iterator<Item=&str>` (as in the OP), every token will in fact be a &str... In any case, this scheme is pretty limited, since you probably want other info in your token span like line numbers.
- estebank 6y agoIn `rustc` we keep `Span`s, which are byte offsets for start and end. We only compute line and columns when about to display diagnostics, not before, and at that we also account for "non-1-byte-to-1-col" chars (some are 0-visible-width, many other are 2 cols wide, and tabs are normalized to 4 cols wide, we need the normalization for accurate underlines). This reduces memory utilization (you don't store line/cols you will likely not need) and computation (you don't compute the line/cols you won't need). Both of these costs are actually quite low, but we also have a ton of `Span`s.