3 ms·
Well, if your tokenizer returns `impl Iterator<Item=&str>` (as in the OP), every token will in fact be a &str... In any case, this scheme is pretty limited, si
by couchand 6y ago
Well, if your tokenizer returns `impl Iterator<Item=&str>` (as in the OP), every token will in fact be a &str...
In any case, this scheme is pretty limited, since you probably want other info in your token span like line numbers.
- estebank 6y agoIn `rustc` we keep `Span`s, which are byte offsets for start and end. We only compute line and columns when about to display diagnostics, not before, and at that we also account for "non-1-byte-to-1-col" chars (some are 0-visible-width, many other are 2 cols wide, and tabs are normalized to 4 cols wide, we need the normalization for accurate underlines). This reduces memory utilization (you don't store line/cols you will likely not need) and computation (you don't compute the line/cols you won't need). Both of these costs are actually quite low, but we also have a ton of `Span`s.