4 ms·
> Rust doesn't appear to have a way to say "give me the location of this &str in that &str" (even though it seems like it would be safe). That's a fantastic id
by couchand 6y ago
> Rust doesn't appear to have a way to say "give me the location of this &str in that &str" (even though it seems like it would be safe).
That's a fantastic idea. It seems like it would be not just safe, but quite efficient, too.
fn offset_of(parent: &str, child: &str) -> Option<usize> {
let cp = child.as_ptr();
let pp = parent.as_ptr();
if pp <= cp && cp <= pp + parent.len() {
Some(cp as usize - pp as usize)
} else {
None
}
}
(Edit: as can be expected, my hastily-thrown-together example code has several bugs in it. I think there's an off-by-one error in the ending condition, and I'm really not sure if it handles multi-byte characters properly.)
- estebank 6y agoYou could do that, with the caveat that a lot of tokens won't be &strs, like numeric literals, so you will need an alternative way of keeping positions/Spans into the source code for those.
- couchand 6y agoWell, if your tokenizer returns `impl Iterator<Item=&str>` (as in the OP), every token will in fact be a &str... In any case, this scheme is pretty limited, since you probably want other info in your token span like line numbers.
- estebank 6y agoIn `rustc` we keep `Span`s, which are byte offsets for start and end. We only compute line and columns when about to display diagnostics, not before, and at that we also account for "non-1-byte-to-1-col" chars (some are 0-visible-width, many other are 2 cols wide, and tabs are normalized to 4 cols wide, we need the normalization for accurate underlines). This reduces memory utilization (you don't store line/cols you will likely not need) and computation (you don't compute the line/cols you won't need). Both of these costs are actually quite low, but we also have a ton of `Span`s.