2 ms·
I'm curious: why can't you do tokenization in linear time? And why not use incremental tokenization for an editor?
by cwzwarich 7y ago
I'm curious: why can't you do tokenization in linear time? And why not use incremental tokenization for an editor?
- jakear 7y agoI think you’re right, the end result was actually linear. And the go extension is a thin wrapper around standard go tooling, we weren’t tokenizing ourselves just converting between their tokens and ones we could process; a large part of that was converting from byte offsets to UTC-8 character offsets.