The Externals API is designed to only let you do things that are compatible with incremental parsing. It’s good for things like indentation based delimiters and

The Externals API is designed to only let you do things that are compatible with incremental parsing. It’s good for things like indentation based delimiters and automatic semicolon insertion. You cannot call back into the parser; it’s just for tokenization. You can produce zero length tokens, and we do use this feature a lot. You dont have access to the parse state directly, but the external scanner is passed an array of Boolean values that indicates which external tokens are expected in the current state.