4 ms·
You can also be smart about memory with lexing for great winnage. Have a struct for your tokens that has an enum for your token type and either pointer or indic
by jonstewart 2y ago
You can also be smart about memory with lexing for great winnage. Have a struct for your tokens that has an enum for your token type and either pointer or indices or a string_view (or a &str but lol lotsa luck with the borrow checker). You can then have a vector of your token structs for fast allocation and iteration and you have a slice for the token back into the original input, no substring copying.
- uecker 2y agoYes, the C FE I write (in C) does exactly this and othen utputs in one pass a flattened intermediate code. I did not see at as AST because it does semantic analysis during parsing, but it sitll has all the information.