2 ms·

You need to use the "lexer trick". While lexing your input data, instead of ignoring whitespace or producing token WS or TAB or whatever, keep track of your ind

You need to use the "lexer trick". While lexing your input data, instead of ignoring whitespace or producing token WS or TAB or whatever, keep track of your indents and newlines. Then it's simple to decide if it's an indent (newline followed by at least 1 more space or tab than previous, add this to your indents stack) or dedent (newline followed by the previous indent (look at your indent stack)). Then produce INDENT and DEDENT seperate tokens. Once you have these tokens produced by tokenizer, it's easy to parse. Do the usual recursive descent. Source: I built my own python compiler (python->C++) as a university projects.

tomp 7y ago
Pretty much this. You need a stack because an single NEWLINE might result in 2 DEDENT tokens: if x: if y: f() g() You only need to keep track of whitespace (spaces and/or tabs) after a newline (and you can skip lines with only whitespace & comments). It's also useful to keep tracks of delimiters (like parentheses and brackets), if you want to ignore newlines/indents inside expressions (like Python). The easiest IMO is writing a lexer using a lexer generator (Lex, Flex, ...) and outputting a NEWLINE(spaces) tokens, then have another class that keeps track of indents, parentheses, etc. I wrote a lexer/parser combo that (1) tracks whitespace in top (file) context, (2) ignores whitespace in expressions (parentheses, [] arrays, {} records), except (3) restarts indentation-tracking in {} blocks (even if they're in parentheses themselves!) - now that's hard to code (and debug LOL).