3 ms·
I've never seen a tokenizer as one big regex that finds all the types/functions/keywords all at once.
by tyingq 8y ago
I've never seen a tokenizer as one big regex that finds all the types/functions/keywords all at once.
- dboon 8y agoTo get a bit theoretical, one way to write a lexer is to use a state machine called a deterministic finite automata - a fancy term for a state machine that is finite and always gives the same output given an input. Each new character moves the state, and there are terminal states when you reach e.g. a space which will spit out what kind of token you just read. You can show that the things a regex can compute are the same things that this kind of state machine can compute. I believe that most regex under the hood use such a state machine, so it's very natural to use regex to tokenize.
- gnulinux 8y agoYou can't do it in general, e.g. to tokenize Python you'd need a layer that'll count tabs (because INDENT and DEDENT are tokens). That it, unless your regex dialect is Turing complete, and if my memory serves, some of them are? I can't really remember. In general though, you can't do it with "normal" regex.