To get a bit theoretical, one way to write a lexer is to use a state machine called a deterministic finite automata - a fancy term for a state machine that is f

To get a bit theoretical, one way to write a lexer is to use a state machine called a deterministic finite automata - a fancy term for a state machine that is finite and always gives the same output given an input. Each new character moves the state, and there are terminal states when you reach e.g. a space which will spit out what kind of token you just read. You can show that the things a regex can compute are the same things that this kind of state machine can compute. I believe that most regex under the hood use such a state machine, so it's very natural to use regex to tokenize.