3 ms·

Peter Norvig did one for Python, and it's pretty cool: https://github.com/norvig/pytudes/blob/master/ipynb/BASIC.ipynb https://github.com/norvig/pytudes/blob/ma

• tyingq 8y ago
  Interesting abuse of a regex as the tokenizer. Cool.
  • elcomet 8y ago
    How is it an abuse? I thought using regex as the tokenizer was the main approach
    • tyingq 8y ago
      I've never seen a tokenizer as one big regex that finds all the types/functions/keywords all at once.
      • dboon 8y ago
        To get a bit theoretical, one way to write a lexer is to use a state machine called a deterministic finite automata - a fancy term for a state machine that is finite and always gives the same output given an input. Each new character moves the state, and there are terminal states when you reach e.g. a space which will spit out what kind of token you just read. You can show that the things a regex can compute are the same things that this kind of state machine can compute. I believe that most regex under the hood use such a state machine, so it's very natural to use regex to tokenize.
      • gnulinux 8y ago
        You can't do it in general, e.g. to tokenize Python you'd need a layer that'll count tabs (because INDENT and DEDENT are tokens). That it, unless your regex dialect is Turing complete, and if my memory serves, some of them are? I can't really remember. In general though, you can't do it with "normal" regex.
    • GordonS 8y ago
      It's the way I always approach it when parsing a DSL, but always line-by-line, rather than a regex across the whole script. I know it isn't going to be the most performant approach, but if you're comfortable with regular expressions it's really simple, and performant enough for most cases.