For example, here below is a CL version of "Enter The Scanner". Instead of a closure, we could instanciate a class, but this is sufficient: (defun create-to

For example, here below is a CL version of "Enter The Scanner". Instead of a closure, we could instanciate a class, but this is sufficient: (defun create-tokenizer (rules) (loop for (token rule) in rules for regex = (if (stringp rule) `(:regex ,rule) rule) collect `(:register ,regex) into alternatives collect token into tokens finally (let ((scanner (ppcre:create-scanner `(:alternation ,@alternatives))) (tokens (coerce tokens 'vector))) (return (lambda (string &key (start 0)) ;; generator-like (lambda () (multiple-value-bind (match-start match-end registers) (ppcre:scan scanner string :start start) (cond (match-start (setf start match-end) (values (aref tokens (position-if-not #'null registers)) match-start match-end)) (t (values)))))))))) The above compiles the regex and returns a closure which accepts a string. That closure returns another closure which generates tokens on-demand for the given string, along with the start and end positions of the token inside the string. Here is a test: (loop with tokenizer = (funcall (create-tokenizer '((ws "\\s+") (plus #\+) (minus #\-) (mult #\*) (div #\/) (num :digit-class) (par-open #\() (par-close #\)))) "(1 + 2) * 3") for token = (multiple-value-list (funcall tokenizer)) while token collect token) => ((par-open 0 1) (num 1 2) (ws 2 3) (plus 3 4) (ws 4 5) (num 5 6) (par-close 6 7) (ws 7 8) (mult 8 9) (ws 9 10) (num 10 11))