4 ms·
It's simple because it doesn't do much. A really tokeniser would have the ability to handle strings, escape characters and comments.
by sjf 16y ago
It's simple because it doesn't do much. A really tokeniser would have the ability to handle strings, escape characters and comments.
- pufuwozu 16y agoAnd that's why it was changed to this kind-of ugly function in lispy2: def tokenize(s): """Separate string s into tokens. A token can be: a comment (which is ignored); a paren or ,@ or , or quote or quasiquote; a non-string atom; or a string consisting of quotes around (non-quotes or backslash plus anything).""" tokens = re.findall(r"""\s*(;.*|,@|[('`,)]|[^\s('");]+|"(?:[\\].|[^\\"])*")\s*""", s) return [t for t in tokens if not t.startswith(';')]