5 ms·
I love how short and simple the tokeniser is: def tokenize(s): "Convert a string into a list of tokens." return s.replace('(',' ( ').replac
by pufuwozu 16y ago
I love how short and simple the tokeniser is:
def tokenize(s):
"Convert a string into a list of tokens."
return s.replace('(',' ( ').replace(')',' ) ').split()
- sjf 16y agoIt's simple because it doesn't do much. A really tokeniser would have the ability to handle strings, escape characters and comments.
- pufuwozu 16y agoAnd that's why it was changed to this kind-of ugly function in lispy2: def tokenize(s): """Separate string s into tokens. A token can be: a comment (which is ignored); a paren or ,@ or , or quote or quasiquote; a non-string atom; or a string consisting of quotes around (non-quotes or backslash plus anything).""" tokens = re.findall(r"""\s*(;.*|,@|[('`,)]|[^\s('");]+|"(?:[\\].|[^\\"])*")\s*""", s) return [t for t in tokens if not t.startswith(';')]