3 ms·

Were you going to parse S-expressions with regular expressions? I guess that saves you from learning how to write loops and conditionals, and what is substr() c

Were you going to parse S-expressions with regular expressions? I guess that saves you from learning how to write loops and conditionals, and what is substr() called (and what arguments it takes) in this new language, but is not that against the point of learning the new language?

bgorman 6y ago
The make a lisp tutorial provides a PCRE regular expression to generate the tokens that are later fed into the reader.
- bjoli 6y ago
  Now, I am a grumpy old fart, but I would suggest to make a one-pass recursive descent lexer+reader. That should be trivial for the MAL lisp. WRiting an ocaml recursive descent parser should be amazingly straightforward, especially since you can just tailcall the different states.
  - Joker_vD 6y ago
    While re-using an existing regexp like [\s,]*(~@|[\[\]{}()'`~^@]|"(?:\\.|[^\\"])*"?|;.*|[^\s\[\]{}('"`,;)]*) is easier than writing a tokenizer manually (keeping track offsets and looping and stuff), writing that regexp is definitely harder than writing the tokenizer like this: curr, end = 0, len(s) while True: while curr < end and isspace(s[curr]): curr += 1 if curr >= end: break if s[curr:curr + 2] == "~@": yield s[curr:curr + 2] curr += 2 elif isspecial(s[curr]): # isspecial(c) matches c against []{}()'`~^@ yield s[curr] curr += 1 elif isquote(s[curr]): # isquote(c) matches c against " start = curr curr += 1 # check this condition out: you can totally support several quotes, # and accurately match the closing and opening ones. Imagine doing it # with a regexp: either duplicate it, or use some back-referencing magic while curr < end and not (s[curr] == s[start] and s[curr-1] != '\\'): curr += 1 curr += 1 # we want to include the closing quote yield s[start:curr] elif s[curr] == ';': yield s[curr:] break else: start = curr while curr < end and not (isspace(s[curr]) or isspecial(s[curr]) or isquote(s[curr])): curr += 1 yield s[start:curr] Yeah, it's more verbose and somewhat repetitive, but on the other hand, it's way more readable, and debuggable too: with regexps, it's always a mystery which part of it exactly didn't match what you wanted or captured something you didn't want to match. Here, the loop invariants and preconditions are almost immediately obvious.
    - bjoli 6y ago
      Not only that: having a generator generate the tokens means you can do it in one pass, while writitng code that has the clarity of 2 passes.

bgorman 6y ago

The make a lisp tutorial provides a PCRE regular expression to generate the tokens that are later fed into the reader.

bjoli 6y ago
Now, I am a grumpy old fart, but I would suggest to make a one-pass recursive descent lexer+reader. That should be trivial for the MAL lisp. WRiting an ocaml recursive descent parser should be amazingly straightforward, especially since you can just tailcall the different states.
- Joker_vD 6y ago
  While re-using an existing regexp like [\s,]*(~@|[\[\]{}()'`~^@]|"(?:\\.|[^\\"])*"?|;.*|[^\s\[\]{}('"`,;)]*) is easier than writing a tokenizer manually (keeping track offsets and looping and stuff), writing that regexp is definitely harder than writing the tokenizer like this: curr, end = 0, len(s) while True: while curr < end and isspace(s[curr]): curr += 1 if curr >= end: break if s[curr:curr + 2] == "~@": yield s[curr:curr + 2] curr += 2 elif isspecial(s[curr]): # isspecial(c) matches c against []{}()'`~^@ yield s[curr] curr += 1 elif isquote(s[curr]): # isquote(c) matches c against " start = curr curr += 1 # check this condition out: you can totally support several quotes, # and accurately match the closing and opening ones. Imagine doing it # with a regexp: either duplicate it, or use some back-referencing magic while curr < end and not (s[curr] == s[start] and s[curr-1] != '\\'): curr += 1 curr += 1 # we want to include the closing quote yield s[start:curr] elif s[curr] == ';': yield s[curr:] break else: start = curr while curr < end and not (isspace(s[curr]) or isspecial(s[curr]) or isquote(s[curr])): curr += 1 yield s[start:curr] Yeah, it's more verbose and somewhat repetitive, but on the other hand, it's way more readable, and debuggable too: with regexps, it's always a mystery which part of it exactly didn't match what you wanted or captured something you didn't want to match. Here, the loop invariants and preconditions are almost immediately obvious.
  - bjoli 6y ago
    Not only that: having a generator generate the tokens means you can do it in one pass, while writitng code that has the clarity of 2 passes.