3 ms·
Yes, as long as you include punctuation in your definition of "word". I like to think of it like this: A lexer takes a sequence of characters and produces a se
by munificent 12y ago
Yes, as long as you include punctuation in your definition of "word". I like to think of it like this:
A lexer takes a sequence of characters and produces a sequence of tokens. List in, list out. It just chunks runs of characters together.
A parser takes in a sequence of tokens and produces a tree. It produces a nested data structure from a flat input.
- ch 12y agoRight, a lexer is a mapping function. A parser is a fold.
- nostrademons 12y agoA lexer is a fold as well - there are usually fewer elements in the output list than in the input list. A mapping function implies a 1:1 correspondence, because the recursion is abstracted out into map() and the map function can't accumulate any state during the computation. (At least in traditional FP technology; the Mapper in MapReduce isn't actually a true map function, it's more like an unfold.)
- ch 12y agoAh true I am incorrect on the lexer as a map. Had I thought on that half a second more I would have saved myself some embarrassment. :)