4 ms·
A bit off-topic question: why do people use "parsing" instead of "lexing" or "tokenization" for denoting the analysis of strings from regular languages; conside
by nsajko 6y ago
A bit off-topic question: why do people use "parsing" instead of "lexing" or "tokenization" for denoting the analysis of strings from regular languages; considering lexing is just a trivial sub-case of parsing anything that includes the regular languages?
Perhaps more on topic: I think this can be even faster if one is willing to offload the heavy lifting to an FPGA, here's a result from 2016: https://www.miraclelinux.com/labs/pdf/fpga-en https://www.miraclelinux.com/labs/pdf/fpga-en
- pubby 6y agoLexing/tokenization implies creating tokens to pass to another parser. This is more general than that.
- gameswithgo 6y agoHuman language is flexible and imprecise. There is no compiler. English in particular is often interpreted in a descriptivist manner (meaning is defined based on how the public uses the words) rather than prescriptivist (meaning is defined by central authority). Parsing is sometimes used to describe the translation of tokens to abstract syntax tree, sometimes it is used to describe both tokenzation and tree formation together, sometimes it is used to describe any process of taking in text and producing something more structured (Turbo Pascal for instance went straight from text -> machine code, was that parsing?) Tokenzation is taking in text and producing something more structured, so it may be appropriate to call that a form of parsing too. Many programmers would like for human language to be as strict and precise as programming, but absent some plan to cause that to happen it is better to go with the flow, and simply be as clear in your own communication as you can rather than quibbling about definitions of others all the time. You won't ever win.