2 ms·
> All of the lexer and parser code in perl5 are purely custom and intertwined. I had given myself a torture by taking a look at perl5/toke.c [1]. Among dozens
by lifthrasiir 6y ago
> All of the lexer and parser code in perl5 are purely custom and intertwined.
I had given myself a torture by taking a look at perl5/toke.c [1]. Among dozens of lexical analyzers I've ever seen, it is the only tokenizer that uses a probablistic estimator to determine the extent of the current token (grep for the comment "this is terrifying, and it works"). Perl 5 is my go-to language for counterexamples in programming language grammers and syntaxes.
[1] https://github.com/Perl/perl5/blob/blead/toke.c https://github.com/Perl/perl5/blob/blead/toke.c