5 ms·

> When I got started, I had just learned how to use yacc (really Bison) and lex (really flex) I've dug into parsers a few times, but everything I encountered s

> When I got started, I had just learned how to use yacc (really Bison) and lex (really flex) I've dug into parsers a few times, but everything I encountered seemed to think that once I had a parsed tree of commands it was obvious how to consume it...and it wasn't (for me). I've never had the free time to dedicate to experimenting that abstractly, so anytime I'm tempted to write a DSL or similar for a current problem I punt and do something else. Does anyone have advice on how to find nice ways to use parsers/lexers in a practical way that doesn't involve a massive investment of time or assume I have a lot of abstract comp sci background? Every thing I've seen has been more of a compilers course, which seems overkill.

lmm 8y ago
I could never understand why you have a parser as a separate thing when this was presented as an up-front decision. Reading this series of posts, where something like a traditional compiler architecture emerges in the same way that normal program architecture emerges via gradual refactoring, made it make a lot more sense to me: https://hokstad.com/compiler https://hokstad.com/compiler .
jerf 8y ago
What skills/languages do you have? It can help tune the suggestions. Barring that, you can google "compiler tutorial" and just start looking for one you like. One nice thing about compilers is they consist of a lot of stages and while the whole is arguably greater than the sum of its parts, the parts are all pretty darned useful on their own, too.
- ergothus 8y ago
  Javascript is my strongest, and I have enough Python to be comfortable - I had a lot of Perl back in the day but that knowledge is fairly rusty (though Parse::RecDescent was my first exposure to parsers). I did Java for a few years, but I refer to those as my Dark Times and wouldn't want to venture back there. (Experienced Perl dev doing Java is NOT a good time for either the dev or the code) I can't argue about the broad applicability of compilers, but the main reason I've avoided such tutorials is that I'm looking for something I can get the basics of in hours/days rather than a weeks/months.
  - rictic 8y ago
    If you got to the point of having a tree, but found them clumsy to work with, the first thing I'd suggest is to play with an AST explorer. https://astexplorer.net/ https://astexplorer.net/ is quite good. Picture the task of what you want to do in your head. Maybe you're writing a linter, and you want to enforce a rule like "never use the 'var' keyword". Write some example code and poke around in the explorer. Another thing to keep in mind is that once you have a tree, usually you also want to have some tree traversal utilities, so that you can walk over the tree and optionally transform it.
  - james_s_tayler 8y ago
    For the record neither is experienced Java dev doing Perl.
adrianN 8y ago
IMHO you're often best off with writing a simple recursive descent parser by hand. For lexing you can often get away with splitting the input string at word boundaries. For slightly more advanced needs, parser combinator libraries make parsing and lexing quite straightforward. I honestly wouldn't use a parser generator.
- angara 8y ago
  Regular expressions are also useful for lexing. I like to work from this example code from the Python standard library docs: https://docs.python.org/3.6/library/re.html#writing-a-tokenizer https://docs.python.org/3.6/library/re.html#writing-a-tokeni...
  - fanf2 8y ago
    Remember, `lex` is just a pile of regular expressions, so it is great for simple parsing in C. There are good reasons it was known as the Swiss Army knife of Unix before scripting took over :-)
- swiley 8y ago
  IMHO recursive ascent is both easier to write and faster.
amenod 8y ago
Actually "compilers course" is exactly what you should be looking for imho. Compilers are basically apps that parse input and output something in response to it - which is what this is all about. Lex and yacc are very simple to use once you understand the core principles.
UK-Al05 8y ago
Tbh parser combinators and PEG parsers are the modern way of doing these sorts of things.
blattimwind 8y ago
PEGs (parsing expression grammars) are probably the by far easiest way currently.

lmm 8y ago

I could never understand why you have a parser as a separate thing when this was presented as an up-front decision. Reading this series of posts, where something like a traditional compiler architecture emerges in the same way that normal program architecture emerges via gradual refactoring, made it make a lot more sense to me: https://hokstad.com/compiler https://hokstad.com/compiler .

jerf 8y ago

What skills/languages do you have? It can help tune the suggestions. Barring that, you can google "compiler tutorial" and just start looking for one you like. One nice thing about compilers is they consist of a lot of stages and while the whole is arguably greater than the sum of its parts, the parts are all pretty darned useful on their own, too.

ergothus 8y ago
Javascript is my strongest, and I have enough Python to be comfortable - I had a lot of Perl back in the day but that knowledge is fairly rusty (though Parse::RecDescent was my first exposure to parsers). I did Java for a few years, but I refer to those as my Dark Times and wouldn't want to venture back there. (Experienced Perl dev doing Java is NOT a good time for either the dev or the code) I can't argue about the broad applicability of compilers, but the main reason I've avoided such tutorials is that I'm looking for something I can get the basics of in hours/days rather than a weeks/months.
- rictic 8y ago
  If you got to the point of having a tree, but found them clumsy to work with, the first thing I'd suggest is to play with an AST explorer. https://astexplorer.net/ https://astexplorer.net/ is quite good. Picture the task of what you want to do in your head. Maybe you're writing a linter, and you want to enforce a rule like "never use the 'var' keyword". Write some example code and poke around in the explorer. Another thing to keep in mind is that once you have a tree, usually you also want to have some tree traversal utilities, so that you can walk over the tree and optionally transform it.
- james_s_tayler 8y ago
  For the record neither is experienced Java dev doing Perl.

adrianN 8y ago

IMHO you're often best off with writing a simple recursive descent parser by hand. For lexing you can often get away with splitting the input string at word boundaries. For slightly more advanced needs, parser combinator libraries make parsing and lexing quite straightforward. I honestly wouldn't use a parser generator.

angara 8y ago
Regular expressions are also useful for lexing. I like to work from this example code from the Python standard library docs: https://docs.python.org/3.6/library/re.html#writing-a-tokenizer https://docs.python.org/3.6/library/re.html#writing-a-tokeni...
- fanf2 8y ago
  Remember, `lex` is just a pile of regular expressions, so it is great for simple parsing in C. There are good reasons it was known as the Swiss Army knife of Unix before scripting took over :-)
swiley 8y ago
IMHO recursive ascent is both easier to write and faster.

amenod 8y ago

Actually "compilers course" is exactly what you should be looking for imho. Compilers are basically apps that parse input and output something in response to it - which is what this is all about. Lex and yacc are very simple to use once you understand the core principles.

UK-Al05 8y ago

Tbh parser combinators and PEG parsers are the modern way of doing these sorts of things.

blattimwind 8y ago

PEGs (parsing expression grammars) are probably the by far easiest way currently.