5 ms·

This is just an anecdote, but I had to write a SQL DDL and DML parser during my last semester. I wrote the DDL parser by hand, and it wasn't as bad as I expecte

by jjice 5y ago

This is just an anecdote, but I had to write a SQL DDL and DML parser during my last semester. I wrote the DDL parser by hand, and it wasn't as bad as I expected, but it was time consuming. I managed to convince the professor to give us the option of using a parser generator for the next phase (DML) since the point of the class wasn't parsing context free grammars and more focused on executing the SQL. I used Flex and Bison since the project was in C. Getting up and running and understanding how the tools have to be set up with different options was a bit tricky, but after that, my parser was up and running in about two hours, compared to probably four times that for the hand written DDL. Our DML subset was also much larger and more complex than our DDL, so I was very happy with the development speed increase. I had this idea that using a parser generator was slow and wasteful since many modern tutorials online write them by hand and speak against parser generators (possibly because there isn't a catch all for all languages). Turns out dev speed is way more important to me up front, because in the case that I notice parsing speed actually being an issue I should be happy that my MVP has gotten enough use. It's also nice because a lexer and parser can be pretty easily black-boxed and swapped out for your hand written, just keep the AST and API the same and you should be good. All that said, that's personal preferences and writing the parser by hand is definitely good experience and more extensible, especially for error handling. Nice work!

brundolf 5y ago
To add an anecdote, I've gotten into a hobby of writing parsers and parser-related things, and I've developed a pattern and a handful of utility functions (which are small enough to be re-written from memory) that together make hand-writing a pretty breezy task. Just the other week at work I hand-wrote a GraphQL schema parser from scratch in about 2 hours using this technique (GraphQL's schemas have a relatively simple syntax, but I still think it's a decent data point)
- qorrect 5y ago
  Is it hosted somewhere we can see it?
  - brundolf 5y ago
    It's not, though I've thought about open-sourcing or doing a write-up on my pattern, but I've just assumed there are tons of libraries out there for this that are probably better than my homegrown version. Could still make an interesting blog post though, I suppose
dboreham 5y ago
Your post came through a wormhole from 1979.
WalterBright 5y ago
> dev speed is way more important to me up front Having written many parsers, I can attest that the time spent writing and debugging the parser is about 0.0000001% of the time you'll invest in the compiler. Heck, I spent more time trying to figure out how to integrate the tag name space from ImportC into the host D compiler's symbol table mechanism, than I did writing the C parser.
- codr7 5y ago
  Second that, I think the divide is simply experience writing parsers.
- neilv 5y ago
  In a university compilers class, I suppose that implementing the parser (or the tokenizer part of it) might be used as a weedout assignment. Or to tell students without much programming experience that they are going to have to put a lot of energy into the class (so don't, say, take compilers, operating systems, organic chemistry, and epic novel-writing all in the same term). (When I took compilers, the initial weedout/attention-getting assignment was artificial: write a string/symbol table manager module that could handle strings of arbitrary size, up to megabytes or larger for each entry. Though I don't know how well this first assignment worked for setting expectations, or if it could've been improved. After that school term, I heard from someone else in the class that he and I had been the only two people in the class to actually "finish" our compilers. Not that we were smarter or harder-working, but we both had a lot of prior programming experience, so a big head start. Setting expectations for a project-heavy systems CS class seems hard to do, without discouraging people who could do it if they invest the time, and I suppose just noticing a couple nerds who had a big head start could be discouraging to everyone else. One of the secret keys to learning/accomplishing something hard is to feel you're doing well enough to stick with it long enough to do well.)
- WalterBright 5y ago
  P.S. Having a parallel tag symbol table means doubling the work in creating and tearing down symbol tables. Adding another member to the Symbol struct means lots of extra memory allocated. Tag symbols are relatively rare, so I was willing to trade lookup times for reduced memory consumption. Wound up adding a pointer to a hash table in each symbol table, where the tag symbols would be stored. Only when tag symbols are actually used is it ever accessed or even initialized. It works a treat.

by jjice 5y ago

brundolf 5y ago

To add an anecdote, I've gotten into a hobby of writing parsers and parser-related things, and I've developed a pattern and a handful of utility functions (which are small enough to be re-written from memory) that together make hand-writing a pretty breezy task. Just the other week at work I hand-wrote a GraphQL schema parser from scratch in about 2 hours using this technique (GraphQL's schemas have a relatively simple syntax, but I still think it's a decent data point)

qorrect 5y ago
Is it hosted somewhere we can see it?
- brundolf 5y ago
  It's not, though I've thought about open-sourcing or doing a write-up on my pattern, but I've just assumed there are tons of libraries out there for this that are probably better than my homegrown version. Could still make an interesting blog post though, I suppose

dboreham 5y ago

Your post came through a wormhole from 1979.

WalterBright 5y ago

> dev speed is way more important to me up front Having written many parsers, I can attest that the time spent writing and debugging the parser is about 0.0000001% of the time you'll invest in the compiler. Heck, I spent more time trying to figure out how to integrate the tag name space from ImportC into the host D compiler's symbol table mechanism, than I did writing the C parser.

codr7 5y ago
Second that, I think the divide is simply experience writing parsers.
neilv 5y ago
In a university compilers class, I suppose that implementing the parser (or the tokenizer part of it) might be used as a weedout assignment. Or to tell students without much programming experience that they are going to have to put a lot of energy into the class (so don't, say, take compilers, operating systems, organic chemistry, and epic novel-writing all in the same term). (When I took compilers, the initial weedout/attention-getting assignment was artificial: write a string/symbol table manager module that could handle strings of arbitrary size, up to megabytes or larger for each entry. Though I don't know how well this first assignment worked for setting expectations, or if it could've been improved. After that school term, I heard from someone else in the class that he and I had been the only two people in the class to actually "finish" our compilers. Not that we were smarter or harder-working, but we both had a lot of prior programming experience, so a big head start. Setting expectations for a project-heavy systems CS class seems hard to do, without discouraging people who could do it if they invest the time, and I suppose just noticing a couple nerds who had a big head start could be discouraging to everyone else. One of the secret keys to learning/accomplishing something hard is to feel you're doing well enough to stick with it long enough to do well.)
WalterBright 5y ago
P.S. Having a parallel tag symbol table means doubling the work in creating and tearing down symbol tables. Adding another member to the Symbol struct means lots of extra memory allocated. Tag symbols are relatively rare, so I was willing to trade lookup times for reduced memory consumption. Wound up adding a pointer to a hash table in each symbol table, where the tag symbols would be stored. Only when tag symbols are actually used is it ever accessed or even initialized. It works a treat.