When implementing lexers and parsers for language specifications, I have taken the approach of testing each token (keyword, number, etc.) for the lexer for each

When implementing lexers and parsers for language specifications, I have taken the approach of testing each token (keyword, number, etc.) for the lexer for each EBNF symbol. This repeats tests for common keywords and other constructions, but means I can see that the lexer handles all the tokens in each EBNF symbol. In this way, these tests are more BDD (behavioural driven development) in style, but without things like gherkin/cucumber. With this approach, I do full coverage tests for a given symbol/token, and just check the basic symbol/token case elsewhere. I do a similar thing with the parser tests -- have a test for each valid complete symbol production, and then tests for as many error cases that the parser can handle for that symbol. With that base, I can add additional tests for bugs and additional error recovery cases I implement. I never see unit tests as a waste as they provide a set of regression tests that are invaluable for refactoring and making other changes to the code, like implementing new features or better error handling.