Sure you should definitively not try to do the overkill use case first but I would assume that tree sitter can emit "just" tokens as well? Getting the flexibili

Sure you should definitively not try to do the overkill use case first but I would assume that tree sitter can emit "just" tokens as well? Getting the flexibility and control of a tool like tree sitter should allow you to quickly throw away stuff like comments and keywords if you want since you can do syntax aware filtering. Then again I haven't used tree-sitter, can just imagine that this is a strength of it.