5 ms·
Is tree-sitter good enough?
- gravypod 4y agoAnother issue with tree-sitter is how basic functionality is not consistent across all languages. Want to find all comment text in all languages? Each tree-sitter lang support will have a slightly different behavior for every language. When you extract the comment you'll have syntax from the language in there like `/*/` or `{}`. Example from a sourcegraph docs generation project: https://github.com/sourcegraph/doctree/blob/07a2d593fe9855be6a8928394abee72e2543fef9/doctree/indexer/zig/indexer.go#L360 https://github.com/sourcegraph/doctree/blob/07a2d593fe9855be... https://github.com/sourcegraph/doctree/blob/07a2d593fe9855be6a8928394abee72e2543fef9/doctree/indexer/golang/indexer.go#L418 https://github.com/sourcegraph/doctree/blob/07a2d593fe9855be...
- layer8 4y agoRegarding the `foo() { bar() {` example, I wonder if a possible approach would be to determine in which syntactic contexts the local token sequence around the parse error (here: `bar() {`) would be correct, and then try to determine what is missing for that context (here: class scope) to have been established; or just give the error “method definition in local scope”, but still parse it as a method definition. As it currently works, the interpretation as a method definition is not even considered, because the grammar doesn’t have method definitions in that syntactic context. But that’s not how human pattern recognition works: we tend to recognize a method definition regardless of the context. So maybe we could build a parser that would try do determine all possible valid interpretations of all token subsequences in parallel (like a nondeterministic generalized parser), outputting errors where neighboring /overlapping sequences clash which each other (“this starts out like a method body but then transitions into a method definition”).