3 ms·

Ahh my mistake! :-) To be fair, we're cheating a little bit because the Ruby grammar relies so heavily on an external scannar, which is just under 1,000 lines

Ahh my mistake! :-) To be fair, we're cheating a little bit because the Ruby grammar relies so heavily on an external scannar, which is just under 1,000 lines of C++: https://github.com/tree-sitter/tree-sitter-ruby/blob/master/src/scanner.cc https://github.com/tree-sitter/tree-sitter-ruby/blob/master/...

chrisseaton 6y ago
But for example how do you parse the difference between `x = 14; x` and `y = 14; x`? In the latter case `x` is a method call, and in the former it's a local variable read. I can't see where the parser maintains a set of local variables and where it queries this set. Is it somehow done declaratively? If so that's a huge achievement I don't think that's really been done before in a parser generator. I really want to try tree-sitter for using in an actual Ruby implementation because it's so beautiful!
- dcreager 6y ago
  [EDITED to make the example actually line up with OP's test] There's no symbol table in the parser, so at parse time, we don't distinguish those cases: $ cat test.rb module Test def test1 x = 14; x end def test2 y = 14; x end end $ tree-sitter parse test.rb (program [0, 0] - [9, 0] (module [0, 0] - [8, 3] name: (constant [0, 7] - [0, 11]) (method [1, 2] - [3, 5] name: (identifier [1, 6] - [1, 11]) (assignment [2, 4] - [2, 10] left: (identifier [2, 4] - [2, 5]) right: (integer [2, 8] - [2, 10])) (identifier [2, 12] - [2, 13])) (method [5, 2] - [7, 5] name: (identifier [5, 6] - [5, 11]) (assignment [6, 4] - [6, 10] left: (identifier [6, 4] - [6, 5]) right: (integer [6, 8] - [6, 10])) (identifier [6, 12] - [6, 13])))) In both cases the bit after the semicolon just parses as (identifier). For some use cases (e.g. syntax highlighting, depending on your colorization rules) it doesn't matter, and so we don't want to pay the cost. If it does matter (like in an actual implementation), then you'd have to implement this yourself and drive it by the parse tree you get from tree-sitter.
  - chrisseaton 6y ago
    Right you could just have a phase to fix-it-up after parsing. Much better than trying to shoe-horn an imperative action into a nice more-pure parser. Great idea!
- tclem 6y ago
  So what's cool is that while we don't handle that during parsing, you can use another set of tree-sitter features to do tree queries to achieve this. Here's the query for detecting Ruby locals: https://github.com/tree-sitter/tree-sitter-ruby/blob/32cd5a04adb4accb0c121f037ab59df3c3488228/queries/locals.scm https://github.com/tree-sitter/tree-sitter-ruby/blob/32cd5a0... and here's some better documentation for how the query language works: https://tree-sitter.github.io/tree-sitter/syntax-highlighting#local-variables https://tree-sitter.github.io/tree-sitter/syntax-highlightin....

chrisseaton 6y ago

But for example how do you parse the difference between `x = 14; x` and `y = 14; x`? In the latter case `x` is a method call, and in the former it's a local variable read. I can't see where the parser maintains a set of local variables and where it queries this set. Is it somehow done declaratively? If so that's a huge achievement I don't think that's really been done before in a parser generator. I really want to try tree-sitter for using in an actual Ruby implementation because it's so beautiful!

dcreager 6y ago
[EDITED to make the example actually line up with OP's test] There's no symbol table in the parser, so at parse time, we don't distinguish those cases: $ cat test.rb module Test def test1 x = 14; x end def test2 y = 14; x end end $ tree-sitter parse test.rb (program [0, 0] - [9, 0] (module [0, 0] - [8, 3] name: (constant [0, 7] - [0, 11]) (method [1, 2] - [3, 5] name: (identifier [1, 6] - [1, 11]) (assignment [2, 4] - [2, 10] left: (identifier [2, 4] - [2, 5]) right: (integer [2, 8] - [2, 10])) (identifier [2, 12] - [2, 13])) (method [5, 2] - [7, 5] name: (identifier [5, 6] - [5, 11]) (assignment [6, 4] - [6, 10] left: (identifier [6, 4] - [6, 5]) right: (integer [6, 8] - [6, 10])) (identifier [6, 12] - [6, 13])))) In both cases the bit after the semicolon just parses as (identifier). For some use cases (e.g. syntax highlighting, depending on your colorization rules) it doesn't matter, and so we don't want to pay the cost. If it does matter (like in an actual implementation), then you'd have to implement this yourself and drive it by the parse tree you get from tree-sitter.
- chrisseaton 6y ago
  Right you could just have a phase to fix-it-up after parsing. Much better than trying to shoe-horn an imperative action into a nice more-pure parser. Great idea!
tclem 6y ago
So what's cool is that while we don't handle that during parsing, you can use another set of tree-sitter features to do tree queries to achieve this. Here's the query for detecting Ruby locals: https://github.com/tree-sitter/tree-sitter-ruby/blob/32cd5a04adb4accb0c121f037ab59df3c3488228/queries/locals.scm https://github.com/tree-sitter/tree-sitter-ruby/blob/32cd5a0... and here's some better documentation for how the query language works: https://tree-sitter.github.io/tree-sitter/syntax-highlighting#local-variables https://tree-sitter.github.io/tree-sitter/syntax-highlightin....