3 ms·

Thank you for the details; I will study them more. But let me try again with my main question. I want to see what the lexer outputs. How can I view that? I'

Thank you for the details; I will study them more. But let me try again with my main question. I want to see what the lexer outputs. How can I view that? I'd like to be able to hand a string (or file) of what I think is valid (or invalid) Red code and have the lexer return to me that input NOT executed but parsed into lexemes, grouping tokens, and appropriate white space. Again, according to the specification you linked, the lexing is a discrete step to resolve any ambiguity of syntax in this manner ahead of interpretation/execution. So I'd think accessing the lex output would be trivial and transparent -- not to mention invaluable for learning and debugging Red. Why should lexer be crystal clear about exactly what is going to be interpreted, and in what order, while the human is sometimes left in the dark? Then, obviously, I'd like to be able to pass this lexed and "grouped" string directly in to the interpreter for execution as a second step. Showing me the result of interpreting a string isn't nearly so helpful as this would be. Can this be done? Why would I want to do this? Well, I've weird. ;-/ More seriously, at this point you guys don't have a good stepping debugger - it would help me debug my own code. It would also be a rather cool thing to be able to do just for the hell of it, as a great exploratory exercise. I would think I could then play with the lexemes individually once I can see exactly what the lexer sees instead of only what I think - too often mistakenly - the lexer sees. Thanks for entertaining my questions.

9214 8y ago
> I'd like to be able to hand a string (or file) of what I think is valid (or invalid) Red code and have the lexer return to me that input NOT executed but parsed into lexemes, grouping tokens, and appropriate white space. This is what "load" essentially does - it takes a string and returns a concrete syntax tree [1]. "do" then takes that CST and evaluates it. You can do that in a single-step manner with "do/next" (there's also "load/next"). Moreso, you can think of blocks as phrase markers (see wiki example): [S [NP John] [VP [V hit] [NP the [N ball]]]] Now compare it to Red internal data structure written down in similar fashion: <block! [<integer! 1> <word! +> <integer! 2>]> Here I loosely follow phrase marker notation - nonterminals correspond to datatypes, terminals correspond to literal values. But datatype is implied by literal form, and implicitly contained in each value slot (its a datatype ID tag in the header), so I can tidy this up to: [1 + 2] This is what lexer returns - just a block. You can then manipulate it whichever way you want and evaluate. > at this point you guys don't have a good stepping debugger We don't have it for a reason - implementing it in user-space would be extremely limited as you can't really distinguish between code and data, and so can't adequately place debugging hooks. The best alternative is just to spruce up block with debugging "print"s and such, or, in case of syntactic errors use "load/trap" and search for error values. Rebol had some interesting projects in this regard, e.g. Anamonitor comes to mind [2]. I think someone already ported it to Red, or re-implemented using our reactive framework. There was a recent discussion in community chat about that, and we came to conclusion that debugger should rather be implemented at the level of "load" and "do" themselves (e.g. "load" can provide some metainfo, like line numbers and file name, "do" then can single-step it in real time and provide the meaningful info). [1]: https://en.wikipedia.org/wiki/Parse_tree https://en.wikipedia.org/wiki/Parse_tree [2]: http://rebol2.blogspot.com/2011/11/anamonitor-2-check-block-contents.html http://rebol2.blogspot.com/2011/11/anamonitor-2-check-block-...

9214 8y ago

> I'd like to be able to hand a string (or file) of what I think is valid (or invalid) Red code and have the lexer return to me that input NOT executed but parsed into lexemes, grouping tokens, and appropriate white space. This is what "load" essentially does - it takes a string and returns a concrete syntax tree [1]. "do" then takes that CST and evaluates it. You can do that in a single-step manner with "do/next" (there's also "load/next"). Moreso, you can think of blocks as phrase markers (see wiki example): [S [NP John] [VP [V hit] [NP the [N ball]]]] Now compare it to Red internal data structure written down in similar fashion: <block! [<integer! 1> <word! +> <integer! 2>]> Here I loosely follow phrase marker notation - nonterminals correspond to datatypes, terminals correspond to literal values. But datatype is implied by literal form, and implicitly contained in each value slot (its a datatype ID tag in the header), so I can tidy this up to: [1 + 2] This is what lexer returns - just a block. You can then manipulate it whichever way you want and evaluate. > at this point you guys don't have a good stepping debugger We don't have it for a reason - implementing it in user-space would be extremely limited as you can't really distinguish between code and data, and so can't adequately place debugging hooks. The best alternative is just to spruce up block with debugging "print"s and such, or, in case of syntactic errors use "load/trap" and search for error values. Rebol had some interesting projects in this regard, e.g. Anamonitor comes to mind [2]. I think someone already ported it to Red, or re-implemented using our reactive framework. There was a recent discussion in community chat about that, and we came to conclusion that debugger should rather be implemented at the level of "load" and "do" themselves (e.g. "load" can provide some metainfo, like line numbers and file name, "do" then can single-step it in real time and provide the meaningful info). [1]: https://en.wikipedia.org/wiki/Parse_tree https://en.wikipedia.org/wiki/Parse_tree [2]: http://rebol2.blogspot.com/2011/11/anamonitor-2-check-block-contents.html http://rebol2.blogspot.com/2011/11/anamonitor-2-check-block-...