4 ms·

> We can't change the parser Aw :-( That's the bit that I'd really like to have TS macros for! My own approach has been to loosely parse input as something l

by tonyg 3y ago

> We can't change the parser Aw :-( That's the bit that I'd really like to have TS macros for! My own approach has been to loosely parse input as something like D-expressions, transform that, and pipe the result into the TS compiler. Crude but effective. The trickiest part is extending the tsserver support, but even this is doable.

LispSporks22 3y ago
Lisp user here. How come they can’t change their parser?
- goto11 3y ago
  The describe it as "an unsanctioned hack on top of typescript", so I think the point is they don't want to change the typescript parser itself, but provide this as an add-on.
- tonyg 3y ago
  The implementation of the parser is deeply, deeply baked in to the whole of the MS TypeScript compiler and tooling libraries. It's not feasible to, e.g., add new AST nodes, or change or augment the grammar producing them, and there's nothing like S- or D-expressions to allow for read/parse separation like in lisps. (Which is why I added something D-expression like.)
- o1y32 3y ago
  Practically speaking, the TS team wants the language to resemble JS as much as possible and usually avoids completely new syntax, e.g. new expressions. For this reason they haven't added switch expressions even though technically it's probably not hard to do.
  - jsmith45 3y ago
    yeah, at this point, other than type annotations and other type related syntax (like type imports and type declarations), the team mostly view the other differences from Javascript as a historical mistake. Ideally, a transpiler that does not want to type check the code, should merely need to know the syntax for type related code and strip them out. This is especially true if targeting a sufficiently recent version of ECMAScript, or if the transpiler will be doing its own transformation of newer features to older ones for backward compatibility. In practice while many typescript features can be handled like that, even some that are not strictly type features (like method accessibility modifiers), some existing features like enums needs to generate new code, although at least for that one it is an easy pattern to generate. This is especially true of any feature that cannot be made into a trivial re-write or strip rule. Namespaces and const enums are an example of features now discouraged, as simple rewriting engines cannot handle them. Indeed the whole "isolatedModules" compiler flag is specifically about making such separate simple transpilation possible, and its documentation details other scenarios that can be a problem.
sm1ch 3y ago
That sounds great - is your code available anywhere so we can build on it?
- tonyg 3y ago
  Yes: https://git.syndicate-lang.org/syndicate-lang/syndicate-js/src/branch/main/packages/compiler https://git.syndicate-lang.org/syndicate-lang/syndicate-js/s... The `syntax` subdirectory is (or should be!) pretty clean. The `compiler` subdirectory is, uh, functional. Please do get in touch if you have any questions.
dunham 3y ago
Lean4 manages to pull off changing the parser on the fly at compile time. You can add new productions, add new syntax node types, and add new tokens. Then define macros or code to process the additional syntax. Here is a sample I found that adds a simple JSX-like syntax starting around line 93 and then uses it at line 169: https://github.com/leanprover/lean4/blob/master/tests/playground/webserver/Webserver.lean#L93 https://github.com/leanprover/lean4/blob/master/tests/playgr... I believe most of the language is defined this way, although it is pre-compiled. For more details see the lean4 metaprogramming book: https://github.com/arthurpaulino/lean4-metaprogramming-book https://github.com/arthurpaulino/lean4-metaprogramming-book
- tonyg 3y ago
  Yes, I've done this too, but I don't think it's feasible for the MS TypeScript implementation, which is implemented quite differently.

by tonyg 3y ago

LispSporks22 3y ago

Lisp user here. How come they can’t change their parser?

goto11 3y ago
The describe it as "an unsanctioned hack on top of typescript", so I think the point is they don't want to change the typescript parser itself, but provide this as an add-on.
tonyg 3y ago
The implementation of the parser is deeply, deeply baked in to the whole of the MS TypeScript compiler and tooling libraries. It's not feasible to, e.g., add new AST nodes, or change or augment the grammar producing them, and there's nothing like S- or D-expressions to allow for read/parse separation like in lisps. (Which is why I added something D-expression like.)
o1y32 3y ago
Practically speaking, the TS team wants the language to resemble JS as much as possible and usually avoids completely new syntax, e.g. new expressions. For this reason they haven't added switch expressions even though technically it's probably not hard to do.
- jsmith45 3y ago
  yeah, at this point, other than type annotations and other type related syntax (like type imports and type declarations), the team mostly view the other differences from Javascript as a historical mistake. Ideally, a transpiler that does not want to type check the code, should merely need to know the syntax for type related code and strip them out. This is especially true if targeting a sufficiently recent version of ECMAScript, or if the transpiler will be doing its own transformation of newer features to older ones for backward compatibility. In practice while many typescript features can be handled like that, even some that are not strictly type features (like method accessibility modifiers), some existing features like enums needs to generate new code, although at least for that one it is an easy pattern to generate. This is especially true of any feature that cannot be made into a trivial re-write or strip rule. Namespaces and const enums are an example of features now discouraged, as simple rewriting engines cannot handle them. Indeed the whole "isolatedModules" compiler flag is specifically about making such separate simple transpilation possible, and its documentation details other scenarios that can be a problem.

sm1ch 3y ago

That sounds great - is your code available anywhere so we can build on it?

tonyg 3y ago
Yes: https://git.syndicate-lang.org/syndicate-lang/syndicate-js/src/branch/main/packages/compiler https://git.syndicate-lang.org/syndicate-lang/syndicate-js/s... The `syntax` subdirectory is (or should be!) pretty clean. The `compiler` subdirectory is, uh, functional. Please do get in touch if you have any questions.

dunham 3y ago

Lean4 manages to pull off changing the parser on the fly at compile time. You can add new productions, add new syntax node types, and add new tokens. Then define macros or code to process the additional syntax. Here is a sample I found that adds a simple JSX-like syntax starting around line 93 and then uses it at line 169: https://github.com/leanprover/lean4/blob/master/tests/playground/webserver/Webserver.lean#L93 https://github.com/leanprover/lean4/blob/master/tests/playgr... I believe most of the language is defined this way, although it is pre-compiled. For more details see the lean4 metaprogramming book: https://github.com/arthurpaulino/lean4-metaprogramming-book https://github.com/arthurpaulino/lean4-metaprogramming-book

tonyg 3y ago
Yes, I've done this too, but I don't think it's feasible for the MS TypeScript implementation, which is implemented quite differently.