3 ms·

This is true, but in other cases they added keywords in ways that could work with type stripping. For example, the `as` keyword for casts has existed for a long

This is true, but in other cases they added keywords in ways that could work with type stripping. For example, the `as` keyword for casts has existed for a long time, and type stripping could strip everything after the `as` keyword with a minimal grammar. When TypeScript added const declarations, they added it as `as const` so a type stripping could have still worked depending on how loosely it is implemented. I think there is a world where type stripping exists (which the TS team has been in favor of) and the TS team might consider how it affects type stripping in future language design. For example, the `satisfies` keyword could have also been added by piggy-backing on the `as` keyword, like: const foo = { bar: 1 } as subtype of Foo (I think not using `as` is a better fit semantically but this could be a trade-off to make for better type stripping backwards compatibility)

Timon3 2y ago
I don't know a lot about parser theory, and would love to learn more about ways to make parsing resilient in cases like this one. Simple cases like "ignore rest of line" make sense to me, but I'm unsure about "adversarial" examples (in the sense that they are meant to beat simple heuristics). Would you mind explaining how e.g. your `as` stripping could work for one specific adversarial example? function foo<T>() { return bar( null as unknown as T extends boolean ? true /* ): */ : (T extends string ? "string" : false ) ) } function bar(value: any): void {} Any solution I can come up with suffers from at least one of these issues: - "ignore rest of line" will either fail or lead to incorrect results - "find matching parenthesis" would have to parse comments inside types (probably doable, but could break with future TS additions) - "try finding end of non-JS code" will inevitably trip up in some situations, and can get very expensive I'd love a rough outline or links/pointers, if you can find the time! [0] TS Playground link: https://www.typescriptlang.org/play/?#code/AQ4MwVwOwYwFwJYHsriUgPAFQHwAoBKYAbwChQLgAnAUzgitQCMBDKvcyrqCAG1+AsAzsGgBrKEgDuqYcCzAaADzg0oAExFN0vGiyicuFAPzA4VCDWAB6AFTACALmC3rhoyGd4Fy1RpFC5ghQAObuHibAAESBVMEhUeERnuAsvEI0SVwEWZQ5FAC+pO6QsIgowKzsAG5pls76AJ5OwNVICOokRUA https://www.typescriptlang.org/play/?#code/AQ4MwVwOwYwFwJYHs...
- WorldMaker 2y ago
  Most parsers don't actually work with "lines" as a unit, those are for user-formatting. Generally the sort of building blocks you are looking for are more along the lines of "until end of expression" or "until end of statement". What defines an "expression" or a "statement" can be very complex depending on the parser and the language you are trying to parse. In JS, because it is a fun example, "end of statement" is defined in large part by Automatic Semicolon Insertion (ASI), whether or not semicolons even exist in the source input. (Even if you use semicolons regularly in JS, JS will still insert its own semicolons. Semicolons don't protect you from ASI.) ASI is also a useful example because it is an ancient example of a language design intentionally trying to be resilient. Some older JS parsers even would ignore bad statements and continue on the next statement based on ASI determined statement break. We generally like our JS to be much more strict than that today, but early JS was originally built to be a resilient language in some interesting ways. One place to dive into that directly (in the middle of a deeper context of JS parser theory): https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Lexical_grammar#automatic_semicolon_insertion https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
  - Timon3 2y ago
    Thanks for the response, but I'm aware of the basics. My question is pointed towards making language parsers resilient towards separately-evolving standards. How would you build a JS parser so that it correctly parses any new TS syntax, without changing behavior of valid code? The example snippet I added is designed to violate the rules I could come up with. I'd specifically like to know: what are better rules to solve this specific case?
    - thanksgiving 2y ago
      > How would you build a JS parser so that it correctly parses any new TS syntax, without changing behavior of valid code? I don't know anything about parsers besides what I learned from that one semester worth of introduction class I took in college but from what I understand of your question, I think the answer is you can't simply because we can't look into the future.
zarzavat 2y ago
It can’t strip what’s after the as keyword without an up-to-date TS grammar, because `as` is an expression. The parser needs to know how to parse type expressions in order to know when the RHS of the `as` expression ends. Let’s say that typescript adds a new type operator “wobble T”. What does this desugar to? x as wobble T Without knowing about the new wobble syntax this would be parsed as `x as wobble; T` and desugar to `x; T` With the new wobble syntax it would be parsed as `x as (wobble T);` according to JS semicolon insertion rules because the expression wobble is incomplete, and desugar to `x`
- bazoom42 2y ago
  The “as” expression is not valid JavaScript anyway, so the default rule for implicit semicolon does not apply. A grammer for type expressions could define if and how semicolons should be inserted.
  - zarzavat 2y ago
    TypeScript already has such type operators though. For example: type T = keyof { a: null, b: null } Here T is “a”|”b”, no automatic semicolon is inserted after `keyof`. While I don’t personally write code like this, I’m sure that someone does. It’s perfectly within the rules, after all. While it’s true that TS doesn’t have to follow JS rules for semicolon insertion in type expressions, it always has done, and probably always should do.
    - bazoom42 2y ago
      This is just the default. Automatic semicolon insertion only happen in specific well-defined cases, for example after the “return” keyword or when an invalid expression can be made valid by semicolon insertion. Neither applies here.

Timon3 2y ago

I don't know a lot about parser theory, and would love to learn more about ways to make parsing resilient in cases like this one. Simple cases like "ignore rest of line" make sense to me, but I'm unsure about "adversarial" examples (in the sense that they are meant to beat simple heuristics). Would you mind explaining how e.g. your `as` stripping could work for one specific adversarial example? function foo<T>() { return bar( null as unknown as T extends boolean ? true /* ): */ : (T extends string ? "string" : false ) ) } function bar(value: any): void {} Any solution I can come up with suffers from at least one of these issues: - "ignore rest of line" will either fail or lead to incorrect results - "find matching parenthesis" would have to parse comments inside types (probably doable, but could break with future TS additions) - "try finding end of non-JS code" will inevitably trip up in some situations, and can get very expensive I'd love a rough outline or links/pointers, if you can find the time! [0] TS Playground link: https://www.typescriptlang.org/play/?#code/AQ4MwVwOwYwFwJYHsriUgPAFQHwAoBKYAbwChQLgAnAUzgitQCMBDKvcyrqCAG1+AsAzsGgBrKEgDuqYcCzAaADzg0oAExFN0vGiyicuFAPzA4VCDWAB6AFTACALmC3rhoyGd4Fy1RpFC5ghQAObuHibAAESBVMEhUeERnuAsvEI0SVwEWZQ5FAC+pO6QsIgowKzsAG5pls76AJ5OwNVICOokRUA https://www.typescriptlang.org/play/?#code/AQ4MwVwOwYwFwJYHs...

WorldMaker 2y ago
Most parsers don't actually work with "lines" as a unit, those are for user-formatting. Generally the sort of building blocks you are looking for are more along the lines of "until end of expression" or "until end of statement". What defines an "expression" or a "statement" can be very complex depending on the parser and the language you are trying to parse. In JS, because it is a fun example, "end of statement" is defined in large part by Automatic Semicolon Insertion (ASI), whether or not semicolons even exist in the source input. (Even if you use semicolons regularly in JS, JS will still insert its own semicolons. Semicolons don't protect you from ASI.) ASI is also a useful example because it is an ancient example of a language design intentionally trying to be resilient. Some older JS parsers even would ignore bad statements and continue on the next statement based on ASI determined statement break. We generally like our JS to be much more strict than that today, but early JS was originally built to be a resilient language in some interesting ways. One place to dive into that directly (in the middle of a deeper context of JS parser theory): https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Lexical_grammar#automatic_semicolon_insertion https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
- Timon3 2y ago
  Thanks for the response, but I'm aware of the basics. My question is pointed towards making language parsers resilient towards separately-evolving standards. How would you build a JS parser so that it correctly parses any new TS syntax, without changing behavior of valid code? The example snippet I added is designed to violate the rules I could come up with. I'd specifically like to know: what are better rules to solve this specific case?
  - thanksgiving 2y ago
    > How would you build a JS parser so that it correctly parses any new TS syntax, without changing behavior of valid code? I don't know anything about parsers besides what I learned from that one semester worth of introduction class I took in college but from what I understand of your question, I think the answer is you can't simply because we can't look into the future.

zarzavat 2y ago

It can’t strip what’s after the as keyword without an up-to-date TS grammar, because `as` is an expression. The parser needs to know how to parse type expressions in order to know when the RHS of the `as` expression ends. Let’s say that typescript adds a new type operator “wobble T”. What does this desugar to? x as wobble T Without knowing about the new wobble syntax this would be parsed as `x as wobble; T` and desugar to `x; T` With the new wobble syntax it would be parsed as `x as (wobble T);` according to JS semicolon insertion rules because the expression wobble is incomplete, and desugar to `x`

bazoom42 2y ago
The “as” expression is not valid JavaScript anyway, so the default rule for implicit semicolon does not apply. A grammer for type expressions could define if and how semicolons should be inserted.
- zarzavat 2y ago
  TypeScript already has such type operators though. For example: type T = keyof { a: null, b: null } Here T is “a”|”b”, no automatic semicolon is inserted after `keyof`. While I don’t personally write code like this, I’m sure that someone does. It’s perfectly within the rules, after all. While it’s true that TS doesn’t have to follow JS rules for semicolon insertion in type expressions, it always has done, and probably always should do.
  - bazoom42 2y ago
    This is just the default. Automatic semicolon insertion only happen in specific well-defined cases, for example after the “return” keyword or when an invalid expression can be made valid by semicolon insertion. Neither applies here.