7 ms·

Show HN: Zero-codegen, no-compile TypeScript type inference from Protobufs

138 points by 18nleung 1y ago

deleted 1y ago
[deleted]
jillyboel 1y ago
Cool, but I assume not great for performance? Probably better to just stick with codegen
- dtech 1y ago
  Assuming you mean compiler/editor performance then yes I assume this wrecks it. Shouldn't matter for runtime though.
  - jillyboel 1y ago
    Right, I meant editor performance. None of the options should impact runtime performance anyway.
- 18nleung 1y ago
  You're right that IDE/dev-time performance might be slower than using generated types since this relies on "dynamic" TypeScript inference rather than static codegen'd types. That said, depending on how your codegen works and how you're using protos at runtime, this approach might actually be faster at runtime. Types are stripped at compile-time and there’s no generated class or constructor logic — in the compiled output, you're left with plain JS objects which potentially avoids the serialization or class overhead that some proto codegen tools introduce. (FWIW, type inference in VSCode seemed reasonably fast with the toy examples I was playing with)
  - recursive 1y ago
    Typescript never generates classes or constructors that aren't present in source code. Whether or not constructors are present is completely independent from whether you're using code gen.
  - jillyboel 1y ago
    > depending on how your codegen works and how you're using protos at runtime, this approach might actually be faster at runtime If your codegen is introducing runtime overhead you should use a different codegen. > type inference in VSCode seemed reasonably fast with the toy examples I was playing with It usually is. It can become a problem in a real project that has a lot of stuff going on, though.
mubou 1y ago
The fact that the source is so small is wild. I would have expected a huge convoluted parsing library implemented in types. On the other hand, the fact that this is even possible is more wild. Instead of replacing JS with a proper statically-typed language, we're spending all this effort turning a preprocessor's type system into a turing-complete metalanguage. Pretty soon we'll be able to compile TypeScript entirely using types.
- 18nleung 1y ago
  I would have written a shorter source, but I did not have the time.
- sgrove 1y ago
  Or even run doom in TypeScript's type system!
  - mubou 1y ago
    Prepare to have your mind blown: https://www.youtube.com/watch?v=0mCsluv5FXA https://www.youtube.com/watch?v=0mCsluv5FXA
    - IshKebab 1y ago
      Probably not though because he was clearly referring to that.
- throwanem 1y ago
  People have fussed the same of the C preprocessor, around the same time I and maybe you were born. (There's a pretty good chance I'm your parents' age, and nearly no chance you're the age of mine.)
  - NoTeslaThrow 1y ago
    The criticisms were valid then, too. C (including the preprocessor of course) is still not fully parseable if you include things like token concatenation.
tim1994 1y ago
It's always impressive how far people can take TypeScript and even build parsers with it. But this is limited to inlined string literals and cannot read files (a TS limitation). I wonder if the author has a use case in mind for this that I don't see. Like if you are only using TS, what's the point of protobuf? If you are exchanging data with programs written in other languages why avoid the protobuf tooling that you need anyway? Maybe this is just a fun toy project to write a parser in TS?
throwanem 1y ago
I love this, and I bet the compile errors it produces on malformed protobuf are wild.
porridgeraisin 1y ago
Cool. Once the linked TS issue is resolved it will be able to resolve from files too which is great
recursive 1y ago
This requires the whole `.proto` declaration inline in source a string constant. I'm not holding my breath on "Import non-js content"[1] getting approved, so that means you still have to use another build dependency, or manually keep the .proto files synchronized across multiple sources truth. In that light, it's not clear when this would be a benefit over straight-forward code gen. Cool POC hack though. [1]: https://github.com/microsoft/TypeScript/issues/42219 https://github.com/microsoft/TypeScript/issues/42219
- cadamsdotcom 1y ago
  That can be done with a `sed` call so it’s not a new dependency.
- ZitchDog 1y ago
  The problem is that TypeScript is terrible at codegen, there are no standard extension points like we have with javac and others. So we are forced to do these crazy hacks at the type level rather than just generating types as you would in other languages.
  - recursive 1y ago
    Not familiar with the capabilities of javac, but in my imagination, I'm referring to a tool that runs prior to the typescript compiler, that just writes the intended source as text. Typescript never knew it wasn't in the repository or anything.
- catapart 1y ago
  It's true that it's another dependency, but this is the entire contents of a file I drop into my project root called `raw-loader.d.ts`: ``` declare module '*?raw' { const rawFileContent: string export default rawFileContent } ``` Then, when I add the file to my types property array of my tsconfig's compilerOptions, I can import anything I want into a typescript file as a string, so long as I add "?raw" to the end of it. I use it to inject HTML and CSS into templates. No reason it couldn't be used to inject a .proto file's contents into the inline template. Again, you're technically correct! But a "import non js content" feature is a pretty solveable problem in TS. Maybe not at the language level, but at the implementation level, at least.
spankalee 1y ago
This would be even nicer if TypeScript added type inference for tagged template literals, like in this issue [1]. Then you could write: const schema = proto` syntax = "proto3"; message Person { ... } `; type Person = typeof schema['Person']; And you could get built-in schema validation with a sophisticated enough type definition for `proto`, nice syntax highlighting in many tools with a nested grammar. We would love to see this feature in TypeScript to be able to have type-safe template in lit-html without an external tool. The issue hasn't seen much activity lately, but it would be good to highlight this library as another use case. [1]: https://github.com/microsoft/TypeScript/issues/33304 https://github.com/microsoft/TypeScript/issues/33304
- deleted 1y ago
  [deleted]
aappleby 1y ago
This is both hilarious and awesome. I think the Typescript devs are just showing off at this point. :D
meindnoch 1y ago
Looks like TypeScript envies Swift's compile times.
pragma_x 1y ago
What's kind of amazing is that Typescript's matching of strings through the type system converges on a first-class PEG in a few places (see string.ts). The rest of the library is really damn succinct for how much lifting it's doing. My hat's off to the author - I attempted something like this for a toy regex engine and got nowhere fast. This is a much better illustration of what I thought _should_ be possible, but I couldn't quite wrap my head around the use of the ternary operator to resolve types.
catapart 1y ago
Very cool work! Also, I hope you expected me to read that output in the same cadence as the Hooli focus groups, because that's exactly what I did.
mherkender 1y ago
This is kinda why I hate advanced type systems, they slowly become their own language. "No compile/no codegen" sounds nice until you get slow compile times because a type system is slow VM, the error messages are so confusing it's hard to tell what's going on, and there's no debugging tools.
jitl 1y ago
It’s pretty rad how flexible template literal types are, but I can’t imagine wanting this kind of shenanigans hanging out in a production app slowing down compile times. I prefer to define types in TypeScript and generate proto from that, since the TypeScript type system is so much more powerful than the Protobuf system. Types are much more composable in TS.
- tantalor 1y ago
  What do you use to go from ts->pb?
  - deleted 1y ago
    [deleted]
  - jitl 1y ago
    I have an old public version here: https://github.com/justjake/ts-simple-type/blob/main/src/compile-to/proto3.ts https://github.com/justjake/ts-simple-type/blob/main/src/com... Ultimately i decided ts-simple-type is too difficult to maintain, so now I just use the TypeScript compiler API directly to introspect types and emit stuff, but most of that code is private to Notion Labs Inc
- h1fra 1y ago
  Can you run Doom in a Typescript string template?
mifydev 1y ago
This makes me wonder if this the way to do schema generation in Typescript. I’m working on Typeconf, and we have a separate step for translating Typespec schema to Typescript, it’ll be cool if we could just load typespec directly.
nikolayasdf123 1y ago
I wish there was other way around. infer binary encoding from native types
anjandutta 1y ago
This is super cool — love the zero-codegen approach. I’ve had to deal with codegen hell in monorepos where a tiny .proto change breaks half the pipeline. Curious how this handles more complex types like nested messages or oneof fields? Also, been building something in a different space (LeetCode prep tool), but the idea of removing build steps for dev speed really resonates. Would love to see how this could plug into a lightweight frontend setup.

138 points by 18nleung 1y ago

deleted 1y ago

[deleted]

jillyboel 1y ago

Cool, but I assume not great for performance? Probably better to just stick with codegen

dtech 1y ago
Assuming you mean compiler/editor performance then yes I assume this wrecks it. Shouldn't matter for runtime though.
- jillyboel 1y ago
  Right, I meant editor performance. None of the options should impact runtime performance anyway.
18nleung 1y ago
You're right that IDE/dev-time performance might be slower than using generated types since this relies on "dynamic" TypeScript inference rather than static codegen'd types. That said, depending on how your codegen works and how you're using protos at runtime, this approach might actually be faster at runtime. Types are stripped at compile-time and there’s no generated class or constructor logic — in the compiled output, you're left with plain JS objects which potentially avoids the serialization or class overhead that some proto codegen tools introduce. (FWIW, type inference in VSCode seemed reasonably fast with the toy examples I was playing with)
- recursive 1y ago
  Typescript never generates classes or constructors that aren't present in source code. Whether or not constructors are present is completely independent from whether you're using code gen.
- jillyboel 1y ago
  > depending on how your codegen works and how you're using protos at runtime, this approach might actually be faster at runtime If your codegen is introducing runtime overhead you should use a different codegen. > type inference in VSCode seemed reasonably fast with the toy examples I was playing with It usually is. It can become a problem in a real project that has a lot of stuff going on, though.

mubou 1y ago

The fact that the source is so small is wild. I would have expected a huge convoluted parsing library implemented in types. On the other hand, the fact that this is even possible is more wild. Instead of replacing JS with a proper statically-typed language, we're spending all this effort turning a preprocessor's type system into a turing-complete metalanguage. Pretty soon we'll be able to compile TypeScript entirely using types.

18nleung 1y ago
I would have written a shorter source, but I did not have the time.
sgrove 1y ago
Or even run doom in TypeScript's type system!
- mubou 1y ago
  Prepare to have your mind blown: https://www.youtube.com/watch?v=0mCsluv5FXA https://www.youtube.com/watch?v=0mCsluv5FXA
  - IshKebab 1y ago
    Probably not though because he was clearly referring to that.
throwanem 1y ago
People have fussed the same of the C preprocessor, around the same time I and maybe you were born. (There's a pretty good chance I'm your parents' age, and nearly no chance you're the age of mine.)
- NoTeslaThrow 1y ago
  The criticisms were valid then, too. C (including the preprocessor of course) is still not fully parseable if you include things like token concatenation.

tim1994 1y ago

It's always impressive how far people can take TypeScript and even build parsers with it. But this is limited to inlined string literals and cannot read files (a TS limitation). I wonder if the author has a use case in mind for this that I don't see. Like if you are only using TS, what's the point of protobuf? If you are exchanging data with programs written in other languages why avoid the protobuf tooling that you need anyway? Maybe this is just a fun toy project to write a parser in TS?

throwanem 1y ago

I love this, and I bet the compile errors it produces on malformed protobuf are wild.

porridgeraisin 1y ago

Cool. Once the linked TS issue is resolved it will be able to resolve from files too which is great

recursive 1y ago

This requires the whole `.proto` declaration inline in source a string constant. I'm not holding my breath on "Import non-js content"[1] getting approved, so that means you still have to use another build dependency, or manually keep the .proto files synchronized across multiple sources truth. In that light, it's not clear when this would be a benefit over straight-forward code gen. Cool POC hack though. [1]: https://github.com/microsoft/TypeScript/issues/42219 https://github.com/microsoft/TypeScript/issues/42219

cadamsdotcom 1y ago
That can be done with a `sed` call so it’s not a new dependency.
ZitchDog 1y ago
The problem is that TypeScript is terrible at codegen, there are no standard extension points like we have with javac and others. So we are forced to do these crazy hacks at the type level rather than just generating types as you would in other languages.
- recursive 1y ago
  Not familiar with the capabilities of javac, but in my imagination, I'm referring to a tool that runs prior to the typescript compiler, that just writes the intended source as text. Typescript never knew it wasn't in the repository or anything.
catapart 1y ago
It's true that it's another dependency, but this is the entire contents of a file I drop into my project root called `raw-loader.d.ts`: ``` declare module '*?raw' { const rawFileContent: string export default rawFileContent } ``` Then, when I add the file to my types property array of my tsconfig's compilerOptions, I can import anything I want into a typescript file as a string, so long as I add "?raw" to the end of it. I use it to inject HTML and CSS into templates. No reason it couldn't be used to inject a .proto file's contents into the inline template. Again, you're technically correct! But a "import non js content" feature is a pretty solveable problem in TS. Maybe not at the language level, but at the implementation level, at least.

spankalee 1y ago

This would be even nicer if TypeScript added type inference for tagged template literals, like in this issue [1]. Then you could write: const schema = proto` syntax = "proto3"; message Person { ... } `; type Person = typeof schema['Person']; And you could get built-in schema validation with a sophisticated enough type definition for `proto`, nice syntax highlighting in many tools with a nested grammar. We would love to see this feature in TypeScript to be able to have type-safe template in lit-html without an external tool. The issue hasn't seen much activity lately, but it would be good to highlight this library as another use case. [1]: https://github.com/microsoft/TypeScript/issues/33304 https://github.com/microsoft/TypeScript/issues/33304

deleted 1y ago
[deleted]

aappleby 1y ago

This is both hilarious and awesome. I think the Typescript devs are just showing off at this point. :D

meindnoch 1y ago

Looks like TypeScript envies Swift's compile times.

pragma_x 1y ago

What's kind of amazing is that Typescript's matching of strings through the type system converges on a first-class PEG in a few places (see string.ts). The rest of the library is really damn succinct for how much lifting it's doing. My hat's off to the author - I attempted something like this for a toy regex engine and got nowhere fast. This is a much better illustration of what I thought _should_ be possible, but I couldn't quite wrap my head around the use of the ternary operator to resolve types.

catapart 1y ago

Very cool work! Also, I hope you expected me to read that output in the same cadence as the Hooli focus groups, because that's exactly what I did.

mherkender 1y ago

This is kinda why I hate advanced type systems, they slowly become their own language. "No compile/no codegen" sounds nice until you get slow compile times because a type system is slow VM, the error messages are so confusing it's hard to tell what's going on, and there's no debugging tools.

jitl 1y ago

It’s pretty rad how flexible template literal types are, but I can’t imagine wanting this kind of shenanigans hanging out in a production app slowing down compile times. I prefer to define types in TypeScript and generate proto from that, since the TypeScript type system is so much more powerful than the Protobuf system. Types are much more composable in TS.

tantalor 1y ago
What do you use to go from ts->pb?
- deleted 1y ago
  [deleted]
- jitl 1y ago
  I have an old public version here: https://github.com/justjake/ts-simple-type/blob/main/src/compile-to/proto3.ts https://github.com/justjake/ts-simple-type/blob/main/src/com... Ultimately i decided ts-simple-type is too difficult to maintain, so now I just use the TypeScript compiler API directly to introspect types and emit stuff, but most of that code is private to Notion Labs Inc
h1fra 1y ago
Can you run Doom in a Typescript string template?

mifydev 1y ago

This makes me wonder if this the way to do schema generation in Typescript. I’m working on Typeconf, and we have a separate step for translating Typespec schema to Typescript, it’ll be cool if we could just load typespec directly.

nikolayasdf123 1y ago

I wish there was other way around. infer binary encoding from native types

anjandutta 1y ago

This is super cool — love the zero-codegen approach. I’ve had to deal with codegen hell in monorepos where a tiny .proto change breaks half the pipeline. Curious how this handles more complex types like nested messages or oneof fields? Also, been building something in a different space (LeetCode prep tool), but the idea of removing build steps for dev speed really resonates. Would love to see how this could plug into a lightweight frontend setup.