4 ms·

After some frustration with the TypeScript schema library ecosystem, I've decided that I'd prefer to declare my types using TypeScript's excellent type syntax,

by jitl 4y ago

After some frustration with the TypeScript schema library ecosystem, I've decided that I'd prefer to declare my types using TypeScript's excellent type syntax, so I can take advantage of generics, mapped types, etc. Then, I'll take those TypeScript types and compile them to whatever alternative schema format I need. There are many libraries that claim to convert your Typescript types to other formats, such as ts-json-schema-generator, ts-to-zod, or typeconv/core-types-ts. These libraries work by interpreting the Typescript AST, essentially re-implementing a bare-bones type system from scratch. Most do not support advanced Typescript features like generic application, mapped types, or string literal types. So, what's the point? To avoid those limitations, I use Typescript's first-party ts.TypeChecker API to analyze types, and an existing library called ts-simple-type (which I've forked) to convert from ts.Type to a more usable intermediate format. Then, I recurse over the intermediate format and emit "AST nodes". It's pretty simple, but seems promising. So far, I have compilers from TypeScript type to Python 3 and Thrift. But I plan to add OpenAPI/JSONSchema, Protobuf (Proto3), Kotlin, Swift, and maybe Zod and Avro. Each target language is around ~300 LoC so they're pretty easy to put together. My ultimate goal is to use this toolkit with Notion's internal types, which are quite a bit more complex than something like ts-to-zod can handle. Repo: https://github.com/justjake/ts-simple-type https://github.com/justjake/ts-simple-type Compiler input and output: https://github.com/justjake/ts-simple-type/blob/jake--compilers-as-features/test/snapshots/compiler.spec.ts.md https://github.com/justjake/ts-simple-type/blob/jake--compil... Thrift compiler: https://github.com/justjake/ts-simple-type/blob/jake--compilers-as-features/src/compile-to/thrift.ts https://github.com/justjake/ts-simple-type/blob/jake--compil... Python compiler: https://github.com/justjake/ts-simple-type/blob/jake--compilers-as-features/src/compile-to/python3.ts https://github.com/justjake/ts-simple-type/blob/jake--compil...

eropple 4y ago
> Most do not support advanced Typescript features like generic application, mapped types, or string literal types. So, what's the point? I use @sinclair/typebox and the point is that I can get a JSON Schema out the other side, usable in-process and as a communicative description of what my code expects, just by evaluating my code and printing out an object. It does support string literal types, FWIW. I'm good with it not supporting generics, because the places where I need an interchange format generally don't. Mapped types would be nice but aren't really critical to where I need to generate interchange formats; dependent types that result from transforming these can use mapped types, so it's close enough. For my money, typebox has far-and-away the best user experience of any of the flavor of libraries you described, and IMO is probably worth studying.
- jitl 4y ago
  I agree that the runtime-validador-with-type-inference library genre is generally enough for API surface area. We have a similar internal library, including JSON schema output, and it’s serviceable for checking our public & private API incoming requests. It does come with some costs. First, the type inference using these large mapped types (at least in our implementation and/or scale) noticeably slows down our Typescript type checking; the public api files with our runtime type declarations always top our build profiling. Second, it’s annoying to need to re-implement an existing type as a runtime-type if you start to need it in an API specification. There’s a tension in the codebase between using the easily-available, language native type system, or making do with the runtime type stuff which increases verbosity/noise and decreases expressiveness. We have thousands of lines of types, including many types inferred from `as const` literals, written in Typescript. Rewriting those types in another schema language, such as typebox or Protobuf, would take a bunch of time and spread that tension from an isolated spot (backend API handlers) into every part of the codebase. It imposes new constraints, and requires devs to learn more things. The libraries I listed and the compiler I’m building try to address this issue by supporting the native typescript declaration syntax. No more red types and blue types, much better gradual adoption pathway, and ideally one less thingy to learn for most developers.
brodo 4y ago
Looks promising. Thanks for sharing. This also needs a CLI. I was thinking about building something like this in Rust using swc.
- jitl 4y ago
  Swc will give you an AST, but you’ll still need to re-implement mapped types, type inference, template string types, etc - a large part of the TS type checker. Although, the author of swc is supposedly building a full type checker. > This also needs a CLI. I’m not planning on implementing one at the moment. The project goal is to greatly expand the use-cases for Typescript types by making it much easier to build custom compile targets. A CLI means directing much more of my energy to configuration logic and file system IO. As a library, it’s clear how users should “configure” behavior - write or subclass a compiler target and implement whatever you want!
  - brodo 4y ago
    Ah - I see. I've started to build a CLI around it and adopted the `SimpleTypeCompiler` example to build simple JSON schema files. I'm struggling with the API a bit, but I think I can manage.
gavinray 4y ago
Wow, holy smokes. This is incredible, thanks for posting.

by jitl 4y ago

eropple 4y ago

> Most do not support advanced Typescript features like generic application, mapped types, or string literal types. So, what's the point? I use @sinclair/typebox and the point is that I can get a JSON Schema out the other side, usable in-process and as a communicative description of what my code expects, just by evaluating my code and printing out an object. It does support string literal types, FWIW. I'm good with it not supporting generics, because the places where I need an interchange format generally don't. Mapped types would be nice but aren't really critical to where I need to generate interchange formats; dependent types that result from transforming these can use mapped types, so it's close enough. For my money, typebox has far-and-away the best user experience of any of the flavor of libraries you described, and IMO is probably worth studying.

jitl 4y ago
I agree that the runtime-validador-with-type-inference library genre is generally enough for API surface area. We have a similar internal library, including JSON schema output, and it’s serviceable for checking our public & private API incoming requests. It does come with some costs. First, the type inference using these large mapped types (at least in our implementation and/or scale) noticeably slows down our Typescript type checking; the public api files with our runtime type declarations always top our build profiling. Second, it’s annoying to need to re-implement an existing type as a runtime-type if you start to need it in an API specification. There’s a tension in the codebase between using the easily-available, language native type system, or making do with the runtime type stuff which increases verbosity/noise and decreases expressiveness. We have thousands of lines of types, including many types inferred from `as const` literals, written in Typescript. Rewriting those types in another schema language, such as typebox or Protobuf, would take a bunch of time and spread that tension from an isolated spot (backend API handlers) into every part of the codebase. It imposes new constraints, and requires devs to learn more things. The libraries I listed and the compiler I’m building try to address this issue by supporting the native typescript declaration syntax. No more red types and blue types, much better gradual adoption pathway, and ideally one less thingy to learn for most developers.

brodo 4y ago

Looks promising. Thanks for sharing. This also needs a CLI. I was thinking about building something like this in Rust using swc.

jitl 4y ago
Swc will give you an AST, but you’ll still need to re-implement mapped types, type inference, template string types, etc - a large part of the TS type checker. Although, the author of swc is supposedly building a full type checker. > This also needs a CLI. I’m not planning on implementing one at the moment. The project goal is to greatly expand the use-cases for Typescript types by making it much easier to build custom compile targets. A CLI means directing much more of my energy to configuration logic and file system IO. As a library, it’s clear how users should “configure” behavior - write or subclass a compiler target and implement whatever you want!
- brodo 4y ago
  Ah - I see. I've started to build a CLI around it and adopted the `SimpleTypeCompiler` example to build simple JSON schema files. I'm struggling with the API a bit, but I think I can manage.

gavinray 4y ago

Wow, holy smokes. This is incredible, thanks for posting.