5 ms·

We’ve use an API style similar to tRPC at Notion, although our API predated tRPC by 4 years or so. You can build this kind of thing yourself easily using Types

by jitl 3y ago

We’ve use an API style similar to tRPC at Notion, although our API predated tRPC by 4 years or so. You can build this kind of thing yourself easily using Typescript’s mapped types, by building an object type where the keys are your API names, and the values are { request, response } types. Structure your “server” bits to define each API handler as a function taking APIs[“addUser”][“request”] and returning Promise<APIs[“addUser”][“response”]>. Then build a client that exposes each API as an async function with those same args and return type. We use this strategy for our HTTPS internal API (transport over POST w/ JSON bodies), real-time APIs (transport over Websockets), Electron<>Webview APIs (transport over electron IPC), and Native (iOS, Android)<>Webview APIs (transport over OS webview ipc). For native APIs, the “server” side of things is Swift or Kotlin, and in those cases we rewrite the request and response types from Typescript by hand. I’m sure at some point we’ll switch to a binary based format with its own IDL, but for a single cross-language API that grows slowly the developer experience overhead of Protobuf or similar hasn’t seemed worth it.

purkka 3y ago
In our current project with a TS frontend and Python backend, we use an OpenAPI schema as the source of truth and openapi-typescript-codegen [0] to interface with it on the client side. While not perfect, it provides a very nice interface to our API with request/response typings. I also wrote a 10-line mock API wrapper that you can call as mockApi<SomeService["operationName"]>((request) => response), and it will type-check that your mock function implements the API correctly and return a function that looks exactly like the real API function. [0]: https://github.com/ferdikoomen/openapi-typescript-codegen https://github.com/ferdikoomen/openapi-typescript-codegen
- iansinnott 3y ago
  Can second this approach. At a past job we did the same, except to connect the frontend to a Go backend. I really like that the openAPI approach is language agnostic, and makes it relatively simple to support SDKs for many other languages if needed. For any company where the API itself is a product, OpenAPI is great.
  - nerdponx 3y ago
    We use OpenAPI internally for communication between systems. It eliminates an entire category of bugs, and we can offload testing of schema conformance entirely to third-party tools. We've never had a bug due to a mistake in calling an internal API that I know of. And we get both internal documentation and a web UI for free via SwaggerUI and Redoc. Good zero-config OpenAPI support is one of the best features of the FastAPI framework in Python. The "fast" part refers to the speed of basic product up and running.
- Cannabat 3y ago
  Another great library to generate TS types from OpenAPI is https://github.com/drwpow/openapi-typescript https://github.com/drwpow/openapi-typescript . It provides the types as single objects you access via indexing, which is pretty nice. There's a partner library to generate a typed fetch client.
  - taswellian 3y ago
    Do you know if there's something similar for Python? Or what approach would you use for Python?
    - Cannabat 3y ago
      Like generating pydantic models or dataclasses for an OpenAPI schema? I haven't needed to go in that direction myself, but this[0] looks promising! Apologies if I've misunderstood your comment https://koxudaxi.github.io/datamodel-code-generator/ https://koxudaxi.github.io/datamodel-code-generator/
      - taswellian 3y ago
        This looks interesting, thank you! I'm basically wondering what would help us produce (automatically?) a Python HTTP client based an OpenAPI spec, and achieve a development experience similar to using drwpow/openapi-typescript. drwpow/openapi-typescript will generate the whole schema, along with paths, request and response models, as well as path and query parameters. In the IDE, all client methods will be type-safe and autocompleted. Example: client = createClient<GeneratedSchema>(); client.get("/my/autocompleted/endpoint/{some_param}", { params: { path: { some_param: "foobar" } } }) koxudaxi/datamodel-code-generator seems to generate the response models, but not much else. It seems we would have to manually write wrapper methods for each endpoint, manually specify the parameters and request models, and use a type hint for the return value of each method with the generated response model. Example: def my_wrapped_endpoint(...) -> ResponseModel: pass client.my_wrapped_endpoint(some_param="foobar") I'd like to avoid any manual wrapping or at least minimize the amount of it. Basically: to achieve a similar experience to what we have in TypeScript. I hope my question is a bit clearer now. I'm not that familiar with Python, so I will appreciate any guidance regarding this problem. :)
scriptsmith 3y ago
Is there some trick to doing validation of request data using this process? That's a valuable part of using something like tRPC, JSON Schema + type generation, zod, etc.
- jitl 3y ago
  We use an internal validator library that we infer request types from. It’s similar to Zod (but also predates it by a year). I’ve also spent some time on a Typescript type to X compiler. My first prototype is open source and targets Thrift, Proto3, Python, and JSON schema: https://github.com/justjake/ts-simple-type/tree/main/src/compile-to https://github.com/justjake/ts-simple-type/tree/main/src/com... I’m not happy with the design decision in that codebase to try to “simplify” Typescript types before compiling, and probably won’t continue that implementation, but we have a few internal code generators that consume TS types and output test data builders and model clases we use in production. I want to open source some of those bits but haven’t found the time.
- chaos_emergent 3y ago
  Deepkit is a fantastic solution for this. It uses a compilation step to inject metadata about types into plain JS. https://deepkit.io/ https://deepkit.io/
  - jitl 3y ago
    Deepkit looks really cool, but it’s so complex on the inside and leverages a forked/patched Typescript and requires full typecheck before emit. What happens if the Deepkit guy retires? What if I want to run my code without waiting for 11 minutes of typechecking? What if there’s a bug somewhere in there? There’s way too much risk for me to consider Deepkit for production.
ushakov 3y ago
For cross-language, I can recommend Fern, which works with OpenAPI http://buildwithfern.com http://buildwithfern.com
- mdaniel 3y ago
  You can recommend it in what context, from openapi (as they claim https://github.com/fern-api/fern#starting-from-openapi https://github.com/fern-api/fern#starting-from-openapi ) or from their ... special ... definition schema? For those wanting less talk, moar code: https://github.com/fern-api/fern-java/blob/0.4.2-rc3/example-project/api/src/api.yml https://github.com/fern-api/fern-java/blob/0.4.2-rc3/example... -> https://github.com/fern-api/fern-java/blob/0.4.2-rc3/example-project/example-spring-server/src/main/java/com/example/ExampleResource.java https://github.com/fern-api/fern-java/blob/0.4.2-rc3/example... Regrettably, I wasn't able to readily find a matching openapi example, likely cause they really seem to believe their kid-gloves format is the future (or lock in, depending on how bitter one is feeling) --- confusingly, their repo has a "YCombinator 2023" badge on it that just links to the badge itself. Some Algolia for Launch HN didn't cough up anything, but there was a Show HN I found: https://news.ycombinator.com/item?id=34346428 https://news.ycombinator.com/item?id=34346428
  - dsinghvi 3y ago
    Disclosure: I'm a contributor to Fern. This is good feedback that we need to provide more examples starting with OpenAPI instead of with the Fern Definition. For some context, we convert the OpenAPI spec into a Fern definition and then pass that into the code generators. If you want to see some real world examples, check out these links: - https://github.com/vellum-ai/vellum-client-generator/tree/main/fern/api/openapi https://github.com/vellum-ai/vellum-client-generator/tree/ma... -> https://github.com/vellum-ai/vellum-client-python https://github.com/vellum-ai/vellum-client-python - https://github.com/Squidex/sdk-fern/tree/main/fern/api/openapi https://github.com/Squidex/sdk-fern/tree/main/fern/api/opena... -> https://github.com/Squidex/sdk-node https://github.com/Squidex/sdk-node Worth calling out that you can go from the Fern Definition -> OpenAPI anytime to prevent lock in.
c-hendricks 3y ago
This works quite well actually: https://github.com/mikew/transmission-material-ui/blob/master/src/api/index.ts#L39-L42 https://github.com/mikew/transmission-material-ui/blob/maste... https://github.com/mikew/transmission-material-ui/blob/master/src/api/types.d.ts#L15-L259 https://github.com/mikew/transmission-material-ui/blob/maste...
- yencabulator 3y ago
  That's seems to be completely missing validation. Typescript types are at their worst when they are lies, and the actual shape of the data is something completely different.
  - c-hendricks 3y ago
    It's copied and adapted directly from their RPC docs, validation would just be busy work. https://github.com/transmission/transmission/blob/main/docs/rpc-spec.md https://github.com/transmission/transmission/blob/main/docs/...
    - yencabulator 3y ago
      For Transmission, maybe. In general, validation is very much not "busywork".
      - c-hendricks 3y ago
        On top of automatically generated types (eg. openapi, graphql), I would say it is. Seems like you're just inserting yourself and ranting about typescript where it's not really applicable.
        yencabulator 3y ago
        Lovely attitude, you have a nice day too. There's others expressing the same concerns, for example https://news.ycombinator.com/item?id=37101393 https://news.ycombinator.com/item?id=37101393
nip 3y ago
Similarly, we built a typed-client for inter-service communication using lambdas at Robocorp. The requests and responses are inferred from the interface (published following semver) defined in Zod (including the errors that are following HTTP-like conventions: 401, 409…). It also includes “hooks” for pre and post processing on the server side (effectively middlewares)
scottmas 3y ago
This is the way. tRPC adds unnecessary complexity over simply inferring types. My theory is that no well maintained and promoted library adopting this approach has emerged, and that’s why you don’t see it discussed very often.
dclowd9901 3y ago
Yeah, I’m reading their sample code and wondering if this is just type-imbuing wrappers on top of XHR calls. It even asks you to provide the generic argument in the invocation (this sucks for trying to keep your dependency tree in order).
danvk 3y ago
I built something that sounds very similar at my last company called crosswalk. Open sourced here: https://github.com/danvk/crosswalk https://github.com/danvk/crosswalk One thing that worked surprisingly well: codegen TypeScript types from your database and use those in your API schema.

by jitl 3y ago

purkka 3y ago

In our current project with a TS frontend and Python backend, we use an OpenAPI schema as the source of truth and openapi-typescript-codegen [0] to interface with it on the client side. While not perfect, it provides a very nice interface to our API with request/response typings. I also wrote a 10-line mock API wrapper that you can call as mockApi<SomeService["operationName"]>((request) => response), and it will type-check that your mock function implements the API correctly and return a function that looks exactly like the real API function. [0]: https://github.com/ferdikoomen/openapi-typescript-codegen https://github.com/ferdikoomen/openapi-typescript-codegen

iansinnott 3y ago
Can second this approach. At a past job we did the same, except to connect the frontend to a Go backend. I really like that the openAPI approach is language agnostic, and makes it relatively simple to support SDKs for many other languages if needed. For any company where the API itself is a product, OpenAPI is great.
- nerdponx 3y ago
  We use OpenAPI internally for communication between systems. It eliminates an entire category of bugs, and we can offload testing of schema conformance entirely to third-party tools. We've never had a bug due to a mistake in calling an internal API that I know of. And we get both internal documentation and a web UI for free via SwaggerUI and Redoc. Good zero-config OpenAPI support is one of the best features of the FastAPI framework in Python. The "fast" part refers to the speed of basic product up and running.
Cannabat 3y ago
Another great library to generate TS types from OpenAPI is https://github.com/drwpow/openapi-typescript https://github.com/drwpow/openapi-typescript . It provides the types as single objects you access via indexing, which is pretty nice. There's a partner library to generate a typed fetch client.
- taswellian 3y ago
  Do you know if there's something similar for Python? Or what approach would you use for Python?
  - Cannabat 3y ago
    Like generating pydantic models or dataclasses for an OpenAPI schema? I haven't needed to go in that direction myself, but this[0] looks promising! Apologies if I've misunderstood your comment https://koxudaxi.github.io/datamodel-code-generator/ https://koxudaxi.github.io/datamodel-code-generator/
    - taswellian 3y ago
      This looks interesting, thank you! I'm basically wondering what would help us produce (automatically?) a Python HTTP client based an OpenAPI spec, and achieve a development experience similar to using drwpow/openapi-typescript. drwpow/openapi-typescript will generate the whole schema, along with paths, request and response models, as well as path and query parameters. In the IDE, all client methods will be type-safe and autocompleted. Example: client = createClient<GeneratedSchema>(); client.get("/my/autocompleted/endpoint/{some_param}", { params: { path: { some_param: "foobar" } } }) koxudaxi/datamodel-code-generator seems to generate the response models, but not much else. It seems we would have to manually write wrapper methods for each endpoint, manually specify the parameters and request models, and use a type hint for the return value of each method with the generated response model. Example: def my_wrapped_endpoint(...) -> ResponseModel: pass client.my_wrapped_endpoint(some_param="foobar") I'd like to avoid any manual wrapping or at least minimize the amount of it. Basically: to achieve a similar experience to what we have in TypeScript. I hope my question is a bit clearer now. I'm not that familiar with Python, so I will appreciate any guidance regarding this problem. :)

scriptsmith 3y ago

Is there some trick to doing validation of request data using this process? That's a valuable part of using something like tRPC, JSON Schema + type generation, zod, etc.

jitl 3y ago
We use an internal validator library that we infer request types from. It’s similar to Zod (but also predates it by a year). I’ve also spent some time on a Typescript type to X compiler. My first prototype is open source and targets Thrift, Proto3, Python, and JSON schema: https://github.com/justjake/ts-simple-type/tree/main/src/compile-to https://github.com/justjake/ts-simple-type/tree/main/src/com... I’m not happy with the design decision in that codebase to try to “simplify” Typescript types before compiling, and probably won’t continue that implementation, but we have a few internal code generators that consume TS types and output test data builders and model clases we use in production. I want to open source some of those bits but haven’t found the time.
chaos_emergent 3y ago
Deepkit is a fantastic solution for this. It uses a compilation step to inject metadata about types into plain JS. https://deepkit.io/ https://deepkit.io/
- jitl 3y ago
  Deepkit looks really cool, but it’s so complex on the inside and leverages a forked/patched Typescript and requires full typecheck before emit. What happens if the Deepkit guy retires? What if I want to run my code without waiting for 11 minutes of typechecking? What if there’s a bug somewhere in there? There’s way too much risk for me to consider Deepkit for production.

ushakov 3y ago

For cross-language, I can recommend Fern, which works with OpenAPI http://buildwithfern.com http://buildwithfern.com

mdaniel 3y ago
You can recommend it in what context, from openapi (as they claim https://github.com/fern-api/fern#starting-from-openapi https://github.com/fern-api/fern#starting-from-openapi ) or from their ... special ... definition schema? For those wanting less talk, moar code: https://github.com/fern-api/fern-java/blob/0.4.2-rc3/example-project/api/src/api.yml https://github.com/fern-api/fern-java/blob/0.4.2-rc3/example... -> https://github.com/fern-api/fern-java/blob/0.4.2-rc3/example-project/example-spring-server/src/main/java/com/example/ExampleResource.java https://github.com/fern-api/fern-java/blob/0.4.2-rc3/example... Regrettably, I wasn't able to readily find a matching openapi example, likely cause they really seem to believe their kid-gloves format is the future (or lock in, depending on how bitter one is feeling) --- confusingly, their repo has a "YCombinator 2023" badge on it that just links to the badge itself. Some Algolia for Launch HN didn't cough up anything, but there was a Show HN I found: https://news.ycombinator.com/item?id=34346428 https://news.ycombinator.com/item?id=34346428
- dsinghvi 3y ago
  Disclosure: I'm a contributor to Fern. This is good feedback that we need to provide more examples starting with OpenAPI instead of with the Fern Definition. For some context, we convert the OpenAPI spec into a Fern definition and then pass that into the code generators. If you want to see some real world examples, check out these links: - https://github.com/vellum-ai/vellum-client-generator/tree/main/fern/api/openapi https://github.com/vellum-ai/vellum-client-generator/tree/ma... -> https://github.com/vellum-ai/vellum-client-python https://github.com/vellum-ai/vellum-client-python - https://github.com/Squidex/sdk-fern/tree/main/fern/api/openapi https://github.com/Squidex/sdk-fern/tree/main/fern/api/opena... -> https://github.com/Squidex/sdk-node https://github.com/Squidex/sdk-node Worth calling out that you can go from the Fern Definition -> OpenAPI anytime to prevent lock in.

c-hendricks 3y ago

This works quite well actually: https://github.com/mikew/transmission-material-ui/blob/master/src/api/index.ts#L39-L42 https://github.com/mikew/transmission-material-ui/blob/maste... https://github.com/mikew/transmission-material-ui/blob/master/src/api/types.d.ts#L15-L259 https://github.com/mikew/transmission-material-ui/blob/maste...

yencabulator 3y ago
That's seems to be completely missing validation. Typescript types are at their worst when they are lies, and the actual shape of the data is something completely different.
- c-hendricks 3y ago
  It's copied and adapted directly from their RPC docs, validation would just be busy work. https://github.com/transmission/transmission/blob/main/docs/rpc-spec.md https://github.com/transmission/transmission/blob/main/docs/...
  - yencabulator 3y ago
    For Transmission, maybe. In general, validation is very much not "busywork".
    - c-hendricks 3y ago
      On top of automatically generated types (eg. openapi, graphql), I would say it is. Seems like you're just inserting yourself and ranting about typescript where it's not really applicable.
      - yencabulator 3y ago
        Lovely attitude, you have a nice day too. There's others expressing the same concerns, for example https://news.ycombinator.com/item?id=37101393 https://news.ycombinator.com/item?id=37101393

nip 3y ago

Similarly, we built a typed-client for inter-service communication using lambdas at Robocorp. The requests and responses are inferred from the interface (published following semver) defined in Zod (including the errors that are following HTTP-like conventions: 401, 409…). It also includes “hooks” for pre and post processing on the server side (effectively middlewares)

scottmas 3y ago

This is the way. tRPC adds unnecessary complexity over simply inferring types. My theory is that no well maintained and promoted library adopting this approach has emerged, and that’s why you don’t see it discussed very often.

dclowd9901 3y ago

Yeah, I’m reading their sample code and wondering if this is just type-imbuing wrappers on top of XHR calls. It even asks you to provide the generic argument in the invocation (this sucks for trying to keep your dependency tree in order).

danvk 3y ago

I built something that sounds very similar at my last company called crosswalk. Open sourced here: https://github.com/danvk/crosswalk https://github.com/danvk/crosswalk One thing that worked surprisingly well: codegen TypeScript types from your database and use those in your API schema.