4 ms·

I would probably go with gRPC + grpc-gateway[1] instead. Declaring your services and models in proto files, annotating your services with google.api.http to hel

by 0xbkt 6y ago

I would probably go with gRPC + grpc-gateway[1] instead. Declaring your services and models in proto files, annotating your services with google.api.http to help grpc-gateway scaffold your HTTP base. Then just implement your services from the interface generated by grpc-go. You can even register your gRPC services to grpc-gateway without actually bringing up a gRPC server. You finally end up having your exact data models injected to your service handlers. Thus, you don't have to repeat yourself preparing the groundwork to call into your services. This is mostly the way today services in big OSS projects are exposed outside for consumption in a RESTful style. One exception I know of is sourcegraph/sourcegraph. [1] https://github.com/grpc-ecosystem/grpc-gateway#usage https://github.com/grpc-ecosystem/grpc-gateway#usage

yolo42 6y ago
This is exactly what I started using in my new open source project (not announced yet) and the experience has been amazing. I was wary about it in the beginning but it quickly became clear that this is the easiest and probably the fastest way to get this job done. The community and tooling has come far ahead.
whoisjohnkid 6y ago
Have you looked into goa design[1]? You define your services with a contract via their DSL and it generates all code for you as well; has some nice validation options out of the box. - similar to grpc gateway you just need to conform to the generated code’s service interface. I believe it supports, http, websocket and grpc. [1]https://github.com/goadesign/goa https://github.com/goadesign/goa
anderspitman 6y ago
These things can be fun to play with, but my experience is most projects don't need them. I would rather spend my time implementing functionality than regenerating and recompiling code from proto files. The large projects you mention often have something most projects don't: a well-defined data model and interface that has been heavily iterated and evolved over time. At that point things like backwards compatibility, auto docs, and performance become more important, and gRPC can be more useful. You only get a certain number of complexity tokens and IMO it's not worth spending any here.
- ammanley 6y ago
  Seconding this. Our team actually decided to implement gRPC + gRPC-gateway in a green-field project at work, as part of an initiative to "PoC" that it could be done and bring us lots of benefits. We got auto-gen'd docs, clients, all the works that gRPC + protobuf in Golang promises. Between a few breaking upstream changes in grpc-gateway (iirc), learning a new framework vs the stdlib + chi router setup that had already been proven in the greater team, feature shipment dropped through the floor. Couple that with the fact that, as stated, these benefits come with well-iterated data models; conversely, our _not_ well-established data models proved to be a constant PITA when having to regenerate this file and that file. No one used the auto-gen'd docs because they were constantly changing anyway. And integrating with our CI/CD pipeline was a nightmare. Between providing value with a team's previously proven tools and spending innovation tokens, I'd definitely suggest weighing carefully if you need this to be one of them (bi-directional streaming and a well-established project seem like good candidates though).
hardwaresofton 6y ago
It feels like an ugly hack that grpc-gateway sits on top of gRPC to support HTTP/1 when gRPC could have been written to support HTTP/1 and HTTP/2 from the beginning. Roughly, gRPC ~= protobuf (serialization & schema enforcement) + HTTP/2 + non-standard HTTP/2 (trailers, etc), and it is hard to believe that just as they standardized the non-standard bits, they couldn't have found a way to work over HTTP/1 (or HTTP/1.1 at least) with the flexibility available there. gRPC feels to me like a violation/break in expectations of how the layers of abstraction are supposed to work -- gRPC works over HTTP/2, and it's weird that it translates a lower level (HTTP/1) at a higher level (gRPC) to re-encode it into something at the lower level. An appropriately robust protocol would simply support both lower levels of abstraction if it wanted to (and they were widespread in daily use), right? The only places I feel like I see this kind of layer violations are in lower level networking and it generally just makes everything worse and more complicated there. Of course, I know that gRPC + gateway is actually not a huge deal in practice -- you've got reverse proxies like envoy that will do it for you automatically[0], but it just... doesn't sit great. The benefits of gRPC are not to be sneezed at (better performance, strict typing at the protocol level, schema enforcement, bidirectional streaming, etc), but it feels like it could have accomplished a lot of those goals without throwing out HTTP/1.1 completely (and then that machinery could have been reused to support HTTP/3). [0]: https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/grpc_http1_bridge_filter https://www.envoyproxy.io/docs/envoy/latest/configuration/ht...
- anderspitman 6y ago
  IMO your bidi streaming layer should should be entirely transport- and data- agnostic. You can build those in a layer up if you actually need them. See rSocket[0] or my own omnistreams[1]. [0]: https://rsocket.io/ https://rsocket.io/ [1]: https://github.com/omnistreams https://github.com/omnistreams
bincyber 6y ago
Can you expand on how you can register gRPC services to grpc-gateway without needing to run a gRPC server?
- 0xbkt 6y ago
  `protoc-gen-grpc-gateway` generates a few public functions which you can use to bind your service to grpc-gateway. You can, - pass a gRPC client instance, - pass a gRPC connection, - pass a gRPC endpoint in string format, - pass a gRPC service instance implementing the service interface generated by `protoc-gen-go-grpc`. These are meant to fit grpc-gateway to scenarios where gRPC server might be implemented in another programming language, or accepting clients at a TCP port or another kind of a socket, possibly under a different network realm than grpc-gateway. Pretty much like a reverse proxy, but can also be integrated with services in the same code level, escaping the overhead of a transport layer.

by 0xbkt 6y ago

yolo42 6y ago

This is exactly what I started using in my new open source project (not announced yet) and the experience has been amazing. I was wary about it in the beginning but it quickly became clear that this is the easiest and probably the fastest way to get this job done. The community and tooling has come far ahead.

whoisjohnkid 6y ago

Have you looked into goa design[1]? You define your services with a contract via their DSL and it generates all code for you as well; has some nice validation options out of the box. - similar to grpc gateway you just need to conform to the generated code’s service interface. I believe it supports, http, websocket and grpc. [1]https://github.com/goadesign/goa https://github.com/goadesign/goa

anderspitman 6y ago

These things can be fun to play with, but my experience is most projects don't need them. I would rather spend my time implementing functionality than regenerating and recompiling code from proto files. The large projects you mention often have something most projects don't: a well-defined data model and interface that has been heavily iterated and evolved over time. At that point things like backwards compatibility, auto docs, and performance become more important, and gRPC can be more useful. You only get a certain number of complexity tokens and IMO it's not worth spending any here.

ammanley 6y ago
Seconding this. Our team actually decided to implement gRPC + gRPC-gateway in a green-field project at work, as part of an initiative to "PoC" that it could be done and bring us lots of benefits. We got auto-gen'd docs, clients, all the works that gRPC + protobuf in Golang promises. Between a few breaking upstream changes in grpc-gateway (iirc), learning a new framework vs the stdlib + chi router setup that had already been proven in the greater team, feature shipment dropped through the floor. Couple that with the fact that, as stated, these benefits come with well-iterated data models; conversely, our _not_ well-established data models proved to be a constant PITA when having to regenerate this file and that file. No one used the auto-gen'd docs because they were constantly changing anyway. And integrating with our CI/CD pipeline was a nightmare. Between providing value with a team's previously proven tools and spending innovation tokens, I'd definitely suggest weighing carefully if you need this to be one of them (bi-directional streaming and a well-established project seem like good candidates though).

hardwaresofton 6y ago

It feels like an ugly hack that grpc-gateway sits on top of gRPC to support HTTP/1 when gRPC could have been written to support HTTP/1 and HTTP/2 from the beginning. Roughly, gRPC ~= protobuf (serialization & schema enforcement) + HTTP/2 + non-standard HTTP/2 (trailers, etc), and it is hard to believe that just as they standardized the non-standard bits, they couldn't have found a way to work over HTTP/1 (or HTTP/1.1 at least) with the flexibility available there. gRPC feels to me like a violation/break in expectations of how the layers of abstraction are supposed to work -- gRPC works over HTTP/2, and it's weird that it translates a lower level (HTTP/1) at a higher level (gRPC) to re-encode it into something at the lower level. An appropriately robust protocol would simply support both lower levels of abstraction if it wanted to (and they were widespread in daily use), right? The only places I feel like I see this kind of layer violations are in lower level networking and it generally just makes everything worse and more complicated there. Of course, I know that gRPC + gateway is actually not a huge deal in practice -- you've got reverse proxies like envoy that will do it for you automatically[0], but it just... doesn't sit great. The benefits of gRPC are not to be sneezed at (better performance, strict typing at the protocol level, schema enforcement, bidirectional streaming, etc), but it feels like it could have accomplished a lot of those goals without throwing out HTTP/1.1 completely (and then that machinery could have been reused to support HTTP/3). [0]: https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/grpc_http1_bridge_filter https://www.envoyproxy.io/docs/envoy/latest/configuration/ht...

anderspitman 6y ago
IMO your bidi streaming layer should should be entirely transport- and data- agnostic. You can build those in a layer up if you actually need them. See rSocket[0] or my own omnistreams[1]. [0]: https://rsocket.io/ https://rsocket.io/ [1]: https://github.com/omnistreams https://github.com/omnistreams

bincyber 6y ago

Can you expand on how you can register gRPC services to grpc-gateway without needing to run a gRPC server?

0xbkt 6y ago
`protoc-gen-grpc-gateway` generates a few public functions which you can use to bind your service to grpc-gateway. You can, - pass a gRPC client instance, - pass a gRPC connection, - pass a gRPC endpoint in string format, - pass a gRPC service instance implementing the service interface generated by `protoc-gen-go-grpc`. These are meant to fit grpc-gateway to scenarios where gRPC server might be implemented in another programming language, or accepting clients at a TCP port or another kind of a socket, possibly under a different network realm than grpc-gateway. Pretty much like a reverse proxy, but can also be integrated with services in the same code level, escaping the overhead of a transport layer.