6 ms·

I recently chose gRPC as a communication protocol between two devices (sort of IoT). Until now it works perfectly as expected. The C++ code generator provides

I recently chose gRPC as a communication protocol between two devices (sort of IoT). Until now it works perfectly as expected. The C++ code generator provides a clean abstraction plus it saved a lot of time (both in programming and debugging). The gRPC proto file syntax also nudge you in the right direction wrt protocol design. When trying to "sell" gRPC it helps that there are generators for plenty of languages and it's backed by a major company.

gravypod 7y ago
I wish that tooling around generating protoc into stubs and client libraries was simpler. I wish there was a single command I could run and turn large collection of proto files into libraries for "all" languages (python, java, C++, Node package, etc). Unfortunately there's no universal approach to this.
- shereadsthenews 7y ago
  bzl build :my_proto_library ?
  - gravypod 7y ago
    In many conversations I've had with Google (or Google-Adjacent) engineers I've revealed a truth to them that was quite shocking: Bazel isn't the only build system in wide scale deployment currently. It's also far from the most used build system currently. While Bazel is a monolithic tool that solves this problem there is no external tool that solves this problem of configuring the different semantics and configuration for protos in different languages.
    - shereadsthenews 7y ago
      You said you wished for a command and I gave you one.
- jsty 7y ago
  What makes that impossible now? I'm probably overlooking something in your use case, but couldn't you just have a simple build script / makefile that generated the libraries with a different protoc call for each library?
  - gravypod 7y ago
    Nothing makes it impossible but comparing it to things like thrift the complexity becomes apparent: thrift --gen <language> <Thrift filename> This handles the following languages: C (depends on GLib), Cocoa, C++, C#, D, delphi, Erlang, Go, Haskell, Java, Javascript, OCaml, Perl, PHP, Python, Ruby, Smalltalk. From this one command I can instantly integrate this into almost every build system I know of. On the other hand gRPC has the same features but it's a slightly different workflow. For each language you go to the language's code generation page, find the command line option for generating your language's code, read up on some decisions that were made for you, etc. All of that is fine, the part that annoys me a bit is each language needs a language module for the compiler (if it's not one of the core few languages). For example in the documentation for generating Go [1] they have you download and install protoc-gen-go from http://github.com/golang/protobuf http://github.com/golang/protobuf assuming that you already have golang installed. gRPC seems much more focused on the idea that I want to define an API for my code, I want that specification to live inside the project that I am writing, and that you can figure out how to generate stubs on a language by language basis. What I want is something I can write a set of system specification files, type one command and get modules built for all languages. From there I can import those modules using my native language's favorite package manager (npm, composer, Hunter for CMake, etc). Ideally the Protocol Specification, the Library Generation, and the Library Usage are three components that are separate. [1] - https://developers.google.com/protocol-buffers/docs/reference/go-generated https://developers.google.com/protocol-buffers/docs/referenc...
    - jsty 7y ago
      With the significant caveat that I haven't used it much myself (I mainly work with Bazel which obviates the need), I think Uber's prototool [0] can manage most if not all of that. Might be worth giving it a look. [0] https://github.com/uber/prototool https://github.com/uber/prototool
- q3k 7y ago
  This seems like an odd requirement. Are you trying to generate stubs for your API users ahead of time? This will likely not work as generated stubs evolve in lockstep with protoc and runtime support libraries, and thus are not guaranteed to work across discrepant versions. Thus, stub code should be generated alongside the consumer/client. It also likely shouldn't be committed into a VCS.
  - gravypod 7y ago
    It would be done in CI. Generate stubs -> package/compile -> push to internal package repo. This way your protocol for your infrastructure is just another library.
    - q3k 7y ago
      Having an explicit 'create client library by generating/compiling proto stubs' is generally also bad mojo from my experience, unless you're also abstracting API stability and service discovery. If not, it will be unnecessarily painful to make a change to either the service discovery method or a non-backwards-compatible proto change, as you will have to lockstep both the service rollout, the library build and the client bump.

gravypod 7y ago

I wish that tooling around generating protoc into stubs and client libraries was simpler. I wish there was a single command I could run and turn large collection of proto files into libraries for "all" languages (python, java, C++, Node package, etc). Unfortunately there's no universal approach to this.

shereadsthenews 7y ago
bzl build :my_proto_library ?
- gravypod 7y ago
  In many conversations I've had with Google (or Google-Adjacent) engineers I've revealed a truth to them that was quite shocking: Bazel isn't the only build system in wide scale deployment currently. It's also far from the most used build system currently. While Bazel is a monolithic tool that solves this problem there is no external tool that solves this problem of configuring the different semantics and configuration for protos in different languages.
  - shereadsthenews 7y ago
    You said you wished for a command and I gave you one.
jsty 7y ago
What makes that impossible now? I'm probably overlooking something in your use case, but couldn't you just have a simple build script / makefile that generated the libraries with a different protoc call for each library?
- gravypod 7y ago
  Nothing makes it impossible but comparing it to things like thrift the complexity becomes apparent: thrift --gen <language> <Thrift filename> This handles the following languages: C (depends on GLib), Cocoa, C++, C#, D, delphi, Erlang, Go, Haskell, Java, Javascript, OCaml, Perl, PHP, Python, Ruby, Smalltalk. From this one command I can instantly integrate this into almost every build system I know of. On the other hand gRPC has the same features but it's a slightly different workflow. For each language you go to the language's code generation page, find the command line option for generating your language's code, read up on some decisions that were made for you, etc. All of that is fine, the part that annoys me a bit is each language needs a language module for the compiler (if it's not one of the core few languages). For example in the documentation for generating Go [1] they have you download and install protoc-gen-go from http://github.com/golang/protobuf http://github.com/golang/protobuf assuming that you already have golang installed. gRPC seems much more focused on the idea that I want to define an API for my code, I want that specification to live inside the project that I am writing, and that you can figure out how to generate stubs on a language by language basis. What I want is something I can write a set of system specification files, type one command and get modules built for all languages. From there I can import those modules using my native language's favorite package manager (npm, composer, Hunter for CMake, etc). Ideally the Protocol Specification, the Library Generation, and the Library Usage are three components that are separate. [1] - https://developers.google.com/protocol-buffers/docs/reference/go-generated https://developers.google.com/protocol-buffers/docs/referenc...
  - jsty 7y ago
    With the significant caveat that I haven't used it much myself (I mainly work with Bazel which obviates the need), I think Uber's prototool [0] can manage most if not all of that. Might be worth giving it a look. [0] https://github.com/uber/prototool https://github.com/uber/prototool
q3k 7y ago
This seems like an odd requirement. Are you trying to generate stubs for your API users ahead of time? This will likely not work as generated stubs evolve in lockstep with protoc and runtime support libraries, and thus are not guaranteed to work across discrepant versions. Thus, stub code should be generated alongside the consumer/client. It also likely shouldn't be committed into a VCS.
- gravypod 7y ago
  It would be done in CI. Generate stubs -> package/compile -> push to internal package repo. This way your protocol for your infrastructure is just another library.
  - q3k 7y ago
    Having an explicit 'create client library by generating/compiling proto stubs' is generally also bad mojo from my experience, unless you're also abstracting API stability and service discovery. If not, it will be unnecessarily painful to make a change to either the service discovery method or a non-backwards-compatible proto change, as you will have to lockstep both the service rollout, the library build and the client bump.