4 ms·

Not to be too Rust evangelist here, but why the emphasis on reflection instead of further support for generics? On the Rust side, macros can[1] transform .prot

Not to be too Rust evangelist here, but why the emphasis on reflection instead of further support for generics? On the Rust side, macros can[1] transform .proto files into data structure definitions at compile time, and instances of the structs passed around have the benefit of strong typing, code completion in editors, lints and tests can easily validate properties of protobuf objects and so on. And all without adding more codegen steps which is nice - the macros can replace a build system having to do codegen as a separate step. How often do you need to mutate any protocol buffer and remove (redact as they say) fields generating a completely invalid protocol buffer object? Will anything downstream match the ".proto" of an object that's missing a bunch of required fields? Wouldn't it be easier to just all of the sensitive data into an optional "sensitive_data" field that's defined by another .proto, and strip that? [1] - https://github.com/danburkert/prost#generated-code-example https://github.com/danburkert/prost#generated-code-example

tantalor 7y ago
Any talk of required fields is a non-starter. Just forget about it. Don't use them. Whether a field is "sensitive" may change after the field is defined and used all over the place, so you can't just move it without changing all the code. But you can easily add a option (like [non_sensitive=true]) to any field so your redaction code picks it up.
uluyol 7y ago
Compile time generic programming is not a replacement for runtime reflection. A great example of this is if you want to use dynamically generated protocol buffers. A great usecase for this is if you want to store each row in a database as a proto message. Instantiate a descriptor based on the table schema and go.
- wyager 7y ago
  > Compile time generic programming is not a replacement for runtime reflection. Why not? > A great example of this is if you want to use dynamically generated protocol buffers. When would you want to do that? If your data isn't statically typed, isn't it more appropriate to use a type designed for dynamic shapes (e.g. a list or hashmap)? > A great usecase for this is if you want to store each row in a database as a proto message. Wouldn't a reasonable approach to this be to turn each row into a list (of variants) and serialize that?
- atilaneves 7y ago
  > Compile time generic programming is not a replacement for runtime reflection. Yes, it is, especially since the former can be used to implement the latter.
kccqzy 7y ago
That's not very scalable. Putting more logic in a macro means the compiler needs to reprocess it each time the file is touched, even when the part that's changed is not the protos. A separate code gen step means the proto compiler doesn't need to be invoked as often, and the generated files don't need to be recompiled either. > Will anything downstream match the ".proto" of an object that's missing a bunch of required fields? You may have missed it, but in proto3, required fields have been removed. All fields are optional now. https://stackoverflow.com/a/31814967/1102119 https://stackoverflow.com/a/31814967/1102119 TL;DR: it's a code evolution problem.
- AaronFriel 7y ago
  That doesn't sound right, I don't think there's any reason a proc macro couldn't cache the .proto file's generated code and use the cache between runs.
stubish 7y ago
A Go API for protobufs can't support generics because Go doesn't support generics, yet. I guess the protobufs team didn't want to wait until Go v2, probably a good call given Go v1.x is going to be around for years. In Go, a code generator converts the .proto files into Go code, which give the same benefits you see with the Rust macros. Its arguable if a code generation step is better or worse than macros. The reflection is needed if you have received a protobuf the code don't have a definition for. You can now implement something like a gRPC proxy or gRPC load balancer in Go, without needing to compile it with code from the specific .proto files. You also appear to be able to access annotations on the message definitions, which are not embedded in the generated Go structs. Rust may well have similar features in its API. Java certainly does. A gRPC proxy is a use case redact sensitive data, for when you use it to create an audit log of the messages in the requests and responses.
- jsnell 7y ago
  If you don't have the proto definition, there's nothing you can do except pass the object through unmodified. And you should not need a reflection API for that (unless the v1 API was totally messed up in other ways). Without the definition (and knowing the type!) there won't be anything around to tell the reflection API what the names, types and annotations are. All you would have is field numbers mapped to opaque blobs of data.
  - uluyol 7y ago
    You can pass around a type descriptor (itself a proto message) along with the opaque message.
- twblalock 7y ago
  > Its arguable if a code generation step is better or worse than macros. I'd rather just have generics. Code generation for protos is nice for several reasons, but using it as an alternative to generics is not nice at all. I work on a number of projects with generics in internal classes that are not exposed as an API, and using code generation just to get those to work seems like a very big hammer for a small nail.
- 7y ago
7532yahoogmail 7y ago
But the protoc compiler already makes data structure definitions in the target, client language. So what's the point of a parallel macro system? Either it's a subset of capability of what protoc does or equal. Former questionable; latter redundant.

tantalor 7y ago

Any talk of required fields is a non-starter. Just forget about it. Don't use them. Whether a field is "sensitive" may change after the field is defined and used all over the place, so you can't just move it without changing all the code. But you can easily add a option (like [non_sensitive=true]) to any field so your redaction code picks it up.

uluyol 7y ago

Compile time generic programming is not a replacement for runtime reflection. A great example of this is if you want to use dynamically generated protocol buffers. A great usecase for this is if you want to store each row in a database as a proto message. Instantiate a descriptor based on the table schema and go.

wyager 7y ago
> Compile time generic programming is not a replacement for runtime reflection. Why not? > A great example of this is if you want to use dynamically generated protocol buffers. When would you want to do that? If your data isn't statically typed, isn't it more appropriate to use a type designed for dynamic shapes (e.g. a list or hashmap)? > A great usecase for this is if you want to store each row in a database as a proto message. Wouldn't a reasonable approach to this be to turn each row into a list (of variants) and serialize that?
atilaneves 7y ago
> Compile time generic programming is not a replacement for runtime reflection. Yes, it is, especially since the former can be used to implement the latter.

kccqzy 7y ago

That's not very scalable. Putting more logic in a macro means the compiler needs to reprocess it each time the file is touched, even when the part that's changed is not the protos. A separate code gen step means the proto compiler doesn't need to be invoked as often, and the generated files don't need to be recompiled either. > Will anything downstream match the ".proto" of an object that's missing a bunch of required fields? You may have missed it, but in proto3, required fields have been removed. All fields are optional now. https://stackoverflow.com/a/31814967/1102119 https://stackoverflow.com/a/31814967/1102119 TL;DR: it's a code evolution problem.

AaronFriel 7y ago
That doesn't sound right, I don't think there's any reason a proc macro couldn't cache the .proto file's generated code and use the cache between runs.

stubish 7y ago

A Go API for protobufs can't support generics because Go doesn't support generics, yet. I guess the protobufs team didn't want to wait until Go v2, probably a good call given Go v1.x is going to be around for years. In Go, a code generator converts the .proto files into Go code, which give the same benefits you see with the Rust macros. Its arguable if a code generation step is better or worse than macros. The reflection is needed if you have received a protobuf the code don't have a definition for. You can now implement something like a gRPC proxy or gRPC load balancer in Go, without needing to compile it with code from the specific .proto files. You also appear to be able to access annotations on the message definitions, which are not embedded in the generated Go structs. Rust may well have similar features in its API. Java certainly does. A gRPC proxy is a use case redact sensitive data, for when you use it to create an audit log of the messages in the requests and responses.

jsnell 7y ago
If you don't have the proto definition, there's nothing you can do except pass the object through unmodified. And you should not need a reflection API for that (unless the v1 API was totally messed up in other ways). Without the definition (and knowing the type!) there won't be anything around to tell the reflection API what the names, types and annotations are. All you would have is field numbers mapped to opaque blobs of data.
- uluyol 7y ago
  You can pass around a type descriptor (itself a proto message) along with the opaque message.
twblalock 7y ago
> Its arguable if a code generation step is better or worse than macros. I'd rather just have generics. Code generation for protos is nice for several reasons, but using it as an alternative to generics is not nice at all. I work on a number of projects with generics in internal classes that are not exposed as an API, and using code generation just to get those to work seems like a very big hammer for a small nail.
7y ago

7532yahoogmail 7y ago

But the protoc compiler already makes data structure definitions in the target, client language. So what's the point of a parallel macro system? Either it's a subset of capability of what protoc does or equal. Former questionable; latter redundant.