4 ms·

I find it interesting that the folks running away screaming from protobuf are using it in conjunction with gRPC. Is the problem really with the wire format or i

• jimbokun 2y ago
  Yes, I wonder if anyone uses Protobuf encoded payloads over plain old HTTP REST calls.
  • matzf 2y ago
    The Connect RPC protocol is pretty much that: https://connectrpc.com/docs/protocol https://connectrpc.com/docs/protocol
  • timvdalen 2y ago
    We use it in a client-facing application to keep state of a complex configuration, primarily as a means of having a URL-safe way to encode that configuration. It works great, very happy with it.
  • vouwfietsman 2y ago
    We do flatbuffers (super similar) over websockets/http rest. Works beautifully. gRPC is the culprit here.
    • tonyarkles 2y ago
      Yeah we stream a bunch of telemetry and GPS/attitude data in protobufs over a websocket and it works beautifully.
  • rochacon 2y ago
    I've used Twitch's Twirp before and it does that. It is a great middleground between gRPC and plain-HTTP services. :) https://github.com/twitchtv/twirp https://github.com/twitchtv/twirp
    • dalyons 2y ago
      Yes! its amazing, it really should be the default people reach for. Vast majority of people dont need or want the complexity of grpc, they just want protos.
  • worldsayshi 2y ago
    Performance and the promise of seamless type safety?
  • pjc50 2y ago
    The one really annoying decision in GRPC's design is tying it to HTTP/2 without an official choice. It optimizes for complicated high-throughput and bidirectional cases, while making it slightly harder to use for simple one-at-a-time "client server unary" RPC calls.
  • dilyevsky 2y ago
    Technically Kubernetes does: https://kubernetes.io/docs/reference/using-api/api-concepts/#alternate-representations-of-resources https://kubernetes.io/docs/reference/using-api/api-concepts/...
  • fl0ki 2y ago
    > plain old HTTP REST calls Please don't add to the confusion around the term REST. These days most people just mean they use the GET/POST/PUT/DELETE verbs specifically, which is just using the HTTP protocol itself, no REST about it. https://htmx.org/essays/how-did-rest-come-to-mean-the-opposite-of-rest/ https://htmx.org/essays/how-did-rest-come-to-mean-the-opposi...
  • neonsunset 2y ago
    I have experience doing so with .NET simply because writing RPC / Contract definition in a .proto file once and then having gRPC tooling generate all boilerplate associated with it is much better than dealing with existing OpenAPI generators - this one slots right into .csproj where you just add a package reference and a reference to .proto file and you are ready to go.
  • dalyons 2y ago
    yes! Twirp - last two companies ive worked at have used it to great success. Protos over plain ol http, without all the weird bespoke network stuff of gRPC. We just use regular dns, loadbalancers, etc etc. It should be way more popular IMHO.
  • secondcoming 2y ago
    Google’s AdX was entirely protobuf, but now they offer json too.
  • burutthrow1234 2y ago
    Yeah protobuf is a good IDL and encoding. Unfortunately gRPC makes some choices that make sense for internal RPCs in a large engineering org, but it's not good for external clients IMO.
  • sbrother 2y ago
    I've done this in a lot of projects -- it works great. Protobuf is nice in a lot of ways and pretty simple, while gRPC is overkill (imo) for a simple web server that doesn't see tons of traffic.
• pjc50 2y ago
  We've been using GRPC to talk between a Java UI and a C# backend for several years now. Apart from some upgrade issues tying it more heavily to ASP.NET, and some around connection management, it's been completely fine. Mind you, I can see why people used to weakly typed languages would prefer to just slam everything into JSON.
• seiferteric 2y ago
  Generally agree. I know of at least two protocols using protobuf payloads over raw UDP and seems pretty good that way (dnstap and some networking asic's in-band telemetry). My biggest gripe was protobuf v3 changing things like not including default values and not being able detect field presence which I found very annoying.
  • robert-brown 2y ago
    Fortunately, you can continue to use proto2, if you decide it's superior, or use a more recent version of proto3 that supports field presence.
• fl0ki 2y ago
  Context: I started with proto2 in C++ internal to Google and still use proto3 from some Go and Rust projects over a decade later. I can't say the wire format has ever been a problem for me directly. Newer formats have reduced some CPU overheads, but haven't pulled it all together the way official protobuf and gRPC ecosystems have. From what I've seen the biggest problem with the wire format is that the framing for a nested message requires a varint size. You don't know how many bytes to set aside for that integer until you know how many bytes the nested message will serialize to, and this applies recursively. Without a hacky size cache [1], you get exponential runtime. Even BSON did better here; its nested document framing is fixed-size so it can always go back and patch it later with just an external size stack, no need for an intrusive size cache. There are still benefits to the wire format, especially over JSON. For example, you get real 64-bit unsigned integers, and you can disable varint encoding for them (fixed64). It gives you a lot of opportunities for both accuracy and efficiency. The bad news is that normal idiomatic use of protobuf and gRPC infect your code. It's designed for the code to be generated in very predictable standard ways, but those aren't necessarily the ways you actually want. Even if you decide to isolate proto to a corner of your project and use your own model the rest of the time, the transformation between proto types and your own types can cost you more memory allocations and copy more memory instead of sharing existing memory. So if you care about performance, you often have to design a whole project around protobuf end-to-end, infecting your code even more than usual. With JSON in either Go or Rust, you can make your own custom types that serialize to JSON and these types instantly feel and work first-class. You own how the schema is mapped to code; often the only living schema is defined in code anyway. In most cases you can use your own types throughout your project and serialize them to JSON as needed without JSON itself infecting your code. This helps even more if you also involve formats like BSON because they can all coexist just fine for the same types, unlike protobuf which insists on its own types and generated code. Even if you fully embrace protobuf throughout your project, there are other problems and limitations with the generated code. For example, in official protobuf for Go, there's no way to avoid heap allocating each individual nested or repeated message, and there's no way to avoid an absurd 5-machine-word overhead in every single message. (Hopefully will be brought down to 3 soon, but it's been 5 for years). If you're designing a proto schema around these problems, you can seriously compromise readability and maintainability just to work around poor implementation decisions in Go protobuf itself. I'm guilty of that kind of optimization, but when my team saw the benchmarks numbers they agreed it was worth it. This is not the kind of decision you want to have to make in a technical project, but I emphasize that it can still be the right choice in many circumstances. The prost crate for Rust is not official but already gives you more control over your schema. You can technically use your own Rust code instead of the generated code, though I don't see anyone actually do this and it doesn't seem to be encouraged. In any case, my biggest issue with prost is that it makes it difficult (or perhaps currently impossible) to share sub-messages with Arc [2], which on the other hand is trivial with Go just using *. While prost avoids allocations in more cases than Go protobuf, my experience has been that it avoids copies less, and some of those copies require allocations anyway. I'm encouraged that Google's upcoming Rust library seems to be modeled after the C++ one and not the Go one. I haven't seen the latest work on it but I trust it's in a good place given how many collective decades of experience in protobuf implementation are going into it. In summary, for a project that was explicitly designed for efficiency, in practice it can limit your code in ways that hurt efficiency more than they ever helped. And while generating code for many languages is a handy feature, that generated code is unlikely to be what you want, and the more you embrace proto throughout your project the more places you pay efficiency and maintainability tolls. [1] https://github.com/protocolbuffers/protobuf-go/blob/1d4293e052f6cb6f021eafba93c865de80274f5f/internal/impl/encode.go#L51-L58 https://github.com/protocolbuffers/protobuf-go/blob/1d4293e0... [2] Not within one message to be sent to a client, because that information would be redundant. More for sharing some nested messages across bigger messages sent to multiple clients.
• deleted 2y ago
  [deleted]
• Ferret7446 2y ago
  Protobuf has a JSON serialization format. You can use it to help define and validate JSON schemas, which is quite nice. And of course, clients don't need protobuf bindings to read the resulting JSON objects (although you lose the automatic type checking/field unpacking, etc).