35 ms·
Go Protobuf: The New Opaque API
- favflam 2y agoOh, this is great. I just did an implementation in gRPC in Go whereby I had to churn through 10MB/s of data. I could not implement any kind of memory pool and thus I had a lot of memory allocation issues which lead to bad memory usage and garbage collection eating up my CPU.
- alecthomas 2y agoThis is probably what you want: https://github.com/planetscale/vtprotobuf https://github.com/planetscale/vtprotobuf
- h4ch1 2y agoSurprisingly I saw this on the front page mere minutes after deciding to use protobufs in my new project. Currently I'm not quite sold on RPC since the performance benefits seem to show up on a much larger scale than what I am aiming for, so I'm using a proto schema to define my types and using protoc codegen to generate only JSON marshaling/unmarshaling + types for my golang backed and typescript frontend, with JSON transferred between the two using REST endpoints. Seems to give me good typesafety along with 0 headache in serializing/deserializing after transport. One thing I also wanted to do was generate SQL schemas from my proto definitions or SQL migrations but haven't found a tool to do so yet, might end up making one. Would love to know if any HN folk have ideas/critique regarding this approach.
- the_gipsy 2y ago> syntax = "proto2" uses explicit presence by default > syntax = "proto3" used implicit presence by default (where cases 2 and 3 cannot be distinguished and are both represented by an empty string), but was later extended to allow opting into explicit presence with the optional keyword > edition = "2023", the successor to both proto2 and proto3, uses explicit presence by default The root of the problem seems to be go's zero-values. It's like putting makeup on a pig, your get rid of null-panics, but the null-ish values are still everywhere, you just have bad data creeping into every last corner of your code. There is no amount of validation that can fix the lack of decoding errors. And it's not runtime errors instead of compile-time errors, which can be kept in check with unit tests to some degree. It's just bad data and defaulting to carry on no matter what, like PHP back in the day.
- delusional 2y agoI don't think the reason for zero values has anything to do with "avoiding null panics". If you want to inline the types, that is avoid using most of your runtime on pointer chasing, you can't universally encode a null value. If I'm unclear, ask yourself: What would a null int look like? If what you wanted was to avoid null-panics, you can define the elementary operations on null. Generally null has always been defined as aggressively erroring, but there's nothing stopping a language definition from defining propagation rules like for float NaN.
- the_gipsy 2y agoSorry, I don't follow you. If you don't have zero values, you either have nulls and panics, or you have some kind of sum-type á la Option<T> and cannot possibly construct null or zero-ish values. Is there a way to have your cake and eat it too, and are there real world examples of it?
- masklinn 2y ago> or you have some kind of sum-type á la Option<T> and cannot possibly construct null or zero-ish values. Option types specifically allow defaulting (to none) even if the wrapped value is not default-able. You can very much construct null or zero-ish values in such a langage, but it’s not universal, types have to be opted into this capability.
- the_gipsy 2y agoExactly my point, you have to opt-in, and in practice you only do precisely where it's actually necessary. Which is completely different than "every single type can be a [null | zero value]". You cannot possibly construct some type A (that is not Option<A> or A@nullable or whatever) without populating it correctly. Of course you need some way to represent "absence of a value", the matter is how: simple but incorrect, or complex but correct. And, simple/complex here can mean both the language (so performance tradeoff), and (initial) programmer ergonomics. That's why I ask if you can have your cake and eat it too, the answer is no. Or you'll get sick sooner than later, in this case.
- remram 2y ago> version: 2, 3, 2023 (released in 2024) I call this Battlefield versioning, after the Battlefield video game series [1]. I bet the next version will be proto V. [1]: in order: 1942, 2, 2142, 3, 4, 1, V, 2042
- deleted 2y ago[deleted]
- kubb 2y agoI hate this API and Go's handling of protocol buffers in general. Especially preparing test data for it makes for some of the most cumbersome and unwieldy files that you will ever come across. Combined with table driven testing you have thousands upon thousands of lines of data with an unbelievably long identifiers that can't be inferred (e.g. in array literals) that is usually copy pasted around and slightly changed. Updating and understanding all of that is a nightmare and if you miss a coma or a brace somewhere, the compiler isn't smart enough to point you to where so you get lines upon lines of syntax errors. But, being opaque has some advantages for sure.
- alienchow 2y agoThe testing practice I've seen is to have a testdata/ directory with a bunch of textprotos for different test cases. If you're using Bazel, just include the entire directory glob as data dependency for the unit tests. The test tables are essentially just appropriately named textproto filenames that are unmarshaled into the proto message to be tested. Then again I've also seen people do these thousand line in-code literal string protos which really grind my gears.
- atombender 2y agoThe generated Go code situation has always been wild to me. For example, every message embeds protoimpl.MessageState and a bunch of other types, which contain mutexes. That means proto structs cannot be copied or compared byte-for-byte like normal Go structs can. For several years I used the GoGo Protobuf SDK. It was vastly superior to the awful Javaesque Go code that the official compiler generated. It allowed structs to be pure data structs, was much more performant, and supported a bunch of options to generate native-feeling, ergonomic Go code. But the Google team refused to partake in any such improvements, and GoGo was shut down as the burden of following the upstream implementation became too big. [1] I'm not an expert, but as far as I understand, the extra struct junk is mostly to avoid having a parallel set of types for metadata (including reflection). It's unclear to me why these can't simply be generated as internal types with some nice API on top. Clearly the new field metadata adds to this extra information, and the Go team is moving in the opposite direction of what I thought the future was — they're doubling down on stuffing metadata into the structs, and making the structs bigger and even less wieldy. I understand how this might make things more performant, but I was hoping this sort of thing could be solved with the type system, especially now that we have generics. For example, surely lazy field access could be done like this: type Info struct { User LazyProto[User] } userName := user.Get().Name [1] https://x.com/awalterschulze/status/1584553056100057088 https://x.com/awalterschulze/status/1584553056100057088
- alakra 2y agoIs this like the FlatBuffers "zero-copy" deserialization?
- strawhatguy 2y agoGreat, now there's an API per struct/message to learn and communicate throughout the codebase, with all the getters and setters. A given struct is probably faster for protobuf parsing in the new layout, but the complexity of the code probably increases, and I can see this complexity easily negating these gains.
- hellcow 2y agoI'd recommend transforming protobuf types to domain types at your API boundary. Then you have domain types through the whole application.
- mxey 2y agoAt which point I loose all the benefits of lazy decoding that the accessor methods can provide, so I could just decode directly into a sensible struct, except you can’t with Protobuf.
- mort96 2y agoAccessor methods aren't for lazy decoding but for more efficient memory layouts.
- aktau 2y agoBoth, actually. Without accessor methods, laziness couldn't be implemented.
- mxey 2y agoBut that will also not transfer over to the domain struct
- mort96 2y agoWell it depends. If your data model doesn't include "this bool is optional", you can just include the bool directly in the struct and get all the memory layout advantages, and then you decide in your protobuf -> domain type conversion code whether it's an error if that field is missing or if it just defaults to 'false'. You only need to make ways for a field to be optional (such as naming it a pointer where nil represents "missing") when that actually makes sense in your data model.
- jeffbee 2y agoProtobuf 3 was bending over backwards to try to make the Go API make sense, but in the process it screwed up the API for C++, with many compromises. Then they changed course and made presence explicit again in proto 3.1. Now they are saying Go gets a C++-like API. What I'd like is to rewind the time machine and undo all the path-dependent brain damage.
- ein0p 2y agoNice to see my comments on their proto3 design doc vindicated, lol. There were a lot of comments on that doc, far more than what you'd usually see. Some of those comments dealt with the misguided decision to basically drop nullability (that is, the `has_` methods) that proto2 had. The team then just deleted all the comments and disabled commenting on the doc and proceeded with their original design much to the consternation of their primary stakeholders.
- deleted 2y ago[deleted]
- sa46 2y agoWhen I was at Google around 2016, there was a significant push to convince folks that the proto3 implicit presence was superior to explicit presence. Is there a design doc with the rationale for switching back to explicit presence for Edition 2023? The closest docs I've found are https://buf.build/blog/protobuf-editions-are-here https://buf.build/blog/protobuf-editions-are-here and https://github.com/protocolbuffers/protobuf/tree/main/docs/design/editions https://github.com/protocolbuffers/protobuf/tree/main/docs/d....
- jeffbee 2y agoI was only there for the debate you mentioned and not there for the reversal, so I dunno.
- IX-103 2y agoI wasn't there for the debate, but was there for the reversal. I don't remember there being anything explicitly said about it. The only thing I can think of is that I know of some important projects that couldn't migrate to proto3 because of this implicit field issue. So some people were still writing new code with proto2.
- kyrra 2y agoThe opaque API brings some niceties that other languages have, specifically about initialization. The Java impl for protobuf will never generate a NullPointerException, as calling `get` on a field would just return the default instance of that field. The Go OpenAPI did not do this. For many primative types, it was fine. But for protobuf maps, you had to check if the map had been initialized yet in Go code before accessing it. Meaning, with the Opaque API, you can start just adding items to a proto map in Go code without thinking about initialization. (as the Opaque impl will init the map for you). This is honestly something I wish Go itself would do. Allowing for nil maps in Go is such a footgun.
- the_gipsy 2y ago> The Java impl for protobuf will never generate a NullPointerException, as calling `get` on a field would just return the default instance of that field. This is NOT the solution lmao
- usrnm 2y agoIt's so fun to watch go devs rediscover all the patterns that they so happily threw out in the beginning. It's like watching a person grow up from a sunny little kid to a mature disgruntled alcoholic.
- rad_gruchalski 2y agoAn alternative explanation is that non-go people got their hands on go and complain that go is not x or y. Like with generics. Now they’re in go. They’re not great, they have some sense but they may as well not exist as far as I’m concerned. I find them pretty useless anyway without lower and upper type bounds.
- mxey 2y agoWhich non-Go people brought generics into Go?
- rad_gruchalski 2y ago
- dpeckett 2y agoTo be honest I kind of find myself drifting away from gRPC/protobuf in my recent projects. I love the idea of an IDL for describing APIs and a great compiler/codegen (protoc) but there's just soo many idiosyncrasies baked into gRPC at this point that it often doesn't feel worth it IMO. Been increasingly using LSP style JSON-RPC 2.0, sure it's got it's quirks and is far from the most wire/marshaling efficient approach but JSON codecs are ubiquitous and JSON-RPC is trivial to implement. In-fact I recently even wrote a stack allocated, server implementation for microcontrollers in Rust https://github.com/OpenPSG/embedded-jsonrpc https://github.com/OpenPSG/embedded-jsonrpc. Varlink (https://varlink.org/ https://varlink.org/) is another interesting approach, there's reasons why they didn't implement the full JSON-RPC spec but their IDL is pretty interesting.
- mirekrusin 2y agoAlso json parsers are crazy fast nowadays, most people don't realize how fast they are.
- Cthulhu_ 2y agoWhile true, it's still a text and usually http/tcp based format; data -> json representation -> compression? -> http -> tcp -> decompression -> parsing -> data. Translating to / from a text just feels inefficient.
- mirekrusin 2y agoWith projects I work on it's over websockets, js/ts has builtin support, easy to log, debug, extend/work with etc. Binary protocols have exactly same steps. Redis is also using text based protocol and people don't seem to be bothered too much about it.
- malkia 2y agoApart from being text format, I'm not sure how well JSON-RPC handles doubles vs long integers and other types, where protobuf can be directed to handle them appropriately. That is a problem in JSON itself, so you may neeed to encode some numbers using... "string"
- tonymet 2y agowhy is code generation under-utilized? protobufs and other go tooling are great for code generation. Yet in practice i see few teams using it at scale. Lots of teams creating rest / json APIs, but very few who use code generation to provide compile-time protection.
- kevmo314 2y agoCode generation leaves a layer of abstraction between the API and the actual implementation which works great if that code generation is bug-free but if it's not, you're like... totally fucked. Most commonly people say you can read the generated code and step backwards but that's like saying you can read the compiled JavaScript and it's basically open source. That layer of abstraction is an underrated mental barrier. Of course, code generation is still practical and I'm a lot more likely to trust a third-party writing a code generator like protobufs, OpenAPI specs, etc, but I would not trust an internal team to do so without a very good reason. I've worked on a few projects that lost hundreds of dev hours trying to maintain their code generator to avoid a tiny bit of copy/paste.
- kccqzy 2y agoCode generation is under utilized because most people don't have a build system good enough for it. Traditional make is fine: you just define dependencies and rules. But a lot of people want to use language-specific build systems and these often don't have good support for code generation and dependency tracking for generated code. Yet another subtlety is that when cross-compiling, you need to build the code generation tool for the local target always even though the main target could be a foreign architecture. And because the code generation tool and the main code could share dependencies, these dependencies need to be built twice for different targets. That again is something many build tools don't support.
- lakomen 2y agoGraphql won the race for me. Grpc is no longer relevant. Too many hurdles, no proper to and from Web support. You have to use some 3rd party non free service.
- deleted 2y ago[deleted]
- nicce 2y agoAren’t their usecases completely different?
- pensatoio 2y agoYes. The use cases are very different, as far as these things go. To say otherwise is borderline misinformation. You can build services internally with gRPC and serve a public graphQL API that aggregates them.
- lakomen 2y agoNot at all.
- _cenw 2y agoIntersects quite heavily if you're defining a schema for your API
- lmm 2y ago
- g0ld3nrati0 2y agojust curious, why do use protobuf instead of flatbuffers?
- tonyhart7 2y agoYeah idk why we didnt just send binary data representation therefore eliminate entire serialize and deserialize part
- tsimionescu 2y agoWhich binary data representation? If I'm sending a Java object, do you think a C program will be able to just use it? Or for that matter, do you think two different C++ implementations, maybe on different platforms, will use the same binary representation of a class object?
- tonyhart7 2y agojust need an standard for that
- tsimionescu 2y agoJava has a standard, each C implementation has a standard, Python has a standard, etc. The problem is that each of these standards is different, and impossible to modify. So, we need something that can serialize one standard to a wire format, and then deserialize from that wire format to another standard. Oh wait...
- tonyhart7 2y agoso we cant create a standard that can eliminate serde part????
- tsimionescu 2y agoI'm assuming you may be trolling me, but no, that can't realistically be done.
- cyberax 2y agoThanks. I hate it. Now you can not use normal Go struct initialization and you'll have to write reams of Set calls.
- xyse53 2y agoIt's not in the post but when this was rolled out internally at Google there was a corresponding builder struct to initialize from.
- oefrha 2y agoLike the sibling said, there's a complimentary _builder struct generated with a Build() method. For instance, for the sample message in the blog post, here's the public API of the generated _builder: type LogEntry_builder struct { BackendServer *string RequestSize *uint32 IpAddress *string // contains filtered or unexported fields } func (b0 LogEntry_builder) Build() *LogEntry
- cyberax 2y agoSo they managed to screw up even that. The naming system is not idiomatic Go. You still will need to create temporary objects (performance...) and for an unclear gain.
- oefrha 2y ago> The naming system is not idiomatic Go. Underscores are commonly used in names in generated code to avoid conflicts (this applies to all sorts of codegen, not just protobuf). You can easily have both Foo and FooBuilder messages in your protobuf. See also generated enum consts since day one of protobuf-gen-go.
- cyberax 2y agoIf a generated name clashes, then you just override the generated name. And in this case, perhaps, the resistance and the ugliness of the solution should have made the Protobuf authors to pause for a bit and rethink it. Several years ago, I used gogoprotobuf instead of regular Go protobuf. It uses normal by-value structures instead of pointers everywhere, with generated code for s11n instead of reflection. It worked about 10x faster that the regular protobuf.
- cyberax 2y agoBTW, if you care so much about performance, then fix the freaking array representation. It should be simple `[]SomeStruct` instead of `[]*SomeStruct`. This one small change can result in an order of magnitude improvement.
- aktau 2y agoIt's true that this would perform better, and greatly reduce allocations. But: - Messages (especially opaque ones) are not supposed to be copied. The recommendation is to use `m := &mypb.Message{}`. - This would make migrating to use the opaque API more difficult, if the getters don't return the same type as the old open API fields, much more code needs to be rewritten, or some wrapper that allocates a new slice on every get. - Users expect that `subm := m.GetSubMessages()[2] ; m.SetSubMessages(append(m.GetSubMessages(), anotherm))) ; subm.SetInt(42) ; assert(subm.GetInt() == m.GetSubMessages()[2].GetInt())`. This would not be the case if the API returned a slice of values. - ... Effectively, a slice of pointers is baked into the API, and the way people use protocol buffers in Go. For these reasons, it's not clear to me this would end up performing better or causing less work. If we had returned an iterator (new in Go 1.23) instead of an actual slice, then it would've been possible to vary the underlying representation (slice-of-pointers, slice-of-value-chunks, ...). But there are other downsides to that too: - Allocations when passing iterators to functions that expect a slice. - Extra API surface for modifying the list (append, getn, len, ...). Not that clear of a win either. Another thing that could be considered is: when decoding, allocate a slice of values ([]mypb.Message), *and* a slice with pointers (or do it lazily): []*mypb.Message. Then initialize: for i := range valuel { ptrl[i] = &valuel[i] // TODO: verify that this escape doesn't cause disjoint allocations. } That might be beneficial due to grouping allocations, and the user would be none the wiser.
- cyberax 2y ago> - Messages (especially opaque ones) are not supposed to be copied. So? > - This would make migrating to use the opaque API more difficult The opaque API is stupid to begin with. Now the objects are no longer threadsafe. You can't just read a message in one thread and process it in two different threads. > Users expect Then don't expect this. If you're breaking the API, then at least break it in a way that makes it better afterwards.
- neonsunset 2y agoThe absolute state of Go dragging down the entire gRPC stack with it. Oh well, at least we have quite a few competent replacements nowadays.
- aktau 2y agoCan you be specific? I'm curious.
- neonsunset 2y agoOf course not because you wouldn't listen :)
- aktau 2y agoI would.
- parhamn 2y agoIt's interesting, to everyone but but the mega shops like Google, protobuf is a schema declaration tool. To the megashops its a performance tool. For most of my projects, I use a web-framework I built on protobuf over the years but slowly got rid of a lot of the protobufy bits (besides the type + method declarations) and just switched to JSON as the wire format. http2, trailing headers, gigantic multi-MB files of getters, setters and embedded binary representations of the schemas, weird import behaviors, no wire error types, etc were too annoying. Almost every project I've tracked that tries to solve the declarative schema problem seems to slowly die. Its a tough problem an opinionated one (what to do with enums? sum types? defaults? etc). Anyone know of any good ones that are chugging along? OpenAPI is too resty and JSONSchema doesn't seem to care about RPC.
- danans 2y ago> It's interesting, to everyone but but the mega shops like Google, protobuf is a schema declaration tool There are lots of other benefits for non performance-oriented teams and projects: the codegen makes it language independent and it's pretty handy that you can share a single data model across all layers of your system. If you don't care about the wire format, the standard JSON representation makes it pair well with JSON native databases, so you can get strict schema management without the need need for any clunky ORM.
- Cthulhu_ 2y agoThat's assuming JSON native databases are fine for your use case though, but in practice it's only good for storing documents that don't need to be edited/queried much by the backend storing them.
- danans 2y ago> in practice it's only good for storing documents that don't need to be edited/queried much by the backend storing them Why aren't they good for that? They can have very high write throughput, they don't require ORMs, and they can be indexed and queried using standard database methods like the SQL language. You can even enforce strict schemas on them if you want to, just as you would with an RDBMS.
- tuetuopay 2y agoI can’t wait to try this new Protobuf Enterprise Edition, with its sea of getters and setters ad nauseam. /s However I can get behind it for the lazy decoding which seems nice, though I doubt its actual usefulness for serious software (tm). As someone else already mentioned, an actual serious api (tm) will have business-scope types to uncouple the api definition from the implementation. And that’s how you keep sane as soon as you have to support multiple versions of the api. Also, a lot of the benefits mentioned for footgun reductions smell like workarounds for the language shortcomings. Memory address comparisons, accidental pointer sharing and mutability, enums, optional handling, etc are already solved problems and where something like rust shines. (Disclaimer: I run several grpc apis written in rust in prod)
- matrix87 2y agoI recently used code-gen'd protobuf deser objects as the value type for an in-memory db and was considering flattening them into a more memory-efficient representation and using bitfields. That was for java though, not sure if they are doing the same thing there Glad to see this change, for that use case it would've been perfect
- abtinf 2y agoThis looks like an attempt to turn Go into Java/C#. I certainly won’t allow this to be used by the engineering teams under me.
- cpuguy83 2y agoIt's attempt to provide a much more efficient and harder to misuse implementation to a project used in tons of places.
- Zababa 2y agoI don't think it is. Effective Go says that Go doesn't provide automatic support for getters and setters but there's nothing wrong with providing them yourself. Since in that case they are actually doing something (checking/updating the bitfield that contains the presence of each field), it makes sense to use them. They are called `GetFoo()` instead of the idiomatic `Foo()`, but that is to ensure compatibility with the API where the fields are directly exposed as `Foo`, which also makes sense.
- pensatoio 2y agoWhy? I'm going to encourage my engineers and other teams to use it. Using this API would 100% have prevented bugs created by accessing the generated structs directly, especially in the presence of an optional value.
- Naru41 2y agoWhy not just use a naive struct from the beginning? memcpy is the fastest way to get serialize into a form that we can use in actual running program.
- schmichael 2y agoThe article goes into great detail about the benefits of an opaque api vs open structs. Somewhat unintuitively open structs are not necessarily the “fastest” largely due to pointers requiring heap allocations. Opaque APIs can also be “faster” due to lazy loading and avoiding memcpy altogether. The latter appears in libraries like flat buffers but not here IIRC.
- akira2501 2y ago> memcpy is the fastest way To bake endianess and alignment requirements into your protocol.