11 ms·
A new Go API for Protocol Buffers
- zdw 7y agoInteresting that even Google decided to do complex versioning gymnastics and introduce a new namespace in order to avoid how `go mod` adds the version as a path component.
- neild 7y agoNah, we just really wanted to stop using an import path tied to a specific hosting provider. Once we decided to change to an entirely different path, we waffled a lot over whether to tag it v1 or v2. There were arguments for and against either, so we eventually picked one and went with it.
- jzwinck 7y agoSure, but why version the new package as v1.20 instead of v2.x?
- farslan 7y agoBecause they would had to add a `/v2` suffix to the import path and seems like they didn't wanted to add that. The question is, why is this not released as v1.0.0 then? Still trying to understand the reasoning for this.
- jzwinck 7y agoOh I didn't realize that was mandatory. I thought omitting v2 from the path was sort of OK until you explained this. Now it just seems wrong.
- neild 7y agoReasoning went something like this: We could tag the new API v2: + Makes the "v1" and "v2" distinction very clear in the import path. - Confusing: google.golang.org/protobuf@v1 doesn't exist, but v2 does. - In ten years, hopefully nobody cares about the old github.com/golang/protobuf and the confusion is gone. We could tag the new API v1: - Less visually distinct in the import path. + Seems to make sense for the first version of google.golang.org/protobuf to be a v1. + If we decide it was a terrible idea, it's easier to go from v1 to v2 than to roll back from v2 to v1. We waffled back and forth for quite a while on this, and eventually decided that the first version of google.golang.org/protobuf would be a v1. Then as we got closer to release (but with a certain amount of usage of v0 in the wild), we decided not to second-guess that decision but to start with a version that wouldn't overlap with any version of github.com/golang/protobuf to avoid confusion when someone reports a bug in "v1.0.1". Maybe it was the wrong choice. If it was the worst choice we've made in the new API, I'll be happy!
- farslan 7y agoThanks for the all the work. And I hope my rant doesn't come something as a bad. But I truly believe the versioning along with the import path is confusing. I hope the team goes over the versioning again and come up with something that makes more sense.
- llimllib 7y agoWhen I read the article, my brain passed right over the different domain names; the two are even the exact same number of characters, set in a monospace font, and my brain passes right over the domain of golang imports to the end of the path. That's where the interesting bit is (usually!) I find this a super confusing and unintuitive choice, but I suppose it's too late for changing it now. Sincerely, an othwerwise mostly happy protobuf user
- rverghese 7y agoWhat are you going to do for APIv3? Version it as google.golang.org@v2? google.golang.org@v3 (but @v2 doesn't exist)? Release an @v2 which is identical to @v1?
- neild 7y agoWe used up our breaking change budget for the next decade with this release, so we'll think about it in 2030.
- deleted 7y ago[deleted]
- loopz 7y agoAdmit it, once Windows 22 is out the door, protobuf 1.22 is shippin' as well! Yieehaaw! Rust .22 with breaking changes, will be breaking 4 years after that as well!
- bouk 7y agoYou could give google.golang.org/protobuf@v1 an //importpath to the github package, so go would warn people about it.
- ifoundthetao 7y agoThat makes the most sense, and throws a great seasoning of context on that decision. Thanks for providing that here.
- endorphone 7y ago"we just really wanted to stop using an import path tied to a specific hosting provider" e.g. github is now owned by Microsoft and it's a little awkward. If Google won out on their attempt to buy Github, they wouldn't wouldn't be talking about a "specific hosting provider". Though it's a little weird that people take this at face value, when the majority of the userbase was questioning this since the first release.
- paulddraper 7y agoSo both old and new are available through the new domain?
- twblalock 7y agoThe whole idea of tying image names to hosting provider urls or other kinds of locators was probably a bad idea. What if those things move to different domains, or different directory structures? Wouldn't that break every Go program that depends on them? I've been using Java, Python, and Ruby for a while. They all have various pain points for managing dependencies but they do get one thing right: packages have names and versions, names are opaque strings that are not coupled to a particular hosting provider, and they don't care what directory on my laptop I use to store my dependencies and my code. The Go dependency model seems like a real step backwards in usability and maintenance. It's like they are forcing one software org's standards on the rest of us. We don't all work at Google! Just let us depend on things that have names and version numbers!
- zerr 7y ago> we just really wanted to stop using an import path tied to a specific hosting provider Are there any plans to fix this for the whole language, for others as well? i.e. use namespaces, module names in the source but move the actual github.com, google.com or geocities.com URLs outside.
- yencabulator 7y agoThe support has been there for a long time. `go help importpath`
- dgellow 7y ago> The github.com/golang/protobuf module is APIv1. > The google.golang.org/protobuf module is APIv2. We have taken advantage of the need to change the import path to switch to one that is not tied to a specific hosting provider. That's such a weird choice, and will be quite confusing. Now every time you see protobuf being imported, you have to be sure to pay extra attention to the domain used, and correctly remember which one is v1 or v2. If only go modules established a clear way to differentiate API versions using the import path!
- kardianos 7y agoThe goal of issuing a major version is to create a separate package namespace so it can be imported concurrently to other major versions. By treating major versions as different packages, it creates efficiencies in VCS history and common namespace. However, the goal is to create a new package name that can be concurrently imported with previous versions. Everything else is just method. Don't confuse the goal with the method.
- cle 7y agoPeople care about versions. If it’s harder or more confusing to understand what version you’re on, that’s more than “just method”. Otherwise we’d just use UUIDs for package names and call it a day.
- mirekrusin 7y agoJulia uses uuids masked with name.
- icholy 7y ago> If only go modules established a clear way to differentiate API versions using the import path! It did, they just chose not to follow it :/ I wonder what rsc thinks about this decisions
- Insanity 7y agoRsc?
- cube2222 7y agoCongrats on the big release! One thing which saddens me a lot though, is returning types you have to type assert because they had to avoid an import cycle. I don’t know if it could be avoided (probably not) but it’s still unergonomic. > (Why the type assertion? Since the generated descriptorpb package depends on protoreflect, the protoreflect package can't return the concrete options type without causing an import cycle.)
- sigzero 7y agoI thought the general consensus on protocol buffers was "Meh"?
- untog 7y agoYou thought wrong?
- sigzero 7y agoSeems so.
- capableweb 7y agoI'm wondering if the whole idea behind "general consensus" really work on technical problems. There are so many factors involved with the choices engineers make that you can't say that general consensus says it's "meh" as it really depends on your use case. No tool will ever fit all general problems, so will always be "meh" for someone.
- myko 7y agoI think for large organizations they probably make sense. They're potentially a lot less data per request than JSON. For the average app? Overkill, sure.
- xmprt 7y agoI also recently learned about grpcurl that makes using protobufs a lot easier. Still not as easy as json but I think protobufs are the future.
- SlowRobotAhead 7y agoCBOR. Binary JSON with some extra features. I like PB, but it was too hard in my application to sync all of the sides with the same proto as it was developing rapidly. For me schemaless was better, and now we have extremely easy JSON conversion... but did have to write the parser myself. I personally can’t agree on PB being “the future”.
- chairmanwow1 7y agoThis article doesn't do a great job explaining exactly what new features are a part of this release. Seems to me like it's finally enabling some of the useful type iteration stuff that is possible in other versions of protobufs.
- atombender 7y agoHopefully this is good news. The old Protobuf package had ossified, with lots of ergonomic issues that the maintainers seemed uninterested in addressing. One perennial annoyance is how awkwardly some Protobuf stuff ends up being represented in Go. For example, "oneof" types: message Event { oneof payload { Create create = 1; Delete delete = 2; } } message Create { string id = 1; } message Delete { string version = 1; } You get these messy types (I've elided most of the yucky marshaling-related stuff): type Event struct { Payload isEvent_Payload `protobuf_oneof:"item"` } type isEvent_Payload interface { isEvent_Payload() MarshalTo([]byte) (int, error) Size() int } type Event_Create struct { Create *Create `protobuf:"bytes,1,opt,name=create,proto3,oneof"` } type Event_Delete struct { Delete *Delete `protobuf:"bytes,1,opt,name=delete,proto3,oneof"` } There are multiple problems here. One: Why the extra struct? For example, to create a single event, you have to do: Event{ Payload: &Event_Create{ Create: &Create{ID: "123"}, }, }, ...instead of just: Event{ Payload: &Create{ID: "123"}, }, Secondly, isEvent_Payload is private! So given a Create or an Delete, you can't store it in a single variable: func nextEvent() isEvent_Payload { // <-- not possible if something { return &Create{...} } else { return &Delete{...} } } The reason for this appears to be so that the generated serialization can be very dumb and not rely on type switches. But I don't buy that this is how it has to be. There are other issues. Overall, the whole package seems designed from the bottom up for machines (mumbles possibly also by machines), not humans.The upshot is that using Protobuf types as "first-class" types — meaning the types you actually use internally in the meat of your app, as opposed to in the controller glue that lives in your API and mediates between the API and the internals — feels super messy. As an aside, anyone know what issues this paragraph refers to? The google.golang.org/protobuf/encoding/protojson package converts protocol buffer messages to and from JSON using the canonical JSON mapping, and fixes a number of issues with the old jsonpb package that were difficult to change without causing problems for existing users. Did they finally fix the zero value problem with jsonpb.go [1]? [1] https://github.com/gogo/protobuf/issues/218 https://github.com/gogo/protobuf/issues/218
- dana321 7y agoIts horrible, i know. Try mixing that with grpc too and it gets even worse.
- AaronFriel 7y agoNot to be too Rust evangelist here, but why the emphasis on reflection instead of further support for generics? On the Rust side, macros can[1] transform .proto files into data structure definitions at compile time, and instances of the structs passed around have the benefit of strong typing, code completion in editors, lints and tests can easily validate properties of protobuf objects and so on. And all without adding more codegen steps which is nice - the macros can replace a build system having to do codegen as a separate step. How often do you need to mutate any protocol buffer and remove (redact as they say) fields generating a completely invalid protocol buffer object? Will anything downstream match the ".proto" of an object that's missing a bunch of required fields? Wouldn't it be easier to just all of the sensitive data into an optional "sensitive_data" field that's defined by another .proto, and strip that? [1] - https://github.com/danburkert/prost#generated-code-example https://github.com/danburkert/prost#generated-code-example
- tantalor 7y agoAny talk of required fields is a non-starter. Just forget about it. Don't use them. Whether a field is "sensitive" may change after the field is defined and used all over the place, so you can't just move it without changing all the code. But you can easily add a option (like [non_sensitive=true]) to any field so your redaction code picks it up.
- uluyol 7y agoCompile time generic programming is not a replacement for runtime reflection. A great example of this is if you want to use dynamically generated protocol buffers. A great usecase for this is if you want to store each row in a database as a proto message. Instantiate a descriptor based on the table schema and go.
- wyager 7y ago> Compile time generic programming is not a replacement for runtime reflection. Why not? > A great example of this is if you want to use dynamically generated protocol buffers. When would you want to do that? If your data isn't statically typed, isn't it more appropriate to use a type designed for dynamic shapes (e.g. a list or hashmap)? > A great usecase for this is if you want to store each row in a database as a proto message. Wouldn't a reasonable approach to this be to turn each row into a list (of variants) and serialize that?
- guessmyname 7y ago> The github.com/golang/protobuf module is APIv1. > The google.golang.org/protobuf module is APIv2. > We have taken advantage of the need to change the import path to switch to one that is not tied to a specific hosting provider. > (Why start at version v1.20.0? To provide clarity. We do not anticipate APIv1 to ever reach v1.20.0, so the version number alone should be enough to unambiguously differentiate between APIv1 and APIv2.) Stuff like this is what makes Go confusing to newcomers. For some time I couldn’t understand why people were complaining so much about —the now deprecated— GOPATH environment variable, and later the confusion of how to use all the different and incompatible dependency managers. Then, suddenly last year, it hits me… Go is confusing to people who haven’t followed the language from the beginning. I started programming in Go in 2013 so for me it was very easy to adapt to every change in the language and the community. I can only imagine all the ambiguous stuff people have to understand when learning the language in 2019-2020.
- EdwardDiego 7y agoI think the bit that got me around GOPATH was last time I'd written Go, that was the only way. Then I fired up Go again, v1.12 and decided to take advantage of a Go module for a new project, which I naturally put beside my existing Go code. And then I got to learn about the settings for GO111MODULE. Having the build tools a core part of your language itself is both a strength and a weakness for Go, I think.
- heydabop 7y ago> the now deprecated— GOPATH environment variable I've been writing Go since 2015ish and this is news to me...
- breakingcups 7y agoI loved Go's simplicity when it came out. However, it has become clear to me that it was a little too simple to handle all the new requirements and additions that came after v1. That's how you get weird comments-as-code (// +build linux,386 darwin,!cgo), strings-as-attributes (tags), tacked-on modules versioning / vendoring, non-typed generics( interface{} everywhere), somewhat-typed generics (go generate), etc. Go was not quite ready to handle all these directions people wanted the language to stretch to and as a result it's now already full of annoying warts. I'll still pick Go if I need a simple service with amazing performance. Funnily enough though, I have no need for any of the (non-performance) additions that came after 1.6.
- 7532yahoogmail 7y agoUmm ... When will pb allow me to decode a field in buffer without having to decode the entire object in the buffer? I'm talking a field in a pb message NOT a byte stream I hand encoded without reference to a message eg more like flat buffers.
- justlexi93 7y agoIt is odd that the Go team is all in on semantic import paths, but basically everyone else is like “I will blow up the moon if it means I don’t have to use v2 in my import path.
- mfcl 7y agov2 is used so that the import path is different. The idea is that by breaking the API, you are basically making a new module instead of improving one (metaphorically of course), and it doesn't make sense for the same module to have the same import path. If you want to change the import path and also release a new version, then simply changing the import path is enough. No need to put v2. See it as them releasing a totally new package that just happens to have the same name, if that helps.
- zadokshi 7y agoI wonder if we will start seeing people add v1.example.com, v2.example.com etc... to their git servers to avoid having to change the path's in all of their private projects.
- Gibbon1 7y agoSeems like you need something like a DNS server for repo's.
- mastrsushi 7y agoHackernews, where the highlight of mentioning common libraries is the fact that they're done so in Go and Rust.
- sethammons 7y agoI find that to be a feature. If I open the article and/or comments and find out this was about a library for Ruby or Java then I wasted my time because I don't use those. Likewise, a non-Go or non-Rust dev can ignore Go and/or Rust tagged library articles.
- eeZah7Ux 7y agoIf only we could filter out stuff like this using some tags...
- cbolton 7y agoDoes someone know if this new version uses fast generated code for (de)serialization? The previous version was pretty slow (because based on reflection I think) so many people used https://github.com/gogo/protobuf https://github.com/gogo/protobuf for good performance.
- DelightOne 7y agoHmm. I think it would be quite simple to build an authentication proxy/firewall based on this without having to know the exact underlying message. Maybe tagging fields with auth_userid and/or auth_token.