10 ms·
A new ProtoBuf generator for Go
- PostThisTooFast 5y agoIs there one for Kotlin yet? It's pretty pathetic that Google's own protocol lacks native support for its most popular operating system.
- hn_go_brrrrr 5y agoYes: https://developers.google.com/protocol-buffers/docs/kotlintutorial https://developers.google.com/protocol-buffers/docs/kotlintu...
- HNLogInsSuckAss 5y agoThose must've been released only in the last couple of years. In 2019 there was still no Kotlin generator. The OP shouldn't have been modded down, because that is indeed pathetic. https://medium.com/digitalfrontiers/a-dance-with-protocols-kotlin-spring-and-protocol-buffers-in-action-ded306546070 https://medium.com/digitalfrontiers/a-dance-with-protocols-k...
- HNLogInsSuckAss 5y agoYes, I was surprised by this. We ended up using the Java ones only two years ago because of the lack of a Kotlin generator. Google had a blog post talking about what a struggle it was for some reason, but meanwhile someone had already created a decent Swift generator.
- PostThisTooFast 5y agoWhere are the other fucking comments? Like the ones pointing out that Kotlin support was absent as late as 2019? Eat shit, Hacker News.
- jzelinskie 5y agoI hadn't realized that Gogo was in such a bad spot with the upstream Go protobuf changes. There was lots of drama when the changes were made and I guess that overshadowed any optics I had on Gogo. Making vtprotobuf an additional protoc plugin seems like the Right Thing™, although it's a shame how complicated protoc commands end up becoming for mature projects. I'm pretty tempted to port Authzed over to this and run some benchmarks -- our entire service requires e2e latency under 20ms, so every little bit counts. The biggest performance win is likely just having an unintrusive interface for pooling allocated protos.
- jeffbee 5y agoProto message unmarshal in Go for a small message should be 5 orders of magnitude below 20ms, shouldn't even begin to matter until you are sweating individual microseconds.
- morelisp 5y agoUntil the GC kicks in and steals a full 200usec + a bunch of your throughput... (Holy shit, who is downvoting this? It's literally the whole article!)
- harikb 5y agoProperly written Go code (or even Java for that matter) will try to minimize allocations. For Java, unless I am mistaken pause-less GC is only offered by Azul - $$
- morelisp 5y agoYeah, the whole point of the article is that gRPC v2 (and frankly v1 for that matter) are not “properly written” to do this.
- RhodesianHunter 5y ago>or even Java Just in case you may be unaware, the latest GCs for Java (Shenandoah, ZGC) are miles ahead of anything available for Go due to sheer age and manpower. Parallel and Pauseless are easily achievable in most cases.
- shoefindortz 5y ago> Arenas are, however, unfeasible to implement in Go because it is a garbage collected language. If you are willing to use cgo, google already implemented one for gapid. https://github.com/google/gapid/tree/master/core/memory/arena https://github.com/google/gapid/tree/master/core/memory/aren...
- pjmlp 5y agoNot only that, there are other garbage collected languages like D, Nim and C# that offer the language features to do arenas without having to touch any C code. There is still so much education to do.
- p_l 5y agoAren't arenas old news in GC languages in general? Most of the time, their non-presence is due to general pools being just as good most of the time, or people simply not needing them that much with modern GC
- pjmlp 5y agoYes, so I really did not got how come such assertion was made. Probably lack of experience with machine friendly code.
- dimitrios1 5y agoI can't believe we've managed to have this lengthy of a discussion about GC languages and speed without anyone mentioning rust. Has HN turned a corner?
- shoefindortz 5y agoRust has an arena allocator too[1], but it is implemented with 165(!!!) usages of unsafe. :) [1] https://github.com/fitzgen/bumpalo https://github.com/fitzgen/bumpalo
- 5y ago
- jen20 5y agoI'm not sure that the phrasing in the article is particularly fair: > The maintainers of Gogo, understandably, were not up to the gigantic task. I'm 99% sure they are "up to" (as in "capable of") doing so, they are just not "up for" it (as in, "will not do it").
- Zababa 5y agoThey could be "not up to" because of lack of resources, probably time and/or money. I think that's what is implied, rather than lack of technical knowledge.
- jahewson 5y agoYes I assume the author meant “not up for”
- lux 5y agoI got the sense that they meant "not willing" but I agree that's one of those English phrases that can easily be misconstrued towards the more negative interpretation. That said, I love the detailed post and the interesting solution, and the commitment to performance!
- jupp0r 5y agoUsing CPU utilization as a performance metric can be extremely misleading. My favorite article on the subject is from Brendan Gregg: http://www.brendangregg.com/blog/2017-05-09/cpu-utilization-is-wrong.html http://www.brendangregg.com/blog/2017-05-09/cpu-utilization-... A much better way to test the influence of the new compiler would be to test the actual throughput at which saturation is achieved (which is what the benchmark in the C++ grpc library measure to assess their performance).
- et1337 5y agoIn this case the regression also caused a 3% decrease in throughput.
- dkhenry 5y agoThere is a fairly robust set of benchmarks that are run to test out performance improvements[1] and macro benchmarks are the ultimate test of holistic improvement. CPU isn't a great proxy, but one of the biggest problems in real world performance on this specific system ( databases in general ) is latency. CPU time is a really good proxy for latency so by taking a look at CPU time we can get an idea of how the system will respond under "normal" conditions. 1.https://benchmark.vitess.io/macrobench https://benchmark.vitess.io/macrobench
- n0x1m 5y agothe biggest current problem with Go and ProtoBuf is swagger support when using it for API returns. Enums are not supported for example. The leniency of protojson can't be used in other languages that built on top of the swagger docs.
- flakiness 5y agoI wonder what Google is thinking about the v2 performance. It's well known that protobuf processing is taxing heavy on their data center [1]. It's hard to imagine they just leave it slow. Or do they? [1] https://research.google/pubs/pub44271/ https://research.google/pubs/pub44271/
- justicezyx 5y agoThere was a project to develop a asic (probably bundled inside NIC) to do protobuf parsing. At some point Sanjay did a change to proto API that rendered that project less appealing. Disclaimer: Google had a lot of internal stuff they considered important to their core tech competencies. For example, no open source about Google paxos APIs and infrastructure, networking, etc.
- gilgad13 5y agoMaybe I'm missing something, but my read of golang/protobuf#364[1] was that part of the motivation for the re-organization in protobuf-go v2 was to allow for optimizations like gogoprotobuf to be developed without requiring a complete fork. I totally understand that the authors of gogoprotobuf do not have the time to re-architect their library to use these hooks, but best I can figure this generator does not use these hooks either. Instead it defines additional member functions, and wrappers that look for those specialized functions and fallback to the generic ones if not found. For example, it looks like pooled decoders could be implemented by setting a custom unmarshaller through the ProtoMethods[2] API. I wonder why not? Did the authors of the vtprotobuf extension not want to bite off that much work? Is the new API not sufficient to do what they want (thus failing some of the goals expressed in golang/protobuf#364? [1]: https://github.com/golang/protobuf/issues/364 https://github.com/golang/protobuf/issues/364 [2]: https://pkg.go.dev/google.golang.org/protobuf@v1.26.0/reflect/protoreflect#Message https://pkg.go.dev/google.golang.org/protobuf@v1.26.0/reflec...
- alecthomas 5y agoI haven't looked in more detail, but one blocker is that `ProtoMethods() *methods` returns a private type, making it effectively unimplementable outside this package.
- zeeboo 5y agoSo, I thought this at one point, too. But it turns out that methods is a type alias to an unnamed type, so there's no package level privacy issues: https://github.com/protocolbuffers/protobuf-go/blob/v1.26.0/reflect/protoreflect/methods.go#L18 https://github.com/protocolbuffers/protobuf-go/blob/v1.26.0/...
- deleted 5y ago[deleted]
- alecthomas 5y agoOh huh, interesting, I've never seen that done before. I'm struggling to understand what the rationale _for_ doing it is though. Maybe it's to avoid an import cycle?
- sa46 5y agoFunny timing, I've just written most of a TypeScript generator for protobufs. I learned about some fun corners of protobufs I didn't expect trying to pass the protouf conformance tests [1] (which this one passes, that's no mean feat!). - If you write the same message multiple times, protobuf implementations should merge fields with a last write wins policy (repeated fields are concatenated). This includes messages in oneofs. - For a boolean array, you're better off using a packed, repeated int64 (if wire size matters a lot). Protobuf bools use varint encoding meaning you need at least 2 bytes for every boolean, 1+ for the tag and type and 1 byte for the 0 or 1 value. With a repeated int64, you'd encode the tag and length in 2 varints, and then you get 64 bools per 8 bytes. - Fun trivia: Varints take up a max of 10 bytes but could be implemented in 9 bytes. You get 7 bits per varint byte, so 9 bytes gets you 63 bits. Then you could use the most significant bit of the last byte to indicate if the last bit is 0 or 1. Learned by reading the Go varint implementation [2]. - Messages can be recursive. This is easy if you represent messages as pointers since you can use nil. It's a fair bit harder if you want to always use a value object for each nested message since you need to break cycles by marking fields as `T | undefined` to avoid blowing the stack. Figuring out the minimal number of fields to break cycles is an NP hard problem called the minimum feedback arc set[3]. - If you're writing a protobuf implementation, the conformance tests are a really nice way to check that you've done a good job. Be wary of implementations that don't implement the conformance tests. [1]: https://github.com/protocolbuffers/protobuf/tree/master/conformance https://github.com/protocolbuffers/protobuf/tree/master/conf... [2]: https://github.com/golang/go/blob/master/src/encoding/binary/varint.go#L18 https://github.com/golang/go/blob/master/src/encoding/binary... [3]: https://en.wikipedia.org/wiki/Feedback_arc_set#Minimum_feedback_arc_set https://en.wikipedia.org/wiki/Feedback_arc_set#Minimum_feedb...
- nly 5y agoThe varint format also isnt as dense on average as it could be and allows for non-canonical encodings. I.e. you can encode any integer in multiple ways (up to 9 or 10 bytes) The solution for this is to subtract 1 from the integer every time you encode a byte (since the existence of the next byte you're adding already indicates that the intermediate value isn't 0)