17 ms·

Don't Use Protobuf for Telemetry

223 points by simonpure 6y ago

dariosalvi78 6y ago
then what should we use? MQTT?
- as-j 6y ago
  MQTT doesn’t specify the payload format. You can easily have a topic with the data in protobuf and another with json.
- toast0 6y ago
  The post seems more like an argument against protobuf in general than against its use in telemetry. The main complaints seem to be that the library is big and that nested serialization is tricky because of design decisions. In regards to binary size, using whatever serializing you're already doing for your main data seems like a win for telemetry. Including telemetry with your main data has pluses and minuses, but using the same serialization is generally fine, unless your serialization can fail, in which case it's hard to report on how many times telemetry serialization fails, because you might not be able to serialize it.
ketamine__ 6y ago
Isn't this just a micro optimization? At what level does one need to care about this?
- deleted 6y ago
  [deleted]
- cpascal 6y ago
  It really depends on your scale and how many messages you are processing per second. For a lot of applications, you’re absolutely correct, but if you’re scale is sufficient, a “micro” optimization like this is actually a “macro” optimization. Also the author of this article works at DataDog and I suspect the number of messages they process each second falls under the sufficient category. For example, suppose you are processing 1M messages per second and you can shave 1 byte off the message size, that shaves off 1MB/sec of data that needs to be processed. If you’re paying for network bandwidth or storing the messages, that saves you something like 2.6TB of data each month. 2.6TB/month is not likely to be a huge deal when it comes to cost savings, but if you keep scaling the messages/sec or the bytes/msg you can start to get some significant savings. Now I used message size as an example, and the article focuses on processing time not message size, but the point still stands. When you can make a micro optimization for something that is done a very large number of times, there are not-insignificant gains to be had.
  - nijave 6y ago
    I think this was for client/application instrumentation where they want to try to have little-to-no overhead for including the datadog agent that ships telemetry back to Datadog's service
  - Thaxll 6y ago
    If you manage scale in the milion messages/sec you really don't care about some TB/month. Most compagnies are still using REST/json in places where a binary protocol "would be better".
- rbanffy 6y ago
  I recommend HTTPS because it's easy to simulate and well understood. Also I like human-readable payloads. Unless you need ridiculously low latencies and ridiculously high volume, you shouldn't worry at all.
xyzzyz 6y ago
His point on protobuf-java library adding nontrivial bloat to his Java app is definitely valid. However, his other argument about protobuf wire format being inefficient is hard to square with decades of practical experience Google had with protobufs, which are used for literally everything, including telemetry, and high throughput, low latency applications. Sure, having to recursively precompute lengths before serialization is a bit of a hassle, but I wouldn’t call it expensive.
- sillysaurusx 6y ago
  Signal boosting this. I am the last person to be swayed by “use our magical design that solves all problems,” but in the case of protobufs, it solves most of them. I didn’t like it until I was forced to use it. Now I’m not sure I’ll ever go back. There’s a handy code snippet to turn any protobuf message into JSON: from google.protobuf.json_format import MessageToJson import json from pprint import pprint as pp pp(json.loads(MessageToJson(msg))) Examples of turning arbitrary tensorflow objects into JSON: https://twitter.com/theshawwn/status/1331625739993575427?s=21 https://twitter.com/theshawwn/status/1331625739993575427?s=2...
  - intricatedetail 6y ago
    Why they use Java like code in Python? "MessageToJson" doesn't look idiomatic.
    - singron 6y ago
      CapWords function names matches the C++ style guide at Google, not java. A lot of python uses autogenerated bindings to C++ code and reuses the same naming conventions. Even pure python code would use that style although their public styleguide doesn't mention it anymore.
    - ipsum2 6y ago
      The official protobuf library for Python is not pythonic at all.
    - 6y ago
fwsgonzo 6y ago
We already have FlatBuffers, which allow you to send structs on the wire without reinventing anything. It sounds like it's a good match here. Protobuf is better if you care about the byte count on the wire and less about the CPU time to pack and unpack it.
- simonh 6y ago
  I'd not come across FlatBuffers before, but they do seem interesting for some use cases, especially when you're highly performance sensitive. https://codeburst.io/json-vs-protocol-buffers-vs-flatbuffers-a4247f8bda6f https://codeburst.io/json-vs-protocol-buffers-vs-flatbuffers...
  - twic 6y ago
    AFAIK, the three main options in this subspace are FlatBuffers, Cap'n Proto, and SBE. Here's a comparison from a Rust perspective: https://speice.io/2019/09/binary-format-shootout.html https://speice.io/2019/09/binary-format-shootout.html And from the Cap'n Proto author: https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-sbe.html https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-...
- tyingq 6y ago
  It also appears to be 236kb and 23 classes for the java api, versus 1.6MB and 700 classes for protobufs.
gregmac 6y ago
> Protobuf-java is a little heavy [...] Just depending on the library adds 1.6MB and nearly 700 classes before you even generate your own message classes. By comparison, protobuf-net [1] is about 260KB and 68 classes. Python's [2] is a 1MB package download (with source). Why's the Java one so big? [1] https://github.com/protobuf-net/protobuf-net https://github.com/protobuf-net/protobuf-net [2] https://pypi.org/project/protobuf https://pypi.org/project/protobuf
- extropy 6y ago
  Or just use the lite version? It removes the heavy features like descriptors and reflection. More details: https://developers.google.com/protocol-buffers/docs/reference/java-generated#message https://developers.google.com/protocol-buffers/docs/referenc...
janders 6y ago
I want to be a fan of Apache Avro (https://avro.apache.org/ https://avro.apache.org/) so much for situations such as this. And while I like Avro as a standard, most of the implementations I have found (specifically in C/C++) are... lacking. I feel Avro would be a great fit for something like this close to zero overhead (assuming a pre-shared schema) but there is little to no support for pre-shared schema and the RPC part of the standard is not a great fit for telemetry. Maybe one day I'll make an Avro library I like, or contribute to an extant one.
- tele_ski 6y ago
  I think AVRO is fairly well designed overall, but JSON schemas? If you need deeply nested objects just look away, it becomes unintelligible when trying to understand the schema with any real nesting. The company I was at used it fairly well as long as you keep things 1 or 2 levels nested at most, but we had a legacy schema that was 5 or 6 levels deep at some points and it was just a disaster. My anecdote anyways.
  - janders 6y ago
    Oh yeah. Schema nesting is a problem. I think a lot of it would be solved by having a schema allow a "types" field which just has a list your named types that can be referenced later in the list and in the actual schema. But yeah, totally concur on that pain point.
beervirus 6y ago
Don’t Use ~~Protobuf for~~ Telemetry
- lstodd 6y ago
  This. Might as well use xmlrpc/SOAP for it and it will be only slightly less ugly.
Cloudef 6y ago
Google's own protobuf runtime is so bloated that even google themselves don't use it in many of their software, but rather nanopb[1], which is done by completely unrelated person to google. 1: https://jpa.kapsi.fi/nanopb/ https://jpa.kapsi.fi/nanopb/
- jeffbee 6y ago
  I don’t think that’s true in google3. People often mistake the scale of Google’s internal code with respect to their visible external code.
  - jmkjaer 6y ago
    For anyone unfamiliar, Google3 is apparently Google's monorepo, according to this HN comment: https://news.ycombinator.com/item?id=17123620 https://news.ycombinator.com/item?id=17123620
    - alexhutcheson 6y ago
      https://research.google/pubs/pub45424/ https://research.google/pubs/pub45424/
- vlovich123 6y ago
  I am only aware of nanopb being used at Google in embedded scenarios. The primary reason for the project I worked on was that the first Pixel Buds didn't have a C++ compiler (at least not a functional one - based on GCC 2 or 3 if I recall correctly & an extremely crippled implementation). Pixel Buds 2 didn't have that limitation (stock GCC 9) but nanopb was kept for the second Pixel Buds because the memory management was saner for an embedded RTOS environment (vs stock upstream that might use STL or otherwise try to allocate memory). I don't think runtime bloat was really on our radar (it could have been the next critical path, but wasn't on our radar at the time). Nanopb itself needed some changes to the codegen for the first Pixel Buds (which I investigated & fixed) due to architectural peculiarities of the CSR8675. It generated a lot of constants but the CSR8675 had limited space there & was made worse because every 8 bits took up 1 16-bit word (e.g. the size of your string literal or any constant byte array is effectively doubled). So I changed the code gen to put those constants in the code section instead. Before this change resolved the issue more fully, random engineers who were unlucky enough to add a constant in some way were hitting limits internally with regularity (as the project was heading toward ship) & working around it by removing characters from log messages (e.g. "Some long message" => "Sme lng msg"). For what it's worth I tried to contribute any meaningful improvements we made back to nanopb but this one didn't make that cut as it was specific to a chip no one in the wider community would be using anyway (8675 is super old). I don't exclude the possibility of it using in more places, but I'd say the average Google engineer does not encounter nanopb in their daily development.
CamouflagedKiwi 6y ago
Is this a thing people care about? From what I've seen of server-side Java applications (we have several at work) 1.6MB and 700 classes are lost in the noise of the endless list of Maven dependencies. I'm sure you can do it a lot more efficiently without using someone else's thing, but what exactly is being optimised for here?
- jeffbee 6y ago
  That was kinda my response as well. If telemetry altogether is using more than 1% of your cpu time you have an architectural problem perhaps.
  - ashtonkem 6y ago
    My experience points to the cost of storing metrics far outweighing the cost of emitting metrics in all but the most extreme of circumstances.
    - RhodesianHunter 6y ago
      Yes, but in this case the author is an employee of DataDog, presumably working on their agents or clients and so is probably acutely focused on not impacting the performance of their customer's systems.
- Yeroc 6y ago
  I thought the author addressed this pretty clearly in the article. For his specific use-case he was delivering a JVM agent.
Aaargh20318 6y ago
Are there any advantages of protobuf over using DER encoded ASN.1 ? The format looks very similar. It looks like they kind of reinvented the wheel here.
- Nvorzula 6y ago
  To my understanding they're both [type, length, content] encoded formats, so they're similar there at least. In my experience people really quite enjoy the IDL aspect of protobuf as the killer feature, and I am not sure any such compiler exists for ASN1.
  - Aaargh20318 6y ago
    ASN.1 is basically an IDL. There are multiple encodings for ASN.1 but DER is one of the most common ones (you can also encode to JSON or XML). DER is indeed TLV (tag-length-value) and there are many code generators available.
  - mianos 6y ago
    ASN.1 is specified as a compilable language. Open source compilers, such as snacc, have existed for 30 years. ASN.1 also defines multiple encodings, DER is used in crypto as you can compare encoded blocks bit by bit. Other encoding rules, such as BER, are tagged. Protobuffers is the same thing re-invented by people who probably never saw ASN.1.
- kevin_thibedeau 6y ago
  ASN.1 has a lot of cruft, particularly with the string encodings.
JoachimSchipper 6y ago
Protobuf's variable-length lengths are indeed annoying, but can't you encode back-to-front as in ASN.1? That lets you write your lengths just after encoding the value, which should work (but may leave your data in e.g. bytes 20+ of a buffer; depending on interfaces, you may need to do one final copy to move the data to the start of the buffer.) (ASN.1 and protobuf are both tag-length-value formats with variable-length integer encodings... h/t to tptacek for mentioning that particular trick.)
- tptacek 6y ago
  Hey, all I did was notice that Microsoft did that for X.509. h/t to Microsoft's cert parsing code team. :)
lxe 6y ago
Ay my previous job we started out using simple StatsD messages that are a few bytes long, easy to implement, easy to debug, and quick. But then possibly due to a full-steam-ahead mandate of "everything should use protobuf (or was it thrift)?", we switched to a much heavier format, where clients needed a codegen'd binary implementation of the protocol, while making messages themselves much heavier.
tomc1985 6y ago
700 classes? To define and implement a wire format? Does it include a kitchen sink too?
tijsvd 6y ago
I have written an in-house implementation of protobuf for C++ (sorry can't share) and studied the wire format extensively. Google's implementations, at least C++ and Java, are a bunch of bloated crap (or maybe they're very good, but for a use case that I haven't yet encountered). Don't shoot down the format because of a specific implementation, find or write a better one and enjoy the fact that every language has at least some implementation available. Then, the author laments the wire format itself for having varint-encoded length prefixes, which can not be fixed up. That is true, but it's not that much of a problem. Most straightforward is to simply go through nested data multiple times, once to calculate the length, and again for the actual encoding (and then again and again for deeper nesting). What makes this bearable is the fact that data will be mostly loaded into L2 cache (L1 for smaller messages) on the first pass, which makes the next pass much faster. The story breaks down for large, deeply nested messages, but then, the topic here is telemetry which I would expect to consist of a stream of small, shallow messages.
- thisiscorrect 6y ago
  Mind elaborating on how the "standard" C++ and Java protobuf implementations are bloated? [edit] I'm genuinely asking. I'm guessing you mean in the generated APIs, i.e. the code "weight", but maybe you meant something else?
  - gabereiser 6y ago
    Probably a complaint of the codegen and it’s wacky name mangling.
  - notretarded 6y ago
    Probably all this "security" nonsense. What could possibly go wrong with implementing a binary serialisation protocol your self...
    - inetknght 6y ago
      Oh look sarcasm! On Hacker News!
Luker88 6y ago
Does anyone have an up-to date comparison protobuf vs captnproto instead? ( https://capnproto.org/ https://capnproto.org/ ) I am starting a new project and it looked really interesting, but I still have to read comments from people who used both in detail
- kjgkjhfkjf 6y ago
  CP was written by Kenton Varda, who spent many years working on protobuf at Google. One massive advantage of protobuf for mainstream Google languages (C++, Java) is that Google has used them extremely heavily for many years, and you can trust that they've been extensively battle tested in Google's enormous high-traffic services, and scrutinized by Google's vast army of engineers. Kenton's experience notwithstanding, the same cannot be said of CP. This argument doesn't apply for languages other than the ones that Google uses for production services.
  - alisonkisk 6y ago
    Part of the reason that Kenton left Google and created CapnProto, was because he was the only person working on proto at Google trying to fix the problems Google found.
    - kentonv 6y ago
      That's not really accurate... I transferred off the Protobuf project three years before quitting Google, based on feedback I received from management suggesting they didn't think my work there was worthwhile. By the time I left Google, there was a new team maintaining Protobuf, and my reasons for leaving were not related to Protobuf. I didn't leave Google to create Cap'n Proto. Rather, having left Google and being free to do whatever I wanted for a while, I created Cap'n Proto mostly for fun, to "scratch an itch" by trying out a different design.
      - sebastos 6y ago
        Damned if you're not the most patient self-advocate for an open source project I've ever seen. I feel like I can always count on finding The Kenton Varda in the comments whenever a protobuf discussion comes up, graciously discussing design trade-offs and dispelling misconceptions. Did you ever consider looking for official / commercial backing for Cap N Proto? It seems demonstrably better than all the alternatives, but I get the sense that it hasn't taken over because "nobody ever got fired for buying IBM". I know for certain I'd have an easier time pushing capnp in my own organization if it was getting that type of backing.
brentru 6y ago
The author mentioned msgpack, what else would someone suggest for low-latency use cases?
- kevin_thibedeau 6y ago
  Pick your poison: https://en.wikipedia.org/wiki/Comparison_of_data-serialization_formats https://en.wikipedia.org/wiki/Comparison_of_data-serializati... The big choice is whether you want a schema or not.
- shaicoleman 6y ago
  Serialization/deserialization is rarely the bottleneck, so don't prematurely optimise for it. The candidates for fast serialization/deserialization that have implementations for all major programming languages are: * Plain JSON using the simdjson library, optionally in NDJSON format (line separated), optionally compressed with LZ4. * CBOR (like msgpack but with more datatypes/features) * msgpack * Protocol Buffers (optionally with gRPC)
pb123456 6y ago
In many cases you do indeed want or need to use Protobuf for telemetry or other time-series data. The Protobuf RecordIO / TFRecord formats are very unhelpful for this use case. Enter Protobag: https://github.com/StandardCyborg/protobag https://github.com/StandardCyborg/protobag Protobag is a library that helps you write and read time-series and other collections of Protobuf data using the standard Tar and Zip archives as containers. Protobag also includes code for leveraging self-describing messages to embed schemas in archives so that you never have data on disk that you can't read. Hope somebody finds Protobag helpful.
stkdump 6y ago
I created a binary serialization format and library that is comparable to json but binary. It uses 1-byte tags to give type information, which often already contains the value (in case of 0,1,true,false,null,empty string, empty list, empty object, single letter string or one of the last 32 interned strings) very similarly to msgpack. But it has the advantage, that strings are interned, which does about the same for performance as a runtime schema-based thing like Avro, but is more generic. Also, like bencode, it uses the convention that keys in objects are sorted, so that linear deserialization directly into the target objects is much simpler. And instead of prepending the size or length of objects and lists, there is a separate end-tag, so that you can start streaming data, before you know how much it will be. Sure, you can't skip without reading the data you skip over, but you (the receiver) can just do an initial pass and store the skip information in a cache, if you need. That seems better than requiring the sender to do multipass without knowing if anyone will ever use the skip info.
bastawhiz 6y ago
Ignoring the bits about Java specifically, one thing that jumps out at me (while reading through the comments) is the distinction between use cases. It seems that the wire format puts some extra burden on the encoding side, but has the benefit of making decoding nice. In a situation where you're using protobufs internally, you're both encoding and decoding the messages. CPU hours aren't fungible, but you're paying a small latency cost somewhere in your system between when the data is produced and when it can be consumed. In the case of telemetry, such as with Datadog, you pretty much only encode the data. That is, if I'm a Datadog customer, I'm literally never decoding the encoded data on my own servers. That being the case, it would seem that the argument is that if encoding performance is a desirable property, protobufs add overhead that doesn't offer any meaningful tradeoff to the user ("you're using more CPU on my server and I don't see tangible benefits"). Is this an incorrect interpretation?
- rstartin 6y ago
  That's exactly the message I intended to get across.
contingencies 6y ago
Optimization: Prototype before polishing. Get it working before you optimize it. - Eric S. Raymond, The Art of Unix Programming (2003) The First Rule of Program Optimization: Don't do it. - Michael Jackson The Second Rule of Program Optimization (for experts only): Don't do it yet. - Michael Jackson Spell create with an 'e'. - Ken Thompson (referring to design regrets on the UNIX creat(2) system call and the fallacy of premature optimization) The No Free Lunch theorem: Any two optimization algorithms are equivalent when their performance is averaged across all possible problems (if an algorithm performs well on a certain class of problems then it necessarily pays for that with degraded performance on the set of all remaining problems). ... quotes via https://github.com/globalcitizen/taoup https://github.com/globalcitizen/taoup FWIW at my company we do embedded robotics, use ridiculously underpowered processors with very limited capabilities, and still prefer use HTTP as it is well understood, has good tooling and is easy for everyone to deal with, which reduces real world business costs vs. custom wire-level formats. I'd gently suggest that unless you have a really particular use case (real time stock market feeds where latency is paramount, very expensive or constrained link layer) it makes little sense to worry about this sort of optimization.
- MauranKilom 6y ago
  What do optimization algorithms have to do with performance optimization? I mean, sure, you can use either to improve the other, but NFL has nothing to do with performance optimization.
  - contingencies 6y ago
    Fair point - I just grepped for optimization. However, it's a bit like a single iteration of any such algorithm, if applied to the domain. Distilled, it means "no single approach works for all environments" which is essentially the feedback people writing this sort of article need to hear (because apparently they missed it). There could be great reasons to use protobuf for telemetry: like, it's good enough for your application, the team is familiar with it and the code is already written, tested and documented.
jnwatson 6y ago
There's a marshalling analog to Greenspun's Tenth Rule. Every sufficiently complicated serialization implementation contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of ASN.1. I'd add that this is true of most ASN.1 implementations as well. The length encoding is a solved problem in ASN.1 (just use CER).
- sargun 6y ago
  Are there any decent CER encoders for C that are free?
  - jnwatson 6y ago
    I'm not aware of any.
0xbadcafebee 6y ago
If Protobuf was just created to make it easy for teams to inter-operate, couldn't you just use a REST API with JSON Schema? If it's gonna be a resource pig you might as well use simple components that are widely supported.
sargun 6y ago
I work with an application that spends something like 70-80% of its time in serialization / deserialization code. This is mostly due to heap allocation / deallocation (in Golang) of objects, and the RPC stacks in use not doing allocation arenas, reuse, or similar. The "state of the art" in de(serialization) isn't awesome. :(.
ricardobeat 6y ago
Interestingly, the 'Github Globe' post that is also in the front page right now shows that GitHub uses protobuf for their event pipelines: https://github.blog/2020-12-21-visualizing-githubs-global-community/ https://github.blog/2020-12-21-visualizing-githubs-global-co...
ganafagol 6y ago
Isn't there an easy solution for this? A varint representation does not mandate that you use its variable length. Just pad with leading zeros and you have a fixed length. If I understand the varint format right, its groups of 7 bits, stuffed into one byte each. Alright, so let's just always use 4 bytes, which gives you a value range from 0 to 2^28-1 for the length, that's up to 256M which should suffice if you are that concerned about telemetry speed. Now your length field is always 4 bytes. Even if you could shrink it to 1 or 2 most of the time, but since you care more about encoding speed than size on the wire that should be fine. Nothing forces you to encode value 42 in 1 byte. Just use 4. Am I missing something? Honest question.
anuraaga 6y ago
I work on OpenTelemetry Java, including a Java Agent that is originally a fork from Datadog's excellent base which they contributed to the project. Protobuf definitely has a sizable footprint, but I guess it's still ok compared to other popular libraries like Guava and Jackson. I suspect Richard is saving an article for gRPC+Netty which is extremely large :-) Lots of classes, and every shaded Netty on a classpath has its own arena of pooled buffers. Wire format I like well enough though. I think protobuf does provide good expressiveness to allow nesting if that fits the bill, or more flat structures, where it is a joy to output repeated fields completely independently and even interleaved in a stream. It's unclear to me if protobuf itself is a bad wire format for telemetry or heavy use of nesting is. The OpenTelemetry Protocol is highly nested to be completely denormalized. This reduces data on the wire (or not so much compared to gzip, but can be considered a hand-coded gzip), but it means all data has to be traversed before any can be sent to compute the length prefix. I don't know which wins in practice - I've just never seen any comparison of the exact same data with different formats. I'm hoping the CPU cache alleviates the double traversal though. But a less nested format in protobuf would easily be possible.
deleted 6y ago
[deleted]

223 points by simonpure 6y ago

dariosalvi78 6y ago

then what should we use? MQTT?

as-j 6y ago
MQTT doesn’t specify the payload format. You can easily have a topic with the data in protobuf and another with json.
toast0 6y ago
The post seems more like an argument against protobuf in general than against its use in telemetry. The main complaints seem to be that the library is big and that nested serialization is tricky because of design decisions. In regards to binary size, using whatever serializing you're already doing for your main data seems like a win for telemetry. Including telemetry with your main data has pluses and minuses, but using the same serialization is generally fine, unless your serialization can fail, in which case it's hard to report on how many times telemetry serialization fails, because you might not be able to serialize it.

ketamine__ 6y ago

Isn't this just a micro optimization? At what level does one need to care about this?

deleted 6y ago
[deleted]
cpascal 6y ago
It really depends on your scale and how many messages you are processing per second. For a lot of applications, you’re absolutely correct, but if you’re scale is sufficient, a “micro” optimization like this is actually a “macro” optimization. Also the author of this article works at DataDog and I suspect the number of messages they process each second falls under the sufficient category. For example, suppose you are processing 1M messages per second and you can shave 1 byte off the message size, that shaves off 1MB/sec of data that needs to be processed. If you’re paying for network bandwidth or storing the messages, that saves you something like 2.6TB of data each month. 2.6TB/month is not likely to be a huge deal when it comes to cost savings, but if you keep scaling the messages/sec or the bytes/msg you can start to get some significant savings. Now I used message size as an example, and the article focuses on processing time not message size, but the point still stands. When you can make a micro optimization for something that is done a very large number of times, there are not-insignificant gains to be had.
- nijave 6y ago
  I think this was for client/application instrumentation where they want to try to have little-to-no overhead for including the datadog agent that ships telemetry back to Datadog's service
- Thaxll 6y ago
  If you manage scale in the milion messages/sec you really don't care about some TB/month. Most compagnies are still using REST/json in places where a binary protocol "would be better".
rbanffy 6y ago
I recommend HTTPS because it's easy to simulate and well understood. Also I like human-readable payloads. Unless you need ridiculously low latencies and ridiculously high volume, you shouldn't worry at all.

xyzzyz 6y ago

His point on protobuf-java library adding nontrivial bloat to his Java app is definitely valid. However, his other argument about protobuf wire format being inefficient is hard to square with decades of practical experience Google had with protobufs, which are used for literally everything, including telemetry, and high throughput, low latency applications. Sure, having to recursively precompute lengths before serialization is a bit of a hassle, but I wouldn’t call it expensive.

sillysaurusx 6y ago
Signal boosting this. I am the last person to be swayed by “use our magical design that solves all problems,” but in the case of protobufs, it solves most of them. I didn’t like it until I was forced to use it. Now I’m not sure I’ll ever go back. There’s a handy code snippet to turn any protobuf message into JSON: from google.protobuf.json_format import MessageToJson import json from pprint import pprint as pp pp(json.loads(MessageToJson(msg))) Examples of turning arbitrary tensorflow objects into JSON: https://twitter.com/theshawwn/status/1331625739993575427?s=21 https://twitter.com/theshawwn/status/1331625739993575427?s=2...
- intricatedetail 6y ago
  Why they use Java like code in Python? "MessageToJson" doesn't look idiomatic.
  - singron 6y ago
    CapWords function names matches the C++ style guide at Google, not java. A lot of python uses autogenerated bindings to C++ code and reuses the same naming conventions. Even pure python code would use that style although their public styleguide doesn't mention it anymore.
  - ipsum2 6y ago
    The official protobuf library for Python is not pythonic at all.
  - 6y ago

fwsgonzo 6y ago

We already have FlatBuffers, which allow you to send structs on the wire without reinventing anything. It sounds like it's a good match here. Protobuf is better if you care about the byte count on the wire and less about the CPU time to pack and unpack it.

simonh 6y ago
I'd not come across FlatBuffers before, but they do seem interesting for some use cases, especially when you're highly performance sensitive. https://codeburst.io/json-vs-protocol-buffers-vs-flatbuffers-a4247f8bda6f https://codeburst.io/json-vs-protocol-buffers-vs-flatbuffers...
- twic 6y ago
  AFAIK, the three main options in this subspace are FlatBuffers, Cap'n Proto, and SBE. Here's a comparison from a Rust perspective: https://speice.io/2019/09/binary-format-shootout.html https://speice.io/2019/09/binary-format-shootout.html And from the Cap'n Proto author: https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-sbe.html https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-...
tyingq 6y ago
It also appears to be 236kb and 23 classes for the java api, versus 1.6MB and 700 classes for protobufs.

gregmac 6y ago

> Protobuf-java is a little heavy [...] Just depending on the library adds 1.6MB and nearly 700 classes before you even generate your own message classes. By comparison, protobuf-net [1] is about 260KB and 68 classes. Python's [2] is a 1MB package download (with source). Why's the Java one so big? [1] https://github.com/protobuf-net/protobuf-net https://github.com/protobuf-net/protobuf-net [2] https://pypi.org/project/protobuf https://pypi.org/project/protobuf

extropy 6y ago
Or just use the lite version? It removes the heavy features like descriptors and reflection. More details: https://developers.google.com/protocol-buffers/docs/reference/java-generated#message https://developers.google.com/protocol-buffers/docs/referenc...

janders 6y ago

I want to be a fan of Apache Avro (https://avro.apache.org/ https://avro.apache.org/) so much for situations such as this. And while I like Avro as a standard, most of the implementations I have found (specifically in C/C++) are... lacking. I feel Avro would be a great fit for something like this close to zero overhead (assuming a pre-shared schema) but there is little to no support for pre-shared schema and the RPC part of the standard is not a great fit for telemetry. Maybe one day I'll make an Avro library I like, or contribute to an extant one.

tele_ski 6y ago
I think AVRO is fairly well designed overall, but JSON schemas? If you need deeply nested objects just look away, it becomes unintelligible when trying to understand the schema with any real nesting. The company I was at used it fairly well as long as you keep things 1 or 2 levels nested at most, but we had a legacy schema that was 5 or 6 levels deep at some points and it was just a disaster. My anecdote anyways.
- janders 6y ago
  Oh yeah. Schema nesting is a problem. I think a lot of it would be solved by having a schema allow a "types" field which just has a list your named types that can be referenced later in the list and in the actual schema. But yeah, totally concur on that pain point.

beervirus 6y ago

Don’t Use ~~Protobuf for~~ Telemetry

lstodd 6y ago
This. Might as well use xmlrpc/SOAP for it and it will be only slightly less ugly.

Cloudef 6y ago

Google's own protobuf runtime is so bloated that even google themselves don't use it in many of their software, but rather nanopb[1], which is done by completely unrelated person to google. 1: https://jpa.kapsi.fi/nanopb/ https://jpa.kapsi.fi/nanopb/

jeffbee 6y ago
I don’t think that’s true in google3. People often mistake the scale of Google’s internal code with respect to their visible external code.
- jmkjaer 6y ago
  For anyone unfamiliar, Google3 is apparently Google's monorepo, according to this HN comment: https://news.ycombinator.com/item?id=17123620 https://news.ycombinator.com/item?id=17123620
  - alexhutcheson 6y ago
    https://research.google/pubs/pub45424/ https://research.google/pubs/pub45424/
vlovich123 6y ago
I am only aware of nanopb being used at Google in embedded scenarios. The primary reason for the project I worked on was that the first Pixel Buds didn't have a C++ compiler (at least not a functional one - based on GCC 2 or 3 if I recall correctly & an extremely crippled implementation). Pixel Buds 2 didn't have that limitation (stock GCC 9) but nanopb was kept for the second Pixel Buds because the memory management was saner for an embedded RTOS environment (vs stock upstream that might use STL or otherwise try to allocate memory). I don't think runtime bloat was really on our radar (it could have been the next critical path, but wasn't on our radar at the time). Nanopb itself needed some changes to the codegen for the first Pixel Buds (which I investigated & fixed) due to architectural peculiarities of the CSR8675. It generated a lot of constants but the CSR8675 had limited space there & was made worse because every 8 bits took up 1 16-bit word (e.g. the size of your string literal or any constant byte array is effectively doubled). So I changed the code gen to put those constants in the code section instead. Before this change resolved the issue more fully, random engineers who were unlucky enough to add a constant in some way were hitting limits internally with regularity (as the project was heading toward ship) & working around it by removing characters from log messages (e.g. "Some long message" => "Sme lng msg"). For what it's worth I tried to contribute any meaningful improvements we made back to nanopb but this one didn't make that cut as it was specific to a chip no one in the wider community would be using anyway (8675 is super old). I don't exclude the possibility of it using in more places, but I'd say the average Google engineer does not encounter nanopb in their daily development.

CamouflagedKiwi 6y ago

Is this a thing people care about? From what I've seen of server-side Java applications (we have several at work) 1.6MB and 700 classes are lost in the noise of the endless list of Maven dependencies. I'm sure you can do it a lot more efficiently without using someone else's thing, but what exactly is being optimised for here?

jeffbee 6y ago
That was kinda my response as well. If telemetry altogether is using more than 1% of your cpu time you have an architectural problem perhaps.
- ashtonkem 6y ago
  My experience points to the cost of storing metrics far outweighing the cost of emitting metrics in all but the most extreme of circumstances.
  - RhodesianHunter 6y ago
    Yes, but in this case the author is an employee of DataDog, presumably working on their agents or clients and so is probably acutely focused on not impacting the performance of their customer's systems.
Yeroc 6y ago
I thought the author addressed this pretty clearly in the article. For his specific use-case he was delivering a JVM agent.

Aaargh20318 6y ago

Are there any advantages of protobuf over using DER encoded ASN.1 ? The format looks very similar. It looks like they kind of reinvented the wheel here.

Nvorzula 6y ago
To my understanding they're both [type, length, content] encoded formats, so they're similar there at least. In my experience people really quite enjoy the IDL aspect of protobuf as the killer feature, and I am not sure any such compiler exists for ASN1.
- Aaargh20318 6y ago
  ASN.1 is basically an IDL. There are multiple encodings for ASN.1 but DER is one of the most common ones (you can also encode to JSON or XML). DER is indeed TLV (tag-length-value) and there are many code generators available.
- mianos 6y ago
  ASN.1 is specified as a compilable language. Open source compilers, such as snacc, have existed for 30 years. ASN.1 also defines multiple encodings, DER is used in crypto as you can compare encoded blocks bit by bit. Other encoding rules, such as BER, are tagged. Protobuffers is the same thing re-invented by people who probably never saw ASN.1.
kevin_thibedeau 6y ago
ASN.1 has a lot of cruft, particularly with the string encodings.

JoachimSchipper 6y ago

Protobuf's variable-length lengths are indeed annoying, but can't you encode back-to-front as in ASN.1? That lets you write your lengths just after encoding the value, which should work (but may leave your data in e.g. bytes 20+ of a buffer; depending on interfaces, you may need to do one final copy to move the data to the start of the buffer.) (ASN.1 and protobuf are both tag-length-value formats with variable-length integer encodings... h/t to tptacek for mentioning that particular trick.)

tptacek 6y ago
Hey, all I did was notice that Microsoft did that for X.509. h/t to Microsoft's cert parsing code team. :)

lxe 6y ago

Ay my previous job we started out using simple StatsD messages that are a few bytes long, easy to implement, easy to debug, and quick. But then possibly due to a full-steam-ahead mandate of "everything should use protobuf (or was it thrift)?", we switched to a much heavier format, where clients needed a codegen'd binary implementation of the protocol, while making messages themselves much heavier.

tomc1985 6y ago

700 classes? To define and implement a wire format? Does it include a kitchen sink too?

tijsvd 6y ago

I have written an in-house implementation of protobuf for C++ (sorry can't share) and studied the wire format extensively. Google's implementations, at least C++ and Java, are a bunch of bloated crap (or maybe they're very good, but for a use case that I haven't yet encountered). Don't shoot down the format because of a specific implementation, find or write a better one and enjoy the fact that every language has at least some implementation available. Then, the author laments the wire format itself for having varint-encoded length prefixes, which can not be fixed up. That is true, but it's not that much of a problem. Most straightforward is to simply go through nested data multiple times, once to calculate the length, and again for the actual encoding (and then again and again for deeper nesting). What makes this bearable is the fact that data will be mostly loaded into L2 cache (L1 for smaller messages) on the first pass, which makes the next pass much faster. The story breaks down for large, deeply nested messages, but then, the topic here is telemetry which I would expect to consist of a stream of small, shallow messages.

thisiscorrect 6y ago
Mind elaborating on how the "standard" C++ and Java protobuf implementations are bloated? [edit] I'm genuinely asking. I'm guessing you mean in the generated APIs, i.e. the code "weight", but maybe you meant something else?
- gabereiser 6y ago
  Probably a complaint of the codegen and it’s wacky name mangling.
- notretarded 6y ago
  Probably all this "security" nonsense. What could possibly go wrong with implementing a binary serialisation protocol your self...
  - inetknght 6y ago
    Oh look sarcasm! On Hacker News!

Luker88 6y ago

Does anyone have an up-to date comparison protobuf vs captnproto instead? ( https://capnproto.org/ https://capnproto.org/ ) I am starting a new project and it looked really interesting, but I still have to read comments from people who used both in detail

kjgkjhfkjf 6y ago
CP was written by Kenton Varda, who spent many years working on protobuf at Google. One massive advantage of protobuf for mainstream Google languages (C++, Java) is that Google has used them extremely heavily for many years, and you can trust that they've been extensively battle tested in Google's enormous high-traffic services, and scrutinized by Google's vast army of engineers. Kenton's experience notwithstanding, the same cannot be said of CP. This argument doesn't apply for languages other than the ones that Google uses for production services.
- alisonkisk 6y ago
  Part of the reason that Kenton left Google and created CapnProto, was because he was the only person working on proto at Google trying to fix the problems Google found.
  - kentonv 6y ago
    That's not really accurate... I transferred off the Protobuf project three years before quitting Google, based on feedback I received from management suggesting they didn't think my work there was worthwhile. By the time I left Google, there was a new team maintaining Protobuf, and my reasons for leaving were not related to Protobuf. I didn't leave Google to create Cap'n Proto. Rather, having left Google and being free to do whatever I wanted for a while, I created Cap'n Proto mostly for fun, to "scratch an itch" by trying out a different design.
    - sebastos 6y ago
      Damned if you're not the most patient self-advocate for an open source project I've ever seen. I feel like I can always count on finding The Kenton Varda in the comments whenever a protobuf discussion comes up, graciously discussing design trade-offs and dispelling misconceptions. Did you ever consider looking for official / commercial backing for Cap N Proto? It seems demonstrably better than all the alternatives, but I get the sense that it hasn't taken over because "nobody ever got fired for buying IBM". I know for certain I'd have an easier time pushing capnp in my own organization if it was getting that type of backing.

brentru 6y ago

The author mentioned msgpack, what else would someone suggest for low-latency use cases?

kevin_thibedeau 6y ago
Pick your poison: https://en.wikipedia.org/wiki/Comparison_of_data-serialization_formats https://en.wikipedia.org/wiki/Comparison_of_data-serializati... The big choice is whether you want a schema or not.
shaicoleman 6y ago
Serialization/deserialization is rarely the bottleneck, so don't prematurely optimise for it. The candidates for fast serialization/deserialization that have implementations for all major programming languages are: * Plain JSON using the simdjson library, optionally in NDJSON format (line separated), optionally compressed with LZ4. * CBOR (like msgpack but with more datatypes/features) * msgpack * Protocol Buffers (optionally with gRPC)

pb123456 6y ago

In many cases you do indeed want or need to use Protobuf for telemetry or other time-series data. The Protobuf RecordIO / TFRecord formats are very unhelpful for this use case. Enter Protobag: https://github.com/StandardCyborg/protobag https://github.com/StandardCyborg/protobag Protobag is a library that helps you write and read time-series and other collections of Protobuf data using the standard Tar and Zip archives as containers. Protobag also includes code for leveraging self-describing messages to embed schemas in archives so that you never have data on disk that you can't read. Hope somebody finds Protobag helpful.

stkdump 6y ago

I created a binary serialization format and library that is comparable to json but binary. It uses 1-byte tags to give type information, which often already contains the value (in case of 0,1,true,false,null,empty string, empty list, empty object, single letter string or one of the last 32 interned strings) very similarly to msgpack. But it has the advantage, that strings are interned, which does about the same for performance as a runtime schema-based thing like Avro, but is more generic. Also, like bencode, it uses the convention that keys in objects are sorted, so that linear deserialization directly into the target objects is much simpler. And instead of prepending the size or length of objects and lists, there is a separate end-tag, so that you can start streaming data, before you know how much it will be. Sure, you can't skip without reading the data you skip over, but you (the receiver) can just do an initial pass and store the skip information in a cache, if you need. That seems better than requiring the sender to do multipass without knowing if anyone will ever use the skip info.

bastawhiz 6y ago

Ignoring the bits about Java specifically, one thing that jumps out at me (while reading through the comments) is the distinction between use cases. It seems that the wire format puts some extra burden on the encoding side, but has the benefit of making decoding nice. In a situation where you're using protobufs internally, you're both encoding and decoding the messages. CPU hours aren't fungible, but you're paying a small latency cost somewhere in your system between when the data is produced and when it can be consumed. In the case of telemetry, such as with Datadog, you pretty much only encode the data. That is, if I'm a Datadog customer, I'm literally never decoding the encoded data on my own servers. That being the case, it would seem that the argument is that if encoding performance is a desirable property, protobufs add overhead that doesn't offer any meaningful tradeoff to the user ("you're using more CPU on my server and I don't see tangible benefits"). Is this an incorrect interpretation?

rstartin 6y ago
That's exactly the message I intended to get across.

contingencies 6y ago

Optimization: Prototype before polishing. Get it working before you optimize it. - Eric S. Raymond, The Art of Unix Programming (2003) The First Rule of Program Optimization: Don't do it. - Michael Jackson The Second Rule of Program Optimization (for experts only): Don't do it yet. - Michael Jackson Spell create with an 'e'. - Ken Thompson (referring to design regrets on the UNIX creat(2) system call and the fallacy of premature optimization) The No Free Lunch theorem: Any two optimization algorithms are equivalent when their performance is averaged across all possible problems (if an algorithm performs well on a certain class of problems then it necessarily pays for that with degraded performance on the set of all remaining problems). ... quotes via https://github.com/globalcitizen/taoup https://github.com/globalcitizen/taoup FWIW at my company we do embedded robotics, use ridiculously underpowered processors with very limited capabilities, and still prefer use HTTP as it is well understood, has good tooling and is easy for everyone to deal with, which reduces real world business costs vs. custom wire-level formats. I'd gently suggest that unless you have a really particular use case (real time stock market feeds where latency is paramount, very expensive or constrained link layer) it makes little sense to worry about this sort of optimization.

MauranKilom 6y ago
What do optimization algorithms have to do with performance optimization? I mean, sure, you can use either to improve the other, but NFL has nothing to do with performance optimization.
- contingencies 6y ago
  Fair point - I just grepped for optimization. However, it's a bit like a single iteration of any such algorithm, if applied to the domain. Distilled, it means "no single approach works for all environments" which is essentially the feedback people writing this sort of article need to hear (because apparently they missed it). There could be great reasons to use protobuf for telemetry: like, it's good enough for your application, the team is familiar with it and the code is already written, tested and documented.

jnwatson 6y ago

There's a marshalling analog to Greenspun's Tenth Rule. Every sufficiently complicated serialization implementation contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of ASN.1. I'd add that this is true of most ASN.1 implementations as well. The length encoding is a solved problem in ASN.1 (just use CER).

sargun 6y ago
Are there any decent CER encoders for C that are free?
- jnwatson 6y ago
  I'm not aware of any.

0xbadcafebee 6y ago

If Protobuf was just created to make it easy for teams to inter-operate, couldn't you just use a REST API with JSON Schema? If it's gonna be a resource pig you might as well use simple components that are widely supported.

sargun 6y ago

I work with an application that spends something like 70-80% of its time in serialization / deserialization code. This is mostly due to heap allocation / deallocation (in Golang) of objects, and the RPC stacks in use not doing allocation arenas, reuse, or similar. The "state of the art" in de(serialization) isn't awesome. :(.

ricardobeat 6y ago

Interestingly, the 'Github Globe' post that is also in the front page right now shows that GitHub uses protobuf for their event pipelines: https://github.blog/2020-12-21-visualizing-githubs-global-community/ https://github.blog/2020-12-21-visualizing-githubs-global-co...

ganafagol 6y ago

Isn't there an easy solution for this? A varint representation does not mandate that you use its variable length. Just pad with leading zeros and you have a fixed length. If I understand the varint format right, its groups of 7 bits, stuffed into one byte each. Alright, so let's just always use 4 bytes, which gives you a value range from 0 to 2^28-1 for the length, that's up to 256M which should suffice if you are that concerned about telemetry speed. Now your length field is always 4 bytes. Even if you could shrink it to 1 or 2 most of the time, but since you care more about encoding speed than size on the wire that should be fine. Nothing forces you to encode value 42 in 1 byte. Just use 4. Am I missing something? Honest question.

anuraaga 6y ago

I work on OpenTelemetry Java, including a Java Agent that is originally a fork from Datadog's excellent base which they contributed to the project. Protobuf definitely has a sizable footprint, but I guess it's still ok compared to other popular libraries like Guava and Jackson. I suspect Richard is saving an article for gRPC+Netty which is extremely large :-) Lots of classes, and every shaded Netty on a classpath has its own arena of pooled buffers. Wire format I like well enough though. I think protobuf does provide good expressiveness to allow nesting if that fits the bill, or more flat structures, where it is a joy to output repeated fields completely independently and even interleaved in a stream. It's unclear to me if protobuf itself is a bad wire format for telemetry or heavy use of nesting is. The OpenTelemetry Protocol is highly nested to be completely denormalized. This reduces data on the wire (or not so much compared to gzip, but can be considered a hand-coded gzip), but it means all data has to be traversed before any can be sent to compute the length prefix. I don't know which wins in practice - I've just never seen any comparison of the exact same data with different formats. I'm hoping the CPU cache alleviates the double traversal though. But a less nested format in protobuf would easily be possible.

deleted 6y ago

[deleted]