22 ms·

Arguing against using protobuffers

• Jo8192 8y ago
  I can not really take someone who calls Jeff Dean an amateur seriously...
• sonnyblarney 8y ago
  Good points, but can someone articulate what the best alternative to protobuffers would be in 2018, you know with 'hindsight' etc.?
  • habitue 8y ago
    There's Avro: https://avro.apache.org/docs/current/ https://avro.apache.org/docs/current/ Not sure if the best, but maybe has different tradeoffs
  • k__ 8y ago
    I know a few companies who use flatbuffers instead.
  • amluto 8y ago
    Depending on your goals, Cap’n Proto is excellent.
    • jeremyjh 8y ago
      Which of the problems listed in the article are solved by it?
      • danharaj 8y ago
        Parameterization of types, off the top of my head.
        • patrec 8y ago
          Also, cap'n proto loses a whole lot of API and semantic weirdness the author complains about: implicit auto-assignment of fields, "repeated" etc. The overall design feels much cleaner, although there are still some surprises.
  • CobrastanJorji 8y ago
    I've heard good things about https://capnproto.org/ https://capnproto.org/ but the author of this piece may strongly disagree, since I think their objections are rather more fundamental, and also because Cap'n Proto went with banning the concept of "required" fields. I'd also be interested in what alternatives the author might suggest.
    • weberc2 8y ago
      Capnproto is built by the guy who originally built protobufs, and he cites capnproto as being better because he was able to learn from his mistakes. Makes me optimistic.
      • enneff 8y ago
        Minor clarification: the author of Capnproto worked on protobufs at Google but was not their original creator.
        • nostrademons 8y ago
          Minor clarification of the clarification: the author of Cap'n'proto was the original creator of Proto2, which was a ground-up (but binary compatible) rewrite of Proto1. The original authors of Proto1 were Jeff Dean & Sanjay Ghemawat. The current version of protobufs is Proto3, which AIUI is maintained by a team at Google (it was released after I left), and is an evolution of the Proto2 codebase with many new features.
          • weberc2 8y ago
            Ah, my mistake. Thanks guys. :+1:
    • patrec 8y ago
      You should definitely also look at Cap'n Proto – it is in many ways a theoretically more elegant design: far less overhead and the "time-traveling" RPC design is quite clever. Another potential important advantage, esp for crypto could be that Cap'n Proto in theory has a canonical representation, but in practice this was neither fully defined nor usable last I looked. Cap'n Proto also suffers from a number of practical disadvantages, which can be quite severe depending on your use case. The main problem is eco system maturity and rate of adoption. So whilst protobufs have excellent cross-language support and a mature RPC system, in cap'n proto everything but C++ feels at bit second class, even other popular languages. Python and Rust, and I believe Java have workable, but not necessarily great bindings for serialization and in theory RPC, but the RPC support for other languages than C++ didn't look that great last I looked (but that was over a year ago) and even with C++ it looks much easier to get something going with gRPC (tooling, http2 based etc). For other languages I looked at (e.g. Ocaml, Lua and Haskell) the bindings looked very immature. There are a few other minor annoyance with capnproto, such as the lack of a timestamp type (you can build your own, incompatible to everyone else's, but this is still an annoying omission). The C++ library is also saddled with the weird home-grown kj library that no one other than Kenton Varda uses. However, it has to be said that the C++ API and also the Python API for Capnproto feel a lot more natural and idiomatic than the protobuf ones. If protobuf is too high overhead for your use case and you don't need the wide language and eco system support, Cap'n Proto is worth a serious look.
  • tlarkworthy 8y ago
    JSON or graphql. Personally I think the typing, API compatibility and extensive language support make protobuffers currently the winner though, despite the warts
    • vthallam 8y ago
      How would graphql help reduce the latency of API calls between services?
      • hardwaresofton 8y ago
        I'm no GraphQL expert/fan but one example is requesting only the data you need -- you can cut down on what you have to send over the wire which is at least guaranteed to help some. Also, the nested query structure can save what would have normally been multiple requests with a by-the-book RESTful HATEOAS setup.
    • FridgeSeal 8y ago
      Neither of those are binary formats though, and if I’m picking up a binary format like Protobuffers it’s probably because I want the performance benefits. Personally, these days I’m taking the approach of just outright avoiding pretty much anything google does. Either because it solves a problem only they have, or they’ll probably just drop it out of the blue one day. Or like Tensorflow, it’s fucking impossible to get your head around because their documentation seems to be written with the goal of being maximally confusing.
    • sonnyblarney 8y ago
      I like GraphQL schema, but it would have to have a standardized serialization aspect. Moreover, it really is designed for querying and has a bunch of stuff totally irrelevant to data. But I agree ... I almost wish they'd finish it off and go for full on serialization. That said, it may not be that-that hard, a serialization spec can be pretty short if need be. Certainly one could 'roll one's own' for a mid-sized project if need be.
  • hardwaresofton 8y ago
    How about json + jsonschema + gzipped over the wire
  • mixmastamyk 8y ago
    Apache Thrift is another one.
    • mlthoughts2018 8y ago
      Thrift suffers pretty much all the same problems as protobufs and has many similar dysfunctional failure modes, like places that rely on serializing into Thrift structs stores in hdfs and treating that like a de facto database, with Thrift struct definitions as the schema. It is so miserable to work in code bases like that.
      • hermanradtke 8y ago
        This is part of the original lambda architecture. What would you recommend the schema be represented in instead? Or are you preferring something like NewSQL?
        • mlthoughts2018 8y ago
          I can’t tell if you mean a schema for the generated RPC or just for data storage. I’m only talking about data storage. But for RPCs, I just think don’t autogenerate them. Just expose an RPC API and let other people write code to consume it in whatever language’s packages for web requests, and don’t ever transmit things that are expected to automatically be treated as any type of object. Just send JSON or an equivalent thing in a more optimized buffer that is never allowed to be anything but a key-value store for primitive types. On the data side, use relational database systems. Yes, even for huge modern webscale event data for apps or services with hundreds of millions of users. Don’t ever use hadoop, period. If you’re bigger than hundreds of millions of users or otherwise are generating hundreds of billions of records per day or more, that’s the size when you might _start_ considering something different than sharded and distributed standard RDBMSs, but you are likely big enough at that point that you need an in-house, highly customized version of a distributed file store that matches your usage patterns and cost optimizations in a way that a one-size-fits-all solution cannot, and so again you should never be using hadoop. As a result, if you find yourself relying on Thrift data serialized on a distribted filesystem as if those “flat files” are a database and map-reduce is like your de facto table scanner, it’s a red alarm bad code smell that you are growing your data scale in a horribly broken way that’s going to cause huge problems the minute you have products which need a data model or some flexibility that cannot be supported when you’ve welded data infrastructure to Thrift objects.
  • chabsf 8y ago
    Furthermore, in the case where I'm using protobufs as a necessary evil of grpc, what's the best alternative here?
    • tathougies 8y ago
      JSON/custom binary format + HTTP2?
      • chabsf 8y ago
        HTTP2 is just the transport, there would be a ton of machinery to implement in each language in the stack.
    • romed 8y ago
      protos are not necessary for grpc. You can use whatever payload format you want to use.
  • abernard1 8y ago
    Transit: https://github.com/cognitect/transit-format https://github.com/cognitect/transit-format Transit has the following features: - self-describing - allows extension types on top of base types - strong value types + dynamic serialization - can transparently pass through data without consumer knowing about schema - has "repeated key" optimization in the encoder to allow compression in the stream - allows transparent binary encoding with msgpack - is stream-based, so processing can begin immediately unlike JSON which has enclosing {} This set of features basically allow transparent proxying to JSON as well, which is something that proto2/3 cannot do without updating the proxy.
• twic 8y ago
  > Despite map fields being able to be parameterized, no user-defined types can be. This means you'll be stuck hand-rolling your own specializations of common data structures. What a pain! Well, at least Google won't make that mistake again!
  • amelius 8y ago
    You can always write a code-generator to ease the pain.
    • k__ 8y ago
      That's what I hear from Go devs all the time, doesn't sound convicing to me.
      • regecks 8y ago
        I think both comments are sarcastic jabs at Go.
  • zamalek 8y ago
    My contention with the quoted text is that you probably shouldn't be using elaborate data structures in streams/files. Using DTOs as heap/stack data has bitten me enough times that I'm fairly certain that it's an anti-pattern. It doesn't matter if you're using a quantum binomial tree in a black hole: save it as a 'stupid' map<k,v> when it hits the network. That way everyone who interacts with your service can decide how they want to represent that structure. You can compose any in-memory data structure you can dream of, you can validate the data with more richness than "missing" or "present", and Protobuf doesn't contaminate your codebase. Option 3 is the correct choice. Serialization libraries should be amateurish by design.
    • derefr 8y ago
      > That way everyone who interacts with your service can decide how they want to represent that structure. Protobufs aren't a message format for publicly specced standard wire protocols. They're a serialization layer for polyglot RPC request and response messages. The whole point of them is that you're transporting the same typed data around between different languages, rather than there having to be two conversions (source language -> "standard format"; "standard format" -> dest language). In this sense, they're a lot like, say, Erlang's External Term Format—driven entirely by the needs of an RPC-oriented distributed-computation wire protocol, doing interchange between nodes that aren't necessarily running the same version of the same software. Except, unlike ETF, Protobufs can be decoded without the backing of a garbage-collected language runtime (e.g. in C++ nodes.) I'm not saying Protobufs are the best, but you have to understand the problem they're solving—the "rivals" to Protobufs are Cap'n Proto and Thrift, not JSON. JSON can't even be used to do a lot of the stuff Protobufs do (like talk to memory-constrained embedded kernels running on SDN infra.) > and Protobuf doesn't contaminate your codebase Just like in HTTP web-app codebases, a codebase properly designed to interact with an https://en.wikipedia.org/wiki/Enterprise_service_bus https://en.wikipedia.org/wiki/Enterprise_service_bus will tend toward Hexagonal architecture—you have your own logic, and then you have an "RPC gateway" that acts as an MVC system, exposing controllers that decode from the RPC format and forward requests into your business logic; and then build responses back into the RPC "view" format. Once you have that, there's no point in having your application's parsed-out representation of the RPC format be different than the on-the-wire RPC format. The only thing that's touching those RPC messages is your RPC gateway's controller code anyway. So why not touch them with as little up-front design required as possible?
• justicezyx 8y ago
  Wow, some minor complaints on language formality issues, then discredit the thing as a whole. But in the end, the author probably does not understand PB solves the inter-language data sharing problem, which makes all its complain secondary and inconsequential. It's like complaining a car that drives fast and has great gas efficiency of cannot talk. (Well in 2018, it might be less of a stretch to demand a car that can talk)
  • wtetzner 8y ago
    > But in the end, the author probably does not understand PB solves the inter-language data sharing problem, which makes all its complain secondary and inconsequential. Huh? It's a serialization format. This problem has been solved many times in many different ways. The author is pointing out that Protobufs were designed poorly, and they didn't have to be.
    • justicezyx 8y ago
      > This problem has been solved many times in many different ways. Mind provide examples of such tools.
      • cakebrewery 8y ago
        Plain old JSON, with types defined.
        • tgsovlerkhgsel 8y ago
          Way too slow even at moderate scale.
      • wtetzner 8y ago
        There's thrift, msgpack, Avro, ASN.1, JSON, XML, BSON, S-Expressions, EDN, etc.
• rb808 8y ago
  Versioning was the biggest disappointment for me. I just want a middleware layer that can handle clients of different versions reliably. Surely everyone has the same problem.
  • aaaaaaaaaab 8y ago
    >Surely everyone has the same problem. Yes, we do. But the appropriate answer in most cases is "it's up to the business logic". Trying to solve it at the transport layer is futile.
  • jcelerier 8y ago
    > Surely everyone has the same problem. it doesn't mean that there is a single solution that will satisfy everyone
  • nly 8y ago
    Avro has a solution to this that I always thought was quite elegant. Schemas are external (serialized data isn't self-describing) but the library exploits this to it easy to read and write old versions. To quote the documentation: The library provides "a resolving decoder, which accepts calls for according to a reader's schema but decodes data corresponding to a different (writer's) schema doing schema resolution according to resolution rules in the Avro specification." Those resolution rules are defined here: https://avro.apache.org/docs/current/spec.html#Schema+Resolution https://avro.apache.org/docs/current/spec.html#Schema+Resolu... The downside, of course, is performance.
• _dmurph 8y ago
  Mostly lost me at "Make all fields in a message required. This makes messages product types." Your constraints on protocols can change over time and context, changing something that is required to something that is optional can cause crashes in prod. Unless somehow the 'optional' design has a way to handle that? You also will have to do custom validation anyways, as your storage format will never be able to enforce all constraints you care about. Also, it would seem like the 'debate' of optional vs required is harmful was resolved, as proto3 makes everything optional.
  • danharaj 8y ago
    Like, 5 lines later: > For example, we can rebuild optional fields:
    • _dmurph 8y ago
      How is this different than having both required and optional.
      • danharaj 8y ago
        It's explained. Later in the article.
  • vl 8y ago
    If you would continue reading, there is an explanation of how optional would work almost immediately after this quote.
    • _dmurph 8y ago
      Right - this is the same as having the possibility of both required and optional. I'm saying there should not be any possibility of a 'required' field
      • vl 8y ago
        In context of current protobuf design required was a mistake and optional is clearly better, but author argues about grand type system that is based on different principles, including stronger validation. Criticizing protobufs is like criticizing C++ or Java. Both have major shortcomings, but solve practical problems and de facto lingua franca with no practical solutions to replace them.
• nemothekid 8y ago
  >Fields with scalar types are always present. Even if you don't set them. Did I mention that (at least in proto3) all protobuffers can be zero-initialized with absolutely no data in them? I don't this complaint. You have to initialize data with something. proto2 had to "solve" this problem in C/C++/Go by making everything a pointer. How is dealing with null the "sane" case against zero-initialization?
  • romed 8y ago
    Scalar primitive fields in proto1/proto2 C++ generated code are not pointers. They are just class member fields returned by accessors (foo()) and their presence is indicated by a presence vector (has_foo()). At least that's what you get if you use Google's proto compiler. The class layout for proto3 is the same (surprise!) but the presence vector is ignored.
  • atombender 8y ago
    To be fair, in C++ you could use std::optional or something else that's type-safe. Go has its own philosophy about zero values which is controversial, but at least with pointers you do get to express optional values. And it's not like you can't work around the ergonomic awkwardness that arises: type Post struct { Title *string } func (p *Post) GetTitle() (string, bool) { if p.Title == nil { return "", false } return *p.Title, true } func (p *Post) SetTitle(s string) { p.Title = &s } Hardly elegant, but this is Go. Go also run into the same issue when encoding and decoding JSON. Most languages do distinguish between empty string and a missing string, but not Go, which makes it hard to validate anything. I wrote a code generator for JSON Schema [1] recently, which applies validations while deserializing, and has to resort to a rather low-tech trick to do so: type Post struct { Title string `json:"title"` } func (v *Post) UnmarshalJSON(b []byte) error { var raw map[string]interface{}{ if err := json.Unmarshal(b, &raw); err != nil { return err } if _, ok := raw["title"]; !ok { return errors.New("field title: must be set") } type plain Post var p plain if err := json.Unmarshal(b, &p); err != nil { return err } *v = Post(p) return nil } (Clearly there are slightly more refined and faster ways to do this, but not that easily without importing third-party code, which I wanted to avoid.) C and Go are in the minority here. This doesn't come up in languages like Rust, Swift, TypeScript, Nim, Haskell, OCaml, C#, F#, or Java >= 8, all of which have some sort of optional support (algebraic data types or built-in). For example, TypeScript: interface Post { title?: string } Or Rust: struct Post { title: Option<&str> } [1] https://github.com/atombender/go-jsonschema https://github.com/atombender/go-jsonschema
• atombender 8y ago
  Though I dislike the hyperbolic tone and personal attacks, the author isn't entirely wrong. There are many design choices in Protocol Buffers that seem directly related to the scale and complexity at which Google operates, and which sacrifice safety, clarity and language integration. The utter awkwardness of Protobuf-generated code is particularly problematic. I've had pretty good results with the TypeScript code generator, but the Go generator is just egregiously bad. There's no way to write a client or server that uses the Protobuf structs directly as first-class data types without ending up looking like a ball of spaghetti, at least not if you venture into the realm of oneofs, timestamps or the "well known types" set of types. I think we're at a juncture where we really desperately need a better, unified way to express schemas and pass type-safe data over the network that flows through these schemas. I'm currently working on a project that involves both gRPC (which uses Protobuf), GraphQL and JSON Schema, with the backends written in Go, and the frontend in TypeScript, and the overlap between all of these is ridiculous. TypeScript is a fresh breath of air that mostly solves the brittleness of JavaScript's type system, and it's absolutely crucial to extend this type-safety all the way to the backend. The challenge is that no language is the same, and a common schema format ends up targeting a lowest-common denominator. For example, GraphQL errs (in my opinion, wrongly) on the side of simplicity and doesn't support maps, so now you have a whole layer that needs to either custom types (JSON as an escape hatch, basically) or emulate maps using arrays of pairs, neither of which is ideal.
  • nirelBRACIT0S 8y ago
    Write your own generator then?
    • hellllllllooo 8y ago
      People should be able to provide constructive and useful criticism of a technology without having to build their own solution.
  • enneff 8y ago
    An approach I've used on the Go side is to build nice native Go types and converters to/from proto representations for wire marshaling. There's a minor performance hit but it's insignificant for what I'm doing. The result is quite nice, and you can largely ignore protobufs except for defining service APIs (which it's actually decent at). Which tool are you using for TypeScript protobuf code generation? This is the next step for me, and I'd be keen to hear more of your experience there.
• imtringued 8y ago
  I will say one thing. gzipped JSON is damn efficient over the wire. It elegantly solves the problem that field names are repeated in every object. The only problem is that adding compression and a serializationd/deserialization step cost CPU time. However if you have this problem then you're often better off by using something like cap'n proto or flatbuffers which do not have an extra serdes step. If you consider this then there is only a very very narrow band in which it makes sense to use protobuffers vs JSON (which is a defacto standard) or lesser known solutions that provide superior performance.
  • packetslave 8y ago
    The problem with JSON as a wire/interop format is the lack of any sort of schema. Formats like proto are nice because if you can unmarshall the bytes successfully, you can be reasonably sure you've got a valid message and reason about its contents. A deserialized JSON object can literally be or contain anything.
    • kidsnow 8y ago
      It could even be a gzip bomb
    • romed 8y ago
      Proto doesn't really have a schema. You can encode a proto of type X and tell your program to decode it as type Y, and it might work. Might not, but might. You can easily imagine messages that are isomorphic on the wire. E.g. message X { optional string foo = 1; } message Y { optional X foo = 1; } Encoded buffers of X will decode as Y just fine.
      • packetslave 8y ago
        Sure, but it has more of a schema than JSON does :-) That's why I said "you can be REASONABLY sure you've got a valid message" (if it decodes successfully). It might not be the actual message you THINK it is, but at least all fields are going to be there. Then again, if you're accepting arbitrary bytes off the wire and blindly assuming they're going to be the correct proto as long as they decode, you're... most stubby services I worked on. :-)
• azernik 8y ago
  > Option 1 is clearly the "right" solution, but its untenable with protobuffers. The language isn't powerful enough to encode types that can perform double-duty as both wire and application formats. Which means you'd need to write a completely separate datatype, evolve it synchronously with the protobuffer, and explicitly write serialization code between the two. Seeing as most people seem to use protobuffers in order to not write serialization code, this is obviously never going to happen. This last is not the case. I've only ever seen it used when people a) already have written serialization code that works with XML/JSON, and b) don't want to write code to serialize to a binary format. JSON also doesn't let you use objects as object keys, or have any form of polymorphism. It seems like the author wants protobufs to be an RPC system with a modern type system, when in fact it is a data interchange format.
  • wtetzner 8y ago
    But the author is only talking about data types. I don't see any reason it shouldn't be designed with composition in mind. The only restriction I can think of that JSON imposes (in terms of composition) is that keys must be strings. And I don't necessarily think JSON is a great serialization format either.
• _dmurph 8y ago
  This feels pretty vitriolic, I wouldn't be surprised if there is some bias here. A lot of these problems seem pretty minor and there's weird stuff like "Unlike most companies in the tech space, paying engineers is one of Google's smallest expenses." According to here https://www.quora.com/How-many-software-engineers-does-Google-have https://www.quora.com/How-many-software-engineers-does-Googl... there are ~28,000 engineers in 2014, which paying at 120,00 a year (glassdoor) would be... $3.3 billion a year? And this was 2014, and doesn't include stock compensation Don't get me wrong - there are definitely problems with protobufs. But this post is citing a lot of stuff to back itself up that doesn't add up. Also, required vs optional - definitely only optional, there should be nothing required. So I flat out disagree with that one.
  • why_only_15 8y ago
    Google's annual revenue was $110 billion in 2017 [1]. Even if headcount doubled and salary has increased, that's $7 billion a year. That's not peanuts, but at a company level it's not a massive expense. [1]: https://www.androidauthority.com/alphabet-q4-2017-earnings-834774/ https://www.androidauthority.com/alphabet-q4-2017-earnings-8...
    • justicezyx 8y ago
      Anytime Pull out the revenue gun. It sounds like the revenue is actually produced by the machines though. I wonder why there isn't another search company competing with Google... Btw, if you use glass door to deduce Google's cost on employee, you probably have no idea how company works...
      • _dmurph 8y ago
        lol yeah I know, but it's at least the low bar. I would guess employees are one of the largest expenses for Google.
    • _dmurph 8y ago
      True. I'm also guessing this number changes a lot with stock grants (that number was just base pay), and that yearly number seems pretty low (NYT has the entry level engineer at 124,000 https://www.nytimes.com/2017/09/08/technology/google-salaries-gender-disparity.html https://www.nytimes.com/2017/09/08/technology/google-salarie...) so glassdoor might not have good data. I wouldn't be surprised if it was much more than 6%.
• romed 8y ago
  It's abundantly clear why this author lasted only a year at Google. Aside from using the non-idiomatic "protobuffers" ... "nothing wants to inspect only some bits of a message and then forward it on unchanged" In my experience it is extremely useful to partially parse a protobuf, or to not parse it at all and simply modify it by appending to it. Also useful is the ability to define a message type that is isomorphic on the wire with a different type, but is much cheaper to parse (example: a deeply nested structure where all of the fields of interest are defined at the top level). Nobody even within Google will argue that no mistakes have been made with proto. Proto3 is generally acknowledged to have been a big mistake and it doesn't have a lot of traction within Google. Map and OneOf are both basically antagonistic to high performance application code, and people avoid those for that reason. But the author's arguments are pretty weak and his dismissal of Google itself as an existence proof is insufficiently supported.
  • tathougies 8y ago
    Ah right... when you can't attack the ideas, attack the author. Sandy is a Haskell enthusiast, and it is not a surprise that his criticism of protobuf is inspired heavily by that. The argument presented is that protobuf does not attempt to even replicate the best practices we already have in terms of data representation. It makes sense why languages like C are constrained in their data representations -- they want to be close to the machine. However, a language meant literally to specify data really ought to be able to handle the basic competencies of its field (like sum and product types and polymorphism) with ease. There is no excuse in 2018 for a data language without polymorphism and without product and co-product types. Compatibility with legacy or substandard languages is not an excuse for poor design. Compatibility with needed legacy languages should be the thing that's tacked on, not basic functionality.
    • romed 8y ago
      Sure that makes sense if you can write off C++ as a "legacy" language, which is fine if you just can't get your mind wrapped around the scale at which Google operates. Due to the nature of weighted averages, you can't just write off something as being a "Google-only problem". Google and its peers like Amazon and Facebook own a very large fraction of the world's computing resources. I think the overall mistake you and the author are making is assuming it is desirable for one program examining an encoded message to respect the type system of the program that produced it. That assumption is not obviously correct. The flexibility to just treat a vector of numbers as a vector of numbers, or to skip it, or ignore it, etc are pretty important in practice. That is why protobuf is defined at the level where it is defined. It encodes numbers and strings on the wire. It happens to be a fairly good way to represent search data. It brings no CS academic wankery to the party. It's very practical.
• docker_up 8y ago
  Put me firmly in the camp of "optional fields are bad." I believe all fields should be required. I come from an ONC/RPC background, which is the original UNIX RPC. Every iteration of an rpc would get versioned, and then you could write a conversion between versions, ex from V1 to V2, from V2 to V3, etc. This allowed for true backwards compatibility. The idea of "forwards compatibility" is a pipedream, in my opinion. Just because something won't crash because a field is or isn't there doesn't mean that it will actually work. Code ages over time, and instead of looking at the spec, which will tell you which fields are required, you need to read the documentation, which may or may not tell you what is required. And especially when dealing with an older client, it simply may not work because what used to be optional-optional fields become required-optional fields. It's the same argument as schemaless NOSQL database like Mongo vs MySQL. Having a schema is a pain in the ass, but it's better over time and ages much more gracefully than schemaless ones, because things won't just break.
  • jeremyjh 8y ago
    If all fields are required, you cannot have middleware that processes multiple versions of the same protobuf, and every application has to be updated when a field is added, even if they do not use that field. This is one of the more important design goals underlying not only protobufs, but most of the non-language specific binary formatters.
    • derefr 8y ago
      > every application has to be updated when a field is added, even if they do not use that field This is when, in a protocol, you reach for the hammer called "extensibility." In this case (decoding to native structs), you'd probably have your FooMessage product-type have an "extensions" field, which is a list of zero or more FooExtensionStructs, where a FooExtensionStruct is a sum type of the known extensions to FooMessage. Then, just make the semantics of your format such that a sum type can have a pragma in its IDL indicating that it isn't required to be decodable for the product-type containing it to be successfully decoded. I.e., the sum type becomes a (DecodeResult = Decoded(sum type) | Opaque(raw wire data)). Opaque (unrecognized) extension data can't be manipulated by the program, but it can losslessly survive a trip back through the wire encoder. So you can choose to pass it through in your middleware, or not. Boom: you've reinvented the "chunks" concept of the https://en.wikipedia.org/wiki/Interchange_File_Format https://en.wikipedia.org/wiki/Interchange_File_Format, as seen in PNG and ELF.
• tathougies 8y ago
  Easiest way to send data over the wire. struct mytype s; s.field1 = something; s.field2 = something else; send(socket, &s, sizeof(s), 0) or, using a language with a good type system like Haskell data MyData = MyData Int Int deriving Generic instance Storable MyData alloca $ \buf -> do let d = MyData field1 field2 poke buf d send socket buf (sizeof d)
  • ericpauley 8y ago
    C doesn't make struct memory layout guarantees, endian guarantees, or size guarantees in some cases, so unless you're running the exact same binary this is a very poor serialization technique.
    • tathougies 8y ago
      C does not, but few compilers implement standard C. Most C compilers do have documentation on how structs are laid out and many options to change the behavior. Most C compilers document this.
  • amluto 8y ago
    This may be easy, but it’s wrong. Endianness issues are just the start. Information leaks due to padding are a big deal. And it straight up doesn’t work if nontrivial data structures are involved.
    • tathougies 8y ago
      Endianness issues are something every programmer should be aware of when sending data over the wire. I'm sorry I didn't insert the htons, htonls in my C code. In the Haskell code, endianness is handled by your Storable implementation (you define it after all, and can customize it however you want). I agree my example is somewhat tongue in cheek. The point though is that most languages have standard libraries for dealing with binary data that are almost universally more straightforward than proto bufs.
      • 8y ago
• joshuamorton 8y ago
  These objections are interesting. I have mixed opinions on protos, I think overall I'm mostly in favor. I'm a bit confused by this set of objections though. While oneof fields cannot be repeated, oneof fields can be arbitrary protos, so they can contain repeated fields. In other words, you can have a (pseudo-proto) oneof { RFoo { repeated Foo; } RBar { repeated Bar; } } so in practice this isn't a restriction. If anything, its a (very minor) api wart. As for maps, I'd say don't use them. I have, I don't think they're very useful, except for prototyping things. `repeated (string key, string value)` is just as useful for prototyping things, and you should be quick to promote things to fields. Optional fields aren't costly. I'm also in the minority who thinks that the `has_foo` methods are a smell, and everything should always be set to a default value[0]. If it really matters, one can define a maybe type via a oneof, but that should be the exception, not the norm. Most people disagree with that though, so maybe I'm crazy and you should ignore me. [0]: But if that's not the case, everything should be optional, nothing required.
  • xyzzyz 8y ago
    While oneof fields cannot be repeated, oneof fields can be arbitrary protos, so they can contain repeated fields. In other words, you can have a oneof, so in practice this isn't a restriction. If anything, its a(very minor) api wart. That's right. Moreover, oneof repeated and repated oneof are two different types: in first, you have a list of X, or a list of Y, or a list of Z etc, and in the second, you have a list such that any element can be either X, Y or Z. Both can be cleanly expressed with existing API, so I see no reason to do any special treatment.
  • aaaaaaaaaab 8y ago
    repeated(oneof(Foo, Bar)) is not the same as oneof(repeated(Foo), repeated(Bar))
    • joshuamorton 8y ago
      Correct, both of those are possible to represent as is though. Each oneof just requires and extra proto to wrap it. So you have repeated OneaofWrapper { oneof { Foo Bar } } Or what I did in my previous comment.
• xyzzy123 8y ago
  Many of the restrictions are there because you need to operate as well as you can at the intersection of all target language type systems.
• the_other_guy 8y ago
  Yet protobuf is probably the most compact, efficient and performant serialization method especially when saving bandwidth is important. I experimented with protofbuf, flatbuffers and messagepack and always found protobuf messages the most compact by a noticeable margin
  • twinkletwinkle 8y ago
    That's the core of the author's argument. Protobuffers optimize for something besides usability and maintainability, because Google cares more about incremental performance than developer-friendliness. Which is a fine thing to care about at Google's scale, but maybe others' calculations should be different.
    • sethammons 8y ago
      note that you could succinctly put in one sentence something that the author took a whole page. I feel there is something jib about compression to be said, but I'll leave that as an exercise to the reader.
    • oh_sigh 8y ago
      That doesn't seem like the author's main argument - they say: "Protobuffers were obviously built by amateurs because they offer bad solutions to widely-known and already-solved problems."
    • UncleMeat 8y ago
      That's a fine argument. But then the author decides to call the people who wrote protobuf idiots and amateurs.
  • nly 8y ago
    That really depends on the nature of your data. The space saving in protobufs really comes from its variable-length zig-zag encoding of numeric fields (including lengths for arrays and strings). Tbh, i'm surprised msgpack didn't produce smaller outputs for you, because it really should (at the cost of being slower)
• dweis 8y ago
  Hi there, I'm an actual author of Protocol Buffers :) I think Sandy's analysis would benefit from considering why Protocol Buffers behave the way they do rather than outright attacking the design because it doesn't appear to make sense from a PL-centric perspective. As with all software systems, there are a number of competing constraints that have been weighed that have led to compromises. - D P.S. I also don't believe the personal attacks to be warranted or productive.
  • lowmagnet 8y ago
    The only thing I was thinking while (half-) reading this article is there's some fundamental misunderstanding about what protobuf is for.
  • batmansmk 8y ago
    Dear D, I m very interested in Protocol Buffers. Could you explain the tradeoffs you made while designing protobuff and what would you change if you were to design it now? Cheers!
    • Cthulhu_ 8y ago
      There's been some Protobuf spinoffs that claim to be "Protobuf but better" or "Protobuf but with changes from experience", like (iirc) Cap'n Proto and FlatBuffers, and full alternatives like Thrift and Fast Buffers.
  • oh_sigh 8y ago
    I thought Kenton Varda was the author of protobufs?
    • kentonv 8y ago
      Nope, I'm not the original author -- that would be Jeff Dean and Sanjay Ghemawat (also often credited with inventing things like MapReduce, BigTable, Spanner, ...). I wrote version 2 (a complete rewrite, but largely following the original design) and open sourced it. I stopped working on Protobuf about 8 years ago. Many others who have been on the Protobuf team since can certainly call themselves "authors".
• malkia 8y ago
  I spent 2.5 years at Google, and most of what I did was pushing one protobuf from one place to another :) - and I loved it... Honestly though, you can complain all day, but some of the decisions made in the list you presented most likely come from experience (daily) not as a user of protobufs (which I simply was), but someone that had to support a plethora of compression formats, how protobufs gets stored in the different databases, proxying, etc. etc. Your "oneoff" and "repeated" might've been (I dunno really), a decision based on metrics where encoding repeated oneoff would've cost more, or made problems, more testing, or who knows what. For one, Google was, and I believe still is open to discussing all kinds of matters, and you can get on to a design doc and comment if you like, if you care, if you have something to say... You can definitely influence, and people really did talk (you can even experience this from their public design docs - like from chromium, etc.). And I love protobufs, so I guess I'm biased...
  • mer1inyx 8y ago
    I only interned at Google but I also really liked protocol buffers, somehow it seems very structured and defined to me, and better than Java. But this could just be that I haven't worked with better things... Lol
  • marssaxman 8y ago
    Wow, different strokes indeed. Discovering that most of what I was supposed to do at google was going to consist of pushing protobufs from one place to another completely destroyed my enthusiasm for working there. I hated it!
    • asdfasgasdgasdg 8y ago
      I got bad news for you but almost all programming is just moving bytes from one place to another, and occasionally multiplying, adding, subtracting, or dividing some of them. ;)
      • malkia 8y ago
        I wasn't expecting really to work on a next breakthrough algorithm, and such (although folks in my team actually worked on some incredible statistics related stuff - and this is where I understood I know nothing about statistics). Plenty for everyone, I think. I'm glad I was part of the experience! As an ex-game tools developer (back then, and back again - current job), it was quite a change for me. If I haven't learned enough, then at least seen things from quite different perspective.
• ninjakeyboard 8y ago
  I've used protocol buffers on a large project, evolving the messages over time. They are a fine choice. The tone of this article is a little less than objective which I feel discredits it a bit. But hey it made it to the front page of hacker news so...
• dkoston 8y ago
  The main point that people are missing is that experienced engineers don’t want to work with people who think like the author of this article. Protocol Buffers are not wrong, they simply have constraints, advantages, and disadvantages. No language, binary format, text format, etc is free from advantages and disadvantages. All of them have different use cases. If you are building a system where your data can be described by protobufs, it may be a good choice. If your data structures don’t mash up well and you have to manipulate them heavily, protobufs may be a bad choice. Instead of pointing out use cases where a different serialization format may be better than protobufs and use cases where protobufs are better, and why, the author is spouting dogma about how protobufs are bad for every use case. Be wary of working with developers who prefer to argue about why they hate certain technologies instead of providing useful data and ways to solve problems. You don’t always have to solve a problem just because you are aware of it but don’t go shouting from the rooftops that a technology sucks for every use case under the planet when that’s obviously not the case. The author’s opinion is more of: I don’t like protobufs. Not: protobufs are wrong
  • derefr 8y ago
    The author isn’t trying to say either of those things, really. I think their real point was something like: many companies that have an Enterprise Service Bus architecture for their pile of polyglot microservices, have a dogma for the encoding of the data flowing over the Bus. And Protobufs, though good for some use-cases, are a particularly bad dogma to be stuck with. That is, when they don’t work for a use-case, they really don’t work for that use-case—and an ESB means having to try to shove a wire format into pretty much every use-case. So “when it’s bad, it’s really bad” is a bad property for a format aimed specifically at being an enterprise’s ESB-interchange-format dogma to have. (Contrast to, say, JSON-RPC, which I’d place on the opposite end of the scale being described here. JSON-RPC is mediocre at best in terms of data fidelity, performance, etc., but it’s inoffensive—when it’s bad, it’s no worse than when it’s good. That makes it a good choice of ESB dogma... even though, in engineering terms, it sucks!)
    • ender7 8y ago
      No, the author is showing off how smart they are, and in an unprofessional and nasty manner to boot. It's kind of you to gift the article with some genuinely thoughtful conclusions, but you're doing all the work there, not the original. As the grandparent says, it's easy to poke holes in something, especially when disregarding important requirements that influenced its design. The fact that the OP can't point to any implementation of the "right" way to do things is telling.
• skylark 8y ago
  Hmmm, I suppose I objectively agree with some of the points the author made, but as someone who works with protocol buffers daily, those issues never actually come to be problematic in practice. In fact, I have nothing but positive things to say about protocol buffers and find them pleasant to work with. Definitely a step up from sending raw JSON down the wire. Granted, the application I'm working on is fairly boring/vanilla so maybe I don't feel the pain points that come from going off the beaten path.
  • ionforce 8y ago
    Maybe for the benefit of everyone could you elaborate more on your vanilla application. So we can understand the context of why your problem-free experience differs from the author's or others'.
• gubby 8y ago
  “Make all fields in a message required. This makes messages product types.” This article was clearly written by amateurs.
• moron4hire 8y ago
  There are enough voices on the net these days that we don't have to put up with this kind of attitude anymore. This sort of stuff was "just the way things were" when we were teenagers visiting forums ran by teenagers. I don't know if the guy is right or wrong, I've just got better things to do than to sift through all the vitriol to get to whatever point he is trying to make.
  • ionforce 8y ago
    I think a lot of people are still attracted to the "hostile genius" persona, or whatever. Maybe it justifies their own poor behavior.
• greesil 8y ago
  I think it was Churchill said that "protobufs are the worst form of data serialization, except for all those other forms that have been tried from time to time." I really don't understand why you would ever want to serialize data without a description of what can be in the serialized data blob. And yes, I do care about the difference between a single precision float, a double precision float, or an int64. Thank you, typing system.
• kentonv 8y ago
  Hello. I didn't invent Protocol Buffers, but I did write version 2 and was responsible for open sourcing it. I believe I am the author of the "manifesto" entitled "required considered harmful" mentioned in the footnote. Note that I mostly haven't touched Protobufs since I left Google in early 2013, but I have created Cap'n Proto since then, which I imagine this guy would criticize in similar ways. This article appears to be written by a programming language design theorist who, unfortunately, does not understand (or, perhaps, does not value) practical software engineering. Type theory is a lot of fun to think about, but being simple and elegant from a type theory perspective does not necessarily translate to real value in real systems. Protobuf has undoubtedly, empirically proven its real value in real systems, despite its admittedly large number of warts. The main thing that the author of this article does not seem to understand -- and, indeed, many PL theorists seem to miss -- is that the main challenge in real-world software engineering is not writing code but changing code once it is written and deployed. In general, type systems can be both helpful and harmful when it comes to changing code -- type systems are invaluable for detecting problems introduced by a change, but an overly-rigid type system can be a hindrance if it means common types of changes are difficult to make. This is especially true when it comes to protocols, because in a distributed system, you cannot update both sides of a protocol simultaneously. I have found that type theorists tend to promote "version negotiation" schemes where the two sides agree on one rigid protocol to follow, but this is extremely painful in practice: you end up needing to maintain parallel code paths, leading to ugly and hard-to-test code. Inevitably, developers are pushed towards hacks in order to avoid protocol changes, which makes things worse. I don't have time to address all the author's points, so let me choose a few that I think are representative of the misunderstanding. > Make all fields in a message required. This makes messages product types. > Promote oneof fields to instead be standalone data types. These are coproduct types. This seems to miss the point of optional fields. Optional fields are not primarily about nullability but about compatibility. Protobuf's single most important feature is the ability to add new fields over time while maintaining compatibility. This has proven -- in real practice, not in theory -- to be an extremely powerful way to allow protocol evolution. It allows developers to build new features with minimal work. Real-world practice has also shown that quite often, fields that originally seemed to be "required" turn out to be optional over time, hence the "required considered harmful" manifesto. In practice, you want to declare all fields optional to give yourself maximum flexibility for change. The author dismisses this later on: > What protobuffers are is permissive. They manage to not shit the bed when receiving messages from the past or from the future because they make absolutely no promises about what your data will look like. Everything is optional! But if you need it anyway, protobuffers will happily cook up and serve you something that typechecks, regardless of whether or not it's meaningful. In real world practice, the permissiveness of Protocol Buffers has proven to be a powerful way to allow for protocols to change over time. Maybe there's an amazing type system idea out there that would be even better, but I don't know what it is. Certainly the usual proposals I see seem like steps backwards. I'd love to be proven wrong, but not on the basis of perceived elegance and simplicity, but rather in real-world use. > oneof fields can't be repeated. (background: A "oneof" is essentially a tagged union -- a "sum type" for type theorists. A "repeated field" is an array.) Two things: 1. It's that way because the "oneof" pattern long-predates the "oneof" language construct. A "oneof" is actually syntax sugar for a bunch of "optional" fields where exactly one is expected to be filled in. Lots of protocols used this pattern before I added "oneof" to the language, and I wanted those protocols to be able to upgrade to the new construct without breaking compatibility. You might argue that this is a side-effect of a system evolving over time rather than being designed, and you'd be right. However, there is no such thing as a successful system which was designed perfectly upfront. All successful systems become successful by evolving, and thus you will always see this kind of wart in anything that works well. You should want a system that thinks about its existing users when creating new features, because once you adopt it, you'll be an existing user. 2. You actually do not want a oneof field to be repeated! Here's the problem: Say you have your repeated "oneof" representing an array of values where each value can be one of 10 different types. For a concrete example, let's say you're writing a parser and they represent tokens (number, identifier, string, operator, etc.). Now, at some point later on, you realize there's some additional piece of data you want to attach to every element. In our example, it could be that you now want to record the original source location (line and column number) where the token appeared. How do you make this change without breaking compatibility? Now you wish that you had defined your array as an array of messages, each containing a oneof, so that you could add a new field to that message. But because you didn't, you're probably stuck creating a parallel array to store your new field. That sucks. In every single case where you might want a repeated oneof, you always want to wrap it in a message (product type), and then repeat that. That's exactly what you can do with the existing design. The author's complaints about several other features have similar stories. > One possible argument here is that protobuffers will hold onto any information present in a message that they don't understand. In principle this means that it's nondestructive to route a message through an intermediary that doesn't understand this version of its schema. Surely that's a win, isn't it? > Granted, on paper it's a cool feature. But I've never once seen an application that will actually preserve that property. OK, well, I've worked on lots of systems -- across three different companies -- where this feature is essential.
• zoggenhoff 8y ago
  Protobuffers always seemed like an interesting approach but every time I've tried to use in a prototype I've ended up deciding against the added overhead of including them. And then I use JSON because I simply wanted versioned serialisation when messaging or persisting data.
• camdenreslink 8y ago
  If protobuffers are wrong, then I don’t want to be right.
• arountheworld 8y ago
  What I was reading? It is easy to badmouth someone without knowing the context. It is not uncommon to end up with a flawed design if you find yourself in circumstances like short deadline, uncertain requirements, lack of resources, small user base, no QA and so on that are outside of the developer means to change.
• romwell 8y ago
  Once I was faced with a task of serializing object hierarchy to a binary on disk[1], with the requirement of transparent backward compatibility (and desired forward compatibility), and the ability to be accessed from several programming languages in the future. Protobufs looked promising because of their forward compatibility promise, but it looked like they only work if you write your code from the get-go using them. Rewiring a 15-year-old codebase would be a nightmare, so that was not an option. Boost::serialization[2] looked promising, because it's non-intrusive -- but even "dumb" forward compatibility (skipping unknown fields) is not supported -- and reading its archives from other languages seems nontrivial. So I went ahead and rolled out a custom framework with C++ template trickery which was non-intrusive, wrote to SQLite (with a fixed ORM schema), and did the job. I wondered for a while if I was re-inventing the wheel, but in the end, I am glad I did it: 1. It wasn't that much work - especially compared to rewiring everything for Protobufs 2. It made me appreciate the goals and the API of Boost::serialization better - mine ended up being not too dissimilar. It was a great way to learn that - or any other- serialization API - without even looking at it! Now it's "let's see how they did ___" instead of "what does it do and why is this here". Reading this article made me glad again: partly, for not depending on Protobuf -- and partly, for it having some of the same design flaws that I have (like having difficulty telling a zero-initialized value from a value not present in the data). But the take-away for me was that sometimes, writing a custom framework is justified. Not all tools are suited for all purposes. Figuring out what one wants, and checking the list against the goals of a project can save a lot of headache. [1] Binary was a necessity because some of the data is large arrays of full-precision floating point numbers, on the scale of 10-100MB. This was scientific data, no precision loss on saving/loading was acceptable. [2]https://www.boost.org/doc/libs/1_67_0/libs/serialization/doc/index.html https://www.boost.org/doc/libs/1_67_0/libs/serialization/doc...
• yongjik 8y ago
  > By far, the biggest problem with protobuffers is their terrible type-system. Fans of Java should feel right at home with protobuffers ... Yeah right, because insulting only Protocol buffer and Google engineering won't be edgy enough for a blog post... :/
• tehlike 8y ago
  I never understood debate over technology that just works You may not prefer it, but it works. Do something better, and make people use it.
  • ionforce 8y ago
    The disagreement is over what "just works" means. Obviously the author things Protobuf isn't suitable for some use cases. They don't work in his view.
• tgsovlerkhgsel 8y ago
  > Make all fields in a message required. This makes messages product types. Now you lost one of the biggest benefits of using protobufs, the possibility to make things backwards/forwards compatible. When you add a field to the proto, a server using the new version of the proto will fail to deserialize a request from a client using an older version that's missing that field. The author mentions this later, but doesn't really address it/propose a better solution, as far as I can see.
• thrower123 8y ago
  I like protobufs. I've only ever used them as a serialization format for save and map files for some C# games that I tinker away at, but I've been pretty happy using protobuf-net[1]. File size is smaller than JSON produced by JSON.NET or using the built-in binary serializer, much less XML. And it couldn't be much easier to use. [1] https://github.com/mgravell/protobuf-net https://github.com/mgravell/protobuf-net
• rdtsc 8y ago
  > Give the ability to parameterize product and coproduct types by other types. This is "worse is better" at work. protobufs are inconsistent and types map to Java. But it's a feature because Java is a language designed for for your average programmer. It avoids words like "coproduct types". Because most people's eyes will glaze over that and their mind will think "this is too complicated, there must be something simpler, like a map or something..." > "Google is filled with the world's best engineers," and that "anything they build is, by definition, not built by amateurs. That's true too though. It's a major factor when picking tools. So protobufs are used because of Google. gRPC is used because of Google. Kubernetes because of Google. Angular because of Google. Go became popular because of Google etc. Of course there are technical advantages to those things and they are great tools, but I think quite often the reason is simply ... because it's Google.
• d--b 8y ago
  Now to the real question: what is the best alternative?
• deleted 8y ago
  [deleted]
• patrec 8y ago
  > All you've managed to do is decentralize sanity-checking logic from a well-defined boundary and push the responsibility of doing it throughout your entire codebase. One wonders if the author has practical experience with evolving a complex system composed of multiple independent services without downtime. Centralized sanity checking conflicts with transitioning services stepwise from e.g. foo.a foo.new_a by first deploying producers that create send both foo.a and foo.new_a and then phasing out foo.a after all consumers are updated. Services which communicate via some general purpose, strongly typed schema language that's still lax enough to evolve backwards-compatibly are far easier to understand, maintain and evolve in my experience than alternatives such as json rest apis, overly tied down static or completely ad hoc protocols.
  • repolfx 8y ago
    He meant "centralised in the serialisation library". I don't see how that conflicts with versioned evolution, especially because protobufs historically did verify the presence of fields. The story behind "required considered harmful" at Google is really quite shameful. I was there at the time and couldn't quite believe people were making that argument. Beyond all the logical problems with it, they were basically saying Jeff Dean didn't know what he was doing when he made that decision (except they never actually said it explicitly because the notion itself would have been subjected to ridicule). I used to like protobufs but I wouldn't use proto3. It seems to have regressed over time instead of getting better. Partly this is because the wire format is not evolvable. Protobufs do not have a "this is protobufs v1" header in them. This also explains the type system issues: Google cannot/will not evolve the wire format so each new version of protobufs is just different tooling and APIs over it.
    • patrec 8y ago
      The argument from Jeff Dean authority suffers from the fact that no matter how smart you are, getting some trade-offs right will require empirical validation. Just having multiple end-points or doing version negotiation might work fine in some cases, but I can't see a particular good way field-replacement evolution would work with exclusively required fields in general, What's your suggested approach?
• mmjaa 8y ago
  I'm using protobufs for an embedded product, and I think they're awesome. Not the solution to everything - certainly - but definitely a highly useful tool for the right circumstances. In my case, I have a single .proto file that serves as the input to two software systems: an embedded OS running on ESP8266 hardware, and the web interface served up by that OS to edit OS-specific features/variables/etc. So, I define in one file, my types that I want shared both with the embedded OS, and with the web interface. On compile, both systems get their relevant generated code, and both systems are therefore able to communicate easily with each other. The user can use the embedded OS to change values coded in the protobuf, and the web interface that the system serves up immediately knows about the change. The web interface, by the way, generates the UI based on the protobuf declarations - meaning that adding a new var to the project is a simple matter of editing the .proto file, and doing a full clean build. Very, very nice: one change, multiple systems updated, and it produces an auto-generated UI for the new vars. This kind of interoperability is very key - I'm sure there are alternative ways of doing this, but I can't think of anything as smooth and easy to integrate as protobufs have been .. maybe if we'd put a Lua VM in the mix somewhere, this'd be better, but .. protobufs are pretty tight.
• perpetualcrayon 8y ago
  I think attacking a widely used open source piece of software rather than trying to help improve it behind the scenes without drawing attention to yourself will always be less work.
• Sir_Cmpwn 8y ago
  I disagree severely with large swaths of this article. The biggest problem is here: >Of course, the actual serialization logic is allowed to do something smarter than pushing linked-lists across the network---after all, implementations and semantics don't need to align one-to-one. I like that I can read a pb dump in hex and have it map quite closely to the protocol definition, rather than being viewed through heaping piles of abstractions. This is only going to make it more confusing and more work to use, with the only gain being to make some functional programmer feel squishy about their beautiful type system. I largely agree with the "The Lie of Backwards- and Forwards-Compatibility" section. You can feed /dev/urandom into nearly any protobuf definition and it'll happily give you nonsense back, no errors. I understand why this is the case, and I'm not sure it should change, but it's bitten me several times.
• TeMPOraL 8y ago
  > Protobuffers correspond to the data you want to send over the wire, which is often related but not identical to the actual data the application would like to work with.(...) > Option 1 is clearly the "right" solution, but its untenable with protobuffers. The language isn't powerful enough to encode types that can perform double-duty as both wire and application formats. Which means you'd need to write a completely separate datatype, evolve it synchronously with the protobuffer, and explicitly write serialization code between the two. I'd say, get used to this. This is exactly the same story you have with ORM/ActiveRecord. You either treat it as a different layer and write the translation code that's needed, or mix it with internal logic and pay the price of messy codebase later. The problem isn't protobuff(or ORM)-specific. It's just a consequence of the fact that your internal business model and the data you want to save/send are usually two different things.
  • ionforce 8y ago
    The worst is when people deny that is the truth. "Then model is like 80% the same, just use the same class!"
    • TeMPOraL 8y ago
      Yeah. It's not always obvious at the beginning, though. In my second job, I didn't notice this until it grew to the point we couldn't really implement anything new without spending 90% of the time on the spaghetti the business logic became. My excuse is that we were all new at this kind of work, but the lesson is clear: just don't do that.
• ionforce 8y ago
  One of the most surprising features of Proto mentioned in this thread (at least for me/my background) is the desire to forward on a message you receive. Possibly a message you can't read/parse, or don't want to. And you want to forward that whole payload to the next person. What kind of systems have this architecture? Why would you forward on a message and not know its contents 100%?
  • killerstorm 8y ago
    > What kind of systems have this architecture? The Web, for example. > Why would you forward on a message and not know its contents 100%? In the web architecture you can observe following cases: * proxying, which itself is an umbrella term for a lot of different things, such as: * request routing * filtering * access control * logging * caching * load balancing For example, a very typical scenario for load balancing is "sticky sessions". Basically, a reverse proxy (e.g. nginx) needs to inspect a HTTP request and analyze certain headers to find the server to forward request to (which already has user's session). Note that nginx can just pass headers without understanding what they mean. And it simply cannot understand the meaning of query part of URL or message body. So, apparently, protobuf gives you a lot of HTTP's flexibility in a binary RPC format. So e.g. you can make a cache server which will cache responses without understanding them, so you can evolve your backend without updating cache server each time you add a new field to a response. What I don't understand, however, is why they couldn't control that behavior using compiler options. Sometimes you want to be lax with validation, sometimes you don't. Why not give programmer a choice...
• deleted 8y ago
  [deleted]
• cromwellian 8y ago
  I’ll give you one example where the protobuf design helped out a lot: GWT Protobuffers. The GWT Compiler has a version of Protobufs that transpiles all of the protos into Overlay types of naked JSON objects. You get a statically typed Java interface, but underneath it a raw JSON object, and deserializing the proto is basically JSON.parse() + a cast. e.g. something like this MyProto foo = MessageDeserializer.deserialize(serializedProto); console.log(foo.getFieldFoo()); compiles to this JS: var foo = JSON.parse(serializedProto); console.log(foo[1]);
• emmanueloga_ 8y ago
  Agree with other ppl here, an unnecessarily violent rant, and the point the author makes is kind of lost in the title "Protobuffers Are Wrong". What they really mean is "I think Protobuffers type system could be better". I recently saw a talk by Rich Hickey about Effective Programs [1]. The talk explains why Hickey favors dynamic types, with EDN [2] and transit [3] proposed as an alternative to more statically typed data exchange formats like protobufs and less structured ones like Json (the talk explains the reasons, nearing the end I think). 1: https://www.youtube.com/watch?v=2V1FtfBDsLU https://www.youtube.com/watch?v=2V1FtfBDsLU 2: https://github.com/edn-format/edn https://github.com/edn-format/edn 3: https://github.com/cognitect/transit-format https://github.com/cognitect/transit-format