5 ms·

Hello. I didn't invent Protocol Buffers, but I did write version 2 and was responsible for open sourcing it. I believe I am the author of the "manifesto" entitl

Hello. I didn't invent Protocol Buffers, but I did write version 2 and was responsible for open sourcing it. I believe I am the author of the "manifesto" entitled "required considered harmful" mentioned in the footnote. Note that I mostly haven't touched Protobufs since I left Google in early 2013, but I have created Cap'n Proto since then, which I imagine this guy would criticize in similar ways. This article appears to be written by a programming language design theorist who, unfortunately, does not understand (or, perhaps, does not value) practical software engineering. Type theory is a lot of fun to think about, but being simple and elegant from a type theory perspective does not necessarily translate to real value in real systems. Protobuf has undoubtedly, empirically proven its real value in real systems, despite its admittedly large number of warts. The main thing that the author of this article does not seem to understand -- and, indeed, many PL theorists seem to miss -- is that the main challenge in real-world software engineering is not writing code but changing code once it is written and deployed. In general, type systems can be both helpful and harmful when it comes to changing code -- type systems are invaluable for detecting problems introduced by a change, but an overly-rigid type system can be a hindrance if it means common types of changes are difficult to make. This is especially true when it comes to protocols, because in a distributed system, you cannot update both sides of a protocol simultaneously. I have found that type theorists tend to promote "version negotiation" schemes where the two sides agree on one rigid protocol to follow, but this is extremely painful in practice: you end up needing to maintain parallel code paths, leading to ugly and hard-to-test code. Inevitably, developers are pushed towards hacks in order to avoid protocol changes, which makes things worse. I don't have time to address all the author's points, so let me choose a few that I think are representative of the misunderstanding. > Make all fields in a message required. This makes messages product types. > Promote oneof fields to instead be standalone data types. These are coproduct types. This seems to miss the point of optional fields. Optional fields are not primarily about nullability but about compatibility. Protobuf's single most important feature is the ability to add new fields over time while maintaining compatibility. This has proven -- in real practice, not in theory -- to be an extremely powerful way to allow protocol evolution. It allows developers to build new features with minimal work. Real-world practice has also shown that quite often, fields that originally seemed to be "required" turn out to be optional over time, hence the "required considered harmful" manifesto. In practice, you want to declare all fields optional to give yourself maximum flexibility for change. The author dismisses this later on: > What protobuffers are is permissive. They manage to not shit the bed when receiving messages from the past or from the future because they make absolutely no promises about what your data will look like. Everything is optional! But if you need it anyway, protobuffers will happily cook up and serve you something that typechecks, regardless of whether or not it's meaningful. In real world practice, the permissiveness of Protocol Buffers has proven to be a powerful way to allow for protocols to change over time. Maybe there's an amazing type system idea out there that would be even better, but I don't know what it is. Certainly the usual proposals I see seem like steps backwards. I'd love to be proven wrong, but not on the basis of perceived elegance and simplicity, but rather in real-world use. > oneof fields can't be repeated. (background: A "oneof" is essentially a tagged union -- a "sum type" for type theorists. A "repeated field" is an array.) Two things: 1. It's that way because the "oneof" pattern long-predates the "oneof" language construct. A "oneof" is actually syntax sugar for a bunch of "optional" fields where exactly one is expected to be filled in. Lots of protocols used this pattern before I added "oneof" to the language, and I wanted those protocols to be able to upgrade to the new construct without breaking compatibility. You might argue that this is a side-effect of a system evolving over time rather than being designed, and you'd be right. However, there is no such thing as a successful system which was designed perfectly upfront. All successful systems become successful by evolving, and thus you will always see this kind of wart in anything that works well. You should want a system that thinks about its existing users when creating new features, because once you adopt it, you'll be an existing user. 2. You actually do not want a oneof field to be repeated! Here's the problem: Say you have your repeated "oneof" representing an array of values where each value can be one of 10 different types. For a concrete example, let's say you're writing a parser and they represent tokens (number, identifier, string, operator, etc.). Now, at some point later on, you realize there's some additional piece of data you want to attach to every element. In our example, it could be that you now want to record the original source location (line and column number) where the token appeared. How do you make this change without breaking compatibility? Now you wish that you had defined your array as an array of messages, each containing a oneof, so that you could add a new field to that message. But because you didn't, you're probably stuck creating a parallel array to store your new field. That sucks. In every single case where you might want a repeated oneof, you always want to wrap it in a message (product type), and then repeat that. That's exactly what you can do with the existing design. The author's complaints about several other features have similar stories. > One possible argument here is that protobuffers will hold onto any information present in a message that they don't understand. In principle this means that it's nondestructive to route a message through an intermediary that doesn't understand this version of its schema. Surely that's a win, isn't it? > Granted, on paper it's a cool feature. But I've never once seen an application that will actually preserve that property. OK, well, I've worked on lots of systems -- across three different companies -- where this feature is essential.

deleted 8y ago
[deleted]
darkkindness 8y ago
> Maybe there's an amazing type system idea out there that would be even better, but I don't know what it is. Certainly the usual proposals I see seem like steps backwards. I'd love to be proven wrong, but not on the basis of perceived elegance and simplicity, but rather in real-world use. Care to elaborate on what these usual proposals are and why they're backwards? Right now I'm thinking of 'row polymorphism', which, to my understanding, is just permissiveness cooked into the type system — you get to specify types that say "I may or may not have other fields, but I definitely have a 'name' and 'email' field" for example.
- kentonv 8y ago
  "Row polymorphism" sounds like exactly what protobuf does, which I think is the right answer. As I mentioned, I've heard people argue that the client and server should pre-negotiate exactly which version of the protocol they are using, and then that version should have a rigidly-defined schema. It may have been unfair to suggest that this is a common opinion among type theorists -- I don't have that many data points.
  - jules 8y ago
    Row polymorphism is a more principled approach that makes the type theorists happy. The point of row polymorphism is not just that you can say "I may or may not have other fields, but I definitely have a 'name' and 'email' field", but that the extra unknown fields have a name that can be used to constrain other types. For example, you can have this function type: {name: string, email: string, &rest} -> {name: string, email: string, &rest} This is different than this function type: {name: string, email: string, &rest} -> {name: string, email: string} The first type can act as a forward compatible passthrough because it says that whatever extra fields are in the input are also in the output. The second type promises that its output only has name and email fields. The same applies to variant types: you can say that a variant has option A, option B, and other options rest: (A | B | &rest) -> (A | B | &rest) vs (A | B | &rest) -> (A | B) Functions of these types are polymorphic in the schema. For instance, the type (A | B | &rest) -> (A | B) can be instantiated with rest = (C | D) to get (A | B | C | D) -> (A | B). So row types are fundamentally different in that it's the functions that that explicitly deal with multiple possible schemas. At the end of the day all of the rest variables get instantiated with explicit types to get a concrete instantiation of a function in which all the rest variables have been replaced by concrete types. A serialisation library could use this to do version negotiation automatically. After it has negotiated a version, the library instantiates the functions so that the rest type variables get instantiated with the actual concrete version of the schema. Different languages implement row types in different ways. Some compile row types in a way akin to Java's type erasure. They compile a single version of each polymorphic function that can be used regardless of how the rest parameters get instantiated. Some compile row types in a way akin to C#'s generics or C++ templates: they compile a separate version for each instantiation. The advantage of the latter is that the data representation can be optimised with full knowledge of the concrete schema. If we have a function of type {name: string, email: string, &rest} -> {name: string, email: string, &rest} instantiated with rest = {age: int} then that compiles to a version of type {name: string, email: string, age: int} -> {name: string, email: string, age: int}. This compiles to faster code because the compiler statically knows the size of the thing. In a client-server situation you wouldn't know the schema of rest until run time, so you'd need either have a JIT compiler that can compile new versions at run time, or specify a fixed number of options for rest at compile time. To update a client-server application you'd need to recompile both the client and server with support for the new version. That's not nice but it does not have a chicken and egg problem because both the client and server still support the old version too. TL;DR: with row types schemas are always rigidly defined, it's the functions that can handle multiple schemas.

deleted 8y ago

[deleted]

darkkindness 8y ago

> Maybe there's an amazing type system idea out there that would be even better, but I don't know what it is. Certainly the usual proposals I see seem like steps backwards. I'd love to be proven wrong, but not on the basis of perceived elegance and simplicity, but rather in real-world use. Care to elaborate on what these usual proposals are and why they're backwards? Right now I'm thinking of 'row polymorphism', which, to my understanding, is just permissiveness cooked into the type system — you get to specify types that say "I may or may not have other fields, but I definitely have a 'name' and 'email' field" for example.

kentonv 8y ago
"Row polymorphism" sounds like exactly what protobuf does, which I think is the right answer. As I mentioned, I've heard people argue that the client and server should pre-negotiate exactly which version of the protocol they are using, and then that version should have a rigidly-defined schema. It may have been unfair to suggest that this is a common opinion among type theorists -- I don't have that many data points.
- jules 8y ago
  Row polymorphism is a more principled approach that makes the type theorists happy. The point of row polymorphism is not just that you can say "I may or may not have other fields, but I definitely have a 'name' and 'email' field", but that the extra unknown fields have a name that can be used to constrain other types. For example, you can have this function type: {name: string, email: string, &rest} -> {name: string, email: string, &rest} This is different than this function type: {name: string, email: string, &rest} -> {name: string, email: string} The first type can act as a forward compatible passthrough because it says that whatever extra fields are in the input are also in the output. The second type promises that its output only has name and email fields. The same applies to variant types: you can say that a variant has option A, option B, and other options rest: (A | B | &rest) -> (A | B | &rest) vs (A | B | &rest) -> (A | B) Functions of these types are polymorphic in the schema. For instance, the type (A | B | &rest) -> (A | B) can be instantiated with rest = (C | D) to get (A | B | C | D) -> (A | B). So row types are fundamentally different in that it's the functions that that explicitly deal with multiple possible schemas. At the end of the day all of the rest variables get instantiated with explicit types to get a concrete instantiation of a function in which all the rest variables have been replaced by concrete types. A serialisation library could use this to do version negotiation automatically. After it has negotiated a version, the library instantiates the functions so that the rest type variables get instantiated with the actual concrete version of the schema. Different languages implement row types in different ways. Some compile row types in a way akin to Java's type erasure. They compile a single version of each polymorphic function that can be used regardless of how the rest parameters get instantiated. Some compile row types in a way akin to C#'s generics or C++ templates: they compile a separate version for each instantiation. The advantage of the latter is that the data representation can be optimised with full knowledge of the concrete schema. If we have a function of type {name: string, email: string, &rest} -> {name: string, email: string, &rest} instantiated with rest = {age: int} then that compiles to a version of type {name: string, email: string, age: int} -> {name: string, email: string, age: int}. This compiles to faster code because the compiler statically knows the size of the thing. In a client-server situation you wouldn't know the schema of rest until run time, so you'd need either have a JIT compiler that can compile new versions at run time, or specify a fixed number of options for rest at compile time. To update a client-server application you'd need to recompile both the client and server with support for the new version. That's not nice but it does not have a chicken and egg problem because both the client and server still support the old version too. TL;DR: with row types schemas are always rigidly defined, it's the functions that can handle multiple schemas.