3 ms·

> It wasn't realistic to define a boundary between what each middleman might need and what they wouldn't need, and somehow push the "not needed" part into neste

> It wasn't realistic to define a boundary between what each middleman might need and what they wouldn't need, and somehow push the "not needed" part into nested encoded blobs. This is an interesting argument that I would like to see more elaboration on, because that's the obvious solution. Effectively you're building a pipeline of data processors and each stage in the pipeline reads its own information and then passes along a payload with the rest of the information to the next stage. This would preserve full static typing with required fields, but I can see how it might inhibit some forms of dynamic instrumentation, eg. turning verbose logging on/off might dynamically reconfigure the pipeline, which would affect all upstream producers if they're wrapping messages for downstream consumers. If this were a programming language I would immediately think of row typing to specify the parts that each stage depends on while being agnostic about the rest of the content, but I'm not sure how that might work for a serialization format. Effectively, you're pulling out a typed "view" over the underlying data that contains offsets to the underlying fields (this is the dictionary-passing transform as found in Haskell).

mike_hearn 3y ago
It's easier to understand in context - some services (iirc web search but it might have been ads or something else very core) had kept adding fields to some core protobufs for years and years. It made sense, was the path of least resistance etc, but inevitably some of those fields became obsolete and they wanted to remove them but found it was hard, because every program that did anything with web search was deserializing these structures. Truly generic middleware like RPC balancers did what you are saying, but there were also a lot of service specific "middlemen" which did need to look at parts of these mega-structures. Now due to how protobufs work, you can do what you suggest and "cast" a byte stream to multiple different types, so they could have defined subsets of the overall structures and maybe they did, I don't remember, but the issue then is code duplication. You end up defining the same structures multiple times, just as subsets. With a more advanced type system you can eliminate the duplication, but there was a strong reluctance to add features to protobufs.
kentonv 3y ago
The particular piece of infrastructure I worked on sat in the middle of the search pipeline, between the front-end that served HTML web pages, and the back-end index. This middle piece would request search results from the back-end, tweak them in a bunch of ways, and forward them on. These "tweaks" could be just about anything. Like: "You searched for Jaguar, but I don't know if you meant the cat or the car. The index decided that pages about the car rank higher so the first three pages of results are about the car. I'm going to pull some results about the cat from page 4 and put them near the top so just in case that's what you really wanted, you'll find it." Google Search, at least when I worked on it, was composed of a huge number of such tweaks. People were constantly proposing them, testing if they led to an improvement, and shipping them if they do. For a variety of reasons, our middleman server was a great place to implement certain kinds of tweaks. But what kinds of information are needed for these "tweaks"? Could be anything! It's a general-purpose platform. Search results were annotated with all kinds of crazy information, and any piece of information might be useful in implementing some sort of middleman tweak at some point. So you couldn't very well say upfront "OK, we're going to put all the info that is only for the frontend into the special 'frontend blob' that doesn't get parsed by the middleman", because you have no idea what fields are only needed by the frontend. In fact, that set would change over time. > If this were a programming language I would immediately think of row typing to specify the parts that each stage depends on while being agnostic about the rest of the content Indeed, perhaps one could develop an elaborate system where in the schemas, we could annotate certain fields as being relevant to certain servers. Anywhere else, those fields would be unavailable (but passed through without modification or validation). If you needed the fields in a new place, you change the annotations. But that sounds... complicated to design and cumbersome to maintain the annotations. Simply banning required fields solved the problem for us, and everything else just worked.
- naasking 3y ago
  > Indeed, perhaps one could develop an elaborate system where in the schemas, we could annotate certain fields as being relevant to certain servers. Anywhere else, those fields would be unavailable (but passed through without modification or validation). If you needed the fields in a new place, you change the annotations. I don't think it has to be elaborate. What I was thinking was something more like, in pseudo-C#: // the framework's general channel type from which messages are read public interface IChannel { T Read<T>() where T : interface; } // clients declare the interface they operate on: public interface IClientFields { public int Foo { get; set; } public string? Name { get; set; } } ... // client middleware function Task MiddlewareFn(IChannel chan) { var client = chan.Read<IClientFields>(); ... // do something with client before resuming at next stage } The client's interface type T must simply be a structural subtype of the underlying message type. As long as the underlying format is somewhat self-descriptive with a name and type map, you can perform the necessary checking that only applies locally to the client. Nothing fancy, and the required fields that client cares about are still there and the rest are ignored because they're never referenced. This could return an interface that contains a series of offsets into the data stream, which I believe is how capnproto already works.

mike_hearn 3y ago

It's easier to understand in context - some services (iirc web search but it might have been ads or something else very core) had kept adding fields to some core protobufs for years and years. It made sense, was the path of least resistance etc, but inevitably some of those fields became obsolete and they wanted to remove them but found it was hard, because every program that did anything with web search was deserializing these structures. Truly generic middleware like RPC balancers did what you are saying, but there were also a lot of service specific "middlemen" which did need to look at parts of these mega-structures. Now due to how protobufs work, you can do what you suggest and "cast" a byte stream to multiple different types, so they could have defined subsets of the overall structures and maybe they did, I don't remember, but the issue then is code duplication. You end up defining the same structures multiple times, just as subsets. With a more advanced type system you can eliminate the duplication, but there was a strong reluctance to add features to protobufs.

kentonv 3y ago

The particular piece of infrastructure I worked on sat in the middle of the search pipeline, between the front-end that served HTML web pages, and the back-end index. This middle piece would request search results from the back-end, tweak them in a bunch of ways, and forward them on. These "tweaks" could be just about anything. Like: "You searched for Jaguar, but I don't know if you meant the cat or the car. The index decided that pages about the car rank higher so the first three pages of results are about the car. I'm going to pull some results about the cat from page 4 and put them near the top so just in case that's what you really wanted, you'll find it." Google Search, at least when I worked on it, was composed of a huge number of such tweaks. People were constantly proposing them, testing if they led to an improvement, and shipping them if they do. For a variety of reasons, our middleman server was a great place to implement certain kinds of tweaks. But what kinds of information are needed for these "tweaks"? Could be anything! It's a general-purpose platform. Search results were annotated with all kinds of crazy information, and any piece of information might be useful in implementing some sort of middleman tweak at some point. So you couldn't very well say upfront "OK, we're going to put all the info that is only for the frontend into the special 'frontend blob' that doesn't get parsed by the middleman", because you have no idea what fields are only needed by the frontend. In fact, that set would change over time. > If this were a programming language I would immediately think of row typing to specify the parts that each stage depends on while being agnostic about the rest of the content Indeed, perhaps one could develop an elaborate system where in the schemas, we could annotate certain fields as being relevant to certain servers. Anywhere else, those fields would be unavailable (but passed through without modification or validation). If you needed the fields in a new place, you change the annotations. But that sounds... complicated to design and cumbersome to maintain the annotations. Simply banning required fields solved the problem for us, and everything else just worked.

naasking 3y ago
> Indeed, perhaps one could develop an elaborate system where in the schemas, we could annotate certain fields as being relevant to certain servers. Anywhere else, those fields would be unavailable (but passed through without modification or validation). If you needed the fields in a new place, you change the annotations. I don't think it has to be elaborate. What I was thinking was something more like, in pseudo-C#: // the framework's general channel type from which messages are read public interface IChannel { T Read<T>() where T : interface; } // clients declare the interface they operate on: public interface IClientFields { public int Foo { get; set; } public string? Name { get; set; } } ... // client middleware function Task MiddlewareFn(IChannel chan) { var client = chan.Read<IClientFields>(); ... // do something with client before resuming at next stage } The client's interface type T must simply be a structural subtype of the underlying message type. As long as the underlying format is somewhat self-descriptive with a name and type map, you can perform the necessary checking that only applies locally to the client. Nothing fancy, and the required fields that client cares about are still there and the rest are ignored because they're never referenced. This could return an interface that contains a series of offsets into the data stream, which I believe is how capnproto already works.