4 ms·

Sadly, capnproto's awesome RPC system doesn't appear to support streaming, though I do believe gRPC supports FlatBuffers.

kentonv 7y ago
> Sadly, capnproto's awesome RPC system doesn't appear to support streaming Sure it does. You can implement "streaming" in Cap'n Proto by introducing a callback object, and making one RPC call for each item / chunk in the stream. In Cap'n Proto, "streaming" is just a design pattern, not something that needs to be explicitly built in, because Cap'n Proto is inherently far more expressive than gRPC. That is, you can define a type like: interface Stream(T) { write @0 (item :T); end @1 (); # signal successful end of stream } Then you can define streaming methods like: streamUp @0 (...params...) -> (stream :Stream(T), ...results...) # Method with client->server stream. streamDown @0 (...params..., stream :Stream(T)) -> (...results...) # Method with server->client stream. Admittedly, this technique has the problem that the application has to do its own flow control -- it has to keep multiple calls in-flight to saturate the connection, but needs to place a cap on the number of calls in order to avoid excess buffering. This is doable, but somewhat inconvenient. So I am actually in the process of extending the implementation to make this logic built-in: https://github.com/capnproto/capnproto/pull/825 https://github.com/capnproto/capnproto/pull/825 Note that PR doesn't add anything new to the RPC protocol; it just provides helpers to tell the app how many concurrent calls to make.
- anderspitman 7y ago
  Interesting. I do tend to favor protocols that are a bit lower level and more flexible for composing higher-level functionality. Though this does sound pretty complicated to implement using only what capnproto offers right now. Would there be a way to jury rig "request(n)" backpressure as described in reactive streams[0] (also implemented by RSocket[1]) on top of capnproto? That's what I'm using for omnistreams and it's proven very simple to reason implement and reason about. [0] https://github.com/reactive-streams/reactive-streams-jvm https://github.com/reactive-streams/reactive-streams-jvm [1] http://rsocket.io/ http://rsocket.io/ [2] https://github.com/omnistreams/omnistreams-spec https://github.com/omnistreams/omnistreams-spec
  - anderspitman 7y ago
    Actually the more I think about it I don't think it would work, since request(n) assumes the producer can send multiple messages for each request.

kentonv 7y ago

> Sadly, capnproto's awesome RPC system doesn't appear to support streaming Sure it does. You can implement "streaming" in Cap'n Proto by introducing a callback object, and making one RPC call for each item / chunk in the stream. In Cap'n Proto, "streaming" is just a design pattern, not something that needs to be explicitly built in, because Cap'n Proto is inherently far more expressive than gRPC. That is, you can define a type like: interface Stream(T) { write @0 (item :T); end @1 (); # signal successful end of stream } Then you can define streaming methods like: streamUp @0 (...params...) -> (stream :Stream(T), ...results...) # Method with client->server stream. streamDown @0 (...params..., stream :Stream(T)) -> (...results...) # Method with server->client stream. Admittedly, this technique has the problem that the application has to do its own flow control -- it has to keep multiple calls in-flight to saturate the connection, but needs to place a cap on the number of calls in order to avoid excess buffering. This is doable, but somewhat inconvenient. So I am actually in the process of extending the implementation to make this logic built-in: https://github.com/capnproto/capnproto/pull/825 https://github.com/capnproto/capnproto/pull/825 Note that PR doesn't add anything new to the RPC protocol; it just provides helpers to tell the app how many concurrent calls to make.

anderspitman 7y ago
Interesting. I do tend to favor protocols that are a bit lower level and more flexible for composing higher-level functionality. Though this does sound pretty complicated to implement using only what capnproto offers right now. Would there be a way to jury rig "request(n)" backpressure as described in reactive streams[0] (also implemented by RSocket[1]) on top of capnproto? That's what I'm using for omnistreams and it's proven very simple to reason implement and reason about. [0] https://github.com/reactive-streams/reactive-streams-jvm https://github.com/reactive-streams/reactive-streams-jvm [1] http://rsocket.io/ http://rsocket.io/ [2] https://github.com/omnistreams/omnistreams-spec https://github.com/omnistreams/omnistreams-spec
- anderspitman 7y ago
  Actually the more I think about it I don't think it would work, since request(n) assumes the producer can send multiple messages for each request.