3 ms·

I am switching all our endpoints from paging to streaming. Currently for the paging endpoints some of the clients are opening hundreds of connections to slurp

I am switching all our endpoints from paging to streaming. Currently for the paging endpoints some of the clients are opening hundreds of connections to slurp tens of gigabytes of data as fast as possible. Switching these to streaming directly from database improved the performance of a single request about 100 times as clients can use 10 times less connections and still fetch whatever they need 10 times faster than before. As to drawbacks, the biggest one is additional effort needed to educate people on proper streaming, design of APIs, implementation and correct use of Reactor (ReactiveX implementation for Java). There are couple of technical drawbacks. Resuming a streaming response is a little bit more involved. My ideal solution, whenever I can, is to produce the report in order of some database index and if the call fails, have the client submit the ID of last received response when retrying the call. Achieving optimum performance requires a bit of thought and planning. A lot of internal processes require individual things to be batched together (we try to never direct calls to database for single items, opting to process everything in batches of at least 100 or 1000). When the endpoint has functionality that needs to be executed multiple times concurrently to get needed performance, sometimes an internal function also has its own concurrency because it needs to run something else multiple times in parallel to achieve its target internally. This can quickly cause the developer to loose track of how much stuff is running concurrently. The solution to this is that rather than components returning Flux<Something> typically they return Flux<Flux<Something>>. This can be thought as function saying "my result can be fetched from a collection of multiple substreams of results, and you can decide however many you are comfortable running concurrently". This allows to give the calling function the control of concurrency at the cost of little bit of mental yoga.

cle 5y ago
What about the other drawbacks mentioned in the article, like connection draining and error messaging?
- lmilcin 5y ago
  Well, connection draining is no harder than with normal connections. I think of this problem following way: you already have to design your system to be able to handle network faults and node crashes. So just kill the node and rely on existing solution to handle the error. This also gives the opportunity to test your fault handling logic. Error messaging is a little bit more complicated. At first sight, in normal request-response you get either successful response or an error. But if you look a little bit closer, in fact you get 200 response header followed by response data, which may itself fail to be transmitted. So technically you already need to be able to resolve an error that happens after you have received 200 header. In case of streaming response the difference is not in how the response is being sent to the client but how the client is processing it. The client has option to wait for entire response (header followed by entire stream of data) and only start processing the response after entire response has been received. This is no different from a normal, non-streamed request response. The client also has option to start processing the response while it is being received. The response does not need to be streamed from the server nor it needs any kind of streaming protocol. I have been stream-processing XML responses two decades ago with no special streaming support. One solution I use is that is that the client architecture treats this data as speculative meaning that an error can undo entire batch received. Another solution is that the design is such that the data makes sense even partially received, the processing of response is idempotent and the client will be asking, ad infinitum, to receive the rest of the response until the call succeeds. This of course has its own reliability problems (what if the error is permanent?)

cle 5y ago

What about the other drawbacks mentioned in the article, like connection draining and error messaging?

lmilcin 5y ago
Well, connection draining is no harder than with normal connections. I think of this problem following way: you already have to design your system to be able to handle network faults and node crashes. So just kill the node and rely on existing solution to handle the error. This also gives the opportunity to test your fault handling logic. Error messaging is a little bit more complicated. At first sight, in normal request-response you get either successful response or an error. But if you look a little bit closer, in fact you get 200 response header followed by response data, which may itself fail to be transmitted. So technically you already need to be able to resolve an error that happens after you have received 200 header. In case of streaming response the difference is not in how the response is being sent to the client but how the client is processing it. The client has option to wait for entire response (header followed by entire stream of data) and only start processing the response after entire response has been received. This is no different from a normal, non-streamed request response. The client also has option to start processing the response while it is being received. The response does not need to be streamed from the server nor it needs any kind of streaming protocol. I have been stream-processing XML responses two decades ago with no special streaming support. One solution I use is that is that the client architecture treats this data as speculative meaning that an error can undo entire batch received. Another solution is that the design is such that the data makes sense even partially received, the processing of response is idempotent and the client will be asking, ad infinitum, to receive the rest of the response until the call succeeds. This of course has its own reliability problems (what if the error is permanent?)