3 ms·
Thanks for that link! We've been heavily inspired by the work out of TU Berlin on Scotty (https://tu-berlin-dima.github.io/scotty-window-processor/ https://tu-b
by necubi 3y ago
Thanks for that link! We've been heavily inspired by the work out of TU Berlin on Scotty (https://tu-berlin-dima.github.io/scotty-window-processor/ https://tu-berlin-dima.github.io/scotty-window-processor/) but exciting to see more work on this problem. Our efficient window handling is a big area of improvement over what Flink is able to do right now.
For Yep, Flink has to do this themselves because they multiplex. My sense is that when Flink was originally designed in ~2012 Linux didn't support large numbers of TCP connections well (and memory was more of an issue), so they built a network stack that multiplexes many logical streams on top of a single TCP connection.
My (not terribly informed) opinion is that TCP behaviors and semantics are not necessarily optimal for these systems and you get benefits from allowing your systems' scheduler to control prioritization rather than handing it off the kernel.
There are also a new class of datacenter-oriented protocols like Homa (https://homa-transport.atlassian.net/wiki/spaces/HOMA/overview https://homa-transport.atlassian.net/wiki/spaces/HOMA/overvi...) that I think are pretty interesting to solve this class of problems.