5 ms·

Large amounts of Google's internal RPC traffic goes over a custom internal framework called Pony Express, which is optimized to work on the internal cluster fab

by dgacmu 4y ago

Large amounts of Google's internal RPC traffic goes over a custom internal framework called Pony Express, which is optimized to work on the internal cluster fabric.

rektide 4y ago
Well that's accidentally hella unfortunate. I've been assuming eventually gRPC over HTTP3[1] would get some traction from big camps. But if Google is using their own internal transport, it seems highly unlikely that these main authors of gRPC are ever going to care. This blog post talks about how many of the features of HOMA are already present in existing specs like QUIC/HTTP3. I'd expect many of the wins in Ousterhout's HOMA benchmarks could probably be replicated elsewhere, with better transports. [1] https://github.com/grpc/grpc/issues/19126 https://github.com/grpc/grpc/issues/19126
- vlovich123 4y ago
  They would still care about it to do client to first Google server handling the request
  - rektide 4y ago
    Alas the web standards world has been a trashfire useless pile of nothing at giving webdevs any use of http2 or http3. Push support for fetch was one of the first issues opened[1], and summarily ignored by nearly every browser & spec author & never made at all accessible, right up until modern times where the browsers said, oh hey, we never really made this feature available to webdevs, but since you all havent used it anyways, we're removing Push from the browser. SMDH. Monstrous shit. In short, the browser failed to make basic modern http usable by anyone so there's no hope of getting off the janky ugly grpc-web trashfire & making grpc from the client just work, like it always should have been able to do. Pathetic pathetic showing by the browsers here. Beyond neglect, how they mishandled http2 and http2 and gave webdevs access to almost none of what the most exciting new thing on the www was. Huge huge miss. Incredible bullshit. Some day we'll be running http3 over webtransport & fixing their wrongs. Thia is such a bullshit long workaround, such a pathetic way to handle the browser getting nowhere with new http standards, end running around their utte unmoving complete inability to advance at all. But we'll start to actually use http intensively again, soon, in spite of the browser & standards community being such ridiculous & farcical impedances against using http2 and http3 at all. What a tragic shit show of uselessness it's been, trying ro actually enjoy the new http standards; resistance on all fronts to real usage. [1] https://github.com/whatwg/fetch/issues/51 https://github.com/whatwg/fetch/issues/51
    - vlovich123 4y ago
      You’re conflating very different things I think. HTTP/push requires browser and server support. Multiple experiments by multiple parties have shown it’s difficult to make work well regardless of a user facing API. It’s not just Google either - I recall multiple failed attempts to get it to work well. It was an interesting idea that never panned out. I think the summary by my peers on the cache team at Cloudflare outlines why server push failed [1] and I know those people first hand to be very thoughtful and principled people without a hidden agenda other than making the web very fast. To be clear I have no agenda or strong opinion either way and I could be misinformed, I just didn’t see any string evidence for benefit and lots of evidence that it is a net harm (+ maintenance effort). grpc-web will require very little work to adopt http/3 and the benefit is pretty obvious. The hard part will be if there’s http/1 servers in the way that have an impedance mismatch. Still, I don’t see why grpc which only requires a JS library and matching deployed server which is all open source and doesn’t really require buyin from multiple stakeholders will struggle here. Your emotional reaction to this seems out of place as there’s no grand conspiracy here. It seemed like a plausible idea. It just never panned out enough to be worth it. Certainly the server push as an API wouldn’t buy you that much in terms of performance because you can emulate it via long poll / websockets unless I’m missing something? [1] https://blog.cloudflare.com/early-hints/ https://blog.cloudflare.com/early-hints/
      - rektide 4y ago
        The people who had any power/control over push (standards & broeser folks) seemed to focus exclusively on using it for initial page load. As you do. Which is just one sliver of what Push could have been used for. The community had begged for some ability to actually increase developer capability, to start using Push as a reactive system, where we could start to replace unstructured websocket schenanigans with a holistic resourceful way to asserting resources. There was a couple very brief moments during fetch's addition of progress where push got a brief bit of attention from big enough names that it seemed like maybe after years there's be some real chance of using http2 push interestingly on the web, but that moment flickered out & died. In general there has been a callous treatment for Push, with blinders on, thinking only of tbe narrowest desires & uses. It's been a completely squandered technological capability that was never opened for use in any interesting form, and it's a shame this sad small vision of http2 push & it's so called failure obstructs us from even considering how many more interesting & powerful uses it could have had. My understanding is grpc requires push and trailers support, and that the browser still has no designs on offering either capability to developers for use. It's been some years since I've looked, but http3 in the browser once again seems to give developers absolutely no new capabilities, even though http added new stuff like Push & Trailers nearly a decade ago. My emotional response is because there is such a small & narrow vision, choking how we might be using http & growing the web. Techniques like long-poll & websocket exist, but there's such a clearer better fitting match for sending http resources as they are generated, Push. The lack of browser exposure of new (decade old) http capabilities is pushing us towards a stupid point where we end up running http3-over-webtransport, and it's absurd & enmisersting to see such a lack of follow-through in the deepest most core heart of the web being given a chance to get used, to do the amazing things it could be doing... as opposed to radically non-web non-resourceful hacks like websockets. This harkens back to the HyBi mailing list, and the sad inability for the web & our exhange of resources to be more bidirectional & asynchronous, and that's not a decade od stagnation, it's now two decades of stagnation, stagnation that we almost got a chance to improve past, were it not for the sad silly limited pretense that Early Hints gives us even a thousandth of what Push gave us in terms of capabilities.
- imtringued 4y ago
  I don't have enough time to inform myself about every obscure network protocol that won't be widely adopted. I don't know if Homa has multihoming but QUIC and SCTP have and if Homa doesn't then I think it is a huge step backwards.
yencabulator 4y ago
I dug up some resources for us non-Googlers: https://blog.acolyer.org/2019/11/11/snap-networking/ https://blog.acolyer.org/2019/11/11/snap-networking/ https://research.google/pubs/pub48630/ https://research.google/pubs/pub48630/ https://sosp19.rcs.uwaterloo.ca/slides/marty.pdf https://sosp19.rcs.uwaterloo.ca/slides/marty.pdf https://news.ycombinator.com/item?id=21374204 https://news.ycombinator.com/item?id=21374204

by dgacmu 4y ago

Large amounts of Google's internal RPC traffic goes over a custom internal framework called Pony Express, which is optimized to work on the internal cluster fabric.

rektide 4y ago

Well that's accidentally hella unfortunate. I've been assuming eventually gRPC over HTTP3[1] would get some traction from big camps. But if Google is using their own internal transport, it seems highly unlikely that these main authors of gRPC are ever going to care. This blog post talks about how many of the features of HOMA are already present in existing specs like QUIC/HTTP3. I'd expect many of the wins in Ousterhout's HOMA benchmarks could probably be replicated elsewhere, with better transports. [1] https://github.com/grpc/grpc/issues/19126 https://github.com/grpc/grpc/issues/19126

vlovich123 4y ago
They would still care about it to do client to first Google server handling the request
- rektide 4y ago
  Alas the web standards world has been a trashfire useless pile of nothing at giving webdevs any use of http2 or http3. Push support for fetch was one of the first issues opened[1], and summarily ignored by nearly every browser & spec author & never made at all accessible, right up until modern times where the browsers said, oh hey, we never really made this feature available to webdevs, but since you all havent used it anyways, we're removing Push from the browser. SMDH. Monstrous shit. In short, the browser failed to make basic modern http usable by anyone so there's no hope of getting off the janky ugly grpc-web trashfire & making grpc from the client just work, like it always should have been able to do. Pathetic pathetic showing by the browsers here. Beyond neglect, how they mishandled http2 and http2 and gave webdevs access to almost none of what the most exciting new thing on the www was. Huge huge miss. Incredible bullshit. Some day we'll be running http3 over webtransport & fixing their wrongs. Thia is such a bullshit long workaround, such a pathetic way to handle the browser getting nowhere with new http standards, end running around their utte unmoving complete inability to advance at all. But we'll start to actually use http intensively again, soon, in spite of the browser & standards community being such ridiculous & farcical impedances against using http2 and http3 at all. What a tragic shit show of uselessness it's been, trying ro actually enjoy the new http standards; resistance on all fronts to real usage. [1] https://github.com/whatwg/fetch/issues/51 https://github.com/whatwg/fetch/issues/51
  - vlovich123 4y ago
    You’re conflating very different things I think. HTTP/push requires browser and server support. Multiple experiments by multiple parties have shown it’s difficult to make work well regardless of a user facing API. It’s not just Google either - I recall multiple failed attempts to get it to work well. It was an interesting idea that never panned out. I think the summary by my peers on the cache team at Cloudflare outlines why server push failed [1] and I know those people first hand to be very thoughtful and principled people without a hidden agenda other than making the web very fast. To be clear I have no agenda or strong opinion either way and I could be misinformed, I just didn’t see any string evidence for benefit and lots of evidence that it is a net harm (+ maintenance effort). grpc-web will require very little work to adopt http/3 and the benefit is pretty obvious. The hard part will be if there’s http/1 servers in the way that have an impedance mismatch. Still, I don’t see why grpc which only requires a JS library and matching deployed server which is all open source and doesn’t really require buyin from multiple stakeholders will struggle here. Your emotional reaction to this seems out of place as there’s no grand conspiracy here. It seemed like a plausible idea. It just never panned out enough to be worth it. Certainly the server push as an API wouldn’t buy you that much in terms of performance because you can emulate it via long poll / websockets unless I’m missing something? [1] https://blog.cloudflare.com/early-hints/ https://blog.cloudflare.com/early-hints/
    - rektide 4y ago
      The people who had any power/control over push (standards & broeser folks) seemed to focus exclusively on using it for initial page load. As you do. Which is just one sliver of what Push could have been used for. The community had begged for some ability to actually increase developer capability, to start using Push as a reactive system, where we could start to replace unstructured websocket schenanigans with a holistic resourceful way to asserting resources. There was a couple very brief moments during fetch's addition of progress where push got a brief bit of attention from big enough names that it seemed like maybe after years there's be some real chance of using http2 push interestingly on the web, but that moment flickered out & died. In general there has been a callous treatment for Push, with blinders on, thinking only of tbe narrowest desires & uses. It's been a completely squandered technological capability that was never opened for use in any interesting form, and it's a shame this sad small vision of http2 push & it's so called failure obstructs us from even considering how many more interesting & powerful uses it could have had. My understanding is grpc requires push and trailers support, and that the browser still has no designs on offering either capability to developers for use. It's been some years since I've looked, but http3 in the browser once again seems to give developers absolutely no new capabilities, even though http added new stuff like Push & Trailers nearly a decade ago. My emotional response is because there is such a small & narrow vision, choking how we might be using http & growing the web. Techniques like long-poll & websocket exist, but there's such a clearer better fitting match for sending http resources as they are generated, Push. The lack of browser exposure of new (decade old) http capabilities is pushing us towards a stupid point where we end up running http3-over-webtransport, and it's absurd & enmisersting to see such a lack of follow-through in the deepest most core heart of the web being given a chance to get used, to do the amazing things it could be doing... as opposed to radically non-web non-resourceful hacks like websockets. This harkens back to the HyBi mailing list, and the sad inability for the web & our exhange of resources to be more bidirectional & asynchronous, and that's not a decade od stagnation, it's now two decades of stagnation, stagnation that we almost got a chance to improve past, were it not for the sad silly limited pretense that Early Hints gives us even a thousandth of what Push gave us in terms of capabilities.
imtringued 4y ago
I don't have enough time to inform myself about every obscure network protocol that won't be widely adopted. I don't know if Homa has multihoming but QUIC and SCTP have and if Homa doesn't then I think it is a huge step backwards.

yencabulator 4y ago

I dug up some resources for us non-Googlers: https://blog.acolyer.org/2019/11/11/snap-networking/ https://blog.acolyer.org/2019/11/11/snap-networking/ https://research.google/pubs/pub48630/ https://research.google/pubs/pub48630/ https://sosp19.rcs.uwaterloo.ca/slides/marty.pdf https://sosp19.rcs.uwaterloo.ca/slides/marty.pdf https://news.ycombinator.com/item?id=21374204 https://news.ycombinator.com/item?id=21374204