6 ms·

This seems like a review of John Ousterhout's work w/ Homa. I highly recommend reading the original. https://arxiv.org/abs/2210.00714 https://arxiv.org/abs/2210

This seems like a review of John Ousterhout's work w/ Homa. I highly recommend reading the original. https://arxiv.org/abs/2210.00714 https://arxiv.org/abs/2210.00714 For those who don't know Ousterhout created the first log structured filesystem and created TCL (used heavily in hardware verification but also forming the backbone of the some of the first large web servers: aolserver). I was actually surprised to find out he co-founded a company with the current CTO of Cloudflare. https://en.wikipedia.org/wiki/John_Ousterhout https://en.wikipedia.org/wiki/John_Ousterhout He has both a candidate replacement as well as benchmarks showing 40% performance improvements with grpc on homa compared to grpc on TCP. https://github.com/PlatformLab/grpc_homa https://github.com/PlatformLab/grpc_homa With that in mind I think nobody will replace TCP and I doubt anything not IP compatible will be able to get off the ground. His argument is essentially that for low latency RPC protocols TCP is a bad choice. We've already seen people build a number of similar systems on UDP including HTTP replacements that have delivered value for clients doing lots of parallel requests on the WAN. I think many big tech companies are essentially already choosing to bypass TCP. I recall facebook doing alot of work with memcache on udp. I can't find any public docs on whether or not Google's internal RPC uses TCP. I wouldn't be surprised at all if in the near future something like grpc/capnproto/twirp/etc had an in-datacenter TCP-free fast path. It would be cool if it was built on Homa.

zarkov99 4y ago
TCP is rarely used in HFT data centers except as required by exchanges.
tyingq 4y ago
Ousterhout was also one of the co-authors of the Raft consensus paper.
killingtime74 4y ago
He also wrote one of my favorite software books https://web.stanford.edu/~ouster/cgi-bin/book.php https://web.stanford.edu/~ouster/cgi-bin/book.php. Presentation https://youtu.be/bmSAYlu0NcY https://youtu.be/bmSAYlu0NcY
wpietri 4y ago
Looking at the Ousterhout paper, I share your skepticism. I'm perfectly willing to believe that a parallel universe with datacenters running Homa would be a better world. But if I were to make a list of actual reasons too much money is getting burned on AWS bills, I suspect TCP inefficiencies wouldn't make the top 10. And worse, some of the things that do make the list, like bad managers, rushed schedules, under-trained developers, changing technical fashions, and "enterprise" development culture, all are huge barriers to replacing something so deep in the stack.
opportune 4y ago
The problem is not just TCP vs not-TCP. It’s using TCP/HTTP to transmit data in JSON, vs picking a stack that is more optimized for service-to-service communication in a datacenter. I am willing to wager that most organizations’ microservices spend the majority of their CPU usage doing serialization and deserialization of JSON.
- teknopaul 4y ago
  N.B. And are quite happy doing so because it makes app develop a breeze. Being able to use tools like tcpdump to debug applications is important to fast problem resolving. Unless everything you do is "Web scale" and development costs are insignificant, simple paradims like stream oriented text protocols will have their place.
- jiggawatts 4y ago
  Inefficient RPC compounds the common error of “SELECT * FROM BigTable”. I regularly see multi-hundred-megabyte result sets bloat out to a gigabyte on the wire Bizarrely, this take just a couple of seconds to transfer over 10 GbE so many devs simply don’t notice or chalk it up to “needing more capacity”. Yes, yes, it’s the stingy sysops hoarding the precious compute that’s to blame…
  - hinkley 4y ago
    I know people who've tried to fix this from time to time but it always seems to go wrong. We could track expected response size, but then every feature launch triggers a bunch of alerts which either causes the expenditure of social capital, or results in alert fatigue which causes us to miss real problems, or both. This is a place where telemetry does particularly well. I don't need to be vigilant to regressions every CD cycle, every day, or even every week. Most times if I catch a problem in a couple of weeks, and I can trace it back to the source, that's sufficient to keep the wheels on.
- ilyt 4y ago
  > I am willing to wager that most organizations’ microservices spend the majority of their CPU usage doing serialization and deserialization of JSON. I'd say in majority of cases the service was made "too small". If you waste majority of CPU to serialize/deserialize/send to network you should probably just "do the job" right there and then (aside from loadbalancers and such for obvious reasons)
KaiserPro 4y ago
> We've already seen people build a number of similar systems on UDP including HTTP replacements that have delivered value for clients doing lots of parallel requests on the WAN. HTTP isn't low latency. Spending all that effort porting everything to a discreet message based network, only to have HTTP semantics running over the top is a massive own goal. as for WAN, thats a whole different kettle of fish. You need to deal with huge latencies, whole integer percentage of packet loss, and lots of other wierd shit. In that instance you need to create a protocol to tailor to your needs. Are you after bandwidth efficiency, or raw speed? or do you need to minimise latency? All three things require completely different layouts and tradeoffs. I used to work for a company that shipped TBs from NZ & Aus to LA via london. We used to use aspera, but thats expensive. So we made a tool that used hundreds of TCP connections to max out our network link. For specialist applications, I can see a need for something like homa. But for 95% of datacenter traffic, its just not worth the effort. That paper is also fundamentally flawed, as the OP has rightly pointed out. Just because Ousterhout is clever, doesn't make him right.
- Karrot_Kream 4y ago
  What about HTTP/2 or HTTP/3 do you see poses large latency problems? Both of those do away with HTTP HOL blocking (not TCP HOL blocking) and allow parallel streams. Homa proposes introducing a transport level protocol with request/response semantics specifically to accommodate protocols like HTTP which operate using requests and responses.
  - KaiserPro 4y ago
    HTTP/2 is the king of HOL blocking. Its a single TCP connection in which everything is multiplexed at the app level. It was almost like it was designed to exploit the worst parts of TCP. It was and is a stupid design, they were clearly told about the drawbacks, both in mobile and for transfer of larger website. They chose to ignore it. The key thing is that HTTP is not designed to be low latency. If latency is important to you, then you need to make your protocol bespoke for that application. HTTP/3 has a much better TLS connection process, which means that the cost of creating a connection is much lower. QUIC is much more configurable in terms of per steam or connection flowcontrol. Yes, homa has request/reply semantics, but not in the way that HTTP needs. Homa is optimised for low latency small messages, in a small hop, transparent network. HTTP is a file access protocol with a general data channel hammered in. Sure it'll work in a DC type network, but its also got to deal with lossy, high latency networks.
ignoramous 4y ago
> With that in mind I think nobody will replace TCP Within the data center? AWS uses SRD with custom built network cards [0]. I'd be surprised if Microsoft, Facebook, and Google aren't doing something similar. [0] https://ieeexplore.ieee.org/document/9167399 https://ieeexplore.ieee.org/document/9167399 / https://archive.is/qZGdC https://archive.is/qZGdC
- actionfromafar 4y ago
  Considering how slow Azure caching is, maybe they use Token Ring? Edit: seriously though, does anyone know what they are up to? We get like 30 megabits from cache to CI.
dgacmu 4y ago
Large amounts of Google's internal RPC traffic goes over a custom internal framework called Pony Express, which is optimized to work on the internal cluster fabric.
- rektide 4y ago
  Well that's accidentally hella unfortunate. I've been assuming eventually gRPC over HTTP3[1] would get some traction from big camps. But if Google is using their own internal transport, it seems highly unlikely that these main authors of gRPC are ever going to care. This blog post talks about how many of the features of HOMA are already present in existing specs like QUIC/HTTP3. I'd expect many of the wins in Ousterhout's HOMA benchmarks could probably be replicated elsewhere, with better transports. [1] https://github.com/grpc/grpc/issues/19126 https://github.com/grpc/grpc/issues/19126
  - vlovich123 4y ago
    They would still care about it to do client to first Google server handling the request
    - rektide 4y ago
      Alas the web standards world has been a trashfire useless pile of nothing at giving webdevs any use of http2 or http3. Push support for fetch was one of the first issues opened[1], and summarily ignored by nearly every browser & spec author & never made at all accessible, right up until modern times where the browsers said, oh hey, we never really made this feature available to webdevs, but since you all havent used it anyways, we're removing Push from the browser. SMDH. Monstrous shit. In short, the browser failed to make basic modern http usable by anyone so there's no hope of getting off the janky ugly grpc-web trashfire & making grpc from the client just work, like it always should have been able to do. Pathetic pathetic showing by the browsers here. Beyond neglect, how they mishandled http2 and http2 and gave webdevs access to almost none of what the most exciting new thing on the www was. Huge huge miss. Incredible bullshit. Some day we'll be running http3 over webtransport & fixing their wrongs. Thia is such a bullshit long workaround, such a pathetic way to handle the browser getting nowhere with new http standards, end running around their utte unmoving complete inability to advance at all. But we'll start to actually use http intensively again, soon, in spite of the browser & standards community being such ridiculous & farcical impedances against using http2 and http3 at all. What a tragic shit show of uselessness it's been, trying ro actually enjoy the new http standards; resistance on all fronts to real usage. [1] https://github.com/whatwg/fetch/issues/51 https://github.com/whatwg/fetch/issues/51
deleted 4y ago
[deleted]
hardwaresofton 4y ago
If you want to watch Outsterhout in talk form: Netdev 0x16 - Keynote: It's time to replace TCP in the datacenter: https://www.youtube.com/watch?v=o2HBHckrdQc https://www.youtube.com/watch?v=o2HBHckrdQc Netdev is an amazing conference btw, it is insane that we get access to the trailblazing being shown off at conferences for free on places like YT.

zarkov99 4y ago

TCP is rarely used in HFT data centers except as required by exchanges.

tyingq 4y ago

Ousterhout was also one of the co-authors of the Raft consensus paper.

killingtime74 4y ago

He also wrote one of my favorite software books https://web.stanford.edu/~ouster/cgi-bin/book.php https://web.stanford.edu/~ouster/cgi-bin/book.php. Presentation https://youtu.be/bmSAYlu0NcY https://youtu.be/bmSAYlu0NcY

wpietri 4y ago

Looking at the Ousterhout paper, I share your skepticism. I'm perfectly willing to believe that a parallel universe with datacenters running Homa would be a better world. But if I were to make a list of actual reasons too much money is getting burned on AWS bills, I suspect TCP inefficiencies wouldn't make the top 10. And worse, some of the things that do make the list, like bad managers, rushed schedules, under-trained developers, changing technical fashions, and "enterprise" development culture, all are huge barriers to replacing something so deep in the stack.

opportune 4y ago

The problem is not just TCP vs not-TCP. It’s using TCP/HTTP to transmit data in JSON, vs picking a stack that is more optimized for service-to-service communication in a datacenter. I am willing to wager that most organizations’ microservices spend the majority of their CPU usage doing serialization and deserialization of JSON.

teknopaul 4y ago
N.B. And are quite happy doing so because it makes app develop a breeze. Being able to use tools like tcpdump to debug applications is important to fast problem resolving. Unless everything you do is "Web scale" and development costs are insignificant, simple paradims like stream oriented text protocols will have their place.
jiggawatts 4y ago
Inefficient RPC compounds the common error of “SELECT * FROM BigTable”. I regularly see multi-hundred-megabyte result sets bloat out to a gigabyte on the wire Bizarrely, this take just a couple of seconds to transfer over 10 GbE so many devs simply don’t notice or chalk it up to “needing more capacity”. Yes, yes, it’s the stingy sysops hoarding the precious compute that’s to blame…
- hinkley 4y ago
  I know people who've tried to fix this from time to time but it always seems to go wrong. We could track expected response size, but then every feature launch triggers a bunch of alerts which either causes the expenditure of social capital, or results in alert fatigue which causes us to miss real problems, or both. This is a place where telemetry does particularly well. I don't need to be vigilant to regressions every CD cycle, every day, or even every week. Most times if I catch a problem in a couple of weeks, and I can trace it back to the source, that's sufficient to keep the wheels on.
ilyt 4y ago
> I am willing to wager that most organizations’ microservices spend the majority of their CPU usage doing serialization and deserialization of JSON. I'd say in majority of cases the service was made "too small". If you waste majority of CPU to serialize/deserialize/send to network you should probably just "do the job" right there and then (aside from loadbalancers and such for obvious reasons)

KaiserPro 4y ago

> We've already seen people build a number of similar systems on UDP including HTTP replacements that have delivered value for clients doing lots of parallel requests on the WAN. HTTP isn't low latency. Spending all that effort porting everything to a discreet message based network, only to have HTTP semantics running over the top is a massive own goal. as for WAN, thats a whole different kettle of fish. You need to deal with huge latencies, whole integer percentage of packet loss, and lots of other wierd shit. In that instance you need to create a protocol to tailor to your needs. Are you after bandwidth efficiency, or raw speed? or do you need to minimise latency? All three things require completely different layouts and tradeoffs. I used to work for a company that shipped TBs from NZ & Aus to LA via london. We used to use aspera, but thats expensive. So we made a tool that used hundreds of TCP connections to max out our network link. For specialist applications, I can see a need for something like homa. But for 95% of datacenter traffic, its just not worth the effort. That paper is also fundamentally flawed, as the OP has rightly pointed out. Just because Ousterhout is clever, doesn't make him right.

Karrot_Kream 4y ago
What about HTTP/2 or HTTP/3 do you see poses large latency problems? Both of those do away with HTTP HOL blocking (not TCP HOL blocking) and allow parallel streams. Homa proposes introducing a transport level protocol with request/response semantics specifically to accommodate protocols like HTTP which operate using requests and responses.
- KaiserPro 4y ago
  HTTP/2 is the king of HOL blocking. Its a single TCP connection in which everything is multiplexed at the app level. It was almost like it was designed to exploit the worst parts of TCP. It was and is a stupid design, they were clearly told about the drawbacks, both in mobile and for transfer of larger website. They chose to ignore it. The key thing is that HTTP is not designed to be low latency. If latency is important to you, then you need to make your protocol bespoke for that application. HTTP/3 has a much better TLS connection process, which means that the cost of creating a connection is much lower. QUIC is much more configurable in terms of per steam or connection flowcontrol. Yes, homa has request/reply semantics, but not in the way that HTTP needs. Homa is optimised for low latency small messages, in a small hop, transparent network. HTTP is a file access protocol with a general data channel hammered in. Sure it'll work in a DC type network, but its also got to deal with lossy, high latency networks.

ignoramous 4y ago

> With that in mind I think nobody will replace TCP Within the data center? AWS uses SRD with custom built network cards [0]. I'd be surprised if Microsoft, Facebook, and Google aren't doing something similar. [0] https://ieeexplore.ieee.org/document/9167399 https://ieeexplore.ieee.org/document/9167399 / https://archive.is/qZGdC https://archive.is/qZGdC

actionfromafar 4y ago
Considering how slow Azure caching is, maybe they use Token Ring? Edit: seriously though, does anyone know what they are up to? We get like 30 megabits from cache to CI.

dgacmu 4y ago

Large amounts of Google's internal RPC traffic goes over a custom internal framework called Pony Express, which is optimized to work on the internal cluster fabric.

rektide 4y ago
Well that's accidentally hella unfortunate. I've been assuming eventually gRPC over HTTP3[1] would get some traction from big camps. But if Google is using their own internal transport, it seems highly unlikely that these main authors of gRPC are ever going to care. This blog post talks about how many of the features of HOMA are already present in existing specs like QUIC/HTTP3. I'd expect many of the wins in Ousterhout's HOMA benchmarks could probably be replicated elsewhere, with better transports. [1] https://github.com/grpc/grpc/issues/19126 https://github.com/grpc/grpc/issues/19126
- vlovich123 4y ago
  They would still care about it to do client to first Google server handling the request
  - rektide 4y ago
    Alas the web standards world has been a trashfire useless pile of nothing at giving webdevs any use of http2 or http3. Push support for fetch was one of the first issues opened[1], and summarily ignored by nearly every browser & spec author & never made at all accessible, right up until modern times where the browsers said, oh hey, we never really made this feature available to webdevs, but since you all havent used it anyways, we're removing Push from the browser. SMDH. Monstrous shit. In short, the browser failed to make basic modern http usable by anyone so there's no hope of getting off the janky ugly grpc-web trashfire & making grpc from the client just work, like it always should have been able to do. Pathetic pathetic showing by the browsers here. Beyond neglect, how they mishandled http2 and http2 and gave webdevs access to almost none of what the most exciting new thing on the www was. Huge huge miss. Incredible bullshit. Some day we'll be running http3 over webtransport & fixing their wrongs. Thia is such a bullshit long workaround, such a pathetic way to handle the browser getting nowhere with new http standards, end running around their utte unmoving complete inability to advance at all. But we'll start to actually use http intensively again, soon, in spite of the browser & standards community being such ridiculous & farcical impedances against using http2 and http3 at all. What a tragic shit show of uselessness it's been, trying ro actually enjoy the new http standards; resistance on all fronts to real usage. [1] https://github.com/whatwg/fetch/issues/51 https://github.com/whatwg/fetch/issues/51

deleted 4y ago

[deleted]

hardwaresofton 4y ago

If you want to watch Outsterhout in talk form: Netdev 0x16 - Keynote: It's time to replace TCP in the datacenter: https://www.youtube.com/watch?v=o2HBHckrdQc https://www.youtube.com/watch?v=o2HBHckrdQc Netdev is an amazing conference btw, it is insane that we get access to the trailblazing being shown off at conferences for free on places like YT.