3 ms·

If you'd like to watch Ousterhoust explain Homa and his vision of replacing TCP: https://www.youtube.com/watch?v=o2HBHckrdQc https://www.youtube.com/watch?v=o2

If you'd like to watch Ousterhoust explain Homa and his vision of replacing TCP: https://www.youtube.com/watch?v=o2HBHckrdQc https://www.youtube.com/watch?v=o2HBHckrdQc Spoiler: Ousterhout is right, and the data shows he is right. Maybe not everyone knows it yet, or it's the kind of "right" that not everyone will ever get to (and we'll settle for "good enough" for much longer), but he is almost certainly right. Not to disparage the poster of this article of course -- I always love a good hot take or contrarian piece, but if you can't come with the same amount of data to support your thesis... Homa is probably dismissable on UX grounds (are people really going to switch to it?), but it's clear that long hard thinking has gone into the case for Homa (and it's implementation).

deathanatos 4y ago
I think you're right that he's "right", but the writing is shooting itself in the foot here. I agree that there's a mostly-unexplored middle between TCP & UDP, but you need to not utter things like "HTTP is bundled with HTML". I also think RPC is a poor label for "request/response" protocol (HTTP is request/response, and many of the problems here apply to HTTP too … but it isn't RPC). I'd like more discussion of the contenders in the space: why do they not work? QUIC is barely mentioned beyond "we tried it and nope". SCTP isn't mentioned on the OP at all, but if you click through the Homa stuff a bit, it's written off as "a WAN protocol" — whatever that is? I get that his focus is datacenters, but "WAN protocol" is what I feel most of us are interested in, and I don't immediately see why WAN vs. DC matters¹. What I'm also not getting from the Homa page is … what is the protocol, what are it's packet structures? Is there no documentation? (I've not hit all the third-degree links because I'm not a spider, but nothing seemed relevant or it seemed a little relevant but didn't hold the answers.) The old adage about data structures over algorithms seems to apply: > "Show me your flowchart and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won't usually need your flowchart; it'll be obvious." -- Fred Brooks, The Mythical Man Month (1975) ¹I'd also want to know why a good "WAN" protocol that solves the stated problems wouldn't equally apply to the datacenter, modulo needing some adjustments around assumptions made about the network's latency and maybe reliability: a DC network, I'd hope, is going to have much lower latency. But like Raft timers, it seems like a by-use-case adjustable constant.
- rezonant 4y ago
  On the topic of WAN vs DC it's about reliability. A lot of the engineering behind TCP assumes an unreliable transport, but in a data center context a lot of the unreliability doesn't exist, or can be engineered out of the network infrastructure you use. A good example of this in my space is video over IP (ST 2110), which is over RTP/UDP, but RTCP (for retransmission and rate control) is much more of a vestigial component, and if I remember correctly is entirely optional in the specs. A lot of the SDN solutions even default to not carrying the RTCP back channel. Now, in the video over IP space, there is essentially zero time available for retransmission, so such packet loss would be better handled as a dropped frame or audio packet, but it's not like this is an expected outcome of using UDP in such a controlled environment as a dedicated video transport network
KaiserPro 4y ago
> Spoiler: Ousterhout is right, and the data shows he is right. Maybe not everyone knows it yet, or it's the kind of "right" that not everyone will ever get to (and we'll settle for "good enough" for much longer), but he is almost certainly right. No, Ousterhout is not "right", Ousterhout has put forward an hypothesis, backed up by _some_ data that's not really borne out from real world experiences in the datacentre. Infiniband, despite his claims to the contrary solves a whole bunch of the problems he's describing. Moreover its a proven solution that deploys at scale. not only that, but you can run enthernet/IP over the top. Having proper flow control inside your switching fabric is bloody amazing if you want real-time with QOS over the top. After all Fibre Channel was designed to be fast, guaranteed deliver and low latency. The problem is, ethernet was cheaper and good enough to cover 80% of usecases.
- hardwaresofton 4y ago
  A bit late but for the average (small, let’s say) data center that isn’t interested, then he’s offering a valid (if not unlikely) solution right?
  - KaiserPro 4y ago
    I'm not sure. Having run a smaller datacenter (~2 megawatts) We got far more bang for our buck by upgrading the core switch fabric and moving to dual 40gig(from dual 10 gig) ethernet for the "heavy" nodes. Sure it cost more in terms of capex, but once it was installed we didn't need to change anything else. The cost of having an engineering team porting mission critical stuff to a new transport mechanism would have been astronomical, not only in opex, but in lost capacity to do other business critical tasks.

deathanatos 4y ago

I think you're right that he's "right", but the writing is shooting itself in the foot here. I agree that there's a mostly-unexplored middle between TCP & UDP, but you need to not utter things like "HTTP is bundled with HTML". I also think RPC is a poor label for "request/response" protocol (HTTP is request/response, and many of the problems here apply to HTTP too … but it isn't RPC). I'd like more discussion of the contenders in the space: why do they not work? QUIC is barely mentioned beyond "we tried it and nope". SCTP isn't mentioned on the OP at all, but if you click through the Homa stuff a bit, it's written off as "a WAN protocol" — whatever that is? I get that his focus is datacenters, but "WAN protocol" is what I feel most of us are interested in, and I don't immediately see why WAN vs. DC matters¹. What I'm also not getting from the Homa page is … what is the protocol, what are it's packet structures? Is there no documentation? (I've not hit all the third-degree links because I'm not a spider, but nothing seemed relevant or it seemed a little relevant but didn't hold the answers.) The old adage about data structures over algorithms seems to apply: > "Show me your flowchart and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won't usually need your flowchart; it'll be obvious." -- Fred Brooks, The Mythical Man Month (1975) ¹I'd also want to know why a good "WAN" protocol that solves the stated problems wouldn't equally apply to the datacenter, modulo needing some adjustments around assumptions made about the network's latency and maybe reliability: a DC network, I'd hope, is going to have much lower latency. But like Raft timers, it seems like a by-use-case adjustable constant.

rezonant 4y ago
On the topic of WAN vs DC it's about reliability. A lot of the engineering behind TCP assumes an unreliable transport, but in a data center context a lot of the unreliability doesn't exist, or can be engineered out of the network infrastructure you use. A good example of this in my space is video over IP (ST 2110), which is over RTP/UDP, but RTCP (for retransmission and rate control) is much more of a vestigial component, and if I remember correctly is entirely optional in the specs. A lot of the SDN solutions even default to not carrying the RTCP back channel. Now, in the video over IP space, there is essentially zero time available for retransmission, so such packet loss would be better handled as a dropped frame or audio packet, but it's not like this is an expected outcome of using UDP in such a controlled environment as a dedicated video transport network

KaiserPro 4y ago

> Spoiler: Ousterhout is right, and the data shows he is right. Maybe not everyone knows it yet, or it's the kind of "right" that not everyone will ever get to (and we'll settle for "good enough" for much longer), but he is almost certainly right. No, Ousterhout is not "right", Ousterhout has put forward an hypothesis, backed up by _some_ data that's not really borne out from real world experiences in the datacentre. Infiniband, despite his claims to the contrary solves a whole bunch of the problems he's describing. Moreover its a proven solution that deploys at scale. not only that, but you can run enthernet/IP over the top. Having proper flow control inside your switching fabric is bloody amazing if you want real-time with QOS over the top. After all Fibre Channel was designed to be fast, guaranteed deliver and low latency. The problem is, ethernet was cheaper and good enough to cover 80% of usecases.

hardwaresofton 4y ago
A bit late but for the average (small, let’s say) data center that isn’t interested, then he’s offering a valid (if not unlikely) solution right?
- KaiserPro 4y ago
  I'm not sure. Having run a smaller datacenter (~2 megawatts) We got far more bang for our buck by upgrading the core switch fabric and moving to dual 40gig(from dual 10 gig) ethernet for the "heavy" nodes. Sure it cost more in terms of capex, but once it was installed we didn't need to change anything else. The cost of having an engineering team porting mission critical stuff to a new transport mechanism would have been astronomical, not only in opex, but in lost capacity to do other business critical tasks.