5 ms·

> I wonder why Fibre Channel isn't used as a replacement for TCP in the datacenter But it is often used for block storage in datacenters. Using it for anything

> I wonder why Fibre Channel isn't used as a replacement for TCP in the datacenter But it is often used for block storage in datacenters. Using it for anything else is going to be hard, as it is incompatible with TCP. The problem with not using TCP is the same thing HOMA will face - anything already speaks TCP, nearly all potential hires know TCP and most problems you have with TCP have been solved by smart engineers already. Hardware is also easily available. Once you drop all those advantages, either your scale or your gains need to be massive to make that investment worth it, which is why TCP replacements are so rare outside of FAANG.

ksec 2y ago
I wonder if there are any work on making something similar ( conceptually ) to TCP, super / sub set of TCP while offering 50-80% benefits of HOMA. I guess I am old. Everytime I see new tech that wants to be hyped, completely throw out everything that is widely supported and working for 80-90% of uses cases, not battle tested and may be conceptually complex I will simply pass.
- Sebb767 2y ago
  If you have a sufficiently stable network and/or known failure cases, you can already tune TCP quite a bit with nodelay, large congestion windows etc.. There's also QUIC, which basically is a modern implementation of TCP on top of UDP (with some trade-offs chosen with HTTP in mind). Once you stray too far, you'll loose the ability to use off-the-shelve hardware, though, at which point you'll quickly hit the point of diminishing returns - especially when simply upgrading the speed of the network hardware is usually a cheap alternative.
  - mikepurvis 2y ago
    QUIC feels very pragmatic in terms of being built on UDP. As a lay person I don’t have a sense what additional gains might be on the table if the UDP layer were also up for reconsideration.
    - soneil 2y ago
      UDP has very low cost, the header is pretty much source and dest ports. For this low, low price, you get compatibility with existing routing, firewalling, NAT, etc.
  - mananaysiempre 2y ago
    One issue with QUIC in e.g. C is how heavyweight it feels to use compared to garden-variety BSD sockets (and that’s already not the most ergonomic of APIs). I haven’t encountered a QUIC library that didn’t feel like it would absolutely dominate a simple application in both code size and API-design pressure. Of course, for large applications that’s less relevant, but the result is that switching to QUIC gets perceived as a Big Deal, a step for when you’re doing Serious Stuff. That’s not ideal. I’d love to just play with QUIC a bit because it’s pretty neat, but I always get distracted by this problem and end up reading the RFCs, which so far I haven’t had the patience to get through.
jbverschoor 2y ago
But you might not need TCP. For example, using file-sockets between an app, db, and http server (rails+pgsql+nginx for example) has many benefits. The beauty of OSI layers.
- oneplane 2y ago
  That would work on a single host, but the context of the datacenter probably assumes multihost/manyhost workloads.
- wutwutwat 2y ago
  Unix sockets can use tcp, udp, or be a raw stream https://en.wikipedia.org/wiki/Unix_domain_socket#:~:text=The%20type%20parameter%20should%20be%20one%20of%20two%20common%20socket%20types%3A%20stream%20or%20datagram.%5B10%5D%20A%20third%20socket%20type%20is%20available%20for%20experimental%20design%3A%20raw https://en.wikipedia.org/wiki/Unix_domain_socket#:~:text=The.... Puma creates a `UnixServer` which is a ruby stdlib class, using the defaults, which is extending `UnixSocket` which is also using the defaults https://github.com/puma/puma/blob/fba741b91780224a1db1c456645335b2dd7f27dd/lib/puma/binder.rb#L441 https://github.com/puma/puma/blob/fba741b91780224a1db1c45664... Those defaults are creating a socket of type `SOCK_STREAM`, which is a tcp socket > SOCK_STREAM will create a stream socket. A stream socket provides a reliable, bidirectional, and connection-oriented communication channel between two processes. Data are carried using the Transmission Control Protocol (TCP). https://github.com/ruby/ruby/blob/5124f9ac7513eb590c37717337c430cb93caa151/ext/socket/unixsocket.c#L71C32-L71C43 https://github.com/ruby/ruby/blob/5124f9ac7513eb590c37717337... You still have the tcp overhead when using a local unix socket with puma, but you do not have any network overhead.
  - FaceValuable 2y ago
    Hey! I know it’s muddled but that’s not quite correct. SOCK_STREAM is more general than TCP; SOCK_STREAM just means the socket is a byte stream. You would need to add IPPROTO_TCP on top of that to pull in the TCP stack. UDS using SOCK_STREAM does not do that; ie, it is not using IPPROTO_TCP.
  - shawn_w 2y ago

ksec 2y ago

I wonder if there are any work on making something similar ( conceptually ) to TCP, super / sub set of TCP while offering 50-80% benefits of HOMA. I guess I am old. Everytime I see new tech that wants to be hyped, completely throw out everything that is widely supported and working for 80-90% of uses cases, not battle tested and may be conceptually complex I will simply pass.

Sebb767 2y ago
If you have a sufficiently stable network and/or known failure cases, you can already tune TCP quite a bit with nodelay, large congestion windows etc.. There's also QUIC, which basically is a modern implementation of TCP on top of UDP (with some trade-offs chosen with HTTP in mind). Once you stray too far, you'll loose the ability to use off-the-shelve hardware, though, at which point you'll quickly hit the point of diminishing returns - especially when simply upgrading the speed of the network hardware is usually a cheap alternative.
- mikepurvis 2y ago
  QUIC feels very pragmatic in terms of being built on UDP. As a lay person I don’t have a sense what additional gains might be on the table if the UDP layer were also up for reconsideration.
  - soneil 2y ago
    UDP has very low cost, the header is pretty much source and dest ports. For this low, low price, you get compatibility with existing routing, firewalling, NAT, etc.
- mananaysiempre 2y ago
  One issue with QUIC in e.g. C is how heavyweight it feels to use compared to garden-variety BSD sockets (and that’s already not the most ergonomic of APIs). I haven’t encountered a QUIC library that didn’t feel like it would absolutely dominate a simple application in both code size and API-design pressure. Of course, for large applications that’s less relevant, but the result is that switching to QUIC gets perceived as a Big Deal, a step for when you’re doing Serious Stuff. That’s not ideal. I’d love to just play with QUIC a bit because it’s pretty neat, but I always get distracted by this problem and end up reading the RFCs, which so far I haven’t had the patience to get through.

jbverschoor 2y ago

But you might not need TCP. For example, using file-sockets between an app, db, and http server (rails+pgsql+nginx for example) has many benefits. The beauty of OSI layers.

oneplane 2y ago
That would work on a single host, but the context of the datacenter probably assumes multihost/manyhost workloads.
wutwutwat 2y ago
Unix sockets can use tcp, udp, or be a raw stream https://en.wikipedia.org/wiki/Unix_domain_socket#:~:text=The%20type%20parameter%20should%20be%20one%20of%20two%20common%20socket%20types%3A%20stream%20or%20datagram.%5B10%5D%20A%20third%20socket%20type%20is%20available%20for%20experimental%20design%3A%20raw https://en.wikipedia.org/wiki/Unix_domain_socket#:~:text=The.... Puma creates a `UnixServer` which is a ruby stdlib class, using the defaults, which is extending `UnixSocket` which is also using the defaults https://github.com/puma/puma/blob/fba741b91780224a1db1c456645335b2dd7f27dd/lib/puma/binder.rb#L441 https://github.com/puma/puma/blob/fba741b91780224a1db1c45664... Those defaults are creating a socket of type `SOCK_STREAM`, which is a tcp socket > SOCK_STREAM will create a stream socket. A stream socket provides a reliable, bidirectional, and connection-oriented communication channel between two processes. Data are carried using the Transmission Control Protocol (TCP). https://github.com/ruby/ruby/blob/5124f9ac7513eb590c37717337c430cb93caa151/ext/socket/unixsocket.c#L71C32-L71C43 https://github.com/ruby/ruby/blob/5124f9ac7513eb590c37717337... You still have the tcp overhead when using a local unix socket with puma, but you do not have any network overhead.
- FaceValuable 2y ago
  Hey! I know it’s muddled but that’s not quite correct. SOCK_STREAM is more general than TCP; SOCK_STREAM just means the socket is a byte stream. You would need to add IPPROTO_TCP on top of that to pull in the TCP stack. UDS using SOCK_STREAM does not do that; ie, it is not using IPPROTO_TCP.
- shawn_w 2y ago