5 ms·

I wonder why Fibre Channel isn't used as a replacement for TCP in the datacenter. It is a very robust L3 protocol. It was designed to connect block storage devi

I wonder why Fibre Channel isn't used as a replacement for TCP in the datacenter. It is a very robust L3 protocol. It was designed to connect block storage devices to servers while making the OS think they are directly connected. OSs do NOT tolerate dropped data when reading and writing to block devices and so Fibre Channel has a extremely robust Token Bucket algorithm. The algo prevents congestion by allowing receivers to control how much data senders can send. I have worked with a lot of VMware clusters that use FC to connect servers to storage arrays and it has ALWAYS worked perfectly.

Sebb767 2y ago
> I wonder why Fibre Channel isn't used as a replacement for TCP in the datacenter But it is often used for block storage in datacenters. Using it for anything else is going to be hard, as it is incompatible with TCP. The problem with not using TCP is the same thing HOMA will face - anything already speaks TCP, nearly all potential hires know TCP and most problems you have with TCP have been solved by smart engineers already. Hardware is also easily available. Once you drop all those advantages, either your scale or your gains need to be massive to make that investment worth it, which is why TCP replacements are so rare outside of FAANG.
- ksec 2y ago
  I wonder if there are any work on making something similar ( conceptually ) to TCP, super / sub set of TCP while offering 50-80% benefits of HOMA. I guess I am old. Everytime I see new tech that wants to be hyped, completely throw out everything that is widely supported and working for 80-90% of uses cases, not battle tested and may be conceptually complex I will simply pass.
  - Sebb767 2y ago
    If you have a sufficiently stable network and/or known failure cases, you can already tune TCP quite a bit with nodelay, large congestion windows etc.. There's also QUIC, which basically is a modern implementation of TCP on top of UDP (with some trade-offs chosen with HTTP in mind). Once you stray too far, you'll loose the ability to use off-the-shelve hardware, though, at which point you'll quickly hit the point of diminishing returns - especially when simply upgrading the speed of the network hardware is usually a cheap alternative.
    - mikepurvis 2y ago
      QUIC feels very pragmatic in terms of being built on UDP. As a lay person I don’t have a sense what additional gains might be on the table if the UDP layer were also up for reconsideration.
      - soneil 2y ago
        UDP has very low cost, the header is pretty much source and dest ports. For this low, low price, you get compatibility with existing routing, firewalling, NAT, etc.
    - mananaysiempre 2y ago
      One issue with QUIC in e.g. C is how heavyweight it feels to use compared to garden-variety BSD sockets (and that’s already not the most ergonomic of APIs). I haven’t encountered a QUIC library that didn’t feel like it would absolutely dominate a simple application in both code size and API-design pressure. Of course, for large applications that’s less relevant, but the result is that switching to QUIC gets perceived as a Big Deal, a step for when you’re doing Serious Stuff. That’s not ideal. I’d love to just play with QUIC a bit because it’s pretty neat, but I always get distracted by this problem and end up reading the RFCs, which so far I haven’t had the patience to get through.
- jbverschoor 2y ago
  But you might not need TCP. For example, using file-sockets between an app, db, and http server (rails+pgsql+nginx for example) has many benefits. The beauty of OSI layers.
  - oneplane 2y ago
    That would work on a single host, but the context of the datacenter probably assumes multihost/manyhost workloads.
  - wutwutwat 2y ago
    Unix sockets can use tcp, udp, or be a raw stream https://en.wikipedia.org/wiki/Unix_domain_socket#:~:text=The%20type%20parameter%20should%20be%20one%20of%20two%20common%20socket%20types%3A%20stream%20or%20datagram.%5B10%5D%20A%20third%20socket%20type%20is%20available%20for%20experimental%20design%3A%20raw https://en.wikipedia.org/wiki/Unix_domain_socket#:~:text=The.... Puma creates a `UnixServer` which is a ruby stdlib class, using the defaults, which is extending `UnixSocket` which is also using the defaults https://github.com/puma/puma/blob/fba741b91780224a1db1c456645335b2dd7f27dd/lib/puma/binder.rb#L441 https://github.com/puma/puma/blob/fba741b91780224a1db1c45664... Those defaults are creating a socket of type `SOCK_STREAM`, which is a tcp socket > SOCK_STREAM will create a stream socket. A stream socket provides a reliable, bidirectional, and connection-oriented communication channel between two processes. Data are carried using the Transmission Control Protocol (TCP). https://github.com/ruby/ruby/blob/5124f9ac7513eb590c37717337c430cb93caa151/ext/socket/unixsocket.c#L71C32-L71C43 https://github.com/ruby/ruby/blob/5124f9ac7513eb590c37717337... You still have the tcp overhead when using a local unix socket with puma, but you do not have any network overhead.
    - FaceValuable 2y ago
      Hey! I know it’s muddled but that’s not quite correct. SOCK_STREAM is more general than TCP; SOCK_STREAM just means the socket is a byte stream. You would need to add IPPROTO_TCP on top of that to pull in the TCP stack. UDS using SOCK_STREAM does not do that; ie, it is not using IPPROTO_TCP.
    - shawn_w 2y ago
YZF 2y ago
Are you suggesting some protocol layer of Fibre Channel to be used over IP over Ethernet? TCP (in practice) runs on top of (mostly) routed IP networks and network architectures. E.g. a spine/leaf network with BGP. Fibre Channel as I understand it is mostly used in more or less point to point connections? I do see some mention of "Switched Fabric" but is that very common?
- UltraSane 2y ago
  Fibre Channel is a routed L3 protocol that can support loop-free multi-path typologies.
  - YZF 2y ago
    I'll admit I'm not familiar with the routing protocols used for Fibre Channel. Is there some equivalent of BGP? How well does it scale? What vendors sell FC switches and what's the cost compared to Ethernet/IP/TCP?
    - UltraSane 2y ago
      FC uses Fabric Shortest Path First, which is a lot like OSPF. It can scale to 2^64 ports. There is no FC equivalent of BGP. Broadcom, Cisco, HP, Lenovo, IBM sell FC switches, but some of them are probably rebadged Broadcom switches. The worst thing about FC is that the switches are licensed per port so you might not be able to use all the ports on the device. A brocade G720 with 24 32Gb usable ports is $28,000 on CDW. It has 64 physical ports. a 24 port license is $31,000 on CDW. So it is REALLY freaking expensive. But for servers a company can't make money without it is absolutely worth it. One place I worked had a old EMC FC SAN with 8 years of 100% uptime.
wejick 2y ago
I'm imagining having a shared memory mounted as block storages then do the RPC thru this block. Some synchronization and polling/notifications work will need to be done.
- creshal 2y ago
  The literal version of this is used by sanlock et al. to implement cluster-wide locks. But the whole "pretending to be block devices" overhead makes it not exactly ideal for anything else. Drop the "pretending to be block devices" part and you basically end up with InfiniBand. It works well, if you ignore the small problem of "you need to either reimplement all your software to work with RDMA based IPC, or reimplement Ethernet on top of InfiniBand to remain compatible and throw away most advantages of InfiniBand again".
- fmajid 2y ago
  That’s essentially what RDMA is, except it is usually run over Infiniband although hyperscalers are wary of Nvidia’s control over the technology and looking for cheaper Ethernet-based alternatives. https://blogs.nvidia.com/blog/what-is-rdma/ https://blogs.nvidia.com/blog/what-is-rdma/ https://dl.acm.org/doi/abs/10.1145/3651890.3672233 https://dl.acm.org/doi/abs/10.1145/3651890.3672233
  - hylaride 2y ago
    If it's a secure internal network, RDMA is probably what you want if you need low-latency data transfer. You can do some very performance-oriented things with it and it works over ethernet or infiniband (the quality of switching gear and network cards matters, though). Back in ~2012 I was setting up a high-frequency network for a forex company and at the time we deployed Mellanox and they had some very (at the time) bleeding edge networking drivers that significantly reduced the overhead of writing to TCP/IP sockets (particularily zero-copy which TL;DR meant data didn't get shifted around in memory as much and was written to the ethernet card's buffers almost straight away) that made a huge difference. I eventually left the firm and my successors tried to replace it with cisco gear and Intel NICs and the performance plummeted. That made me laugh as I received so much grief pushing for the Mellanox kit (to be fair, they were a scrappy unheard of Israeli company at the time).
slt2021 2y ago
my take is that within-datacenter traffic is best served by Ethernet. Anything on top of Ethernet, and we no longer know where this host is located (because of software defined networking). Could be next rack server, or could be something in the cloud, could be third party service. And that's a feature, not a bug: because everything speaks TCP: we can arbitrarily cut and slice network just by changing packet forwarding rules. We can partition network however we want. We could have a single global IP space shared by cloud, dc, campus networks, or could have Christmas Tree of NATs. as soon as you introduce something other than TCP to the mix, now you will have gateways: chokepoints where traffic will have to be translated TCP<->Homa and I don't want to be a person troubleshooting a problem at the intersection of TCP and Homa. in my opinion, the lowest level Ethernet should try its best to mirror the actual physical signal flow. Anything on top becomes software-network network
- mafuy 2y ago
  In data centers/HPC, you need to know which data is flowing where and then you design the hardware around that. Not the other way around. What you describe is a lower requirement level that is much easier to handle.
  - marcosdumay 2y ago
    That may be true for HPC, but "datacenter" is a generic name that applies to all kinds of structures.
- gsich 2y ago
  >Anything on top of Ethernet, and we no longer know where this host is located (because of software defined networking). Could be next rack server, or could be something in the cloud, could be third party service. Ping it and you can at least deduce where it's not.
KaiserPro 2y ago
FC was much more expensive than ethernet, so needed a reason to be used. For block storage it is great, if slower than ethernet.
- UltraSane 2y ago
  Is their a fundamental reason for it being more expensive than Ethernet or is just greed?
  - KaiserPro 2y ago
    At the time it was rarer and required more specialised hardware. FC was one of the first non-mainframe specific storage area network fabrics. One of the key differences between FC and ethernet is the collision detection/avoidance and guaranteed throughput. All that extra coordination takes effort on big switches, so it cost more to develop/produce. You could shovel data through it at "line speed" and know that it'll get there, or you'll get an error at the fabric level. Ethernet happily takes packets, and if it can't deliver, then it'll just drop them. (well, kinda, not all ethernet does that, because ethernet is the english language of layer 2 protocols)
holowoodman 2y ago
Fibrechannel is far too expensive, you need expensive switches, cables/transceivers and cards in addition to the Ethernet you'll need anyways. And this Fibrechannel hardware is quite limited in what you can do with it, by far not as capable as the usual Ethernet/IP stuff with regards to routing, encryption, tunneling, filtering and what not. Similar things are happening with stuff like Infiniband, it has become far too expensive and Ethernet/ROCE is making inroads in lower- to medium-end installations. Availability is also an issue, Nvidia is the only Infiniband vendor left.
- bluGill 2y ago
  there is ip over fiber channel. no need for separate ethernet. At least in theory, in practice I'm sure if anyone implemented enough parts to make it useful but the spec exists.
  - UltraSane 2y ago
    No. When I heard about Cisco having FC over Ethernet for their UCS servers I was grossed out because of how Ethernet is a L2 protocol that can't handle multi-path without ugly hacks like Virtual Port Channel and discovered that there is no real support for IP over Fiber Channel. There is a wikipedia page for IPFC but it seems to be completely dead. https://en.wikipedia.org/wiki/IPFC https://en.wikipedia.org/wiki/IPFC
- UltraSane 2y ago
  Is there a fundamental reason why FC is more expensive than Ethernet?
  - convolvatron 2y ago
    since its entire reason to exist was to effect an artificial market segmentation, I guess the answer is .. yes?
    - UltraSane 2y ago
      It isn't an artificial market segmentation. Fibre Channel is a no compromise technology with a single purpose, to connect servers to remote storage with performance and reliability close to directly attached storage, and it does that really, REALLY well. It is by far the single most bullet proof technology I have ever used. In a parallel universe where FC won over ethernet and every Ethernet port in the world was an FC port I don't see why it would be any more expensive that ethernet.
markhahn 2y ago
the question is really: does it have anything vendor-specific, interop-breakers? FC seems to work nicely in a single-vendor stack, or at least among specific sets of big-name vendors. that's OK for the "enterprise" market, where prices are expected to be high, and where some integrator is getting a handsome profit for making sure the vendors match. besides consumer, the original non-enterprise market was HPC, and we want no vendor lock-in. hyperscale is really just HPC-for-VM-hosting - more or less culturally compatible. besides these vendor/price/interop reasons, FC has never done a good job of keeping up. 100/200/400/800 Gb is getting to be common, and is FC there? resolving congestion is not unique to FC. even IB has more, probably better solutions, but these days, datacenter ethernet is pretty powerful.
- UltraSane 2y ago
  FC speeds have really lagged. 64Gbps is available but not widely used and 128Gbps was introduced in 2023. But since by definition 100% of FC bandwidth can only be used for storage it has been adequate. https://fibrechannel.org/preview-the-new-fibre-channel-speed-128gfc/ https://fibrechannel.org/preview-the-new-fibre-channel-speed...
tonetegeatinst 2y ago
Fiber is attractive. As someone who wants to upgrade to fiber, the main barrier to entry is the cost of switches and a router. Granted I'm also trying to find a switch that supports ROCm and rdma. Not easy to find a high bandwidth switch that supports this stuff without breaking the bank.
- jabl 2y ago
  Fiber, as in Fibre Channel (FC, https://en.wikipedia.org/wiki/Fibre_Channel https://en.wikipedia.org/wiki/Fibre_Channel ), not fiber as in "optical fiber" instead of copper cabling.
- _zoltan_ 2y ago
  SN2100/2700 from eBay?
- UltraSane 2y ago
  Fibre Channel is a routed protocol invented specifically to connect block storage arrays to remote servers while making the block storage look locally attached to the OS. And it works REALLY REALLY well.
Sylamore 2y ago
InfiniBand would make more sense than Fibre Channel
- UltraSane 2y ago
  Maybe but InfiniBand is really expensive and InfiniBand switches are in short supply. And it is an L2 protocol while FC is L3

Sebb767 2y ago

> I wonder why Fibre Channel isn't used as a replacement for TCP in the datacenter But it is often used for block storage in datacenters. Using it for anything else is going to be hard, as it is incompatible with TCP. The problem with not using TCP is the same thing HOMA will face - anything already speaks TCP, nearly all potential hires know TCP and most problems you have with TCP have been solved by smart engineers already. Hardware is also easily available. Once you drop all those advantages, either your scale or your gains need to be massive to make that investment worth it, which is why TCP replacements are so rare outside of FAANG.

ksec 2y ago
I wonder if there are any work on making something similar ( conceptually ) to TCP, super / sub set of TCP while offering 50-80% benefits of HOMA. I guess I am old. Everytime I see new tech that wants to be hyped, completely throw out everything that is widely supported and working for 80-90% of uses cases, not battle tested and may be conceptually complex I will simply pass.
- Sebb767 2y ago
  If you have a sufficiently stable network and/or known failure cases, you can already tune TCP quite a bit with nodelay, large congestion windows etc.. There's also QUIC, which basically is a modern implementation of TCP on top of UDP (with some trade-offs chosen with HTTP in mind). Once you stray too far, you'll loose the ability to use off-the-shelve hardware, though, at which point you'll quickly hit the point of diminishing returns - especially when simply upgrading the speed of the network hardware is usually a cheap alternative.
  - mikepurvis 2y ago
    QUIC feels very pragmatic in terms of being built on UDP. As a lay person I don’t have a sense what additional gains might be on the table if the UDP layer were also up for reconsideration.
    - soneil 2y ago
      UDP has very low cost, the header is pretty much source and dest ports. For this low, low price, you get compatibility with existing routing, firewalling, NAT, etc.
  - mananaysiempre 2y ago
    One issue with QUIC in e.g. C is how heavyweight it feels to use compared to garden-variety BSD sockets (and that’s already not the most ergonomic of APIs). I haven’t encountered a QUIC library that didn’t feel like it would absolutely dominate a simple application in both code size and API-design pressure. Of course, for large applications that’s less relevant, but the result is that switching to QUIC gets perceived as a Big Deal, a step for when you’re doing Serious Stuff. That’s not ideal. I’d love to just play with QUIC a bit because it’s pretty neat, but I always get distracted by this problem and end up reading the RFCs, which so far I haven’t had the patience to get through.
jbverschoor 2y ago
But you might not need TCP. For example, using file-sockets between an app, db, and http server (rails+pgsql+nginx for example) has many benefits. The beauty of OSI layers.
- oneplane 2y ago
  That would work on a single host, but the context of the datacenter probably assumes multihost/manyhost workloads.
- wutwutwat 2y ago
  Unix sockets can use tcp, udp, or be a raw stream https://en.wikipedia.org/wiki/Unix_domain_socket#:~:text=The%20type%20parameter%20should%20be%20one%20of%20two%20common%20socket%20types%3A%20stream%20or%20datagram.%5B10%5D%20A%20third%20socket%20type%20is%20available%20for%20experimental%20design%3A%20raw https://en.wikipedia.org/wiki/Unix_domain_socket#:~:text=The.... Puma creates a `UnixServer` which is a ruby stdlib class, using the defaults, which is extending `UnixSocket` which is also using the defaults https://github.com/puma/puma/blob/fba741b91780224a1db1c456645335b2dd7f27dd/lib/puma/binder.rb#L441 https://github.com/puma/puma/blob/fba741b91780224a1db1c45664... Those defaults are creating a socket of type `SOCK_STREAM`, which is a tcp socket > SOCK_STREAM will create a stream socket. A stream socket provides a reliable, bidirectional, and connection-oriented communication channel between two processes. Data are carried using the Transmission Control Protocol (TCP). https://github.com/ruby/ruby/blob/5124f9ac7513eb590c37717337c430cb93caa151/ext/socket/unixsocket.c#L71C32-L71C43 https://github.com/ruby/ruby/blob/5124f9ac7513eb590c37717337... You still have the tcp overhead when using a local unix socket with puma, but you do not have any network overhead.
  - FaceValuable 2y ago
    Hey! I know it’s muddled but that’s not quite correct. SOCK_STREAM is more general than TCP; SOCK_STREAM just means the socket is a byte stream. You would need to add IPPROTO_TCP on top of that to pull in the TCP stack. UDS using SOCK_STREAM does not do that; ie, it is not using IPPROTO_TCP.
  - shawn_w 2y ago

YZF 2y ago

Are you suggesting some protocol layer of Fibre Channel to be used over IP over Ethernet? TCP (in practice) runs on top of (mostly) routed IP networks and network architectures. E.g. a spine/leaf network with BGP. Fibre Channel as I understand it is mostly used in more or less point to point connections? I do see some mention of "Switched Fabric" but is that very common?

UltraSane 2y ago
Fibre Channel is a routed L3 protocol that can support loop-free multi-path typologies.
- YZF 2y ago
  I'll admit I'm not familiar with the routing protocols used for Fibre Channel. Is there some equivalent of BGP? How well does it scale? What vendors sell FC switches and what's the cost compared to Ethernet/IP/TCP?
  - UltraSane 2y ago
    FC uses Fabric Shortest Path First, which is a lot like OSPF. It can scale to 2^64 ports. There is no FC equivalent of BGP. Broadcom, Cisco, HP, Lenovo, IBM sell FC switches, but some of them are probably rebadged Broadcom switches. The worst thing about FC is that the switches are licensed per port so you might not be able to use all the ports on the device. A brocade G720 with 24 32Gb usable ports is $28,000 on CDW. It has 64 physical ports. a 24 port license is $31,000 on CDW. So it is REALLY freaking expensive. But for servers a company can't make money without it is absolutely worth it. One place I worked had a old EMC FC SAN with 8 years of 100% uptime.

wejick 2y ago

I'm imagining having a shared memory mounted as block storages then do the RPC thru this block. Some synchronization and polling/notifications work will need to be done.

creshal 2y ago
The literal version of this is used by sanlock et al. to implement cluster-wide locks. But the whole "pretending to be block devices" overhead makes it not exactly ideal for anything else. Drop the "pretending to be block devices" part and you basically end up with InfiniBand. It works well, if you ignore the small problem of "you need to either reimplement all your software to work with RDMA based IPC, or reimplement Ethernet on top of InfiniBand to remain compatible and throw away most advantages of InfiniBand again".
fmajid 2y ago
That’s essentially what RDMA is, except it is usually run over Infiniband although hyperscalers are wary of Nvidia’s control over the technology and looking for cheaper Ethernet-based alternatives. https://blogs.nvidia.com/blog/what-is-rdma/ https://blogs.nvidia.com/blog/what-is-rdma/ https://dl.acm.org/doi/abs/10.1145/3651890.3672233 https://dl.acm.org/doi/abs/10.1145/3651890.3672233
- hylaride 2y ago
  If it's a secure internal network, RDMA is probably what you want if you need low-latency data transfer. You can do some very performance-oriented things with it and it works over ethernet or infiniband (the quality of switching gear and network cards matters, though). Back in ~2012 I was setting up a high-frequency network for a forex company and at the time we deployed Mellanox and they had some very (at the time) bleeding edge networking drivers that significantly reduced the overhead of writing to TCP/IP sockets (particularily zero-copy which TL;DR meant data didn't get shifted around in memory as much and was written to the ethernet card's buffers almost straight away) that made a huge difference. I eventually left the firm and my successors tried to replace it with cisco gear and Intel NICs and the performance plummeted. That made me laugh as I received so much grief pushing for the Mellanox kit (to be fair, they were a scrappy unheard of Israeli company at the time).

slt2021 2y ago

my take is that within-datacenter traffic is best served by Ethernet. Anything on top of Ethernet, and we no longer know where this host is located (because of software defined networking). Could be next rack server, or could be something in the cloud, could be third party service. And that's a feature, not a bug: because everything speaks TCP: we can arbitrarily cut and slice network just by changing packet forwarding rules. We can partition network however we want. We could have a single global IP space shared by cloud, dc, campus networks, or could have Christmas Tree of NATs. as soon as you introduce something other than TCP to the mix, now you will have gateways: chokepoints where traffic will have to be translated TCP<->Homa and I don't want to be a person troubleshooting a problem at the intersection of TCP and Homa. in my opinion, the lowest level Ethernet should try its best to mirror the actual physical signal flow. Anything on top becomes software-network network

mafuy 2y ago
In data centers/HPC, you need to know which data is flowing where and then you design the hardware around that. Not the other way around. What you describe is a lower requirement level that is much easier to handle.
- marcosdumay 2y ago
  That may be true for HPC, but "datacenter" is a generic name that applies to all kinds of structures.
gsich 2y ago
>Anything on top of Ethernet, and we no longer know where this host is located (because of software defined networking). Could be next rack server, or could be something in the cloud, could be third party service. Ping it and you can at least deduce where it's not.

KaiserPro 2y ago

FC was much more expensive than ethernet, so needed a reason to be used. For block storage it is great, if slower than ethernet.

UltraSane 2y ago
Is their a fundamental reason for it being more expensive than Ethernet or is just greed?
- KaiserPro 2y ago
  At the time it was rarer and required more specialised hardware. FC was one of the first non-mainframe specific storage area network fabrics. One of the key differences between FC and ethernet is the collision detection/avoidance and guaranteed throughput. All that extra coordination takes effort on big switches, so it cost more to develop/produce. You could shovel data through it at "line speed" and know that it'll get there, or you'll get an error at the fabric level. Ethernet happily takes packets, and if it can't deliver, then it'll just drop them. (well, kinda, not all ethernet does that, because ethernet is the english language of layer 2 protocols)

holowoodman 2y ago

Fibrechannel is far too expensive, you need expensive switches, cables/transceivers and cards in addition to the Ethernet you'll need anyways. And this Fibrechannel hardware is quite limited in what you can do with it, by far not as capable as the usual Ethernet/IP stuff with regards to routing, encryption, tunneling, filtering and what not. Similar things are happening with stuff like Infiniband, it has become far too expensive and Ethernet/ROCE is making inroads in lower- to medium-end installations. Availability is also an issue, Nvidia is the only Infiniband vendor left.

bluGill 2y ago
there is ip over fiber channel. no need for separate ethernet. At least in theory, in practice I'm sure if anyone implemented enough parts to make it useful but the spec exists.
- UltraSane 2y ago
  No. When I heard about Cisco having FC over Ethernet for their UCS servers I was grossed out because of how Ethernet is a L2 protocol that can't handle multi-path without ugly hacks like Virtual Port Channel and discovered that there is no real support for IP over Fiber Channel. There is a wikipedia page for IPFC but it seems to be completely dead. https://en.wikipedia.org/wiki/IPFC https://en.wikipedia.org/wiki/IPFC
UltraSane 2y ago
Is there a fundamental reason why FC is more expensive than Ethernet?
- convolvatron 2y ago
  since its entire reason to exist was to effect an artificial market segmentation, I guess the answer is .. yes?
  - UltraSane 2y ago
    It isn't an artificial market segmentation. Fibre Channel is a no compromise technology with a single purpose, to connect servers to remote storage with performance and reliability close to directly attached storage, and it does that really, REALLY well. It is by far the single most bullet proof technology I have ever used. In a parallel universe where FC won over ethernet and every Ethernet port in the world was an FC port I don't see why it would be any more expensive that ethernet.

markhahn 2y ago

the question is really: does it have anything vendor-specific, interop-breakers? FC seems to work nicely in a single-vendor stack, or at least among specific sets of big-name vendors. that's OK for the "enterprise" market, where prices are expected to be high, and where some integrator is getting a handsome profit for making sure the vendors match. besides consumer, the original non-enterprise market was HPC, and we want no vendor lock-in. hyperscale is really just HPC-for-VM-hosting - more or less culturally compatible. besides these vendor/price/interop reasons, FC has never done a good job of keeping up. 100/200/400/800 Gb is getting to be common, and is FC there? resolving congestion is not unique to FC. even IB has more, probably better solutions, but these days, datacenter ethernet is pretty powerful.

UltraSane 2y ago
FC speeds have really lagged. 64Gbps is available but not widely used and 128Gbps was introduced in 2023. But since by definition 100% of FC bandwidth can only be used for storage it has been adequate. https://fibrechannel.org/preview-the-new-fibre-channel-speed-128gfc/ https://fibrechannel.org/preview-the-new-fibre-channel-speed...

tonetegeatinst 2y ago

Fiber is attractive. As someone who wants to upgrade to fiber, the main barrier to entry is the cost of switches and a router. Granted I'm also trying to find a switch that supports ROCm and rdma. Not easy to find a high bandwidth switch that supports this stuff without breaking the bank.

jabl 2y ago
Fiber, as in Fibre Channel (FC, https://en.wikipedia.org/wiki/Fibre_Channel https://en.wikipedia.org/wiki/Fibre_Channel ), not fiber as in "optical fiber" instead of copper cabling.
_zoltan_ 2y ago
SN2100/2700 from eBay?
UltraSane 2y ago
Fibre Channel is a routed protocol invented specifically to connect block storage arrays to remote servers while making the block storage look locally attached to the OS. And it works REALLY REALLY well.

Sylamore 2y ago

InfiniBand would make more sense than Fibre Channel

UltraSane 2y ago
Maybe but InfiniBand is really expensive and InfiniBand switches are in short supply. And it is an L2 protocol while FC is L3