5 ms·

Back when I looked into high-speed networking in small spaces, my idea was doing it over PCI directly bypassing Ethernet. I really wanted to do something like S

Back when I looked into high-speed networking in small spaces, my idea was doing it over PCI directly bypassing Ethernet. I really wanted to do something like SGI's NUMAlink but PCI be cheaper with commodity parts available. You'd need a switch for it, though. I ended up finding a company selling them. So, there's always that kind of thing to consider. I wonder if 10-100Gbps have better price/performance than TCP/IP over PCI by now. I cant even rember the company or tech's name.

wmf 8y ago
That's possible with PLX/Broadcom ExpressFabric but those switches have fallen far behind Ethernet. The fastest PCIe switch is ~768 Gbps while ~6500 Gbps Ethernet switches are available.
- namibj 8y ago
  You need to consider that these PCIe switches you talk about are (afaik, at least there are big ones) single-die, compared to the probably even multi-board construction of the Ethernet switches. They do support and are in at least one document advertised to be used in a switching fabric, where you create a mesh between the nodes you try to connect. Basically, instead of plugging every computer into the same PCIe switch, you fill the PCIe switches half full with computers, maybe even a little less/add some that don't connect directly to computers, and add some interconnections between the switches. The mathematics that describe how one would want to arrange the small switches date back at least to when automated telephone switches became common. A similar version, though with separated inputs and outputs, is known as a Crossbar Switch. Did you by chance come across something smaller than a full-blown, PCIe3x4 per node, TOR switch? I don't need that much capacity, but a cheap enough high speed switch would allow using some slower (individually) servers that got kicked out from the previous owner, instead of getting new, larger machines. A certain level of microservice-thinking/scale of (distributed) Haskell/Erlang does often require much faster connections than one would like to afford with Ethernet, to keep the coding overhead for the sharding to a minimum. At a certain point one can stop worrying about minimizing transfers, and just rely on page-faulting and similar things for less-critical (in a performance sense) data accesses.
  - wmf 8y ago
    I was talking about single-chip switches. The thing about Clos is that it has non-linear scaling, so if you use a chip with lower capacity you end up needing far more chips. All of these chips have smaller sizes available.
rasz 8y ago
There was a product in late nineties that did P2P links over ATA66/100 IDE interface. No drivers needed, just write/read to a virtual filesystem on simulated drive.
daveguy 8y ago
I wanted to do the same thing with PCI!! I thought the bandwidth was large enough that it should be a great direct interface. Unfortunately synchronizing between two pci masters was going to be tricky and the high frequency development required for PCIe requires some pricey diagnostic equipment. I ran across those off the shelf switches / interfaces, but they were a lot more expensive that 10gb ethernet at the time and they weren't even available for high lane count pcie 2 and pcie 3. I think now that 10gb Ethernet prices are coming down to about $100 per card and switch port there's no way specialized direct pcie would compete off the shelf. Still I think it would be cool to have a direct 32 lane pcie 3 of 4. It could operate as fast as ram speeds. If you connected two 2P beasts it would be close to having a 4P. Of course there would probably be protocol issues and overhead as the no-two-masters issue would mean you can't use the native protocol directly.
- nickpsecurity 8y ago
  I didnt think about the specialized, diagnostic gear. That couldve priced me out. Far as your other idea, a Distributed, Shared Memory setup was one of my potential applications. I was looking at Beowulf clusters and old MPP systems trying to find ways to get more CPU's with large RAM and simplified programming. I found quite a few software implementations of DSM model. Infiniband and Myrinet were main thing in networking. So, PCI idea was partly about connecting them with something that already had DMA. That at least led me to NUMAscale's products for scaling AMD systems.
  - exikyut 8y ago
    Note this comment in this subthread: https://news.ycombinator.com/item?id=17276861 https://news.ycombinator.com/item?id=17276861
- namibj 8y ago
  Don't worry, the chip you are searching is branded ExpressFabric and sold/made by Broadcom. It's AFAIK only PCIe4, and you get ones with up to iirc 96 lanes per chip. They support to be connected into a mesh, even adding shared PCIe devices like network cards, GPUs, NVMe and SAS/SATA HBAs is possible, though you might have difficulties with the GPUs in particular. The others, if you are careful when buying, should do well if they can handle SR-IOV, as (from their view), the mesh switch they are attached to is the root, and the computers attached to the mesh take the place a VM would normally occupy. If you could get the smallest chip they make, it should not be hard to get a PCB made for it that fans out to e.g. a set of USB-C receptacles. One such receptacle can handle 2 PCIe3 lanes, and some power/USB-2 on the side. The main reason being the relatively low price, compared to e.g. mini SAS HD, which has a similar density and twice the lanes per connector, but also a slightly higher target impedance at the upper end of the PCIe spec, whereas USB-C is specifically targeting the perfect PCIe impedance.

wmf 8y ago

That's possible with PLX/Broadcom ExpressFabric but those switches have fallen far behind Ethernet. The fastest PCIe switch is ~768 Gbps while ~6500 Gbps Ethernet switches are available.

namibj 8y ago
You need to consider that these PCIe switches you talk about are (afaik, at least there are big ones) single-die, compared to the probably even multi-board construction of the Ethernet switches. They do support and are in at least one document advertised to be used in a switching fabric, where you create a mesh between the nodes you try to connect. Basically, instead of plugging every computer into the same PCIe switch, you fill the PCIe switches half full with computers, maybe even a little less/add some that don't connect directly to computers, and add some interconnections between the switches. The mathematics that describe how one would want to arrange the small switches date back at least to when automated telephone switches became common. A similar version, though with separated inputs and outputs, is known as a Crossbar Switch. Did you by chance come across something smaller than a full-blown, PCIe3x4 per node, TOR switch? I don't need that much capacity, but a cheap enough high speed switch would allow using some slower (individually) servers that got kicked out from the previous owner, instead of getting new, larger machines. A certain level of microservice-thinking/scale of (distributed) Haskell/Erlang does often require much faster connections than one would like to afford with Ethernet, to keep the coding overhead for the sharding to a minimum. At a certain point one can stop worrying about minimizing transfers, and just rely on page-faulting and similar things for less-critical (in a performance sense) data accesses.
- wmf 8y ago
  I was talking about single-chip switches. The thing about Clos is that it has non-linear scaling, so if you use a chip with lower capacity you end up needing far more chips. All of these chips have smaller sizes available.

rasz 8y ago

There was a product in late nineties that did P2P links over ATA66/100 IDE interface. No drivers needed, just write/read to a virtual filesystem on simulated drive.

daveguy 8y ago

I wanted to do the same thing with PCI!! I thought the bandwidth was large enough that it should be a great direct interface. Unfortunately synchronizing between two pci masters was going to be tricky and the high frequency development required for PCIe requires some pricey diagnostic equipment. I ran across those off the shelf switches / interfaces, but they were a lot more expensive that 10gb ethernet at the time and they weren't even available for high lane count pcie 2 and pcie 3. I think now that 10gb Ethernet prices are coming down to about $100 per card and switch port there's no way specialized direct pcie would compete off the shelf. Still I think it would be cool to have a direct 32 lane pcie 3 of 4. It could operate as fast as ram speeds. If you connected two 2P beasts it would be close to having a 4P. Of course there would probably be protocol issues and overhead as the no-two-masters issue would mean you can't use the native protocol directly.

nickpsecurity 8y ago
I didnt think about the specialized, diagnostic gear. That couldve priced me out. Far as your other idea, a Distributed, Shared Memory setup was one of my potential applications. I was looking at Beowulf clusters and old MPP systems trying to find ways to get more CPU's with large RAM and simplified programming. I found quite a few software implementations of DSM model. Infiniband and Myrinet were main thing in networking. So, PCI idea was partly about connecting them with something that already had DMA. That at least led me to NUMAscale's products for scaling AMD systems.
- exikyut 8y ago
  Note this comment in this subthread: https://news.ycombinator.com/item?id=17276861 https://news.ycombinator.com/item?id=17276861
namibj 8y ago
Don't worry, the chip you are searching is branded ExpressFabric and sold/made by Broadcom. It's AFAIK only PCIe4, and you get ones with up to iirc 96 lanes per chip. They support to be connected into a mesh, even adding shared PCIe devices like network cards, GPUs, NVMe and SAS/SATA HBAs is possible, though you might have difficulties with the GPUs in particular. The others, if you are careful when buying, should do well if they can handle SR-IOV, as (from their view), the mesh switch they are attached to is the root, and the computers attached to the mesh take the place a VM would normally occupy. If you could get the smallest chip they make, it should not be hard to get a PCB made for it that fans out to e.g. a set of USB-C receptacles. One such receptacle can handle 2 PCIe3 lanes, and some power/USB-2 on the side. The main reason being the relatively low price, compared to e.g. mini SAS HD, which has a similar density and twice the lanes per connector, but also a slightly higher target impedance at the upper end of the PCIe spec, whereas USB-C is specifically targeting the perfect PCIe impedance.