You need to consider that these PCIe switches you talk about are (afaik, at least there are big ones) single-die, compared to the probably even multi-board cons

You need to consider that these PCIe switches you talk about are (afaik, at least there are big ones) single-die, compared to the probably even multi-board construction of the Ethernet switches. They do support and are in at least one document advertised to be used in a switching fabric, where you create a mesh between the nodes you try to connect. Basically, instead of plugging every computer into the same PCIe switch, you fill the PCIe switches half full with computers, maybe even a little less/add some that don't connect directly to computers, and add some interconnections between the switches. The mathematics that describe how one would want to arrange the small switches date back at least to when automated telephone switches became common. A similar version, though with separated inputs and outputs, is known as a Crossbar Switch. Did you by chance come across something smaller than a full-blown, PCIe3x4 per node, TOR switch? I don't need that much capacity, but a cheap enough high speed switch would allow using some slower (individually) servers that got kicked out from the previous owner, instead of getting new, larger machines. A certain level of microservice-thinking/scale of (distributed) Haskell/Erlang does often require much faster connections than one would like to afford with Ethernet, to keep the coding overhead for the sharding to a minimum. At a certain point one can stop worrying about minimizing transfers, and just rely on page-faulting and similar things for less-critical (in a performance sense) data accesses.