3 ms·

That is a neat setup. I wish someone would do this but just run RMII out to an edge connector on the back. Connect them to a jelly bean switch chip (8 port GbE

That is a neat setup. I wish someone would do this but just run RMII out to an edge connector on the back. Connect them to a jelly bean switch chip (8 port GbE are like $8 in qty) Signal integrity on, at most 4" of PCB trace should not be a problem. You could bring the network "port status" lines to the front if you're interested in seeing the blinky lights of network traffic. The big win here would be that all of the network wiring is "built in" and compact. Blade replacement it trivial. Have your fans blow up from the bottom and stagger "slots" on each row and if you do 32 slots per row, you probably build a kilocore cluster in a 6U box. Ah the fun I would have with a lab with an nice budget.

nine_k 4y ago
What kind of fun might that be?
- ChuckMcM 4y ago
  Well for one, I'd build a system architecture I first imagined back at Sun in the early 90's which is a NUMA fabric attached compute/storage/io/memory scalable compute node. Then I'd take a shared nothing cluster (typical network attached Linux cluster) and refactor a couple of algorithms that can "only" run on super computers and have them run faster on a complex that costs 1/10th as much. That would be based on an idea that was generated by listening to IBM and Google talk about their quantum computers and explaining how they were going to be so great. Imagine replacing every branch in a program with an assert that aborts the program on fail. You send 10,000 copies of the program to 10,000 cores with the asserts set uniquely on each copy. The core that completes kicks off the next round.
themoonisachees 4y ago
Couldn't you do 1ki cores /4U with just Epyc CPUs in normal servers? At that point surely for cheaper, also significantly easier to build, and faster since the cores don't talk over Ethernet?
zokier 4y ago
> That is a neat setup. I wish someone would do this but just run RMII out to an edge connector on the back That stuck out to me too, they are making custom boards and custom chassis, surely it would be cleaner to route the networking and power through backplane instead of having gazillion tiny patch cables and random switch just hanging in there. Could also avoid the need for PoE by just having power buses in the backplane. Overall imho the point of blades is that some stuff gets offloaded to the chassis, but here the chassis doesn't seem to be doing much at all.
sitkack 4y ago
> jelly bean switch chip What do you have in mind? I couldn't find this part. Really am asking.
- ChuckMcM 4y ago
  Exemplar: https://octopart.com/ksz9477stxi-microchip-80980651 https://octopart.com/ksz9477stxi-microchip-80980651 The Chinese made ones are even cheaper, open up a TP-Link "desktop 8 port Gigabit Switch" and you will find the current "leader" in that market. Those datasheets though will be in Chinese so it helps to be able to read Chinese. (various translate apps are not well suited to datasheets in my experience)
  - sitkack 4y ago
    Thank you. Yeah, I found the ones on mouser and digikey. $20 is a bit much (not for a one off, but if you are aggregating low end processors you will need a lot of them). I'd love something like a 12-20 port 1Ge with a 10Ge uplink. If you find a super cheap 1Ge switch chip and docs (I suppose you could just reverse engineer the pcb from a tp-link switch), please post it.
    - ChuckMcM 4y ago
      No worries, the key though is cross section bandwidth. The "super cheap" GbE switch chips can have as little as 2.5 GBPS of cross section bandwidth which makes them ill suited for cluster operations.

nine_k 4y ago

What kind of fun might that be?

ChuckMcM 4y ago
Well for one, I'd build a system architecture I first imagined back at Sun in the early 90's which is a NUMA fabric attached compute/storage/io/memory scalable compute node. Then I'd take a shared nothing cluster (typical network attached Linux cluster) and refactor a couple of algorithms that can "only" run on super computers and have them run faster on a complex that costs 1/10th as much. That would be based on an idea that was generated by listening to IBM and Google talk about their quantum computers and explaining how they were going to be so great. Imagine replacing every branch in a program with an assert that aborts the program on fail. You send 10,000 copies of the program to 10,000 cores with the asserts set uniquely on each copy. The core that completes kicks off the next round.

themoonisachees 4y ago

Couldn't you do 1ki cores /4U with just Epyc CPUs in normal servers? At that point surely for cheaper, also significantly easier to build, and faster since the cores don't talk over Ethernet?

zokier 4y ago

> That is a neat setup. I wish someone would do this but just run RMII out to an edge connector on the back That stuck out to me too, they are making custom boards and custom chassis, surely it would be cleaner to route the networking and power through backplane instead of having gazillion tiny patch cables and random switch just hanging in there. Could also avoid the need for PoE by just having power buses in the backplane. Overall imho the point of blades is that some stuff gets offloaded to the chassis, but here the chassis doesn't seem to be doing much at all.

sitkack 4y ago

> jelly bean switch chip What do you have in mind? I couldn't find this part. Really am asking.

ChuckMcM 4y ago
Exemplar: https://octopart.com/ksz9477stxi-microchip-80980651 https://octopart.com/ksz9477stxi-microchip-80980651 The Chinese made ones are even cheaper, open up a TP-Link "desktop 8 port Gigabit Switch" and you will find the current "leader" in that market. Those datasheets though will be in Chinese so it helps to be able to read Chinese. (various translate apps are not well suited to datasheets in my experience)
- sitkack 4y ago
  Thank you. Yeah, I found the ones on mouser and digikey. $20 is a bit much (not for a one off, but if you are aggregating low end processors you will need a lot of them). I'd love something like a 12-20 port 1Ge with a 10Ge uplink. If you find a super cheap 1Ge switch chip and docs (I suppose you could just reverse engineer the pcb from a tp-link switch), please post it.
  - ChuckMcM 4y ago
    No worries, the key though is cross section bandwidth. The "super cheap" GbE switch chips can have as little as 2.5 GBPS of cross section bandwidth which makes them ill suited for cluster operations.