3 ms·

That’s pretty awesome! Though only 5gig Ethernet? Can’t they do usb-c / thunderbolt 40 Gb/s connections like Macs?

evanjrowley 8mo ago
I appreciate them showcasing Framework Computer hardware, but they would have got 40 GB/s network performance if they had chosen the Minisorum MS-S1 Max: https://www.minisforum.com/pages/new-release-ms-s1-max-ryzen%E2%84%A2-ai-max-395-mini-workstation-minisforum https://www.minisforum.com/pages/new-release-ms-s1-max-ryzen...
- mycall 7mo ago
  The FC hardware has room for a PCIe x4 with maybe ConnectX-7 @ 100GbE mode.
jauntywundrkind 8mo ago
I really wonder if AMD is going to keep getting walloped on the interconnect or if they'll start upping what's available to consumers, at some point.
TacticalCoder 8mo ago
> Though only 5gig Ethernet? Can’t they do usb-c / thunderbolt 40 Gb/s connections like Macs? Does the network speed matter that much when TFA talks about outputting a few tens of tokens per second? Ain't 5 Gbit/s plenty for that? (I understand the need to load the model but that'd be local already right?)
- elcritch 7mo ago
  Running inference requires sharing intermediate matrix results between nodes. Faster networking speeds that up.
  - wokkel 7mo ago
    I read (but cannot find this anymore) that the information sent from layer to layer is minimal. The actual matrix work happens within a layer. They are not doing matrix multiplication over the netwerk (that would be insane latency wise).
    - elcritch 7mo ago
      The LLM/transformers attention layers require an O(n^2) operation between all tokens, which does require significant bandwidth. Yes the latency hurts performance, that why it’s only achieving ~8tok/s.
namibj 8mo ago
It's sad that NDA fetishist Broadcom has a de-facto monopoly on PCIe fabric switches; notably we would have functional open source drivers for at least simpler topologies for a while now, and could just set up cheap FNN topologies by using (usually NMVe targeted) bifurcation support on hosts to get several x4 ports with only a comparatively cheap retimer out into "mini SAS hd" (the square shaped 4-Lane connectors) or QSFP+ ports; and then have a few meters reach on generic DAC cables from such standards (even Skylake-era SAS ones (nominally 12 GT/s; PCIe4.0 is 16 GT/s) should typically manage PCIe4; that's just under 64 Gbit/s from each link, with typical desktop/gaming systems delivering 3~5 links without complaints next to a dGPU (that one at fewer than full lanes).

I appreciate them showcasing Framework Computer hardware, but they would have got 40 GB/s network performance if they had chosen the Minisorum MS-S1 Max: https://www.minisforum.com/pages/new-release-ms-s1-max-ryzen%E2%84%A2-ai-max-395-mini-workstation-minisforum https://www.minisforum.com/pages/new-release-ms-s1-max-ryzen...

mycall 7mo ago
The FC hardware has room for a PCIe x4 with maybe ConnectX-7 @ 100GbE mode.

jauntywundrkind 8mo ago

I really wonder if AMD is going to keep getting walloped on the interconnect or if they'll start upping what's available to consumers, at some point.

TacticalCoder 8mo ago

> Though only 5gig Ethernet? Can’t they do usb-c / thunderbolt 40 Gb/s connections like Macs? Does the network speed matter that much when TFA talks about outputting a few tens of tokens per second? Ain't 5 Gbit/s plenty for that? (I understand the need to load the model but that'd be local already right?)

elcritch 7mo ago
Running inference requires sharing intermediate matrix results between nodes. Faster networking speeds that up.
- wokkel 7mo ago
  I read (but cannot find this anymore) that the information sent from layer to layer is minimal. The actual matrix work happens within a layer. They are not doing matrix multiplication over the netwerk (that would be insane latency wise).
  - elcritch 7mo ago
    The LLM/transformers attention layers require an O(n^2) operation between all tokens, which does require significant bandwidth. Yes the latency hurts performance, that why it’s only achieving ~8tok/s.

namibj 8mo ago

It's sad that NDA fetishist Broadcom has a de-facto monopoly on PCIe fabric switches; notably we would have functional open source drivers for at least simpler topologies for a while now, and could just set up cheap FNN topologies by using (usually NMVe targeted) bifurcation support on hosts to get several x4 ports with only a comparatively cheap retimer out into "mini SAS hd" (the square shaped 4-Lane connectors) or QSFP+ ports; and then have a few meters reach on generic DAC cables from such standards (even Skylake-era SAS ones (nominally 12 GT/s; PCIe4.0 is 16 GT/s) should typically manage PCIe4; that's just under 64 Gbit/s from each link, with typical desktop/gaming systems delivering 3~5 links without complaints next to a dGPU (that one at fewer than full lanes).