2 ms·
So would 40x RPi 5 get 130 token/s?
by varispeed 1y ago
So would 40x RPi 5 get 130 token/s?
- SillyUsername 1y agoI imagine it might be limited by number of layers and you'll get diminishing returns as well at some point caused by network latency.
- VHRanger 1y agoMost likely not because of NUMA bottlenecks
- reilly3000 1y agoIt has to be 2^n nodes and limited to one per attention head that the model has.