3 ms·
I haven't benchmarked against a pro 6000, it's more that i have 4 3090s and i don't have a pro 6000.
by mips_avatar 6mo ago
I haven't benchmarked against a pro 6000, it's more that i have 4 3090s and i don't have a pro 6000.
- embedding-shape 6mo agoYes, that's why I'm asking you what exactly 4 3090s get in prompt-processing and generation, sorry if I was unclear.
- mips_avatar 6mo agoMaxes out around 4K tok/s output. Each pair of 3090s has its own instance of the model with parallelism across the nvlink bridge. Though nvlink is only 2x over pcie5