3 ms·

This is fantastic. it's really nice to see the different speed comparisons. Does anyone know how a large model performs on some serious hardware? How much faste

This is fantastic. it's really nice to see the different speed comparisons. Does anyone know how a large model performs on some serious hardware? How much faster are these top-grade Nvidia cards than a 4090? Do models that require multiple cards to run in memory run with equivalent performance of the multiple cards or are they throttled by the memory access?

brucethemoose2 3y ago
> Do models that require multiple cards to run in memory run with equivalent performance of the multiple cards or are they throttled by the memory access? It depends, but some frameworks just run the layers sequentially. So you get ~50% utilization from 2 GPUs unless you pipeline requests from multiple users