2 ms·
FlexGen can already run GPT-3 size models on commodity hardware, albeit with high latency and fairly slow throughput (order of 1 token/s).
by sebzim4500 4y ago
FlexGen can already run GPT-3 size models on commodity hardware, albeit with high latency and fairly slow throughput (order of 1 token/s).