3 ms·
I agree. Their 70B parameter benchmarks are better than 7B. Something is wrong with their software and they need to fix it. They get 4000 token/s on Falcon 7B,
by imtringued 2y ago
I agree. Their 70B parameter benchmarks are better than 7B. Something is wrong with their software and they need to fix it. They get 4000 token/s on Falcon 7B, which is what you should expect from processing a batch of prompts. The per user performance is atrocious and their hardware should be at least twice as fast as a moderately overclocked DDR5 system.
- qeternity 2y agoYou can do 4k t/s at large batch on last gen consumer gear. You’re right their 70B are more competitive but still very much off the mark. If these cards cost $500 instead of $1400 then maybe it would be more compelling.