3 ms·
IMO the relevant benchmark for now is a mixed stream of requests with 50 (20%), 500 (50%), 2000 (10%) and 50k (20%) input tokens, ignore EOS and decode until yo
by rfoo 2y ago
IMO the relevant benchmark for now is a mixed stream of requests with 50 (20%), 500 (50%), 2000 (10%) and 50k (20%) input tokens, ignore EOS and decode until you get around 300 output tokens.
- SuchAnonMuchWow 2y agoI'm really interested, do you have a source for those percentages ? I tried to look for some service provider to publish this kind of metrics, but haven't found any.
- rfoo 2y agoSorry, I can't. My employer doesn't publish this kind of metrics, either. What I posted was definitely just some very rough number off my brain.