4 ms·
Yeah, 63ms/token (number from the post) is totally usable. Didn't the CPU version that was recently posted take 8min/token? But I think that was a larger mode
by tmalsburg2 4y ago
Yeah, 63ms/token (number from the post) is totally usable. Didn't the CPU version that was recently posted take 8min/token? But I think that was a larger model.
- aledalgrande 4y agoUnless I'm misunderstanding, it would be usable for tests/dev but not for production at that speed. If you feed in a complex prompt of 500 tokens it would take way too long.