4 ms·
I was able to get this running on an 8gb M2. It’s a lot faster than I expected. It’s not lighting fast, but, it is moderately usable.
by zoba 4y ago
I was able to get this running on an 8gb M2. It’s a lot faster than I expected. It’s not lighting fast, but, it is moderately usable.
- tmalsburg2 4y agoYeah, 63ms/token (number from the post) is totally usable. Didn't the CPU version that was recently posted take 8min/token? But I think that was a larger model.
- aledalgrande 4y agoUnless I'm misunderstanding, it would be usable for tests/dev but not for production at that speed. If you feed in a complex prompt of 500 tokens it would take way too long.