2 ms·
What will be the approx token/s prompt processing and generation speed with this setup on RTX 4090?
by Jayakumark 1y ago
What will be the approx token/s prompt processing and generation speed with this setup on RTX 4090?
- danielhanchen 1y agoI also just made IQ1_M which needs 160GB! If you have 160-24 = 136 ish of RAM as well, then you should get 3 tokens to 5 ish per second. If you don't have enough RAM, then < 1 token / s