aarushsah

4 ms·

1. 1.▲
   by aarushsah 3y ago

   Just tried it - doesn't seem to be working. In fact, I'm getting 1.4 t/s with a Quadro P4000 (8 GB) running a 7B at 3 bits per weight. Are you changing anything other than the 8 bit cache and context? For reference, I'm
2. 2.▲
   by aarushsah 3y ago

   How so? I'm only getting 12 t/s using Mistral in LM Studio.
3. 3.▲
   High Schooler looking for a SWE internship

   2 points by aarushsah 4y ago|2 comments