4 ms·
Importantly they note that using a draft model screws it up and this was my experience. I was initially impressed, then started seeing problems, but after disab
by jasonjmcghee 1y ago
Importantly they note that using a draft model screws it up and this was my experience. I was initially impressed, then started seeing problems, but after disabling my draft model it started working much better. Very cool stuff- it's fast too as you note.
The /think and /no_think commands are very convenient.
- marcalc 1y agoWhat do you mean by draft model? And how would one disable it? Cheers
- _neil 1y agoA draft model is something that you would explicitly enable. It uses a smaller model to speculatively generate next tokens, in theory speeding up generation. Here’s the LM Studio docs on it: https://lmstudio.ai/docs/app/advanced/speculative-decoding https://lmstudio.ai/docs/app/advanced/speculative-decoding
- woadwarrior01 1y agoThat should not be the case. Speculative decoding is trading off compute for memory bandwidth. The model's output is guaranteed to be the same, with or without it. Perhaps there's a bug in the implementation that you're using.