4 ms·
I'm impressed by not only how good deepseek r1 is, but also how good the smaller distillations are. qwen-based 7b distillation of deepseek r1 is a great model
by buyucu 2y ago
I'm impressed by not only how good deepseek r1 is, but also how good the smaller distillations are. qwen-based 7b distillation of deepseek r1 is a great model too.
the 32b distillation just became the default model for my home server.
- OCHackr 2y agoHow much VRAM is needed for the 32B distillation?
- jadbox 2y agoDepends on compression, I think 24gb can hold a 32B at around 3b-4b compression.
- deleted 2y ago[deleted]
- brandall10 2y agoDepends on the quant used and the context size. On a 24gb card you should be able to load about a 5 bit if you keep the context small. In general, if you're using 8bit which is virtually lossless, any dense model will require roughly the same amount as the number of params w/ a small context, and a bit more as you increase context.
- buyucu 2y agoI had no problems running the 32b at q4 quantization with 24GB of ram.
- magicalhippo 2y agoI just tries the distilled 8b Llama variant, and it had very poor prompt adherence. It also reasoned its way to an incorrect answer, to a question plain Llama 3.1 8b got fairly correct. So far not impressed, but will play with the qwen ones tomorrow.
- singularity2001 2y agonot adhering to system prompts is even officially mentioned as one of the caveats of the distilled models I wonder if this has to do with their censorship agenda but other report that it can be easily circumvented
- magicalhippo 2y agoI didn't have time to dig into the details of the models, but that makes sense I guess. I tried the Qwen 7B variant and it was indeed much better than the base Qwen 7B model at various math word problems.
- ThouYS 2y agotried the 7b, it switched to chinese mid-response
- popinman322 2y agoAssuming you're doing local inference, have you tried setting a token filter on the model?
- brookst 2y agoGreat as long as you’re not interested in Tiananmen Square or the Uighurs.
- buyucu 2y agoI just tried asking ChatGPT how many civilians Israel murdered in Gaza. It didn't answer.
- whimsicalism 2y agoamerican models have their own bugbears like around evolution and intellectual property
- miohtama 2y agoFor sensitive topics, it is good that we canknow cross ask Grok, DeepSeek and ChatGPT to avoid any kind of biases or no-reply answers.
- lurking_swe 2y agoi can’t think of a single commercial use case, outside of education, where that’s even relevant. But i agree it’s messed up from an ethical / moral perspective.
- brookst 2y agoWell those are the overt political biases. Would you trust DeepSeek to advise on negotiating with a Chinese business? I’m no xenophobe, but seeing the internal reasoning of DeepSeek explicitly planning to ensure alignment with the government give me pause.
- ideashower 2y agocan I ask, what do you do with it on your home server?