4 ms·
From the title I misunderstood what it does. However, now I'm wondering if what I thought is was (don't ask my why I thought it) is possible: I have a PC that
by katzenversteher 3y ago
From the title I misunderstood what it does. However, now I'm wondering if what I thought is was (don't ask my why I thought it) is possible:
I have a PC that is able to run e.g. Mistral Instruct 7B Q4 inference with around 30 token/s.
How (computation and memory) expensive would it be to also run backpropagation in addition to inference?
I'm aware that the models are typically fed with much more and better data than what is typically provided during normal conversations but on the other hand if I could finetune my local model a teeny tiny bit during during / after each conversation I have with it anyways, it would after a while be perfectly customize for me.
I'm also aware that this could be problematic for models that are used by multiple users but my intended use case would be personal use by a single user.
- dartos 3y agoVery expensive. AFAIK the model can’t be quantized during backprop, so right there you’d need a ton of RAM. Backprop is faster bc it can be parallelized, but IIRC you need to hold an entire copy of the model for each backprop process.
- scribu 3y agoActually, there have been attempts to do quantized backprop, but not sure how successfully.
- tveita 3y agoFor an idea of what's possible you might be interested in this story that was just on HN where they fine tune a quantized 70b model in 48GB VRAM: https://www.answer.ai/posts/2024-03-06-fsdp-qlora.html https://www.answer.ai/posts/2024-03-06-fsdp-qlora.html