4 ms·
I have similar set-up - can you help out with running it? Was it in ollama? EDIT: It seems that original authors provided a nice write-up: https://unsloth.ai/
by eurekin 2y ago
I have similar set-up - can you help out with running it? Was it in ollama?
EDIT: It seems that original authors provided a nice write-up:
https://unsloth.ai/blog/deepseekr1-dynamic#:~:text=%F0%9F%96%A5%EF%B8%8F%20Running%20Dynamic%20Quants https://unsloth.ai/blog/deepseekr1-dynamic#:~:text=%F0%9F%96...
- smcleod 2y agoYep that's pretty much what I did, their calculation for the layers was slightly off though, I found I could offload an extra 1-2 layers to the GPUs
- danielhanchen 2y agoOh yes I reduced it by 4 for just in case :) I found sometimes the formula doesn't work, so in the worst case -4 was used - glad at least it ran!