4 ms·
I just ran it up on 48gb (2x 3090) + overflow into CPU RAM and it runs at around 4tk/s (only a little 8k context size though) which while absolutely not somethi
by smcleod 2y ago
I just ran it up on 48gb (2x 3090) + overflow into CPU RAM and it runs at around 4tk/s (only a little 8k context size though) which while absolutely not something I'd personally use daily - it is actually usable.
- eurekin 2y agoI have similar set-up - can you help out with running it? Was it in ollama? EDIT: It seems that original authors provided a nice write-up: https://unsloth.ai/blog/deepseekr1-dynamic#:~:text=%F0%9F%96%A5%EF%B8%8F%20Running%20Dynamic%20Quants https://unsloth.ai/blog/deepseekr1-dynamic#:~:text=%F0%9F%96...
- smcleod 2y agoYep that's pretty much what I did, their calculation for the layers was slightly off though, I found I could offload an extra 1-2 layers to the GPUs
- danielhanchen 2y agoOh yes I reduced it by 4 for just in case :) I found sometimes the formula doesn't work, so in the worst case -4 was used - glad at least it ran!
- ryao 2y agoWhich one did you run?