3 ms·
Does that entire model fit in gpu memory? How's it run? I tried running a model larger than ram size and it loads some layers into the gpu but offloads to the
by larntz 2y ago
Does that entire model fit in gpu memory? How's it run?
I tried running a model larger than ram size and it loads some layers into the gpu but offloads to the cpu also. It's faster than cpu alone for me, but not by a lot.
- shosca 2y agoyou're right, actually noticed gpu clocking up and down with 32b, 14b clocks up fully and actually runs faster