4 ms·
I have an M3 chip in my laptop, it has more memory than my 4090 but it's still way slower when inferencing. So as long as the model fits in memory, Nvidia GPUs
by saberience 2y ago
I have an M3 chip in my laptop, it has more memory than my 4090 but it's still way slower when inferencing. So as long as the model fits in memory, Nvidia GPUs are going to be way faster just because they have more/faster compute cores.
Of course, if the model fits in memory on your M chip and doesn't in your Nvidia chip, the M chip wins by default. However, I would say, if you load a 70B model in your M chip, while it WILL work, the tokens/sec will be slow as fuck... so it kinda doesn't matter anyway.
- throw_nbvc1234 2y agoHow does the inference for LLMs impact battery life? For SD it can be 5% battery per image at times.
- NBJack 2y agoThe latest Nvidia drivers offer an option to start using system memory when VRAM is insufficient. It certainly slows things down, but it does work. It's not perfect in my experience, but it is an alternative for large models.
- wiradikusuma 2y agoSounds great! Do you have source maybe a wiki?
- NBJack 2y agoThis might be the best place to start: https://nvidia.custhelp.com/app/answers/detail/a_id/5490/~/system-memory-fallback-for-stable-diffusion https://nvidia.custhelp.com/app/answers/detail/a_id/5490/~/s...