4 ms·
“Caniuse” equivalent for LLMs depending on machine specs would be extremely useful!
by andy_ppp 2y ago
“Caniuse” equivalent for LLMs depending on machine specs would be extremely useful!
- abdullin 2y agoThere are too many variables at play, unfortunately. One can ran local LLMs even on RaspberryPi, although it will be horribly slow.
- andy_ppp 2y agoMaybe it wouldn’t be an algorithm, maybe it would be a reporting site where you can review your experience if there’s no way to calculate it.
- abdullin 2y agoLocalLLaMA subreddit usually has some interesting benchmarks and reports. Here is one example, testing performance of different GPUs and Macs with various flavours of Llama: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inferen...
- Terretta 2y agoLM Studio on MacOS provides an estimate of whether a model will run on the GPU, also lets you partially offload. The underlying CLI tools do this, the app makes it easier to see and manage.