3 ms·
Note that it's possible to use things like llama.cpp, ggllm.cpp, Kobold, etc. to run these at usable performance on fast CPUs with a good amount of RAM. This is
by api 3y ago
Note that it's possible to use things like llama.cpp, ggllm.cpp, Kobold, etc. to run these at usable performance on fast CPUs with a good amount of RAM. This is far, far more reasonable if you just want to play around. If you want to seriously use these or experiment with any non-trivial training, that's when you need the kind of power being talked about here.
Apple Silicon is interesting for AI because the GPU shares main memory, so you can get one with a lot of RAM and use either CPU or GPU. Less support for Apple GPUs though, at least so far. Work is being done. M1/M2 Macs with 64GiB of RAM are cheaper than many of these GPUs.
Also wow does NVIDIA have near monopoly on this space. Prices will probably come down once Intel and AMD get their acts together.
- tikkun 3y agoYes. For anyone looking for more info on running models locally, https://www.reddit.com/r/LocalLLaMA/wiki/index/ https://www.reddit.com/r/LocalLLaMA/wiki/index/ is the best place to start.
- brucethemoose2 3y agoUnfortunately the AMD MI300 will not be cheap, Intel's Xeon Max GPUs are more-or-less limited to HPC land until at least 2025, and Intel's Gaudi line is apparently being discontinued. There are some worrying reports about Intel Arc gen 2, and AMD is just matching price/VRAM with Nvidia these days. Some of the AI startups are really exotic/business scale only (like Cerebras) or MIA in the consumer/cheap cloud space (like Tenstorrent). ...There are some rumors of an M2 Pro Like APU from AMD/Intel? I hate so sound so grim, but thats about the only positive news I got.
- api 3y agoHrm... yeah I've thought Xeon Max with AVX512 might be competitive with GPUs but these things are as expensive as GPUs or even more: https://ark.intel.com/content/www/us/en/ark/products/232592/intel-xeon-cpu-max-9480-processor-112-5m-cache-1-90-ghz.html https://ark.intel.com/content/www/us/en/ark/products/232592/... I've seen some interesting work on deploying models to FPGAs, but high-end FPGAs are also expensive.