5 ms·
A Mac with a lot of unified RAM can do it, or a dual 3090/4090 setup gets you 48gb of VRAM.
by mlsu 3y ago
A Mac with a lot of unified RAM can do it, or a dual 3090/4090 setup gets you 48gb of VRAM.
- jadbox 3y agoDoes this actually work? I had thought that you can't use SLI to increase your net memory for the modal?
- speedgoose 3y agoIt works. I use ollama these days, with litellm for the api compatibility, and it seems to use both 24GB GPUs on the server.
- benjaminwootton 3y agoI’ve got a 64gb Mac M2. All of the openllm models seem to hang on startup or on API calls. I got them working through GCP colab. Not sure if it’s a configuration issue or if the hardware just isn’t up to it?
- benreesman 3y agoValiant et al work great on my 64Gb Studio at Q4_K_M. Happy to answer questions.
- wahnfrieden 3y agoTry llama.cpp with Metal (critical) and GGUF models from TheBloke Or wait another month or so for https://ChatOnMac.com https://ChatOnMac.com