4 ms·
LMStudio is great, if a bit daunting. If you’re on Mac and want a native open source interface, try out FreeChat https://www.freechat.run https://www.freechat.r
by rgbrgb 3y ago
LMStudio is great, if a bit daunting. If you’re on Mac and want a native open source interface, try out FreeChat https://www.freechat.run https://www.freechat.run
- NoMoreNicksLeft 3y agoThanks for the link. I expected it to not let me run this. I have an intel Macbook, was expecting that I'd need Apple Silicon... am I misunderstanding something? I get fairly fast results at the prompt with the default model. How's this thing running with whatever shitty GPU I have in my laptop?
- rgbrgb 3y agothat's the magic of llama.cpp! I include a universal binary of llama.cpp's server example to do inference. What's your machine? The lowest spec I've heard it running on is a 2017 iMac with 8GB RAM (~5.5 tokens/s). On my m1 with 64GB RAM I get ~30 tokens per second on the default 7B model.
- NoMoreNicksLeft 3y agoMacbook Pro 2020 with 16gb of system ram. I think the gpu is Iris Plus? But I don't much keep up on those. I'm now delving into getting this running in Terminal... there are a few things I want to try that I don't think the simple interface allows. Also, I've noticed that when chats get a few kilobytes long, it just seizes up and can't go further. I complained to it, it spent a sentence apologizing, started up where it left off... and got about 12 words further.
- rgbrgb 3y agohm yeah i think I need to update llama.cpp to get this fix https://github.com/ggerganov/llama.cpp/pull/3996 https://github.com/ggerganov/llama.cpp/pull/3996 Thanks for trying it!