3 ms·
On my M1 Pro/16GB RAM mac I get decently fast, fully local LLMs which are good enough to do this sort of thing. I use them in scripts all the time. Granted, I h
by d3nj4l 3y ago
On my M1 Pro/16GB RAM mac I get decently fast, fully local LLMs which are good enough to do this sort of thing. I use them in scripts all the time. Granted, I haven’t checked the impact on the battery life I get, but I definitely haven’t noticed any differences in my regular use.
- mosselman 3y agoWhich models do you run and how?
- _joel 3y agonot op, but this is handy https://lmstudio.ai/ https://lmstudio.ai/
- mosselman 3y agoThanks!
- verdverm 3y agohttps://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp is a popular local first approach. LLaMa is a good place to start, though I typically use a model from Vertex AI via API
- mosselman 3y agoThanks. I have llama.cpp locally. How do you use it in scripts? As in how do you specifically, not how would one.
- d3nj4l 3y agoI have ollama's server running, and I interact with it via the REST API. My preferred model right now is Intel's neural chat, but I'm going to experiment with a few more over the holidays.
- verdverm 3y agoI tried ollama today and it is super easy, finding good models is definitely going to be challenging. I tried a few on some sample (JSON) tasks and it is... frustrating... how they ellide or are unable to follow instructions. Is there a good fine-tuning workflow with ollama?
- d3nj4l 3y agoI haven’t tried any fine tuning so I can’t help there, sorry. Though I will say that neural chat has been pretty good to me, even though I have definitely observed it ignoring instructions at times, like a “here’s your json:” preamble in response to a query that specifically requests only json.
- d3nj4l 3y agoI use ollama (https://ollama.ai/ https://ollama.ai/) which supports most of the big new local models you might've heard of: llama2, mistral vicuna etc. Since I have 16GB of RAM, I stick to the 7b models.