4 ms·
Nice idea, I do the same with Ollama and local models, except my client code is in Common Lisp, Clojure, and Racket. I have three books for these languages with
by mark_l_watson 3y ago
Nice idea, I do the same with Ollama and local models, except my client code is in Common Lisp, Clojure, and Racket. I have three books for these languages with Ollama examples, all can be read free online https://leanpub.com/u/markwatson https://leanpub.com/u/markwatson
I have been paid to do so-called “AI work” since 1982, lots of early work with neural networks and symbolic AI, then more recently deep learning. I have never been as excited about any technology in my life as I am about LLMs.
Commercial APIs from Anthropic, Mistral, and OpenAI are great tools, but I get off more on running smaller models locally myself.
- emersonmacro 3y agoWhat models do you get the best results with locally?
- mark_l_watson 3y agoI like mistral:7b-instruct, yi:34b, and wizard-vicuna-uncensored:30b. I think the so-called "uncensored" models tend to work better for general purpose, but mistral and yi aren't available uncensored. I have a M2 Pro 32G memory so I need to use 3-bit quantization to run mixtral: dolphin-mixtral:8x7b-v2.5-q3_K_S. In general I don't like to go below 4-bit quantization.