5 ms·
It’s mind bogglingly crazy that language models rivaling ones that used to require huge GPUs with a ton of VRAM to run now run on my upper-mid-range laptop from
by solarkraft 2y ago
It’s mind bogglingly crazy that language models rivaling ones that used to require huge GPUs with a ton of VRAM to run now run on my upper-mid-range laptop from 4 years ago. At usable speed. Crazy.
I didn’t expect capable language models to be practical/possible to run loyally, much less on hardware I already have.
- baq 2y agoYou have a sota multi-modal LLM running in your head at 20W, shared with best in class sensor package and top performing robotics control unit. There’s soooo much more to optimize.
- fecal_henge 2y agoBut can it know love?
- baq 2y agoWord on the street is researchers looked at the weights and weights looked back. You’ll have to ask the weights.
- myrmidon 2y agoI would argue that our sensor package is losing its lead very quickly-- audio performance is already on par with current tech and image processing is closing the gap very quickly as well (it helps a lot that silicon-based technology is much less constrained on bandwidth). Tactile sensing is still lightyears ahead, and I don't see that situation improving anytime soon...
- IWeldMelons 2y agochemical sensing is still quite good though.
- deleted 2y ago[deleted]
- prox 2y agoI might have a go at installing one, what is a good source or install at the moment?
- wint3rmute 2y agoOllama was the easiest way to set up local LLMs for me. https://ollama.com/ https://ollama.com/
- solarkraft 2y agoWith llama3.2:1b, llama3.2:3b and llama 3.1:8b being the main ones I tried and found impressive.
- spacemanspiff01 2y agolmstudio is very easy if you are running on local desktop.
- Alifatisk 2y agomsty.app is good
- Eisenstein 2y agoIf you don't care about docker packages being used as installers and your home directory invisibly used to store massive weight files in exchange for not having to deal with learning any configuration: ollama or lmstudio. If you just want to play for a bit: llamafile If you want granular control with ease of execution in exchange for having to figure out what the settings mean and figure out which weights to download: koboldcpp. (check out bartowski on huggingface for the weights) These are all based on llamacpp as a backend, by the way.
- gavmor 2y agoI run ollama off a symlink to an external volume. It just feels neater that way, and can run any GGUF off of HuggingFace. I would like to know what configuration I'm missing out on, though.
- mandmandam 2y ago> I didn’t expect capable language models to be practical/possible to run loyally Now there's a fun typo. Hopefully not too much fun.
- momojo 2y agoThat sentence triggered memories of reading the huge paperback Asimov compilation my mom kept on the bookshelf as a kid.