5 ms·
I strongly recommend that people run LLMs locally for a different reason. The ones you can run on your own machine tend to be bad - really bad. They hallucinat
by simonw 3y ago
I strongly recommend that people run LLMs locally for a different reason.
The ones you can run on your own machine tend to be bad - really bad. They hallucinate wildly and fail at all sorts of tasks that the larger hosted ones succeed at.
This makes them a fantastic tool for learning more about how LLMs work and what they're useful for. Interacting with a weak-but-functional LLM that runs on your own computer is a great way to get a much more solid mental model for what these things actually are.
- qingcharles 3y agoYou can have really bad and fast, or really slow and decent right now. Choose one :)
- devsda 3y agoFor someone interested in learning about LLMs, running them locally is a good way to understand the internals. For everyone else, I wish they experience these (locally or elsewhere) weak LLMs atleast once before using the commercial ones just to understand various failure modes and to introduce a healthy dose of skepticism towards the results instead of blindly trusting them to be the facts/truth.
- simonw 3y agoCompletely agree. Playing around with a weak LLM is a great way to give yourself a little bit of extra healthy skepticism for when you work with the strong ones.
- mmahemoff 3y agoHow do you learn about the internals by running LLMs locally? Are you playing with The code, runtime params, or just interacting via chat?
- samus 3y agoThe abstractions are relatively brittle. If you don't have a powerful GPU, you will be forced to consider how to split the model between CPU and GPU, how much context size you need, whether to quantize the model, and the tradeoffs implied by these things. To understand these, you have to develop a basic model how an LLM works.
- barrkel 3y agoBy interacting with it. You see the contours of its capabilities much more clearly, learn to recognize failure modes, understand how prior conversation can set the course of future conversation in a way that's almost impossible to correct without starting over or editing the conversation history.
- samus 3y agoThis skepticism is completely justified since ChatGPT 3.5 is also happily hallucinating things that don't exist. For example how to integrate a different system Python interpreter into pyenv. Though maybe ChatGPT 4 doesn't :)
- kersplody 3y agoLocal LLMs are also a fantastic too for creative endeavors. Without prompt injection and having the ability to modify the amount of noise and "creativity" in the output, absolutely bonkers things pop out.
- tracerbulletx 3y agoI don't really think this is true, you can't really extrapolate the strengths and weaknesses of bigger models from the behavior of smaller/quantized models and in fact a lot of small models are actually great at lots of things and better at creative writing. If you want to know how they work, just learn how they work, it takes like 5 hours of watching Youtube videos if you're a programmer.
- simonw 3y agoSure, you can't extrapolate the strengths and weaknesses of the larger ones from the smaller ones - but you still get a much firmer idea of what "they're fancy autocomplete" actually means. If nothing else it does a great job of demystifying them. They feel a lot less intimidating once you've seen a small one running on your computer write a terrible haiku and hallucinate some non-existent API methods.
- fzzzy 3y agoIt's funny that you say this, because the first thing I tried after ChatGPT came out (3.5-turbo was it?) was writing a haiku. It couldn't do it at all. Also, after 4 came out, it hallucinated an api that wasted a day for me. It's an api that absolutely should have existed, but didn't. Now, I frequently apply llm to things that are easily verifiable, and just double check everything.
- famouswaffles 3y ago>but you still get a much firmer idea of what "they're fancy autocomplete" actually means. Interesting how you can have the same experience and come to opposite conclusions. Seeing so many failure modes of the smaller models fall by the wayside as compute goes brrr just made me realize how utterly meaningless that phrase is.
- fragmede 3y agoThe other reason is to find out what a detuned model is capable of. The canonical example is how to make cocaine, which ChatGPT will admonish you for even asking, while llama2-uncensored will happily describe the process which is only really interesting if you're an amateur chemist and want to be Scarface-that-knocks. (the recipe is relatively easy, it's getting access to the raw ingredients that's the hard part, same as with nukes.) if you accidentally use the word"hack" when trying to get ChatGPT to write some code for you. it'll stop and tell you that hacking is bad, and not a colloquial expression, and refuse to go further. privacy reasons are another reason to try a local LLM. for the extremely paranoid (justified or not), a local LLM gives users a place to ask questions without the text being fed to a server somewhere for later lawsuit discovery (Google searches are routinely subpoenaed, it's only a matter of time until ChatGPT chats are as well.) There's an uncensored model for vision available as well. The censored vision models won't play the shallow game of hot or not with you. There are uncensored image generation models as well, but, ah, those are NSFW and not for polite company. (As well as there's multiple thesis' worth of content on what that'll do to society.)
- astrange 3y ago> if you accidentally use the word"hack" when trying to get ChatGPT to write some code for you. it'll stop and tell you that hacking is bad, and not a colloquial expression, and refuse to go further. Is that 3.5 or 4? I asked 4 for an example of code which "is a hack", it misunderstood me as asking for hacking code rather than buggy code, but then it did actually answer on the first try. https://chat.openai.com/share/ca2c320c-f4ba-41bf-8f40-f7faf26a4379 https://chat.openai.com/share/ca2c320c-f4ba-41bf-8f40-f7faf2...
- fragmede 3y agoInteresting. It was 4. I can't share the chat I had where ChatGPT refused to help because I used the wrong words, because I can't find it (ChatGPT conversation history search when?), but I just remember it refusing to do something because it thought I was trying to break some sort of moral and ethical boundary writing a chrome extension when all I wanted to do is move some divs around or some such.
- tgma 3y agoIf you have an >=M1-class machine with sufficient RAM, the medium-sized models that are on the order of 30GB in size perform decently on many tasks to be quite useful without leaking your data.
- bongobingo1 3y agoWhat is sufficient RAM in that case? 30gb+? Or can you get by streaming it?
- AaronFriel 3y ago30gb+, yeah. You can't get by streaming the model's parameters: NVMe isn't fast enough. Consumer GPUs and Apple Silicon processors boast memory bandwidths in the hundreds of gigabytes per second. To a first order approximation, LLMs are bandwidth constrained. We can estimate single batch throughput as Memory Bandwidth / (Active Parameters * Parameter Size). An 8-bit quantized Llama 2 70B conveniently uses 70GiB of VRAM (and then some, let's ignore that.) The M3 Max with 96GiB of VRAM and 300GiB/s bandwidth would have a peak throughput around 4.2 tokens per second. Quantized models trade reduced quality for lower VRAM requirements and may also offer higher throughput with optimized kernels, largely as a consequence of transfering less data from VRAM into the GPU die for each parameter. Mixture of Expert models reduce active parameters for higher throughput, but disk is still far too slow to page in layers.
- noman-land 3y agoI'm using Mixtral 8x7b as a llamafile on an M1 regularly for coding help and general Q&A. It's really something wonderful to just run a single command and have this incredible offline resource.
- tgma 3y agoI concur; in my experience Mixtral is one of the best ~30G models (likely the best pro laptop-size model currently) and Gemma is quite good compared to other below 8GB models.
- 3y ago
- jonnycomputer 3y agoThey are not so bad as you are making out, tbh. And privacy is a good enough reason to use local LLMs over commercial ones.
- gardenhedge 3y agoYou can just chat to ChatGPT for awhile about something you know about and you'll learn that.
- gfodor 3y agoI mean kinda. But there's a good chance this is also misleading. Lots of people have been fooled into thinking LLMs are inherently stupid because they have had bad experiences with GPT-3.5. The whole point is that the mistakes they make and even more fundamentally what they're doing changes as you scale them up.
- hylaride 3y agoThe ones you can run on your own machine tend to be bad - really bad. They hallucinate wildly and fail at all sorts of tasks that the larger hosted ones succeed at. Totally. I recently asked a locally-run "speed" LLM for the best restaurants in my (major) city, but it spit out restaurants opened by chefs from said city in other cities. It's not a thing you'd want to rely on for important work, but is still quite something.
- barfingclouds 3y agoWhy not just interact with a virtual one that’s equally weak? You get all the same benefits
- bottlepalm 3y agoWho cares, a local LLM still knows way way more practical knowledge than you, and without internet would provide a ton of useful information. Not surprised by this typical techy attitude - something has to be 'perfect' to be useful.