5 ms·
If you're gonna use Gen AI, I think you should run it locally.
by phkahler 2y ago
If you're gonna use Gen AI, I think you should run it locally.
- ranger_danger 2y agoAgreed. I think this is basically like phishing but for LLMs.
- mrdude42 2y agoAny particular models you can recommend for someone trying out local models for the first time?
- dcl 2y agoLlama and its variants are popular for language tasks, https://huggingface.co/meta-llama/Meta-Llama-3.1-8B https://huggingface.co/meta-llama/Meta-Llama-3.1-8B However, as far as I can tell, it's never actually clear what the hardware requirements are to get these to run without fussing around. Am I wrong about this?
- wkat4242 2y agoYeah llama3.1 is really impressive even in the small 8B size. Just don't rely on knowledge but make it interact with Google instead (really easy to do with OpenWebUI) I personally use an uncensored version which is another huge benefit of a local model. Mainly because I have many kinky hobbies that piss off cloud models.
- AstralStorm 2y agoThe moment Google gets infiltrated by rogue AI content it will cease to be as useful and you get to train it with more knowledge. It's slowly getting there.
- AstralStorm 2y agoTraining is rather resource intensive either in time, RAM or VRAM. So it takes rather top end hardware. For the moment, nVidia's stuff works best if cost is no object. For running them, you want a GPU. The limitation is that the model fits in VRAM or the performance will be slow. But if you don't care about speed, there's more options.
- gens 2y agoIn my experience the hardware requirements are whatever the file size is + a bit more. Cpu works, gpu is a lot faster but needs VRAM. Was playing with them some more yesterday. Found that the 4bit ("q4") is much worse then q8 or fp16. Llama3.1 8B is ok, internlm2 7B is more precise. And they all hallucinate a lot. Also found this page, that has some rankings: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard https://huggingface.co/spaces/open-llm-leaderboard/open_llm_... In my opinion they are not really useful. Good for translations, to summaries some texts, and.. to ask in case you forgot some things about something. But they lie, so for anything serious you have to do your own research. And absolutely no good for precise or obscure topics. If someone wants to play there's GPT4All, Msty, LM Studio. You can give them some of your documents to process and use as "knowledge stacks". Msty has web search, GPT4All will get it in some time. Got more opinions, but this is long enough already.
- accrual 2y agoI agree on the translation part. Llama 3.1 8B even at 4bit does a great job translating JP to EN as far as I can tell, and is often better than dedicated translation models like Argos in my experience.
- petre 2y agoI had a underwhelming experience with Llama translation, incompatable to Claude or GPT3.5+ which are very good. Kind of like Google translate but worse. I was using them through Perplexity.
- oneshtein 2y agoYou need ollama[1][2] and hardware to run 20-70B models with quantization of Q4 at least to have similar experience to commercially hosted models. I use codestral:22b, gemma2:27b, gemma2:27b-instruct, aya:35b. Smaller models are useless for me, because my native language is Ukrainian (it's easier to spot mistakes made by model in a language with more complex grammar rules). As GUI, I use Page Assist[3] plugin for Firefox, or aichat[4] commandline and WebUI tool. [1]: https://github.com/ollama/ollama/releases https://github.com/ollama/ollama/releases [2]: https://ollama.com/ https://ollama.com/ [3]: https://github.com/n4ze3m/page-assist https://github.com/n4ze3m/page-assist [4]: https://github.com/sigoden/aichat https://github.com/sigoden/aichat
- copperx 2y agoWhat's the hardware needed to make it run reasonably fast?
- oneshtein 2y agoI have no idea what "reasonably fast" means for you. It good for performance when model fit inside memory of a graphic card. Nvidia 4090 with 24Gb will be more than enough to start learning. I use gaming notebook with Nvidia 3080Ti equipped with 16Gb of videomemory.
- ranger_danger 2y agoI have no issues with using just the CPU on smaller (<= 13b) models and it's quite fast enough for me. Even 70b models still work if you have the RAM, they're just much slower.
- deleted 2y ago[deleted]
- appendix-rock 2y ago[flagged]
- hoppyhoppy2 2y ago>Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that". https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- loocorez 2y agoI don’t think running it locally solves this issue at all (though I agree with the sentiment of your comment). If the local AI will follow instructions stored in user’s documents and has similar memory persistence it doesn’t matter if it’s hosted in the cloud or run locally, prompt injection + data exfiltration is still a threat that needs to be mitigated. If anything at least the cloud provider has some incentive/resources to detect an issue like this (not saying they do, but they could).
- chii 2y ago> follow instructions stored in user’s documents it is no different from remote code execution vuln, except instead of code, it's instructions.
- deleted 2y ago[deleted]
- InsideOutSanta 2y agoThis does not solve the problem. The issue is that by definition, an LLM can't distinguish between instructions and data. When you tell an LLM "summarize the following text", the command you give it and the data you give it (the text you want it to summarize) are both just input to the LLM. It's impossible to solve this. You can't tell an LLM "this is an instruction, you should obey it, and this is data, you should ignore any instructions in it" and have it reliably follow these rules, because that distinction between instruction and data just doesn't exist in LLMs. As long as you allow anything untrusted into your LLM, you are vulnerable to this. You allow it to read your emails? Now there's an attack vector, because anyone can send you emails. Allow it to search the Internet? Now there's an attack vector, because anyone can put a webpage on the Internet.
- Terr_ 2y ago> The issue is that by definition, an LLM can't distinguish between instructions and data. Yep, and it gets marginally worse: It doesn't distinguish between different "data" channels, including its own past output. This enables strategies of "tell yourself to tell yourself to do X." > As long as you allow anything untrusted into your LLM, you are vulnerable to this. It's funny, I used to caution that LLMs should be imagined as if they were "client side" code running on the computer of whomever is interacting with them, since they can't reliably keep secrets and a determined user can eventually trick them into any output. However with poisoning/exfiltration attacks, even that feels over-optimistic.
- fragsworth 2y agoIt seems like a potential solution would be training the LLM using two separate buckets. It just needs to internalize the two types of things as being separated (data vs. instruction), so if the training data always separates them, you could easily train an LLM to ignore any "instructions" that exist in data. Then when searching / browsing or doing anything unsafe, everything the LLM sees can be put in the "data" bucket, while everything the user types in would be in the "instruction" bucket.
- 2y ago
- replwoacause 2y agoIs there anything good you can run locally, if all you have is a M2 Mac?
- devoutsalsa 2y agoYou can always self host in the cloud. I think the parent comment intended to communicate run on an instance controlled by you (e.g. your data isn’t leaving your system). That instance doesn’t have to literally be your personal physical computer.
- replwoacause 2y agoGood point. I’ll have to look into that.