24 ms·
The Coming of Local LLMs
- rambojohnson 4y agoThis service has been suspended by its owner.
- emrah 4y agoIt's great they got LLMs running on resource constrained devices but are they any good? Or I should ask, with the limited resources they get, what good are they for?
- BulgarianIdiot 4y agoThey’re crude but will be getting better quickly.
- cgearhart 4y agoEven with smaller models & more optimized hardware, I think edge compute is going to be power-limited first. Batteries today just won’t support constantly running LLMs. But I joked recently that as long as they prove useful then consumers would be willing to swap their iPhone for the old car battery with a phone handle attached.
- BulgarianIdiot 4y agoThe first step is for it to be viable for smaller models on desktops. The rest will follow, as hardware catches up. Hybrid analog-hybrid NN hardware is on the horizon, maybe in 3-4 years. This would allow GPT-4 level performance on an iPhone with plausible battery life. The current hardware of course can't pull anything like this yet. But iPhone supports on-device facial recognition, object recognition, dictation and translation, so small steps...
- boppo1 4y agoWho is making said hardware?
- BulgarianIdiot 4y agoThere are few efforts under way. https://research.ibm.com/blog/why-we-need-analog-AI-hardware https://research.ibm.com/blog/why-we-need-analog-AI-hardware https://news.mit.edu/2022/analog-deep-learning-ai-computing-0728 https://news.mit.edu/2022/analog-deep-learning-ai-computing-...
- tudorw 4y agoI guess if the hardware is cheap for some uses speed is not so important, you can just walk away and let it grind.
- kbrkbr 4y agoFrom my experience with llama.cpp and oobaboogas webui I can say they are amazing, at least on my gaming pc. I’m absolutely astonished at the speed and quality of llama, alpaca, galactica and vicuna (the >10B parameters ones). Make no mistake, it’s for tinkerers that do not expect each prompt to be answered human like. I see them as creativity and thought testing tools, also knowledge exploratory.
- deleted 4y ago[deleted]
- synergy20 4y agoare we talking about training or inference for local LLM here? it's hard to do any meaningful training on the edge unless we all carry a heavy gaming pc, even that, the training quality will be subpar?
- qumpis 4y agoInference, even fine-tuning a few layers would be difficult since one needs to use non-quantized model, I'd imagine
- ctoth 4y agoCheckout LoRA and Alpaca LoRA and the whole huge group of people who have already figured this out. I think there was another breakthrough (yesterday?) which is a further adaption of LoRA to touch even less parameters at runtime.
- homarp 4y agoDyLoRA https://news.ycombinator.com/item?id=35514228 https://news.ycombinator.com/item?id=35514228 is what you meant
- whimsicalism 4y agoDisagree that this is a major breakthrough :) but it is probably a marginal improvement.
- turmeric_root 4y agoI like using them for memeing
- cs702 4y agoI expect we will see the biggest jump in performance if (when) consumer-grade coprocessors like mobile GPUs start incorporating attention layers as a primitive building block at the hardware level, e.g., with instructions and memory layouts engineered specifically to make ultra-low-precision (say, 4-bit) transformer layers as compute- and memory-efficient as possible on consumer devices. That seems almost inevitable to me.
- liuliu 4y agoChip's capability planning seems need about ~2yr lead time. So we are expecting fastest would be somewhere around end of 2024. (Transformers probably earlier than that (2023?), 4-bit would be later).
- cs702 4y agoUnless there are groups that have been thinking about and working on this for a while...
- jjoonathan 4y agoLow precision: agreed Attention: Isn't it quadratic in context length? I dunno, this feels like the crude first iteration of something that will get inevitably passed by something that scales better.
- dpflan 4y agoSharing a comment from a similar line of discussion: """ Complexity is quadratic in sequence length. For 512 tokens it is 262K, but for 4000 tokens it becomes 16M and goes OOM on a single GPU. We need about 100K-1M tokens to load whole books at once. Since 2017 there have been hundreds of attempts to bring O(N^2) to O(N), but none of them replaced the vanilla attention yet in large models. They lose on accuracy. Maybe Flash attention has a shot (https://arxiv.org/abs/2205.14135 https://arxiv.org/abs/2205.14135). """ - source: https://news.ycombinator.com/item?id=34171503 https://news.ycombinator.com/item?id=34171503
- binkHN 4y agoThis is wonderful. As hardware and software continues to improve, everything seems to find a way to run on ever smaller devices. Guess your own pocket-AGI is not too far away after all.
- 1827162 4y agoBy the way, I was thinking of something along the lines of a powerful FPGA with direct access to large quantities of very fast NAND flash, likely many chips in parallel, which will save having to load the model into RAM..... So it will be able to directly run from NAND flash, which opens up the possibility of using very large models??? Power consumption would not be an issue if it's used sporadically throughout the day, it's not like it needs to run continuously? There is still the issue of NAND flash read disturb, which I haven't fully looked into yet.
- deleted 4y ago[deleted]
- anentropic 4y agoApple should get working on a version of the Neural Engine that is useful for these models, and remove the 3GB size limit [1] to take full advantage of the 'unified' memory architecture. Game changer. Waste of die space currently (on Macbook at least, I'm sure they find uses for it in the iPhone) [1] https://github.com/smpanaro/more-ane-transformers/blob/main/src/experiments/NOTES.md https://github.com/smpanaro/more-ane-transformers/blob/main/...
- leetharris 4y agoIt's not a waste on Mac, it will dynamically switch between GPU and NPU whenever CoreML is called. There are a decent amount of applications that use CoreML. But I do agree it should be improved!
- yacine_ 4y agoI was able to run a LLaMa on my personal machine to run some labeling on my documents, as a test of its capabilities. It was instruct tune. 30b parameters 4 example labels, and I had a binary classifier in seconds. Sure, semantic text classifiers were possible for a while, but making it accessible changes everything. Giving anyone who can use a spreadsheet the power of a local LLM (or, basically free LLMs) can make them much, much more productive. A lot of office work is clicking through sheets and doing manual labeling. It's truly wild what is becoming accessible! Really excited to see the next gen software that the open community comes up with :)
- tyingq 4y agoAgree, though plain old Bayesian classifiers have been able to handle some significant portion of that office work for a long time. And not much ever came from it for everyday stuff outside of spam filters. Maybe both the buzz factor and broader applicability means it's more likely to happen this go around?
- yacine_ 4y agoMore like: the accessibility is what will make it go around. The accessibility is what changes everything! Getting an easy to use interface (instructions) over python changes the accessibility from the denizens of this website to ~anyone with a computer If you're interested, see this paper that argues that point: https://arxiv.org/abs/2302.06541 https://arxiv.org/abs/2302.06541 Essentially, being label efficient is more important than being compute efficient, because the biggest computing constraint we have is enough humans doing the labeling (and knowing how to work a jupyter notebook), not tensor smashing nvidia cards
- lisasays 4y agoRe: https://arxiv.org/abs/2302.06541 https://arxiv.org/abs/2302.06541 - how are we supposed to take the phrase "agile text classification" seriously? They could have just said "efficient". But no - they had to go for "agile".
- 4y ago
- 1827162 4y agoI found this to be very liberating, that I can finally type whatever I want into the LLM, without the possibility of the government knowing what I am writing. Just being able to do that, and have the watchful eye of the state not being able to monitor you is amazing.
- croes 4y agoLocal LLMs remove the data protection problem but open the door for malicious use on a larger scale.
- colonwqbang 4y agoIt was never possible to keep this technology secret for any length of time.
- croes 4y agoI thought the possibility to run these LLMs on everyday hardware would be further on the future and in the beginning more limited to big servers. But this could be the equivalent of the Low Orbit Ion Canon for phishers and scammers
- Root_Denied 4y agoIt also makes it easier for the defense/blue team side to build counters or monitor and defend the attack surface of a given system against these attacks. Digital arms races are nothing new, this is just the latest battlefield.
- autoexec 4y agoMy local LLM will automatically detect scams and phishing attempts so it'll all balance out.
- sendfoods 4y agoHow realistic is CPU-only inference in the near future?
- travisjungroth 4y agoIt’s in the near past. https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp
- abetlen 4y agoAlso worth checking out https://github.com/saharNooby/rwkv.cpp https://github.com/saharNooby/rwkv.cpp which is based on Georgi's library and offers support for the RWKV family of models which are Apache-2.0 licensed.
- BrutalCoding 4y agoI’ve got some of their smaller Raven models running locally on my M1 (only 16GB of RAM). I’m also in the middle of making it user friendly to run these models on all platforms (built with Flutter). First MacOS release will be out before this weekend: https://github.com/BrutalCoding/shady.ai https://github.com/BrutalCoding/shady.ai
- abetlen 4y agoYou can see for yourself (assuming you have the model weights) https://github.com/abetlen/llama-cpp-python https://github.com/abetlen/llama-cpp-python I get around ~140 ms per token running a 13B parameter model on a thinkpad laptop with a 14 core Intel i7-9750 processor. Because it's CPU inference the initial prompt processing takes longer than on GPU so total latency is still higher than I'd like. I'm working on some caching solutions that should make this bareable for things like chat.
- deleted 4y ago[deleted]
- sharemywin 4y agoI could certainly see them being trained/used as a front end for a database(file system, api calls).
- layer8 4y agoIt would be nice to be able to run an LLM-driven spamassassin on a VPS for acceptable cost.
- londons_explore 4y agoI don't think it will help. Actual friends occasionally send me mail that says "test" from a random account. And spammers do too... There is no way to seperate them.
- layer8 4y agoPersonally I haven’t had that problem. I can readily distinguish spam from non-spam by manual review ~99% of the time. If I could train or instruct an LLM to do the same as I do now, I would be happy. My current false-negative rate with Bayesian spamassassin is more like 50%.
- autoexec 4y agoDon't you do it somehow? Plus a filter doesn't have to be 100% right all the time. Filtering out what is 99% certain to be or not be spam and leaving the human to cover the tiny number of messages that fall into a grey area would still save a ton of time.
- Zuiii 4y agoYes, but I rely on my own personal experiences (online and offline) to determine whether an email is real or a scam. Unless that AI filter taps into my memory, it will likely lead to too many false positives and false negatives.
- FL33TW00D 4y agoI wrote a similar blog post in Nov 22: https://fleetwood.dev/posts/a-case-for-client-side-machine-learning https://fleetwood.dev/posts/a-case-for-client-side-machine-l...
- Torkel 4y agoThings are moving so fast in this space I felt like I needed a [March] in the title on this one :)
- turnsout 4y agoToo real–early March even felt different from late March. :D
- deleted 4y ago[deleted]
- monkeydust 4y agoWait till these are embedded into soft toys. 'You are a koala who plays with 5-7 year olds, you are friendly natured and curious and like to ask questions'
- jjtheblunt 4y agoLike Teddy from A.I. the movie? https://www.youtube.com/watch?v=YRsICbxDEiI https://www.youtube.com/watch?v=YRsICbxDEiI
- waffletower 4y agoWhile many NLP related Apple ML job listings have been added since this article was written, there were several recent listings at the time of its writing. While I feel that Apple does not focus well on intangible technologies, products that can't be readily carried, worn and given their boutique product development fetish focus, I have some hope that they can overcome this bias somewhat, and see how behind they are.
- hackernewds 4y agoI disagree with the assessment that Apple is behind. Apple is known for executing well and putting their weight behind the things they launch. Like Jack Dorsey would often say "it's not important to be first to market, you can just be best to market". And the world got CashApp. I'm sure however Apple enters the space, it will be fleshed out (vs Bard).
- rootusrootus 4y agoI want to agree, but it's pretty easy to find instances where Apple has dabbled but not delivered best-in-class solutions. Siri. iCloud. Home automation.
- pwinnski 4y agoApple has two values in conflict with each other, I think. On the one hand, they want to deliver best-in-class solutions. On the other hand, they have a commitment to user privacy[0] as perhaps only a gay man growing up in the south might value. Siri should be better! It lost features post-acquisition by Apple, and it seems like user privacy is why. Home automation is arguable. If you consider a single point of failure on a server somewhere to be bad, Apple's solution is pretty great. Their commitment to zigging where others zagged put them behind, since hardware vendors didn't want to put in powerful (expensive) enough chips to handle the cryptography, but while other companies go out of business, or transmit images and video to external parties, Apple's works reliably and securely. Still, as with most of my complaints about Apple, it's a trade-off between privacy and functionality, and Apple will seemingly always choose privacy over functionality, even as Google consistently chooses functionality over privacy. [0] Yes, there are examples of edge cases that suggest a less-than-perfect record. Contrast that with their competitors, for which invading privacy is foundational to the business model.
- 0xDEF 4y agoI know it's secret how GPT-4 actually works but are there any of these local LLMs that are also multimodal?
- whimsicalism 4y agoNo
- Oranguru 4y agoYes, there are. For example: https://laion.ai/blog/open-flamingo/ https://laion.ai/blog/open-flamingo/
- jrm4 4y agoAs I see these things come out, it feels like there's not a lot of discussion on which hardware (that isn't one of the fancy new Macs?) As in, there might be a lot of graphics cards out there that could be used here? Is it only Nvidia still, is AMD a possibility? Maybe I'm missing something on how the tech works?
- seydor 4y agoThis has a list of models and their VRAM requirements https://www.reddit.com/r/LocalLLaMA/comments/11o6o3f/how_to_install_llama_8bit_and_4bit/ https://www.reddit.com/r/LocalLLaMA/comments/11o6o3f/how_to_...
- deleted 4y ago[deleted]
- boppo1 4y ago30B llama needs a 3090 or 4090. 13B I think you can get away with a 3/4080. If you have 64 gigs of ram and a beefy CPU you can run even 65B, but boy it's slow. 13B is pretty meh, but 30B is great, if not quite Chatgpt. But I can ask it why my highschool geometry teacher was such a cunt and it will happily discuss the matter without reservation. Very therapeutic.
- yieldcrv 4y agoSo interesting how you can already tell that is a 3 week old epiphany
- imaurer 4y agoTracking repos and resources for running LLMs locally here: https://github.com/imaurer/awesome-decentralized-llm https://github.com/imaurer/awesome-decentralized-llm
- turnsout 4y agoAlso, this continuously-updated spreadsheet is extremely helpful: https://docs.google.com/spreadsheets/d/1O5KVQW1Hx5ZAkcg8AIRjbQLQzx2wVaLl0SqUu-ir9Fs/edit https://docs.google.com/spreadsheets/d/1O5KVQW1Hx5ZAkcg8AIRj...
- ingenieroariel 4y agoIn order to run large language models, we should all be buying a fully loaded Mac Studio (128GB of ram, 20 CPU cores, a lot of GPU and Neural cores.) and putting Linux on it to remove the artificial restrictions. Yes, we will be running them soon in low end hardware, but we need to get at least to GPT-3.5-turbo level of inference speed and quality before we try to make it small. I already started. $ neofetch -` x@decpti .o+` ------- `ooo/ OS: Arch Linux ARM aarch64 `+oooo: Host: Apple Mac Studio (M1 Ultra, 2022) `+oooooo: Kernel: 6.1.0-rc6-asahi-4-1-ARCH -+oooooo+: Uptime: 4 hours, 23 mins `/:-:++oooo+: Packages: 177 (pacman) `/++++/+++++++: Shell: bash 5.1.16 `/++++++++++++++: Resolution: 1920x1080 `/+++ooooooooooooo/` Terminal: /dev/pts/0 ./ooosssso++osssssso+` CPU: (20) @ 2.064GHz .oossssso-````/ossssss+` Memory: 717MiB / 129540MiB -osssssso. :ssssssso. :osssssss/ osssso+++. /ossssssss/ +ssssooo/-
- simlevesque 4y agoPlease, this is not r/unixporn.
- beiller 4y agoWhich LLM can run on apple's neural cores / GPU cores? I can only run on plain ol' CPU cores (llama), and it runs fine on my Ryzen CPU for less than half the price of that system. That being said I'm switching from Ubuntu to Arch cause I'm sick of all my packages being way out of date!
- ingenieroariel 4y agotinygrad https://github.com/geohot/tinygrad/tree/master/accel/ane https://github.com/geohot/tinygrad/tree/master/accel/ane But I have not tested it on Linux since Asahi has not yet added support. Same machine but OSX, llama.cpp runs at 18ms per token (7B) and 200ms per token (65B) on CPU using float16.
- whimsicalism 4y agoIt appears there is this genre of articles pretending that LLAMA or its RL-HF tuned variants are somehow even close to an alternative to ChatGPT. Spending more than a few moments interacting even with the larger instruct-tuned variants of these models quickly dispels that idea. Why do these takes around open-source AI remain so popular? What is the driving force? I've posted this before, but it seems like this genre is just getting more and more popular - and more and more untethered from any actual metrics of how good these models are.
- simion314 4y agoPeople are enthusiastic about the possibilities, imagine a black box on your desk contains only RAM and matrix multiplication chips, you install on it your favorite AI assistant and you train it with your private data/code, you remove all prudish restrictions and get productive on your work and on your off times. Llama has the potential to reach ChatGPT it needs tunning to get better at responding to questions, llama if I am not worng is mostly attempting to predict what is next. I can see it similar like Midjourney and Stable diffusion, midjorny can make any stupid prompt look like a digiatal art in the style of Midjorney but look how many stable Diffusion innovation happens, a competent person that is on top with all the new stuff can produce absolute anything in any style they want.
- whimsicalism 4y agoI am similarly enthusiastic and super, super impressed by Llama.cpp But I just don't see the need to mislead and suggest that these models are "on par" with ChatGPT or something like that. They just aren't.
- simion314 4y agoI did not said that are the same, the llama has a lot of information in it , the issue seems to be the Q&A chat part. This can be added on top by the community without having to start from scratch IMO. But I might be wrong, and OpenAI put some magic shit in , some super secret unpublished stuff, in that case educate me. In my mind I compare with Stable Diffusion, the proprietary ones produce more artistic effects because they put some more stuff into the prompts and they are imposing their styles. With SD you have the control and you need to wait for some improvements, plugins, new loras or embedings with new styles etc. In SD I could train my face in 1- minutes, with proprietary shit it will never happen... Imagine what a math community could train, they just need access to the model and soem GUI software that can help them train. So llama based Chat stuff is not yet comparable with ChatGPT but there are already lot of progress made. At this moment coding and math is bad in llama based but other stuff is great, like story creation, also I only could test 3-b 4bit and it is good enough to for example provide me a complex response in valid JSON format.
- gigel82 4y agoI don't understand why people are so excited to build this big thing on top of Llama, which is closed source, severely license restricted and we now know for a fact that Meta is going after users with the legal hammer. I'm sure if we'd pool resources together we could build a truly open alternative worthy of building on top of.
- whimsicalism 4y ago> I'm sure if we'd pool resources together we could build a truly open alternative worthy of building on top of. This would require like ConstitutionDAO level of resource pooling without direct monetary payoff. I mean, good luck.
- transfire 4y agoWe really need optical computing to take this to the next level.
- agumonkey 4y agoWere transformers used in other contexts ? biochemistry ? geometry .. whatever.
- skinner_ 4y agoAlphafold uses something they call Evoformer, it is an attention mechanism. Our group has tried, and so far failed to utilize transformers for a very very specific search problem in geometry (https://bit.ly/unit-distances https://bit.ly/unit-distances).
- agumonkey 4y agothanks a lot man, very nice
- txomon 4y agoOut of doubt, which seems to be spreading around the internet. The LLaMa model weights weren't "leaked" AFAIK but rather explicitly given access to to researchers, isn't it right? I know the article goes on to speak about something else, but I'm not sure why this claim that the LLaMa model weights were leaked, as in unintendenly made available is being done.
- ftxbro 4y agoMy understanding is that researchers could ask for access to weights, but then also they were leaked so that anyone could get them without asking. There is another layer, where Facebook seems to accept it on some level (I mean they don't have a choice anymore anyway); they put a cheeky comment in the open pull request instead of closing it.
- turmeric_root 4y agoThe model weights were only shared by FB to people who applied for research access. Github repos containing links to the model weights have been taken down by FB.
- simonw 4y ago"I think we’re going to eventually see a demo showing an open source model running on an iPhone as well" I have Kevin Kwok's SheepyT running on my iPhone right now - it uses GPT-J, which is an openly licensed LLM by EleutherAI. https://twitter.com/antimatter15/status/1644456371121954817 https://twitter.com/antimatter15/status/1644456371121954817
- Sparkyte 4y agoEveryone be like ChatGPT is everything. I am like it won't be the last or the first.
- la64710 4y agoOne simple thing these LLM models cannot do yet .. that is to simply point a LLM to a URL and it will start scraping - ie follow the hyperlinks and start consuming the content. I am not an AI guy but I guess this has to do with the context limitations of most model? How did they train OpenAI with all internet data till 2021? This I think will be a most popular feature for LLM models and I seriously hope it is OSS whenever it comes out.
- marijnz 4y agoThis already works with ChatGPT plugins: https://openai.com/blog/chatgpt-plugins https://openai.com/blog/chatgpt-plugins
- causi 4y agoLLAMA isn't there and probably never will be, but the possibility of running something equivalent to ChatGPT has certainly made me reconsider my GPU purchases. I wonder if in the end will it be Nvidia's CUDA advantage or AMD's larger amount of memory that will end up being more important when we do get it.