14 ms·
Forget ChatGPT: why researchers now run small AIs on their laptops
- tripzilch 2y agoit doesn't fill me with confidence that an MD calls themselves a "ML researcher" when clearly they've only been "into ML" since recently LLMs became popular. I could believe that this MD became an "expert in using LLMs" in the past few years (like, we all became), but this guy is training networks based on medical data, without experience in ML going back further than 2022. a little knowledge is dangerous, there's a whole bunch of boring stuff I'd want someone to learn about ML before going "hey LLMs are cool for medical use" ...
- mikestaub 2y agoThis is a great use-case for WebGPU: https://chat.webllm.ai https://chat.webllm.ai
- HPsquared 2y agoPC: Personal Chatbot
- pella 2y agoNext year, devices equipped with AMD's Strix Halo APU will be available, capable of using ~96GB of VRAM across 4 relatively fast channels from a total of 128GB unified memory, along with a 50 TOPS NPU. This could partially serve as an alternative to the MacBook Pro models with M2/M3/M4 chips, featuring 128GB or 196GB unified memory. - https://videocardz.com/newz/amd-ryzen-ai-max-395-to-feature-16-zen5-cores-and-40-rdna3-5-cus-strix-halo-lineup-leaks-out https://videocardz.com/newz/amd-ryzen-ai-max-395-to-feature-...
- diggan 2y agoAccording to Tom's (https://www.tomshardware.com/pc-components/cpus/amd-pushes-ryzen-to-the-max-ryzen-ai-max-300-strix-halo-reportedly-has-up-to-16-zen-5-cores-and-40-rdna-3-cus https://www.tomshardware.com/pc-components/cpus/amd-pushes-r...), those are supposed to be laptop CPUs, which makes me wonder what AMD has planned for us desktop users.
- adrian_b 2y agoThey are laptop CPUs for bigger laptops, like those that now use both a CPU and a discrete GPU, i.e. gaming laptops or mobile workstations. It seems that the thermal design power for Strix Halo can be configured between 55 W and 120 W, which is similar to the power used now by a combo laptop CPU + discrete GPU.
- MobiusHorizons 2y agoIf I remember right, in the press conference they suggested desktop users would use a gpu because desktop uses are less power sensitive. That doesn’t address the vram limitations of discrete GPUs though.
- wkat4242 2y agoTrue but try to find a 96GB GPU.
- teaearlgraycold 2y agoH100 NVL is easily available. It’s just that it’s close to $20k.
- andersa 2y agoIt is actually much more expensive than 20k. I can find them between 25-30k only.
- teaearlgraycold 2y agoMy bad. Looks like it’s $29k at the local store.
- VikingCoder 2y agoThis drives me insane. PC Part Picker, DDR5-8400 48 GB (2x24GB) is... $340 right now. For $680 you can get 96 GB of very fast RAM. How about someone make an NVidia GPU with 96 GB of RAM at a reasonable price? Please?
- TibbityFlanders 2y ago[dead]
- aurareturn 2y agoIt will have around 250GB/s of bandwidth which makes it nearly unusable for 70b models. So the high amount of RAM doesn’t help with large models.
- Dalewyn 2y agoFast. Large. Cheap. You may only pick two.
- meiraleal 2y agoFor a few years ago standard the current "small" models like mistral and phind are fast, large and cheap.
- throwaway314155 2y ago> nearly unusable for 70b models Can Apple Silicon manage this? Would it be feasible to do with some quantization perhaps?
- aurareturn 2y agoYes at around 8 tokens/s. Also quite slow.
- bearjaws 2y agoAny of the newer M2+ Max chips runs 400GB/s and can run 70b pretty well. It's not fast though, 3-4 token/s. You can get better performance using a good CPU + 4090 + offloading layers to GPU. However one is a laptop and the other is a desktop...
- staticman2 2y agoApparently Mac purchasers like to talk about tokens per second without talking about Mac's atrocious time to first token. They also like to enthusiastically talk about tokens per second asking a 200 token question rather than a longer prompt. I'm not sure what the impact is on a 70b model but it seems there's a lot of exaggeration going on in this space by Mac fans.
- jstummbillig 2y agoAlso, next year, there will be GPT 5. I find it fascinating how much attention small models get, when at the same time the big models just get bigger and prohibitively expensive to train. No leading lab would do that if they thought it a decent chance that small models were able to compete. So who will be interested in a shitty assistant next year when you can have an amazing one, is what I wonder? Is this just the biggest cup of wishful thinking that we have ever seen?
- jabroni_salad 2y agoOne of the reasons I run local is that the models are completely uncensored and unfiltered. If you're doing anything slightly 'risky' the only thing APIs are good for is a slew of very politely written apology letters, and the definition of 'risky' will change randomly without notice or fail to accommodate novel situations. It is also evident in the moderation that your usage is subject to human review and I don't think that should even be possible.
- Tempest1981 2y agoThere is also a long time-window before most laptops are upgraded to screaming-fast 128GB AI monsters. Either way, it will be fun to watch the battle.
- Larrikin 2y agoWhy would anyone buy a Raspberry Pi when they can get a fully decked out Mac Pro? There are different use cases and computers are already pretty powerful. Maybe your local model won't be able to produce tests that check all the corner cases of the class you just wrote for work in your massive code base. But the small model is perfectly capable of summarizing the weather from an API call and maybe tack on a joke that can be read out to you on your speakers in the morning.
- talldayo 2y ago> Why would anyone buy a Raspberry Pi when they can get a fully decked out Mac Pro? They want compliant Linux drivers?
- diggan 2y agoSummary: It's cheaper, safer for handling sensitive data, easier to reproduce results (only way to be 100% sure it's reproduce even, as "external" models can change anytime), higher degree of customization, no internet connectivity requirements, more efficient, more flexible.
- alexander2002 2y agoAn AI chip on laptop devices would be amazing!
- ta988 2y agoThey already exist. Nvidia GPUs on laptops, M series CPUs from Apple, NPUs...
- alexander2002 2y agooh damn guess i am so uninformed
- viraptor 2y agoIt's pretty much happening already. Apple devices have MPS. Both new Intels and Snapdragon X have some form of NPU.
- moffkalast 2y agoIt would be great if any NPU that currently exists was any good at LLM acceleration, but they all have really bad memory bottlenecks.
- aurareturn 2y agoFirst NPU arrived 7 years ago in an iPhone SoC. GPUs are also “AI” chips. Local LLM community has been using Apple Silicon Mac GPUs to do inference. I’m sure Apple Intelligence uses the NPU and maybe the GPU sometimes.
- bionhoward 2y ago
- tetrisgm 2y agoI like self hosting random stuff on docker. Ollama has been a great addition. I know it's not, but it feels on par with ChatGPT. It works perfectly on my 4090, but I've also seen it work perfectly on my friend's M3 laptop. It feels like an excellent alternative for when you don't need the heavy weights, but want something bespoke and private. I've integrated it with my Obsidian notes for 1) note generation 2) fuzzy search. I've used it as an assistant for mental health and medical questions. I'd much rather use it to query things about my music or photos than whatever the big players have planned.
- exe34 2y agowhich model are you using? what size/quant/etc? thanks!
- axpy906 2y agoAgree. Please provide more details on this setup or a link.
- deegles 2y agoJust try a few models on your machine? It takes seconds plus however long it takes to download the model.
- exe34 2y agoI would prefer to have some personal recommendations - I've had some success with Llama3.1-8B/8bits and Llama3.1-70B/1bit, but this is a fast moving field, so I think it's worth the details.
- NortySpock 2y agoNew LLM Prompt: Write a reddit post as though you were a human, extolling how fast and intelligent and useful $THIS_LLM_VERSION is... Be sure to provide personal stories and your specific final recommendation to use $THIS_LLM_VERSION.
- simion314 2y agoOpenAI APIs for GPT and Dalle have issues like non determnism, and their special prompt injection where they add stuff or modify your prompt (with no option to turn that off. Makes it impossible to do research or to debug as a developer variations of things.
- throwaway314155 2y agoWhile that's true for their ChatGPT SaaS, the API they provide doesn't impose as many restrictions.
- simion314 2y ago>While that's true for their ChatGPT SaaS, the API they provide doesn't impose as many restrictions. There are same issues with GPT API, 1. non reproducible is there in the API 2. even after we ensure we do a moderation check on the input prompt, soemtimes GPT will produce "unsafe" output and accuse itself of "unsafe" stuff and we get an error but we pay for GPT "un-safeness" IMO if the GPT is producing unsafe stuff then I should not pay for it's problems. 3. dalle gives no seed so no reproducible, and no option to opt out on their GPT modifying the prompt , so images are sometimes absurdly enhanced with extreme amount of details or extreme diversity, so you need to fight against their GPT enhancing. What extra option we have with the APIs that is useful ?
- SoothingSorbet 2y ago> 1. non reproducible is there in the API It should be reproducible if you set the temperature to 1, have you tried that?
- throwaway314155 2y agoRespectfully, this just seems like a few reasons LLM's are frustrating at the moment. Having said that, there is indeed a seed and temperature parameter in the chat/assistant API that will enable (much stronger) determinism. The reason it's not 100% guaranteed to be deterministic is because they may run their model across different hardware, and hardware-level mistakes may accumulate. With regard to DALLE - that's a fair complaint I didn't realize they don't have a seed for their API. You should really try switching to an open model if you can. You'll have complete control. I recommend flux-schnell or flux-dev.
- McBainiel 2y ago> Microsoft used LLMs to write millions of short stories and textbooks in which one thing builds on another. The result of training on this text, Bubeck says, is a model that fits on a mobile phone but has the power of the initial 2022 version of ChatGPT. I thought training LLMs on content created by LLMs was ill-advised but this would suggest otherwise
- brap 2y agoI think it can be a tradeoff to get to smaller models. Use larger models trained on the whole internet to produce output that would train the smaller model.
- gugagore 2y agoGenerally (not just for LLMs) this is called student-teacher training and/or knowledge distillation.
- andai 2y agoLook into Microsoft's Phi papers. The whole idea here is that if you train models on higher quality data (i.e. textbooks instead of blogspam) you get higher quality results. The exact training is proprietary but they seem to use a lot of GPT-4 generated training data. On that note... I've often wondered if broad memorization of trivia is really a sensible use of precious neurons. It seems like a system trained on a narrower range of high quality inputs would be much more useful (to me) than one that memorized billions of things I have no interest in. At least at the small model scale, the general knowledge aspect seems to be very unreliable anyways -- so why not throw it out entirely?
- mrfinn 2y agoIt's kinda funny how nowadays an AI with 8 billion parameters is something "small". Specially when just two years back entire racks were needed to run something giving way worst performance.
- atemerev 2y agoIDK, 8B-class quantized models run pretty fast on commodity laptops, with CPU-only inference. Thanks to the people who figured out quantization and reimplemented everything in C++, instead of academic-grade Python.
- actualwitch 2y agoA solid chunk of python is just wrappers around C/C++, most tensor frameworks included.
- atemerev 2y agoI know, and yet early model implementations were quite unoptimized compared to the modern ones.
- wslh 2y agoWhat's the current cost of building a DIY bare-bones machine setup to run the top LLaMA 3.1 models? I understand that two nodes are typically required for this. Has anyone built something similar recently, and what hardware specs would you recommend for optimal performance? Also, do you suggest waiting for any upcoming hardware releases before making a purchase?
- atemerev 2y ago405B is beyond homelab-scale. I recently obtained a 4x4090 rig, and I am comfortable running 70B and occasionally 128B-class models. For 405B, you need 8xH100 or better. A single H100 costs around $40k.
- HPsquared 2y agoHere is someone running 405b on 12x3090 (4.5bpw). Total cost around $10k. https://www.reddit.com/r/LocalLLaMA/comments/1ej9uzh/local_llama_31_405b_setup/ https://www.reddit.com/r/LocalLLaMA/comments/1ej9uzh/local_l... Admittedly it's slow (3.5 token/sec)
- wslh 2y agoApproximately, how many tokens per second would the (edited) >~ $ 40k x 8 >=~ $320k version process? Would this result in a >~32x boost in performance compared to other setups? Thanks!
- andersa 2y agoIf you really want to know an exact number for a specific use case, you can rent an 8xH100 node on RunPod and benchmark it. You should expect somewhere around 30t/s for a single response, if running the FP8 rowwise quant that would typically be used on such a node, with TensorRT-LLM. Massively more in total with batching. That quant is twice the size as the 4.5bpw one used on the Mac though. A lower quality one would be faster.
- andersa 2y ago
- brap 2y agoSome companies (OpenAI, Anthropic…) base their whole business on hosted closed source models. What’s going to happen when all of this inevitably gets commoditized? This is why I’m putting my money on Google in the long run. They have the reach to make it useful and the monetization behemoth to make it profitable.
- csmpltn 2y agoThere's plenty of competition in this space already, and it'll only get accelerated with time. There's not enough "moat" in building proprietary LLMs - you can tell by how the leading companies in this space are basically down to fighting over patents and regulatory capture (ie. mounting legal and technical barriers to scraping, procuring hardware, locking down datasets, releasing less information to the public about how the models actually work behind the scenes, lobbying for scary-yet-vague AI regulation, etc). It's fizzling out. The current incumbents are sitting on multi-billion dollar valuations and juicy funding rounds. This buys runtime for a good couple of years, but it won't last forever. There's a limit to what can be achieved with scraped datasets and deep Markov chains. Over time, it will become difficult to judge what makes one general-purpose LLM be any better than another general-purpose LLM. A new release isn't necessarily performing better or producing better quality results, and it may even regress for many use-cases (we're already seeing this with OpenAI's latest releases). Competitors will have caught up to eachother, and there shouldn't be any major differences between Claude, ChatGPT, Gemini, etc - after-all, they should all produce near-identical answers, given identical scenarios. Pace of innovation flattens out. Eventually, the technology will become wide-spread, cheap and ubiquitous. Building a (basic, but functional) LLM will be condensed down to a course you take at university (the same way people build basic operating systems and basic compilers in school). The search for AGI will continue, until the next big hype cycle comes up in 5-10 years, rinse and repeat. You'll have products geared at lawyers, office workers, creatives, virtual assistants, support departments, etc. We're already there, and it's working great for many use-cases - but it just becomes one more tool in the toolbox, the way Visual Studio, Blender and Photoshop are. The big money is in the datasets used to build, train and evaluate the LLMs. LLMs today are only as good as the data they were trained on. The competition on good, high-quality, up-to-date and clean data will accelerate. With time, it will become more difficult, expensive (and perhaps illegal) to obtain world-scale data, clean it up, and use it to train and evaluate new models. This is the real goldmine, and the only moat such companies can really have.
- andai 2y agoWhat local models is everyone using? The last one I used was Llama 3.1 8B which was pretty good (I have an old laptop). Has there been any major development since then?
- demarq 2y agoNada to be honest. I keep trying every new model, and invariably go back to llama 8b. Llama8b is the new mistral.
- moffkalast 2y agoQwen 2.5 has just released, with a surprising amount of sizes. The 14B and 32B look pretty promising for their size class but it's hard to tell yet.
- esoltys 2y agoI like [mistral-nemo](https://ollama.com/library/mistral-nemo https://ollama.com/library/mistral-nemo) "A state-of-the-art 12B model with 128k context length, built by Mistral AI in collaboration with NVIDIA."
- alanzhuly 2y agoI like the latest qwen2.5 (https://nexaai.com/Qwen/Qwen2.5-0.5B-Instruct/gguf-q4_0/readme https://nexaai.com/Qwen/Qwen2.5-0.5B-Instruct/gguf-q4_0/read...). It was just released last week. It is one of the best small langauge models right now according to benchmarks. And it is small and fast!
- oneshtein 2y agocodestral 22b, aya-23 35b, gemma2 27b-instruct.
- swah 2y agoI saw this demo a few months back - and lost it, of LLM autocompletion that was a few milliseconds - it opened a how new way on how to explore it... any ideas?
- JPLeRouzic 2y agohttps://groq.com https://groq.com is very fast. (this is not the same as Grok)
- theodorthe5 2y agoLocal LLMs are terrible compared to Claude/ChatGPT. They are useful to use as APIs for applications: much cheaper than paying for OpenAI services, and can be fine tuned to do many useful (and less useful, even illegal) things. But for the casual user, they suck compared to the very large LLMs OpenAI/Anthropic deliver.
- 78m78k7i8k 2y agoI don't think local LLM's are being marketed "for the casual user", nor do I think the casual user will care at all about running LLM's locally so I am not sure why this comparison matters.
- 123yawaworht456 2y agothey are the only thing you can use if you don't want to or aren't allowed to hand over your data to US corporations and intelligence agencies. every single query to ChatGPT/Claude/Gemini/etc will be used for any purpose, by any party, at any time. shamelessly so, because this is the new normal. Welcome to 2024. I own nothing, have no privacy, and life has never been better. >(and less useful, even illegal) things the same illegal things you can do with Notepad, or a pencil and a piece of paper.
- maxnevermind 2y agoYep, unfortunately those local models are noticeably worse. Also models are getting bigger, so even if a local basement rig for a higher quality model is possible right now, that might not be so in the future. Also Zuck and others might stop releasing their weights for the next gen models, then what, just hope they plateau, what if they don't?
- miguelaeh 2y agoI am betting on local AI and building offload.fyi to make it easy to implement in any app
- rib3ye 2y agoAm I the only one seeing obvious ads in llama3 results?
- deleted 2y ago[deleted]
- HexDecOctBin 2y agoMay as well ask here: what is the best way to use something like an LLM as a personal knowledge base? I have a few thousand book, papers and articles collected over the last decade. And while I have meticulously categorised them for fast lookup, it's getting harder and harder to search for the desired info, especially in categories which I might not have explored recently. I do have a 4070 (12 GB VRAM), so I thought that LLMs might be a solutions. But trying to figure out the whats and hows hase proven to be extremely complicated, what with deluge of techniques (fine-tuning, RAG, quantisation) that might not might not be obsolete, too many grifters hawking their own startups with thin wrappers, and a general sense that the "new shiny object" is prioritised more than actual stable solutions to real problems.
- routerl 2y agoImho opinion, and I'm no expert, but this has been working well for me: Segment the texts into chunks that make sense (i.e. into the lengths of text you'll want to find, whether this means chapters, sub-chapters, paragraphs, etc), create embeddings of each chunk, and store the resultant vectors in a vector database. Your search workflow will then be to create an embedding of your query, and perform a distance comparison (e.g. cosine similarity) which returns ranked results. This way you can now semantically search your texts. Everything I've mentioned above is fairly easily doable with existing LLM libraries like langchain or llamaindex. For reference, this is an RAG workflow.
- dockerd 2y agoWhat spec people recommend here to run small models like Llama3.1 or mistral-nemo etc. Also is it sensible to wait for newer mac, amd, nvidia hardware releasing soon?
- noman-land 2y agoYou basically need as much RAM as the size of the model.
- zozbot234 2y agoYou actually need a lot less than that if you use the mmap option, because then only activations need to be stored in RAM, the model itself can be read from disk.
- noman-land 2y agoCan you say a bit more about this? Based on my non-scientific personal experience on an M1 with 64gb memory, that's approximately what it seems to be. If the model is 4gb in size, loading it up and doing inference takes about 4gb of memory. I've used LM Studio and llamafiles directly and both seem to exhibit this behavior. I believe llamafiles use mmap by default based on what I've seen jart talk about. LM Studio allows you to "GPU offload" the model by loading it partially or completely into GPU memory, so not sure what that means.
- ycombinatrix 2y agoHow does one set this up?
- ignoramous 2y agoWith ggml the mmap part is the default. It isn't a panacea though [0]. Note that most runtimes (like MLX, ONNX, TensorFlow, JAX/XLA etc) will employ a number of techniques for efficient inference and mmap is just one part of it. [0] https://news.ycombinator.com/item?id=35455930 https://news.ycombinator.com/item?id=35455930
- albertgoeswoof 2y agoI have a three year old M1 Max, 32gb RAM. Llama 8bn runs at 25 tokens/sec, that’s fast enough, and covers 80% of what I need. On my ryzen 5600h machine, I get about 10 tokens/second, which is slow enough to be annoying. If I get stuck on a problem, switch to chat gpt or phind.com and see what that gives. Sometimes, it’s not the LLM that helps, but changing the context and rewriting the question. However I cannot use the online providers for anything remotely sensitive, which is more often than you might think. Local LLMs are the future, it’s like having your own private Google running locally.
- meiraleal 2y agoWe need browser and OS level API (mobile) integration to the local LLM.
- evilduck 2y agoBoth are in their early stages: https://developer.chrome.com/docs/ai https://developer.chrome.com/docs/ai https://developer.apple.com/documentation/AppIntents/Integrating-your-app-with-siri-and-apple-intelligence https://developer.apple.com/documentation/AppIntents/Integra...
- fsmv 2y agoA small model necessarily is missing many facts. The large model is the one that has memorized the whole internet, the small one is just trained to mimic the big one. You simply cannot compress the whole internet under 10gb without throwing out a lot of information. Please be careful about what you take as fact coming from the local model output. Small models are better suited to summarization.
- albertgoeswoof 2y agoI don’t trust anything as fact coming out of these models. I ask it for how to structure solutions, with examples. Then I read the output and research the specifics before using anything further. I wouldn’t copy and paste from even the smartest minds, nevermind a model output.
- jmount 2y agoI think this is a big deal. In my opinion, many money making stable AI services are going to be deliberately of limited ability on limited domains. One doesn't want one's site help bot answering political questions. So this could really pull much of the revenue away from AI/LLMs as service.
- pella 2y agoLlama 3.1 405B "2 MacBooks is all you need. Llama 3.1 405B running distributed across 2 MacBooks using @exolabs_ home AI cluster" https://x.com/AIatMeta/status/1834633042339741961 https://x.com/AIatMeta/status/1834633042339741961
- IshKebab 2y ago"All you need is £10k of Apple laptops..."
- deleted 2y ago[deleted]
- nurettin 2y agoThat is... probable, if you bought a newish m2 to replace your 5-6 year old macbook pro which is now just lying around. Or maybe you and your spouse can share cpu hours.
- svnt 2y agoNo, you need two of the newest M3 Macbook Pros with maxed RAM, which in practice some people might have, but it is not gettable by using old hardware. And not having tried it, I’m guessing it will probably run at 1-2 tokens per second or less since the 70b model on one of these runs at 3-4, and now we are distributing the process over the network, which is best case maybe 40-80Gb/s It is possible, and that’s about the most you can say about it.
- earslap 2y agoyes but still, a local model, a lightning in a bottle that is between GPT3.5 and GPT4 (closer to 4), yours forever, for about that price is pretty good deal today. probably won't be a good deal in a couple years but for the value, it is not that unsettling. When ChatGPT first launched 2 years ago we all wondered what it would take to have something close to that locally with no strings attached, and turns out it is "a couple years and about $10k" (all due to open weights provided by some companies, training such a model still costs millions) which is neat. It will never be more expensive.
- vessenes 2y agoAll this will be an interesting side note in the history of language models in the next eight months when roughly 1.5 billion iPhone users will get a local language model tied seamlessly to a mid-tier cloud based language model native in their OS. What I think will be interesting is seeing which of the open models stick around and for how long when we have super easy ‘good enough’ models that provide quality integration. My bet is not many, sadly. I’m sure Llama will continue to be developed, and perhaps Mistral will get additional European government support, and we’ll have at least one offering from China like Qwen, and Bytedance and Tencent will continue to compete a-la Google and co. But, I don’t know if there’s a market for ten separately trained open foundation models long term. I’d like to make sure there’s some diversity in research and implementation of these in the open access space. It’s a critical tool for humans, and it seems possible to me that leaders will be able to keep extending the gap for a while; when you’re using that gap not just to build faster AI, but do other things, the future feels pretty high volatility right now. Which is interesting! But, I’d prefer we come out of it with people all over the world having access to these.
- jannyfer 2y ago> in the next eight months when roughly 1.5 billion iPhone users will get a local language model tied seamlessly to a mid-tier cloud based language model native in their OS. Only iPhone 15 Pro or later will get Apple Intelligence, so the number will be wayyy smaller.
- visarga 2y agoNot in EU they won't.
- deleted 2y ago[deleted]
- darby_nine 2y agoI expect people will just ship with their own model where the built-in one isn't sufficient. When people describe it as a "critical tool" i feel like I'm missing basic information about how people use computers and interact with the world. In what way is it critical for anything? It's still just a toy at this point.
- Anunayj 2y agoI recently experimented with running llama-3.1-8b-instruct locally on my Consumer hardware, aka my Nvidia RTX 4060 with 8GB VRAM, as I wanted to experiment with prompting pdfs with a large context which is extremely expensive with how LLMs are priced. I was able to fit the model with decent speeds (30 tokens/seconds) and a 20k token context completely on the GPU. For summarization, the performance of these models are decent enough. However unfortunately in my use case I felt using Gemini's Free Tier with it's multimodal capabilities and much better quality output made running local LLMs not really worth it as of right now, atleast for consumers.
- mistrial9 2y agoyou moved the goalposts when you add 'multimodal' there; another item is, no one reads PDF tables and illustrations perfectly, at any price AFAIK
- ComputerGuru 2y agoSupposedly submitting screenshots of pdfs (at a large enough zoom per tile/page) to OpenAI gtp4o or Google’s whatever is currently the best way of handling charts and tables.
- binary132 2y agoI really get the feeling with these models that what we need is a very memory-first hardware architecture that is not necessarily the fastest at crunching.... that seems like it shouldn’t necessarily be a terrifically expensive product
- shrubble 2y agoThe newest laptops are supposed to have 40-50 TOPS performance with the new AI/NPU features. Wondering what that will mean in practice.
- toddmorey 2y agoI narrate notes to myself on my morning walks[1] and then run whisper locally to turn the audio into text... before having an LLM clean up my ramblings into organized notes and todo lists. I have it pretty much all local now, but I don't mind waiting a few extra seconds for it to process since it's once a day. I like the privacy because I was never comfortable telling my entire life to a remote AI company. [1] It feels super strange to talk to yourself, but luckily I'm out early enough that I'm often alone. Worst case, I pretend I'm talking to someone on the phone.
- alyandon 2y agoI would be greatly interested in knowing how you set all that up if you felt like sharing the specifics.
- toddmorey 2y agoMy hope is to make this easy with a GH repo or at least detailed instructions. I'm on a Mac and I found the easiest way to run & use local models is Ollama as it has a rest interface: https://github.com/ollama/ollama/blob/main/docs/api.md https://github.com/ollama/ollama/blob/main/docs/api.md I just have a local script that pulls the audio file from Voice Memos (after it syncs from my iPhone), runs it through openai's whisper (really the best at voice to speech; excellent results) and then makes sense of it all with a prompt that asks for organized summary notes and todos in GH flavored markdown. That final output goes into my Obsidian vault. The model I use is llama3.1 but haven't spent much time testing others. I find you don't really need the largest models since the task is to organize text rather than augment it with a lot of external knowledge. Humorously the harder part of the process was finding where the hell Voice Memos actually stores these audio files. I wish you could set the location yourself! They live deep inside ~/Library/Containers. Voice Memos has no export feature, but I found you can drag any audio recording out of the left sidebar to the desktop or a folder. So I just drag the voice memo into a folder my script watches and then it runs the automation. If anyone has another, better option for recording your voice on an iPhone, let me know! The nice thing about all this is you don't even have to start / stop the recording ever on your walk... just leave it going. Dead space and side conversations and commands to your dog are all well handled and never seem to pollute my notes.
- wazdra 2y agoI'd like to point out that llama 3.1 is not open source[1] (I was recently made aware of that fact by [2], when it was on HN front page) While it's very nice to see a peak of interest for local, "open-weights" LLMs, this is an unfortunate choice of words, as it undermines the quite important differences between llama's license model and open-source. The license question does not seem to be addressed at all in the article. [1]: https://www.llama.com/llama3_1/license/ https://www.llama.com/llama3_1/license/ [2]: https://csvbase.com/blog/14 https://csvbase.com/blog/14
- sergiotapia 2y agothat ship sailed 13 years ago dude.
- gptisms 2y agoIt's not an open source licence, but to Meta's credit it's also not a licence you have to sell your soul to use. No more than 700 million monthly active users without requesting a licence, and don't use the model for illegal purposes and follow an AUP. It's as good as open source if you're using the models locally. If you're serious about building a commercial product with llama the AUP is almost certainly going to line up with your own terms of service for your product. Compare this with other models that are also billed as 'open source': yi: previously non-commercial but Apache 2.0 now? deepseek: usage policy larger gemma models: usage policy databricks: similar to llama: no more than 700 million MAUs, usage policy, additional restrictions on using outputs to train other models qwen: no more than 100 million MAUs mistral: non-commercial command-r: CC BY-NC starling: CC BY-NC There are a handful of niche models released under MIT/Apache but the norm is licences similar to or more restrictive than the Llama Community Licence, and I really doubt the situation would be better if Meta wasn't first. >"open-weights" LLMs I doubt this is the point you're making, but the training data really isn't useful even if it could be released under a permissive licence. Most models use similar datasets: reddit (no licence afaik, copyright belongs to comment authors), stackoverflow (CC BY-SA), wikipedia (CC BY-SA), Project Gutenberg (public domain?), previously books3 (books under copyright by publishers with more money and lawyers than reddit users), etc, with various degrees of filtering to remove harmful data. You can't do much with this much data unless you have millions of dollars worth of compute laying around, and you can't rebuild llama any more than any other company using the same data have 'rebuilt llama' - all models trained in a similar manner on the same data are going to converge in outputs eventually. Compare with Linux distributions, they all use the same packages but you're not going to get the same results.
- pimeys 2y agoHas anybody found a good way to utilize ollama with an editor such as zed to do things like "generate rustdoc to this method" etc. I use ollama daily for a ton of things, but for code generation, completion and documentation 4o is still much better than any of the local models...
- navbaker 2y agoThe Continue extension for VSCode is pretty good and has native connectivity to a local install of Ollama
- pimeys 2y agoZed has also support for ollama, but all the local models I tried do not really work so well to things like "write docs for this method"... Also local editor autocomplete in the style of github copilot would be great, without needing to use proprietary Microsoft tooling...
- vunderba 2y agoThere's a lot of plugins/IDEs for assistant style LLMs, but the only TAB style autocompletion ones I know of are either proprietary (Github Copilot), or you need to get an API key (Codestral). If anyone knows of a local autocomplete model I'd love to hear about it. The Continue extension (Jetbrains, VSCodium) lets you set up assistant and autocompletion independently with different API keys.
- antifa 2y agohttps://tabby.tabbyml.com/ https://tabby.tabbyml.com/ https://sourcegraph.com/cody https://sourcegraph.com/cody
- statenjason 2y agoI use gen.nvim[1] with for small tasks, like “write a type definition for this JSON” . Running locally avoids the concern of sending IP or PII to third parties. [1]: https://github.com/David-Kunz/gen.nvim https://github.com/David-Kunz/gen.nvim
- api 2y agoI use ollama through a Mac app called BoltAI quite a bit. It’s like having a smart portable sci-fi “computer assistant” for research and it’s all local. It is about the only thing I can do on my M1 Pro to spin up the fans and make the bottom of the case hot. Llama3.1, Deepseek Coder v2, and some of the Mistral models are good. ChatGPT and Claude top tier models are still better for very hard stuff.
- drio 2y agoIt looks good. Thank you for sharing. Is anyone aware of a similar tool for Linux?
- noman-land 2y agoFor anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducing-llamafile/ https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a step further and use this exact setup for a local-only co-pilot style code autocomplete and chat using Twinny. I use this every day. It's free, private, and offline. https://github.com/twinnydotdev/twinny https://github.com/twinnydotdev/twinny Local LLMs are the only future worth living in.
- christkv 2y agoI recommend llmstudio for this usually
- _kidlike 2y agoOr https://ollama.com/ https://ollama.com/
- fortyseven 2y agoThis has been my go-to for all of my local LLM interaction: it easy to get going, manages all of the models easily. Nice clean API for projects. Updated regularly; works across Windows, Mac, Linux. It's a wrapper around LlamaCpp, but it's a damned good one.
- brewtide 2y agoSame here, however minimal. I've also installed openwebui so the instance has a local web interface, and then use tailscale to access my at home LAN when put and about on the cellphone. (Goes16 weather data, ollama, a speed cam setup, and esphome temp sensors around the home / property). It's been pretty flawless, and honestly pretty darn useful here and there. The big guns go faster and do more, but I'd prefer not having every interaction logged etc. 6core 8th gen i7 I think, with a 1050ti. Old stuff. And it's quick enough on the smaller 7/8b models for sure.
- stainablesteel 2y agoi think this is laughable, the only good 8B models are the llama ones, phi is terrible, even codestral can barely code and that's 22B iirc but truthfully the 8B just aren't that great yet, they can provide some decent info if you're just investigating things but a google search is still faster
- aledalgrande 2y agoDoes anyone know of a local "Siri" implementation? Whisper + Llama (or Phi or something else), that can run shortcuts, take notes, read web pages etc.? PS: for reading web pages I know there's voices integrated in the browser/OS but those are horrible
- xenospn 2y agoApple intelligence?
- aledalgrande 2y agoIt isn't clear if you can know when the task gets handed off to their servers. But yeah that'd be the closest I know. I'm not sure it would build a local knowledge base though.
- gardnr 2y agoEdit: I just found this. I'll give it a try today: https://github.com/0ssamaak0/SiriLLama https://github.com/0ssamaak0/SiriLLama --- Open WebUI has a voice chat but the voices are not great. I'm sure they'd love a PR that integrates StyleTTS2. You can give it a Serper API Key and it will search the web to use as context. It connects to ollama running on a linux box with a $300 RTX 3060 with 12GB of VRAM. The 4bit quant of Llama 3.1 8B takes up a bit more than 6GB of VRAM which means it can run embedding models and STT on the card at the same time. 12GB is the minimum I'd recommend for running quantized models. The RTX 4070 Ti Super is 3x the cost but 7 times "faster" on matmuls. The AMD cards do inference OK but they are a constant source of frustration when trying to do anything else. I bought one and tried for 3 months before selling it. It's not worth the effort. I don't have any interest in allowing it to run shortcuts. Open WebUI has pipelines for integrating function calling. HomeAssistant has some integrations if that's the kind of thing you are thinking about.
- aledalgrande 2y agoThat SiriLLama project looks awesome! I'll give it a try. I also just spun up https://github.com/ItzCrazyKns/Perplexica https://github.com/ItzCrazyKns/Perplexica to try a local Perplexity alternative.
- jsemrau 2y agoThe Mistral models are not half as bad for this.
- shahzaibmushtaq 2y agoI need to have two things of my own that work offline for privacy concerns and cost savings: 1. Local LLM AI models with GUI and command line 2. > Local LLM-based coding tools do exist (such as Google DeepMind’s CodeGemma and one from California-based developers Continue)
- create-username 2y agothere's no small AI that I know of and masters ancient Greek, Latin, English, German and French and that I can run on my 18 GB macbook pro. Please correct me if I'm wrong. It would make my life slightly more comfortable
- sparkybrap 2y agoI agree. Even bi-lingual (English+1) small models would be very useful to process localized data, for ex english-french, english-german, etc. Right now the small models (llama 8B) can't handle this type of task, although they could if they were trained for bi lingual data.
- sandspar 2y agoWhat advantages do local models have over exterior models? Why would I run one locally if ChatGPT works well?
- pieix 2y ago1) Offline connectivity — pretty cool to be able to debug technical problems while flying (or otherwise off grid) with a local LLM, and current 8B models are usually good enough for the first line of questions that you otherwise would have googled. 2) Privacy 3) Removing safety filters — there are some great “abliterated” models out there that have had their refusal behavior removed. Running these locally and never having your request refused due to corporate risk aversion is a very different experience to calling a safety-neutered API. Depending on your use case some, all, or none of these will be relevant, but they are undeniable benefits that are very much within reach using a laptop and the current crop of models.
- pilooch 2y agoI run a fineruned mulmodal LLM as a spam filter (reads emails as images). Game changer. Removes all the stuff I wouldn't read anyways, not only spam.
- peterweyand38 2y ago[dead]
- Archit3ch 2y agoOne use case I've found very convenient: partial screenshot |> minicpm-v Covers 90% of OCR needs with 10% of the effort. No API keys, scripting, or network required.
- trash_cat 2y agoI think it REALLY depends on your use case. Do you want to brainstorm, clear out some thoughts, search or solve complex tasks?
- Anthony1321 2y ago[flagged]
- RevEng 2y agoI'm currently working on an LLM-based product for a large company that's used in circuit design. Our customers have very strict confidentiality requirements since the field is very competitive and they all have trade secret technologies that give them significant competitive advantages. Using something public like ChatGPT is simply out of the question. Their design environments are often completely disconnected from the public internet, so our tools need to run local models. Llama 3 has worked well for us so far and we're looking at other models too. We also prefer not being locked in to a specific vendor like OpenAI, since our reliance on the model puts us in a poor position to negotiate and the future of AI companies isn't guaranteed. For my personal use, I also prefer to use local models. I'm not a fan of OpenAI's shenanigans and Google already abuses its customers data. I also want the ability to make queries on my own local files without having to upload all my information to a third party cloud service. Finally, fine tuning is very valuable for improving performance in niche domains where public data isn't generally available. While online providers do support fine tuning through their services, this results in significant lock in as you have to pay them to do the tuning in their servers, you have to provide them with all your confidential data, and they own the resulting model which you can only use through their service. It might be convenient at first, but it's a significant business risk.
- fsndz 2y agoMy Thesis: Small language models (SLM)— models so compact that you can run them on a computer with just 4GB of RAM — are the future. SLMs are efficient enough to be deployed on edge devices, while still maintaining enough intelligence to be useful. https://www.lycee.ai/blog/why-small-language-models-are-the-future https://www.lycee.ai/blog/why-small-language-models-are-the-...
- bookofbiden 2y ago[flagged]
- alanzhuly 2y agoFor anyone looking for a simple alternative for running local models beyond just text, Nexa AI has built an SDK that supports text, audio (STT, TTS), image generation (e.g., Stable Diffusion), and multimodal models! It also has a model hub to help you easily find local models optimized for your device. Nexa AI local model hub: https://nexaai.com/ https://nexaai.com/ Toolkit: https://github.com/NexaAI/nexa-sdk https://github.com/NexaAI/nexa-sdk It also comes with a built-in local UI to get started with local models easily and OpenAI-compatible API (with JSON schema for function calling and streaming) for starting local development easily. You can run the Nexa SDK on any device with a Python environment—and GPU acceleration is supported! Local LLMs, and especially multimodal local models are the future. It is the only way to make AI accessible (cost-efficient) and safe.
- rthaswrea 2y agoAnother solid option https://www.anaconda.com/products/ai-navigator https://www.anaconda.com/products/ai-navigator
- atentaten 2y agoI'm interested in running locally, but I haven't found consistent advice on hardware specs for optimal performance. I would like to build a server with the best GPU and tons of RAM to run and experiment with these models.
- Havoc 2y agoEither get a Mac with loads of mem or build a rig with a 3090 or three
- ein0p 2y agoLately I’ve been submitting the same requests to GPT 4o and Mistral Large running on my laptop or one of my own servers that has 2xA6000. I get about the same quality from both and probably would not be able to tell which is which in a blind test. There are a couple disadvantages to either of the two local solutions, however. Both of them are slower than 4o. Both of them require a substantial amount of very expensive hardware (which I happed to already have, but still). And on a laptop a single long generation can drain a few percent of the battery. Neural Engine, which would be more energy efficient cannot currently run models this large, and smaller ones do not do as well for the chatbot use case. I do also use smaller LLMs locally, for the more “narrow” tasks.
- teleforce 2y agoRecently in HN there's a repost on how big data is dead (2023) and both HN posts had garnered more than a combined 800 comments and 1000 points, both very substantial in term of interests on this controversial topic [1], [2]. I think someone will post the same topic and link again next year and probably will get similar number of comments and points. In comments sections for repost, I did mentioned that big data is not dead but just having its AI winter moment, and big data is not only about volume (of storage) but also on the memory requirements (of RAM) [3]. Fast forward a few months, it seems the AI LLM is reviving the big data scenario and is seems that big data is far from dead but it's very much healthy in the era of LLMs, cloud or local. Wait until IoT and machine-to-machine (M2M) based systems are in full effects and big data will not just surviving but will be thriving. [1] Big data is dead (2023) - original https://news.ycombinator.com/item?id=34694926 https://news.ycombinator.com/item?id=34694926 [2] Big data is dead (2023) - repost https://news.ycombinator.com/item?id=40488844 https://news.ycombinator.com/item?id=40488844 [3] Comments on: Big data is dead (2023): https://news.ycombinator.com/item?id=40489607 https://news.ycombinator.com/item?id=40489607
- dharma1 2y agoWhat’s the best local codegen model that gets close to Claude 3.5? Deepseek coder? Don’t want to pay for Claude API and Cursor sub gets past the limits quick. Gonna get a M4 Mac with maxed out ram when they come out next month