6 ms·
Having a 20GB file that lets you ask an offline computer any question is amazing
- schoen 3y agoI was pretty impressed by the power and accuracy of the Google Translate offline models (which predate some of the amazing new LLMs), also in a seemingly amazingly small space—tens of megabytes per language, if I remember correctly?
- cassianoleal 3y agoFirefox's builtin translation is not bad either. I've seen it make a mistake here and there, or fail to translate certain words and expressions but it works generally well.
- schoen 3y agoIs it an offline system, not using an API?
- cassianoleal 3y agoYes. "The Firefox Translations extension can automatically translate content from the web pages you visit. Unlike some cloud-based alternatives, this extension translates text locally in Firefox, so the content you’re translating doesn’t leave your machine." https://addons.mozilla.org/en-GB/firefox/addon/firefox-translations/ https://addons.mozilla.org/en-GB/firefox/addon/firefox-trans...
- schoen 3y agoCool, I'll give that a try! I didn't know about it before.
- martin-adams 3y agoI feel the same way with running stable diffusion on my MacBook (DiffuseBee app). The idea that I can generate an image of almost anything and it has references to what it means in the real world, and carry that in my backpack, is little hard to comprehend.
- noobermin 3y agoThe attitude in the comments is amazing. The LLM chatbot hype cycle bears a lot of resemblance to the crypto hype before the crash.
- NicoJuicy 3y agoThe utility of LLM's is a very different question as the profitablity of LLM's. That was something that was severely overlapping with crypto. Most people wanted to believe it's useful because it would mean $$$ LLM 's are here to stay, that's for sure.
- kgeist 3y agoIt seems like everyone in our company uses generative AI at the moment: marketing department coming up with ideas, designers creating illustrations, programmers writing code etc. I.e. there's a lot of useful real life application of generative AI right now, people actually use it for work. Crypto, for most part, in my opinion, did not provide as much value to the average person. And it's now possible to run LLMs locally, so you shouldn't be afraid of exit scams :)
- petesergeant 3y agoI would pay several hundred dollars a month to keep access to GPT-4, and never found any use at all for crypto. There might be hype, but there’s also an insane amount of utility
- mnd999 3y agoSome examples please. I’ve yet to find any actual utility, mostly I’ve got a lot of convincing bullshit out of it.
- vidarh 3y agoI use it to speed up writing boilerplate code. That alone saves me enough time to justify paying for access many times over.
- petesergeant 3y ago
- rvba 3y agoYou can download a local version of Wikipedia and yet barely anyone does this. (And I know wikipedia has a ton of problems)
- bmacho 3y agoOr an Encyclopedia Britannica on the shelf/in pdf. People don't know that they need it, but it indeed feels amazing when you need an information, and it is there.
- lawn 3y agoWikipedia is static, a local AI can be customized and adapted to your needs. For example I would love an AI that could filter, summarize and prioritize news that I'm interested in.
- EVa5I7bHFq9mnYK 3y agoAI doesn't know anything past September 2021. Now who is static?
- MacsHeadroom 3y agoChatGPT doesn't know anything past September 2021. Local LLMs can be as recent as yesterday. Plus they can access a daily updated offline version of Wikipedia as well as news RSS feeds via LLamaIndex and/or Langchain.
- marginalia_nu 3y agoYeah it's a pretty fun dataset that you can do a lot of stuff with. Very slept on.
- lpuerto 3y agoWhy would I do something like that?? Every time I ask something to chatGPT I get wrong answers…
- JEDI-HACKER 3y ago[dead]
- _neil 3y agoMaybe you’re asking the wrong questions
- beebeepka 3y agoHe knows the answer but not the question
- avian 3y agoCorrect. Parent commenter seems to be asking questions they already know the answer to. Those are the wrong kind of questions. They should only ask the questions about things they don't know anything about. ChatGPT gets those right every time.
- cromka 3y agoI appreciate the facetiousness.
- mango7283 3y agoThis is the funniest yet most meaningful comment I've read this year!
- m463 3y agofun laws - the Gell-Mann amnesia effect / Knoll's law of media accuracy: https://en.wikipedia.org/wiki/Michael_Crichton#GellMannAmnesiaEffect https://en.wikipedia.org/wiki/Michael_Crichton#GellMannAmnes...
- smcl 3y ago
- behnamoh 3y agoMy goal: download the entire Wikipedia and store it in a vector db. Then have a local LLM answer questions based on Wikipedia. Best of both worlds!
- thestral 3y agoThere's this by Cohere - https://txt.cohere.com/embedding-archives-wikipedia/ https://txt.cohere.com/embedding-archives-wikipedia/
- tetris11 3y agoI wonder what personality it would take on, when you ask it write fiction
- the_third_wave 3y agoGiven the biased nature of many if not most Wikipedia articles it would be certain that such a system would bend the truth to an unacceptable degree. Train the thing on more than just Wikipedia, e.g add Encyclopedia Britannica to the training set. This is the one area where "diversity" really matters: diversity of opinion. Using a single biased source as your oracle will turn you into a pawn for those who control the bias.
- _ZeD_ 3y agowhat? since when wikipedia is biased? and about what? (and why EB isn't?)
- dharmab 3y agoAt the risk of conflict of interest, there's a wikipedia page that acknowledges that niche sections of the site have biases. Of particular interest is the section on Croatia. https://en.m.wikipedia.org/wiki/Ideological_bias_on_Wikipedia https://en.m.wikipedia.org/wiki/Ideological_bias_on_Wikipedi...
- karmakurtisaani 3y ago
- kramerger 3y agoWe can do that, but then we will need to close all applications to run it because it will need 20GB RAM or VRAM to run. It is still more practical to visit stackoverflow when I have a question. (still impressive how much this improves every week. Maybe in 2-3 months we have something more useful)
- moonchrome 3y agoI got 64gb ram in my desktop for few hundred €. Laptops overprice RAM (as a price discriminator I guess). But unlike Apple SoC, most GPUs have separate memory - getting a GPU with 24gb ram is very very expensive in comparison.
- midasuni 3y agoI just upgraded my 2020 era thinkpad (t470, which I bought this time last year for less than £300) to 32g memory for £70. 20g of ram doesn’t feel like a lot today.
- 0x6c6f6c 3y agoAlso today: soldered memory and 8GB MBPs that START at $1300.
- deleted 3y ago[deleted]
- fancyfredbot 3y agoThe magic of swap means you can leave those applications open and as long as they're idle they'll just quietly wait for you on disk until you've finished your LLM adventure.
- kramerger 3y agoOn many modern systems swap is quite small and not large enough to hold the entire system. For a 32GB machine, Ubuntu recommends 6GB and redhat recommends 6.4GB (20%). In a VM that number is much much lower. Besides, some libraries allocate memory in such way that you get artificial memory spikes in physical memory. I know this because I spent this weekend debugging crashes in an LLM pipeline :(
- samwillis 3y agoI am completely convinced the future is local LLMs, privet, on your device. But then also fine tuned with your own data. All your documents, email, movements, browser history - maybe it even gets trained on your calls. Effectively an extension to your own brain. I really don't want my AI assistant in the cloud, controlled by a corporation, reading and seeing everything I do. I would go as far as saying I want local AI more and open source AI. Another comment said they want it to be an offline Wikipedia, we don't need that, we need it to know all our own stuff, that's what we work with every day.
- sircastor 3y agoI also don’t want my AI in the cloud favoring a corporation’s values and goals. But these tools are almost certainly going to come from them. And they want you to store all your data in the cloud too. And they want to keep you from running the software you want.
- samwillis 3y agoI think there is one company very well placed to do this, Apple. Their revenue is driven by hardware sales, rather than advertising, and they have committed to some level of "privacy" focussed design (although obviously there are criticisms). On top of that their hardware is looking almost like it was design for exactly these use cases. Apple silicone with its unified memory architecture, GPU and Neural cores is very well placed for local LLMs. Apple have a history of doing stuff locally, on the hardware they designed, that other companies have offloaded to the cloud. They are looking for "the next big thing" that lets them sell hardware. Some people thinks that's going to be their VR/AR headset, I don't. It's going to be local AI running privately on your device. Putting the AI in the cloud doesn't help with hardware sales. My only hesitation is that they have such a crap spell checker, surly you would solve first...
- synctext 3y ago> Apple silicone with its unified memory architecture, GPU and Neural cores is very well placed for local LLMs. Yes, university research from 2011 already showed a privacy focused design to machine learning[1]. But will this be private or public like Wikipedia, Bitcoin and Bittorrent? > Effectively an extension to your own brain. Correct. After 1996 fantasy/visionary publications about this, we now know roughly how to do this. Most difficult problem is who should own it, no corporation in control is desired. That means full decentralisation, federated learning is not enough. But fully decentralised learning is hard. Try to apply machine learning in permissionless, byzantine, unsupervised, decentralised, adversarial, continuous learning context. See my lab at Delft University focused for a decade already on "The Global Brain"[2]. With 2.3 million download and crowd-sourcing, we might get there.. [1] "Gossip Learning with Linear Models on Fully Distributed Data", 2011, https://arxiv.org/abs/1109.1396 https://arxiv.org/abs/1109.1396 [2] The Global Brain - the roadmap, https://github.com/Tribler/tribler/issues/7064 https://github.com/Tribler/tribler/issues/7064
- narrator 3y agoThe thing about transformer models that are impressive is that they compress all sorts of stuff about human understanding down into a incredibly tiny space. They asked GPT-4 to explain GPT-2 and it found things like "something done right" or "metaphors" corresponded to neurons. It's this meta understanding of thought, just like diffusion models are this meta understanding of art. The real innovation is one can turn a crank and it keeps getting deeper and more subtle insights into the whole thing. If you turn that crank with A100s for a few days it gets very deep and doesn't hit a wall. In the earlier days of machine learning that didn't used to happen. Understanding was bounded at some point and didn't increase, but it seems transformers just keep going.
- siraben 3y agoI currently have a copy of offline Wikipedia (and Stack Overflow, WikiVoyage, etc.) via Kiwix on my laptop and also LLaMA-style models at the same time (since I like things being local), and with respect to asking an LLM questions, it would be great if it could leverage the offline Wikipedias to ensure it stays grounded and not make up facts.
- irjustin 3y agoThis is why I also like to ask phind.com and chat-4. Helps keep things grounded.
- mike503 3y agoHow much space does that take up, especially SO? There's so much junk on there.
- meghan_rain 3y agoAround 159 GB, it's surprisingly small.
- keeptrying 3y agoLearning is going to be revolutionized.
- analognoise 3y agoYes, this. Once they get a smidge better and can answer questions? Every student will have the King's Tutor - it will be Blooms Two Sigma in action. ...which will drive down the cost of all knowledge work and rather than reconfigure our society we'll simply have no entry level jobs anymore and the few people who still have jobs will still be expected to work full time but have insane outputs. This whole thing is a Monkey's Paw.
- MitPitt 3y agoWhy have a job when you have all the knowledge to sustainably live off the grid? I expect self-employment to skyrocket.
- Aerbil313 3y agoYou had all that knowledge before too, free from the internet. It’s just easier and common thing to do to live within the system with all the other people. But this is changing also with the changing world conditions.
- pmontra 3y agoOne of the comments on reddit > i have no clue why everyone isn't talking about this all the time. Talking maybe, not doing it because it's still too out of reach for most of us? I've got a laptop with 32 GB RAM and a 2GB NVIDIA card. At which speed can I run that model, if I can run it at all?
- teaearlgraycold 3y ago2GB VRAM means you can run things comparable to GPT2, a glorified Markov chain. On your CPU you could run much larger models at far from real time speeds.
- born-jre 3y ago2gb vram is not enough for running LLM but you can run on CPU gpt4all provides gui just click click download model click and you are ready, just give it a try start with `mpt-chat7b`, my suggestion https://gpt4all.io/index.html https://gpt4all.io/index.html
- stranded22 3y agoThank you for that, I’m installing now
- _xnmw 3y agoNow I have an excuse to purchase high powered machines. As a programmer who doesn't play games, edit video, or do anything requiring a high-powered graphics card, I always eyed high-end rigs as a wasteful indulgence. Not anymore! Now I can't wait to train my own models on my own data and make the computer do things for me, like its 1982 all over again.
- ProllyInfamous 3y agoI recently purchased my first new computer since 2008 — a Mac Mini M2Pro, stock. It's incredible what this $1300 machine can do, almost out of the box (e.g. DiffusionBee). From my understanding the device shares RAM/VRAM, and has a total of 16GB (up to 32), 14.5~GB of which can be used for LLM/genAI processess. I am just a layperson, a non-gamer; and this purchase has been really wild for local generations.
- lloydatkinson 3y agoHow restricted are these models? ChatGPT often refuses even simple questions
- EVa5I7bHFq9mnYK 3y agoWhat's the best model for 16GB GPU on my laptop?
- MacsHeadroom 3y agoWizardLM-13B-Uncensored
- EVa5I7bHFq9mnYK 3y agothank you!
- pentagrama 3y agoI think this will be more mainstream if someone create an installable app (win/mac/linux), right now I did a bit of research and I have to install several stuff (with https://huggingface.co/ https://huggingface.co/), it was confusing to a non advance user, I wasn't able to do it.