12 ms·
I think it's a good idea for everyone to download and be able to run a LLM locally, even if you have the minimum of requirements. As a pseudo-backup of a large
by bottlepalm 3y ago
I think it's a good idea for everyone to download and be able to run a LLM locally, even if you have the minimum of requirements. As a pseudo-backup of a large chunk of human knowledge.
- TaylorAlexander 3y agoI contend that most human knowledge is not written down or if it is written down it’s not publicly available on the internet and so does not exist in these datasets. There’s so much subtle knowledge like the way a mother learns to calm her child or the way a carpenter learns to work different kinds of wood which may be written down in part, but may also be learned through lived experience or transferred from human to human such that little of it gets written down and posted online.
- mickdarling 3y agoWait till all the videos ever created are tokenized and ingested into a training dataset. Carpentry techniques are certainly there. The subtleties of parenting maybe harder to derive from that, but maybe lots of little snippets of people’s lives will add up to a general understanding of parenting. There have certainly been bigger surprises in the field.
- oblio 3y agoWhat about smells or tastes? Or feelings? I can't help but feel we're at the "aliens watch people eat from space and recreate chemically identical food that has no taste" phase of AI development.
- skeledrew 3y agoIf the food is chemically identical then the taste would be the same though, since taste (and smell) is about chemistry. I do get what you're saying though.
- nyokodo 3y ago> If the food is chemically identical… If it were 99.9% chemically identical but they left out the salt and spices…
- skeledrew 3y agoI'd say that, when it comes to chemistry, only 100% reproduction can be considered identical. Anything less is to be deemed similar to some degree. And so without the correct amount of salt and/or spices, we're talking about food that's very similar, and not identical.
- samus 3y agoTheir perception is very likely to be totally different. * They might not perceive some substances at all, others that we don't notice might make it unpalatable. * Some substances might be perceived differently than us, or be indistinguishable from others. * And some might require getting used to. Note that all of the above phenomena also occur in humans because of genetics, cultural background, or experiences!
- skeledrew 3y agoThis may come off as pedantic, but "identical" is a very strong term when it comes to something like chemistry. The smallest chemical difference can manifest as a large physical difference. Consider that genetically, humans are about 60% similar to the fruit fly, yet phenotically, the similarity could be considered under 1%.
- dekhn 3y agohttps://en.wikipedia.org/wiki/Knowledge_argument https://en.wikipedia.org/wiki/Knowledge_argument
- skeledrew 3y agoAn interesting thought experiment, but there's a flaw in it, an implicit fallacy that's probably a straw man. On its own, the argument would likely stand that Mary gains new knowledge on actually being exposed to color. However, there is a broader context: this is supporting an argument against physicalism, and in this light it falls apart. There are a couple of missing bits required to complete the experiment in this context. The understanding that knowledge comes in 2 varieties: direct (actual experience) and indirect (description by one with the actual experience using shared language). This understanding brings proper clarity to the original argument, as we are aware - I think - that language is used to create compressed representations of things; something like a perceptual hash function. The other key bit, which I guess we've only considered and extensively explored after the argument was formulated, is that all information coming in via the senses goes to the brain as electrical signals. And we actually have experimental data showing that sensory information can be emulated using machines. Thus, the original argument, to be relevant to the context, should be completed by giving Mary access to a machine that she can program to emulate the electrical signals that represent color experience. I posit that without access to that hypothetical machine, given the context of the experiment, it cannot be said that Mary has "learned everything there is to learn about color". And once she has comprehensively and correctly utilized said machine on herself, she will gain no new knowledge when she is exposed to the world of color. Therefore this experiment cannot be used as an argument against physicalism as originally intended.
- mickdarling 3y agoWell, I have synesthetic smell/color senses, so I don’t even know what other humans experience, nor they me. But, I have described it in detail to many people and they seem to get the idea, and can even predict how certain smells will “look” to me. All that took was using words to describe things.
- nyokodo 3y ago> All that took was using words to describe things. All that took was words and a shared experience of smelling.
- mickdarling 3y agoHow rude, what do our bathing habits have to do with this? ;-) But, fair point. The gist I was trying to get across is that I don't even know what a plant smells like to you, and you don't know what a plant smells like to me. Those aren't comparable with any objective data. We make guesses, and we try to get close with our descriptions, which are in words. That's the best we can do and we share our senses. Asking more from computers seems overly picky to me.
- visarga 3y agoI think we can safely say that any taste, smell, sensation or emotion of any importance has been described 1000 times over in the text corpus of GPT. Even though it is fragmented, by sheer volume there is enough signal in the training set, otherwise it would not be able to generate coherent text. In this case I think the map (language) is asymptotically close to the territory (sensations & experience in general).
- andersa 3y agoWhat makes you think they aren't already?
- spacephysics 3y agoFor sure agree, however as the storage of information evolves, it’s becoming more efficient over time From oral tradition to tablets to scrolls to books to mass produced books to digital and now these LLMs, I think it’s still a good idea to preserve what we have the best we can. Not as a replacement, but a hedge against a potential library of Alexandria incident. I could imagine a time in the near future where the models are domain-specific, and just like there are trusted encyclopedia publishers there are trusted model publishers that guarantee a certain level of accuracy. It’s not like reading a book, but I for sure had an easier time learning golang talking with ChatGPT than a book
- nyokodo 3y ago> a hedge against a potential library of Alexandria incident What would cause a Library of Alexandria incident wiping out all human knowledge elsewhere, that would also allow you to run a local LLM?
- deleted 3y ago[deleted]
- AnthonyMouse 3y agoTo run a local LLM you need the device it currently runs on and electricity. There are actually quite a lot of ways to generate electricity, but to name one, a diesel generator that can run on vegetable oil. What you're really asking is, what could cause a modern Library of Alexandria incident? But the fact is we keep the only copy of too many things on the servers of the major cloud providers. Which are then intended to have their own internal redundancy, but that doesn't protect you against a targeted attack or a systemic failure when all the copies are under the same roof and you lose every redundant copy at once from a single mistake replicated in a monoculture.
- spacephysics 3y agoA more dooms-day prepping would call for some heavy lead-faraday cage to store the storage mediums in the event of an EMP/major solar flare. Or more Sci-fi related, some hyper computer virus that ends up infecting all internet connected devices. Not too far fetched if we can conceive of some AI enabled worm that mutates depending on the target, I could imagine a model of sorts being feasible within the next 5-10 years
- _ache_ 3y agoI think you underestimate the amount of information contained in books and the extent to which our society (as a whole) depends on them.
- Barrin92 3y agosociety depends much more on social networks, mentorship and tacit knowledge than books. It's easy to test this. Just run the thought experiment by a few people, if you could get only one, would you take an Ivy league degree without the education or the education without the degree? Venture capital in tech is a good example of this. The book knowledge is effectively globally distributed and almost free, effectively success happens in a few geographically concentrated counties.
- _ache_ 3y agoBy book, I mean, written in any form, study paper, blog, theses, books, etc. I don't understand you comparison. Same for your example, no logical link between the effect and the consequences.
- skeledrew 3y agoI'd content that those are skills (gained through experience) rather than knowledge (gained through rote learning).
- TaylorAlexander 3y agoI think it’s worth expanding your definition of knowledge.
- bamboozled 3y agoYes but it contains enough hints to help someone find their way on the these types of tasks.
- nicklecompte 3y agoIt's not even "human knowledge" that can't be written down - it seems all vertebrates understand causality, quantity (in the sense of intuitively understanding what numbers are), and object permanence. Good luck writing those concepts down in a way that GPT can use! In general AI in 2024 is not even close to understanding these ideas, nor does any AI developer have a clue how to build an AI with this understanding. The best we can do is imitating object permanence for a small subset of perceptible objects, a limitation not found in dogs or spiders.
- wruza 3y agoThat's where humans suck. The classic "you're not doing it right" then proceeds to quickly show how to do it without verbalizing any info on learning process, pitfalls, failure modes, etc, as if just showing it was enough for themselves to learn. Most people do[n't do] that, not even a sign of reflection. My worst case was with a guy who asked me to write an arbitrage betting bot. When I asked how to calculate coeffs, he pointed at two values and said "look, there <x>, there <y> thinks for a minute then it's <z>!". When I asked how exactly did he calculate it, he simply repeated with different numbers.
- samus 3y ago> When I asked how exactly did he calculate it, he simply repeated with different numbers. Now you know how an LLM feels during training!
- stavros 3y agoProbably during inference, as well.
- Aerroon 3y agoPeople often don't know how to verbalize them in the first place. Some of these topics are very complex, but our intuition gets us halfway there. Once upon a time I was good at a video game. Everyone realized that positioning is extremely important in this game. I have good positioning in that game and was asked many times to make a guide about positioning. I never did, because I don't really know how. There is too much information they you need to convey to cover all the various situations. I think you would first have to come up with a framework on positioning to be able to really teach this to someone else. Some kind of base truths/patterns that you can then use to convey the meaning. I believe the same thing applies to a lot of these processes that aren't verbalized.
- snovv_crash 3y agoOften for this kind of problem writing a closed form solution is simply intractable. However, it's often still possible to express the cost function of at least a big portion of what goes into a human-optimal solution. From here you can sample your space, do gradient descent or whatever to find some acceptable solution that has a more human-intuitive property.
- HarHarVeryFunny 3y ago> I contend that most human knowledge is not written down Yes - the available training data is essentially mostly a combination of declarative knowledge (facts - including human-generated artifacts) and procedural knowledge (how to do things). What is missing is the learning process of taking a description of how to do something, and trying to apply that yourself in a specific situation. No amount of reading books, or reading other people's blogs on how they did something, can avoid the need for hands-on experience if you want to learn how to do it yourself. It's not just a matter of information that might be missing or unclear in instructional material, including how to cope with every type of failure and unexpected outcome, but crucially how to do this yourself - if you are to be the actor, then it's the predictive process in your mind that matters. Partly for this reason, and partly because current AI's (transformer-based LLMs) don't support online learning (try & fail skill acquisition), I think we're going to see two distinct phases of AI. 1) The current "GenAI" phase where AI can only produce mash-ups of things it saw in it's pre-training data, augmented by similar "book learning" provided in-context which can be utilized by in-context learning. I'd characterize what this type of AI to be useful for, and capable of, as "automation". Applying that book (incl. anecdotal) knowledge to new situations where mash-up is all you need. 2) The second phase is where we have something closer to AGI, even if still below human level, which is no longer just a pre-trained transformer, but also has online learning and is agentic - taking actions predicated on innate traits like curiosity and boredom, so that given the book knowledge it can (& will!) then learn to apply that by experimentation/practice and learning from its own mistakes. There will no doubt be advances beyond this "phase two" as well, but it seems we're likely to be stuck at "phase one" for a while (even as models become much better at phase one capabilities), until architectures fundamentally advance beyond transformers to allow this type of on-the-job training and skill acquisition.
- texuf 3y agoAny recommendations for the latest and greatest way to run these locally?
- speps 3y agollamafile as per TFA...
- etc-hosts 3y agohttps://justine.lol/oneliners/ https://justine.lol/oneliners/
- threecheese 3y agoThis looks amazing, but the docs mention .llamafiles exceed the Windows executable size limit, and there are workarounds to externalize the weights. Do you think this is an impediment to its becoming popular? Or is MS consumer hardware just far enough behind (w/o dedi gpu) that “there’s time”?
- fragmede 3y agoollama
- slowmotiony 3y agoI use a tool called LM Studio, makes it trivial to run these models on a Mac. You can also use it as a local API so it kinda acts like a drop-in replacement for the openAI API.
- chown 3y agoI am the author of Msty [1]. My goal is to make it as straightforward as possible with just one click (once you download the app). If you end up trying it, I would love to hear your feedback. 1: https://msty.app https://msty.app
- mikewarot 3y agoI don't see LLMs as a large chunk of knowledge, I see them as an emergent alien intelligence snapshotted at the moment it appeared to stop learning. It's further hobbled by the limited context window it has to use, and the probabilistic output structure that allows for outside random influences to pick its next word. Both the context window and output structure are, in my opinion, massive impedance mismatches for the emergent intellect embedded in the weights of the model. If there were a way to match the impedance, I strongly suspect we'd already have AGI on our hands.
- bamboozled 3y agoWhat is alien about them ? LLMs are of this earth and created by our species. Seems quite familiar to me.
- jfoster 3y agoThey can write in a way similar to how a human might write, but they're not human. The chat interfaces (Claude, ChatGPT) certainly have a particular style of writing, but the underlying LLMs are definitely capable of impersonating as our species in the medium of text.
- deleted 3y ago[deleted]
- bamboozled 3y agoBut they're extremely relatable to us because it's regurgitating us. I saw this talk with Geoffrey Hinton the other day and he said he was astonished at the capabilities of ChatGPT-4 because he asked it what the relationship between a compost heap and a nuclear bomb was, and he couldn't believe it answered, he really thought it was proof the thing could reason. Totally mind blown. However I got it right away with zero effort. Either I'm a super genius or this has been discussed before and made it's way into the training data. Usual disclaimer: I don't think this invalidates the usefulness of AI or LLMs, just that we might be bamboozling ourselves into the idea that we've created an alien intelligence.
- gpm 3y agoIf you want to download a backup of a large chunk of human knowledge... download wikipedia. It's a similar size to a small LLM and can actually distinguish between real life and fantasy: https://en.wikipedia.org/wiki/Wikipedia:Database_download https://en.wikipedia.org/wiki/Wikipedia:Database_download If you just want to play around with an LLM though, absolutely.
- int_19h 3y agoKiwix provides prepackaged highly compressed archives of Wikipedia, Project Gutenberg, and many other useful things: https://download.kiwix.org/zim/ https://download.kiwix.org/zim/. Between that and dirt cheap storage prices, it is possible to have a local, offline copy of more human knowledge than one can sensibly consume in a lifetime. Hell, it's possible to have it all on one's smartphone (just get one with an SD card slot and shove a 1+ Tb one in there).
- claritise 3y agoJust create a RAG with wikipedia as the corpus and a low parameter model to run it and you can basically have an instantly queryable corpus of human knowledge runnable on an old raspberry pi.
- CaptainOfCoit 3y ago> a low parameter model > on an old raspberry pi I bet the LLM responses will be great... You're better off just opening up a raw text dump of Wikipedia markup files in vim.
- boywitharupee 3y agobut which model to tokenize with? is there a leaderboard for models that are good for RAG?
- sroussey 3y ago“For RAG” is ambiguous. First there is a leaderboard for embeddings. [1] Even then, it depends how you use them. Some embeddings pack the highest signal in the beginning so you can truncate the vector, while most can not. You might want that truncated version for a fast dirty index. Same with using multiple models of differing vector sizes for the same content. Do you preprocess your text? There will be a model there. Likely the same model you would use to process the query. There is a model for asking questions from context. Sometimes that is a different model. [2]
- simonw 3y agoI strongly recommend that people run LLMs locally for a different reason. The ones you can run on your own machine tend to be bad - really bad. They hallucinate wildly and fail at all sorts of tasks that the larger hosted ones succeed at. This makes them a fantastic tool for learning more about how LLMs work and what they're useful for. Interacting with a weak-but-functional LLM that runs on your own computer is a great way to get a much more solid mental model for what these things actually are.
- qingcharles 3y agoYou can have really bad and fast, or really slow and decent right now. Choose one :)
- devsda 3y agoFor someone interested in learning about LLMs, running them locally is a good way to understand the internals. For everyone else, I wish they experience these (locally or elsewhere) weak LLMs atleast once before using the commercial ones just to understand various failure modes and to introduce a healthy dose of skepticism towards the results instead of blindly trusting them to be the facts/truth.
- simonw 3y agoCompletely agree. Playing around with a weak LLM is a great way to give yourself a little bit of extra healthy skepticism for when you work with the strong ones.
- mmahemoff 3y agoHow do you learn about the internals by running LLMs locally? Are you playing with The code, runtime params, or just interacting via chat?
- samus 3y agoThe abstractions are relatively brittle. If you don't have a powerful GPU, you will be forced to consider how to split the model between CPU and GPU, how much context size you need, whether to quantize the model, and the tradeoffs implied by these things. To understand these, you have to develop a basic model how an LLM works.
- jrflowers 3y agoIt is invaluable to have a chunk of human knowledge that can tell you things like the Brooklyn Nets won the 1986 Cricket World Cup by scoring 46 yards in only 3 frames
- fragmede 3y agoAccording to ChatGPT > Australia won the 1987 Cricket World Cup. The 1986 date is incorrect; there was no Cricket World Cup in 1986. The tournament took place in 1987, and Australia defeated England in the final to win their first title. https://chat.openai.com/share/e9360faa-1157-4806-80ea-563489f946f3 https://chat.openai.com/share/e9360faa-1157-4806-80ea-563489... I'm no cricket fan, so someone will have to correct Wikipedia if that's wrong. If you want to point out that LLMs hallucinate, you might want to speak plainly and just come out and say it, or at least give a real world example and not one where it didn't.
- vlunkr 3y agoWe’re not talking about running chatGPT locally though, are we?
- fragmede 3y agosigh your going to make me open my laptop, aren't you.
- fragmede 3y agoI ran 'who won the 1986 Cricket World Cup' against llama2-uncensored (the local model I have pre-downloaded) and hilarious got 5 different answers asking it 5 times: >>> who won the 1986 Cricket World Cup India >>> who won the 1986 Cricket World Cup Australia >>> who won the 1986 Cricket World Cup New Zealand >>> who won the 1986 Cricket World Cup West Indies >>> who won the 1986 Cricket World Cup England Which proves GP's point about hallucinations, though none of those are > Brooklyn Nets won the 1986 Cricket World Cup by scoring 46 yards in only 3 frames LLM's hallucinations are insidous because they have the ring of truth around them. yards and frames aren't cricket terms, so we're off to the races with them.
- TheCaptain4815 3y agoIt’s kind of crazy really. Before LLMs, any type of world scale disaster you’d hope for what? Wikipedia backups? Now, a single LLM ran locally would be much more effective. Imagine the local models in 5 years!
- danmur 3y agoUh yeah I would, and still am, take the Wikipedia backup for doomsday scenarios. I'm not even sure how that would be a competition
- Zambyte 3y agoThe processing required to run current language models with a useful amount of knowledge encoded in them is way more than I imagine would be available in a "world scale disaster".
- int_19h 3y agoThere's a lot more than just Wikipedia that gets archived, and yes, that is a far more sensible way to go about it. For one thing, the compute required to then read it back is orders of magnitude less (a 15 year old smartphone can handle it just fine). For another, you don't have to wonder how much of what you got back is hallucinated - data is either there or it's corrupted and unreadable.
- creatonez 3y agoMaybe I'm seeing things through a modern lens, but if I were trying to restart civilization and was only left with ChatGPT, I would be enraged and very much not grateful for this.
- nyokodo 3y ago> if I were trying to restart civilization and was only left with ChatGPT In this scenario you’d need to also be left with a big chunk of compute, and power infrastructure. Since ChatGPT is the front end of the model you’d also need to have the internet still going in a minimum capacity.
- deleted 3y ago[deleted]
- CaptainOfCoit 3y agoIf we're playing this game, you forgot to mention that they also need: A monitor, a keyboard, roof over their head (to prevent rain from entering your electronic), etc etc... But really, didn't you catch the meaning of parents message, or are you being purposefully obtuse?
- devsda 3y agoI think re-imagining the "Dr. Stone" series with the main character replaced by an LLM will be a funny & interesting series if we decide to stay true to LLMs nature and make it hallucinate as well. Given the way LLMs are right now, I suspect there will be lot of failed experiments and the kingdom of science will not advance that quick.
- latexr 3y ago> the kingdom of science will not advance that quick. It’s more likely that it wouldn’t even start. The first step to any development was figuring out nitric acid as the cure to the petrification. Good luck getting any LLM to figure that out. Even if it did, good luck getting any of the other characters to know what to do with that information that early on.
- m3kw9 3y agoAnd why would I need to backup human knowledge as an individual
- exe34 3y agoYou remember those fantasies where you got up from your seat at the pub and punched the lights out of this guy for being rude? A lot of us have fantasies of being the all powerful oracle that guides a reboot of civilization using knowledge of science and engineering.
- latexr 3y ago> the all powerful oracle that guides a reboot of civilization using knowledge of science and engineering. https://en.wikipedia.org/wiki/Dr._Stone https://en.wikipedia.org/wiki/Dr._Stone
- raincole 3y agoIt seems to be an unbelievably inefficient way to back up knowledge.
- samus 3y agoAre they though? They are lossy compressing trillions of tokens into a few dozen GB. The decompression action is fuzzy and inefficient though.
- raincole 3y agoAnd it requires massive computational power to decompress, which I don't expect to be available in a catastrophic situation where humans have lost a large chunk of important knowledge.
- samus 3y agoI don't necessarily agree. It requires massive computing power, but running models smaller than 70G parameters is possible on consumer hardware, albeit slowly.
- threecheese 3y agoParent may be thinking more along the lines of a “hope we can print all the knowledge“ type catastrophe. Though if there is zero compute it’ll be tough reading all those disks!
- LunaSea 3y agoI wonder how the Chinese government will manage to sensor LLMs within China?
- kalleboo 3y agoI had downloaded some LLMs to run locally just to experiment when a freak hailstorm suddenly left me without internet for over a week. It was really interesting to use a local LLM as a replacement for Google. It gave me a new mental model for LLMs rather than a "spicy autocomplete" or whatever, I now think of it as "a lossy compressed database of knowledge". Like you ran the internet through JPEG at 30% quality.
- pizzafeelsright 3y agoFeels like that really smart friend who is probably correct but ya just don't know.
- dragonwriter 3y agoLanguage models are an inefficient way to store knowledge; if you want to have a “pseudo-backup of a large chunk of human knowledge,” download a wikipedia dump, not an LLM. If you want a friendly but fallible UI to that dump, download an LLM and build a simple ReAct framework around it with prompting to use the wikipedia dump for reference.
- TrevorJ 3y agoIt's a very underrated side effect of this whole LLM thing: We've created a super compact representation of human knowledge in a form that requires a FAR less complex tech stack to get the information 'out' of in the future. A year ago, a lot of this information only existed on the internet, and would have been nearly impossible to recover in any cohesive unfragmented form if the lights were to ever go out on our civilization. Now the problem space has moved simply to "find a single solitary PC that will still boot up", and boom, you have access to everything. I think we just created our Rosetta stone.