4 ms·
LLMs are language models. Not knowledge models. That's a tremendous breakthrough! Language is really hard and we've basically "solved" it computationally. Incr
by lukev 3y ago
LLMs are language models. Not knowledge models.
That's a tremendous breakthrough! Language is really hard and we've basically "solved" it computationally. Incredible!
But whether via retrieval or some other form of database integration, LLMs will only become "AI" when tightly integrated with an appropriate "knowledge model".
- AnimalMuppet 3y agoNo, LLMs are a bit more than that. We encode knowledge in language. When an LLM trains on language that is not just random words, it also trains on the knowledge encoded in that language. It does so only by training on the language - there's no real understanding there - but it's more than nothing. Do AIs need a better knowledge model? Almost certainly. In this I agree with you.
- eli 3y agoI don't consider that a knowledge model. (Does a calculator have a knowledge model of multiplication?) But I agree that it's something more than Markov chains. I think maybe the scale of these LLMs makes them into something new. Maybe we need a new word to describe this type of AI.
- mannykannot 3y agoI think we have to take seriously the proposition that this is what a Markov chain can do when it is based on statistics from a vast corpus of human language use, and consider the possibility that a similar process plays a greater role in human intelligence than many of us (well, me, at least) would have thought. On the other hand, all our prior experience with language of the quality sometimes produced by LLMs has been produced by humans, so LLMs mess with our intuitions and may lead us to anthropomorphize them excessively.
- eli 3y agoYes, I think the scale that the LLMs operate at makes them interesting and different more than anything else. It's not really a new idea. There's a theory going back decades that consciousness is an emergent property and that if we could build a neural network in a lab that was big and complex enough it would become conscious. (Not really sure I buy it, but it's interesting)
- mannykannot 3y agoYes - emergence is the relatively straightforward observation that large systems can have complex features that are not discernible at the detailed level. My go-to example is Darwinian fitness, which is not discernible at the atomic level. The trickiest part of making a large neural network conscious would be in how you train it. Our brains have been trained by a half-billion years of evolution since the first neurons emerged, and while evolution is slow, that's still a long time.
- causal 3y agoRight - humans encode knowledge in language all the time, but it's certainly not the only way we keep it in our heads. Supposedly Sora is trained to have a built-in physical world model that gives it a huge advantage in its video generation abilities. It will be interesting to see what the same approach would give us with something like GPT-4.
- deleted 3y ago[deleted]
- hammock 3y ago> LLMs are language models Aren't LLMs used to generate all these images like midjourney etc as well? Or is that a different type of model?
- dartos 3y agoA image generator can and is built with the same underlying math, but different training data. LLM literally stands for “large language model”
- dragonwriter 3y agoLLMs can do a lot more than what a normal person would see as language by throwing different encoders and/or decoders than the standard text ones on either end of the model, though, and some models that do things that aren't superficially language work that way, so its a fair question to ask if you don’t specifically know how inage generation models work.
- dartos 3y agoI haven’t seen a model trained for language spit out an image given a pixel decoder. I have seen them make images by writing an svg. Do you have any links for something like a transformer model trained on language generating images with a different head?
- dragonwriter 3y agoNo, I've never seen that, either; I was just saying that given the number of other kinds of not-obviously-language things done that way, its a not-unreasonable question to have about inage generators.
- dartos 3y agoI wouldn’t call any transformer model an LLM. Just ones trained on languages. That’s why I said the same underlying math can be used for languages and images.