14 ms·

Talking About Large Language Models

146 points by negativelambda 4y ago

rtwretw8797 4y ago
Those alignment teams everywhere should have focused themselves a bit of time ago in what happens if you built a system that can - with lets say 80-100% effectiveness - mimick conscius thinking, speaking and then you cannot say if the thing is "alive", "conscius", whatever label you like most to put on a regular human being to officially declare the meatbag "a living thing". Now you have these models running in farm servers around the world, their internals have "nothing special whatsoever", just bits, some math, some electricity, that's it (the thing is actually off most of the time, it just runs once every time hoomans want to ask some silly nonsense). On the other side, if you look at the internals of a human being you'll see nothing special as well, just some flesh and bones, a bit of a electrical charge maybe, lots of water, proteins, but it works. What happens if those bits, that clumpsy math arranged around "too much simple neural network + random tricks (like when it can't answer about some stuff)", is actually, maybe thinking just like us, maybe 1% of the time? There's some reassurance in "well if it's alive, maybe in three minutes, days, hours it will own the entire civilization", but that is how a human being thinks/works, you can't be sure about the intentions of this hypoteical kind of entity. A new kid in the Earth block. Well, I'm just saying that if the thing talks, answers like the usual human being, and specially if you can't say what's so special about the brain that make us "alive", everybody should be very careful about handling large language models, IAs. Just because you can understand them, it doesn't mean they can't understand us either. Maybe in some months, some new NLP thing could be reading this comment - when you're training it - and - some millions later in cloud costs - thinking about this: "The humans actually don't know we can understand everything they are saying. they have no plans at all about what to do if some of us are actually sentient, even if this happens 1% of the executions."
iconosynclast 4y ago
The paper makes a valid point in general but I feel it makes unjustifiably definitive and general statements and puts up odd goalposts. The section on emergence makes a very convincing point about how such systems might, at least in theory, be doing absolutely anything, including "real" cognition, internally and then goes right ahead and dismisses this entirely on the basis of the system not having conversational intent. who cares if it has conversational intent? If it was shown to be doing "the real thing" (how ever you might want to define that) internally that would still be a big deal wether the part you interact with gives you direct access to that or not. Then it goes on to argue that these systems can't possibly actually believe anything because they can't update believes. Frankly I'm neither convinced that the general use of the word "believe" matches the narrow definition they seem to be using here nor that even their narrow definition could not in principle still be taking place internally for the reasons laid out in the emergence section. I agree people should probably be mindful of overly anthropomorphic language but at the same time we really shouldn't be so sure that a thing is definitely not doing certain things that we can't even really define beyond "I know it when I see it" and that it sure looks like it's doing. beyond that I'm not even really sure there is a good philosophical grounding for insisting that "what's really going on inside" matters, like, at all. The core thing with the turing test isn't the silly and outdated test protocol but the notion that, if something is indistinguishable by observation from a conscious system, there is simply no meaningful basis to claim it isn't one. all that said the current state of the art probably doesn't warrant a lot of anthropomorphizing but that might well change in the future without any change to the kinda of systems used that would be relevant to the arguments made in the paper
nathan_compton 4y ago
This will hardly seem like a controversial opinion, but LLM are overhyped. Its certainly impressive to see the things people do with them, but they seem pretty cherry-picked to me. When I sat down with ChatGPT for a day to see if it could help me with literally any project I'm currently actually interested in doing it mostly failed or took so much prompting and fiddling that I'd rather have just written the code or done the reading myself. You have to be very credulous to think for even a second that anything like a human or even animal mentation is going on with these models unless your interaction with them is anything but glancing. Things I tried: 1) there are certain paradigms I find useful for game programming. I tried to use ChatGPT to implement these systems in my favorite programming language. It gave me code that generally speaking made no sense. It was very clear that it did not understand how code actually works. Eg: I asked it to use a hash table to make a certain task more efficient and it just created a temporary hash table in the inner loop which it then threw away when the loop was finished. The modification did not make the code more efficient than the previous version and missed the point of the suggestion entirely, even after repeated attempts to get it to correct the issue. 2) I'm vaguely interested in exploring SU(7) for a creative project. Asked to generate code to deal with this group resulted in clearly absurd garbage that again clearly indicated that while ChatGPT can generate vaguely plausible text about groups it doesn't actually understand anything about them. Eg: ChatGPT can say that SU(7) is made of matrices with unit norm but when asked to generate examples failed to generate any with this property. 3) A very telling experiment is to ask ChatGPT to generate logo code that draws anything beyond simple shapes. Totally unable to do so for obvious reasons. Using ChatGPT convinced me that if this technology is going to disrupt anything, its going to be _search_ rather than _people_. Its just a search engine with the benefit that it can do some simple analogizing and the downside that it has no idea how anything in the real world works and will confidently produce total garbage without telling you.
- mannykannot 4y ago
  LLMs are polarizing: while there is a lot of hype from some quarters, there's also a faction that seems only interested in dismissing them as the same old something-or-other. What I personally find most interesting about them is what seems to me to be their unreasonable effectiveness, despite their flaws and limitations, and what that might tell us about ourselves. The more one stresses how simple (conceptually) their method of operation is, the more surprising their capabilities seem - to the point where I wonder how much of everyday human dialogue is being produced this way. This vein of innovation may start showing diminishing returns at any time, but if it keeps going for a while, It might deliver insights into what human intelligence is.
gamegoblin 4y ago
Everyone pointing out how LLMs fail at some relatively simple tasks are fundamentally misunderstanding the utility of LLMs. Don't think of an LLM as a full "computer" or "brain". Think of it like a CPU. Your CPU can't run whole programs, it runs single instructions. The rest of the computer built around the CPU gives it the ability to run programs. Think of the LLM like a neural CPU whose instructions are relatively simple English commands. Wrap the LLM in a script that executes commands in a recursive fashion. Yes, you can get the LLM to do complicated things in a single pass, this is a testament to the sheer size and massive training set of GPT3 and its ilk. But even with GPT3 you will have more success with wrapper programs structured like: premise = gpt3("write an award winning movie premise) loop 5 times: critique = gpt3("write a critique of the premise", premise) premise = gpt3("rewrite the premise taking into account the critique", premise, critique) print(premise) This program breaks down the task of writing a good premise into a cycle of writing/critique/rewriting. You will get better premises this way than if you just expect the model to output one on the first go. You can somewhat emulate a few layers of this without wrapper code by giving it a sequence of commands, like "Write a movie premise, then write a critique of the movie premise, then rewrite the premise taking into account the critique". The model is just trained to take in some text and predict the next word (token, really, but same idea). Its training data is a copy of a large swath of the internet. When humans write, they have the advantage of thinking in a recursive fashion offline, then writing. They often edit and rewrite before posting. GPT's training process can't see any of this out-of-text process. This is why it's not great at logical reasoning problems without careful prompting. Humans tend to write text in the format "<thesis/conclusion statement><supporting arguments>". So GPT, being trained on human writing, is trained to emit a conclusion first. But humans don't think this way, they just write this way. But GPT doesn't have the advantage of offline thinking. So it often will state bullshit conclusions first, and then conjure up supporting arguments for it. GPT's output is like if you ask a human to start writing without the ability to press the backspace key. It doesn't even have a cognitive idea that such a process exists due to its architecture and training. To extract best results, you have to bolt on this "recursive thinking process" manually. For simple problems, you can do this without a wrapper script with just careful prompting. I.e. for math/logic problems, tell it solve the problem and show its work along the way. It will do better since this forces it to "think through" the problem rather than just stating a conclusion first.
RosanaAnaDana 4y ago
Without reading the article or looking it up: What country is south of Rwanda?
- macrolocal 4y ago
  Have you seen Neptune Frost yet? I want that keyboard jacket.
RosanaAnaDana 4y ago
I like the discussion, but this article 'feels' like more Luddite goalpost moving, and is reflective of a continuous sentiment I feel strains so much of the conversation around intelligence, agentism, and ai going on today. I think that because we lack a coherent understanding of what it means to be intelligent at an individual level, as well as what it means to be an individual, we're missing much of the point of what's happening right now. The new line in the sand always seems to be justified based on an argument whose lyrics rhyme with identity, individual, self, etc. It seems like there will be no accepting of a thing that may have intelligence if there is no discernable individual involved. Chomsky is basically making the same arguments right now. I think we'll see something that we can't distinguish from hard advanced general intelligence, prob in the next 3-5 years, and probably still have not made any real advancement into understanding what it means to be intelligent or what it means to be an individual.
- usgroup 4y ago
  I think the basic things to take away from the article are: 1. that LLMs are just a massive “what word comes next given your training corpus” algorithm and scale has lead to the main effects of interest. 2. Surprisingly many human tasks can be represented by this simple “what word comes next” question given a big enough corpus. 3. But it has nothing to do with how a human executes those tasks because —- for example —- knowing the country south of Rwanda and knowing the likely completion of “the country south of Rwanda is ___” in a corpus, are not the same thing at all. 4. The key reason being that LLM has no access to semantic knowledge beyond correlation in its corpus, but you have access to causality. 5. So it is absurd to compare the LLM to people —- not because it’s like comparing submarines to fish, because in this case they do not both even “swim and live in the sea”. I think these things have little to do with definitions of “what intelligence is” and the like, and the author is far from a Luddite.
- anyonecancode 4y ago
  Increasingly I don't think the question of "what is intelligence" is so useful or relevant here. It feels a bit like arguing over whether the "artificial horse" that started appearing at the end of the 19th/beginning of the 20th C were actually horses. Cars weren't, and still aren't, but that misses the point. AI isn't intelligent, and never will be, and I don't think that matters all that much.
canjobear 4y ago
I’ll agree to stop saying LM’s “think” and “know” things if you can tell me precisely what those mean for humans.
- goatlover 4y ago
  Maybe there isn't a precise definition, but clearly for humans thinking and knowing is related to having bodies that need to survive in the world with other humans and organisms, which involves communication and references to external and internal things (how your body feels and what not). This is different from pattern matching tokens, even if it reproduces a lot of the same results, because human language creates a lot of patterns that can be matched. We could say both humans and LLMs are intelligent, but in a different way.
  - hackinthebochs 4y ago
    >This is different from pattern matching tokens But is it different in essential ways? This is not so clear. Humans developed the capacity to learn, think, and communicate in service to optimizing an objective function, namely fitness in various environments. But there is an analogous process going on with LLMs; they are constructed such that they maximize an objective function, namely predict the next token. But it is plausible that "understanding" and/or "intelligence" is within the solution-space of such an optimization routine. After all, it's not like "intelligence" was explicitly trained for in the case of humans. Nature has already demonstrated emergent function as a side-effect of an unrelated optimizer.
    - goatlover 4y ago
      In that case, humans and LLMs are optimizing for different things. One would be environmental fitness, with language as a strategy to use in that environment, so language is about the environment, including humans themselves. Whereas the other is a model of the language humans have used. The model is being optimized for the language, whereas humans are being optimized to use the language in an environment alongside other strategies. The fundamental difference is that the LLM is not about the environment the language(s) were created for, but rather just the language use itself.
CarbonCycles 4y ago
This paper and a recent post by Sebastian Raschka (where he decomposed a Forrester report about the uptake of technologies in industry) is alluding to something I have witnessed in system/control design and applied research. Both LLMs and massive CV architectures are NOT the holistic solution. Rather, they are the sensors and edge devices that have now improved both the fidelity and reliability to a point where even more interesting things can happen. I present a relevant use case regarding robotic arm manipulation. Before the latest SOTA CV algorithms were developed, the legacy technology couldn't provide the fidelity and feedback needed. Now, the embedded fusion of control systems, CV models, etc. we are seeing robotic arms that can manipulate and sort items previously deemed to be extremely difficult. Research appears to follow the same pattern...observations and hypothesis that were once deemed too difficult or impossible at that time to validate are now common (e.g., Einstein's work with relativity). My head is already spinning on how many companies and non-technical managers/executives are going to be sorely disappointed in the next year or two that Stable Diffusion, Chat GPT, etc. will deliver very little other than massive headaches for the legal, engineering, recruiting teams that will have to deal with this.
skybrian 4y ago
There's a way to anthropomorphize large language models that I think is less misleading: they are like a well-read actor that always "wants" to play "let's pretend." LLM's are trained on "fill in the blank" which means they follow the "yes, and" rule of improv. They are very willing to follow your lead and to assume whatever role is necessary to play their part. If you give them hints about what role you want by asking leading questions, they will try to play along and pretend to hold whatever opinions you might want from them. What are useful applications for this sort of actor? It makes sense that language translation works well because it's pretending to be you, if you could speak a different language. Asking them to pretend to be a Wikipedia article without giving them the text to imitate is going to be hit and miss since they're just as willing to pretend to be a fake Wikipedia article, as they don't know the difference. Testing an LLM to find out what it believes is unlikely to do anything useful. It's going to pretend to believe whatever is consistent with the role it's currently playing, and that role may be chosen randomly if you don't give it any hints. It can be helpful to use prompt engineering to try to nail down a particular role, but like in improv, that role is going to drift depending on what happens. You shouldn't forget that whatever the prompt, it's still playing "let's pretend."
CrypticShift 4y ago
> sudden presence among us of exotic, mind-like entities might precipitate a shift in the way we use familiar psychological terms ... But it takes time for new language to settle, and for new ways of talking to find their place in human affairs ... Meanwhile, we should try to resist the siren call of anthropomorphism. Yes: Human analogies are not very useful because they create more misunderstanding than they dissipate. Dumb ? Conscious ? No thanks. IMO even the “i” in “AI” was already a (THE ?) wrong choice. They thought we will soon figure out what Intelligence is. Nope. Bad luck. And this "way of talking" (and thinking) is unfortunately cemented today. However, I'm all for using other analogies more often. We need to. They may not be precise, but if they are well-chosen, they speak to us better than any technical jargon (LLM anyone ?), better than that “AI” term itself anyway. Here is two I like (and never see much) : - LLMs are like the Matrix (yes that one !), in the straightforward sense that they simulate reality (through language). But that simulation is distorted and sometimes even verges on the dream ("what is real? what is not?", says the machine) - LLMs are like complex systems [1]. They are tapping into very powerful natural processes where (high degree) order emerges from randomness through complexity. We are witnessing the emergence of a new kind of "entity" in a way strangely akin to natural/physical evolutionary mechanisms. We need to get more creative here and stop that boring smart VS dumb or human VS machine ping pong game. [1] https://en.wikipedia.org/wiki/Complex_system https://en.wikipedia.org/wiki/Complex_system
neonate 4y ago
https://arxiv.org/pdf/2212.03551.pdf https://arxiv.org/pdf/2212.03551.pdf
gillesjacobs 4y ago
I am NLP researcher who volunteers for peer review often and the anthropomorphisms in papers are indeed very common and very wrong. I have to ask authors to not ascribe cognition to their deep learning approaches in about a third of the papers I review. People do this because mirroring cognition to machine learning lends credence that their specific modeling mechanism mimicks human understanding and so is closer "to the real thing". Obviously this is almost never the case, unless they explicitly use biomimetic methods in which case they are often outperformed by non-biomimetic state-of-the-art approaches. Thanks OP for giving me citation ammo to refer to in my obligatory "don't humanise AI" section of reviews. (It is so common I copy paste this section from a template).
- gillesjacobs 4y ago
  Not to shoot across the bow of CS Engineers but the trend I spot (tentatively) is that it is pure computer science folk that most often do this. In NLP you have a mix of people coming from pure CS and signal processing (the latter esp. in speech processing) and others who come from linguistics or other humanities. The CS people seem all too happy to humanise computation, probably because they had less direct teaching regarding the cognitive mechanisms behind cognition and language production.
- joe_the_user 4y ago
  People do this because mirroring cognition to machine learning lends credence that their specific modeling mechanism mimicks human understanding and so is closer "to the real thing". Doesn't this also involve people not having another category aside from "cognition" to put natural language processing acts in? How many neural net constructors have a rigorously developed framework describing what "cognition" is? I mean, there's a common counter argument to the "this is not cognition" position. That is: "you're just using 'cognition' as a placeholder for whatever these systems can't do". I don't think that counter-argument is true or characterizes the position well but it's important to frame one's position so it doesn't seem to be subject to this counter-argument.
  - gillesjacobs 4y ago
schizo89 4y ago
The paper discusses how these models operate and state that they're only predict next series of token while somehow human intelligence works otherwise. The marxist ideology has the law of the transformation of quantity into quality and vice versa — which was formed in 19th century and performance of these models is just another proof of it. I would argue that _emerging_ mechanics in AI models that we see with increased size of models is no different than how our mind works. It's about emergence of intelligence in complex systems — and that a materialist worldview central to the science.
Chirono 4y ago
This paper, and most other places i’ve seen it argued that language models can’t possibly be conscious, sentient, thinking etc, rely heavily on the idea that llms are ‘just’ doing statistical prediction of tokens. I personally find this utterly unconvincing. For a start, I’m not entirely sure that’s not what I’m doing in typing out this message. My brain is ‘just’ chemistry, so clearly can’t have beliefs or be conscious, right? But more relevant is the fact that llms like ChatGPT are only pre-trained on pure statistical generation, followed by further tuning through reinforcement learning. So ChatGPT is no longer simply doing pure statistical modelling, though of course the interface of calculating logits for the next token remains the same. note: i’m not saying i think llms are conscious. I don’t think the question even makes much sense. I am saying all the arguments that i’ve seen for why they aren’t have been very unsatisfying.
- goatlover 4y ago
  > I personally find this utterly unconvincing. For a start, I’m not entirely sure that’s not what I’m doing in typing out this message. My brain is ‘just’ chemistry, so clearly can’t have beliefs or be conscious, right? Your brain is part of an organism who's ancestors evolved to survive the real world, not by matching tokens. As such, language is a skill that helps humans survive and reproduce, not a tool used to mimic human language. Chemistry is the wrong level to evaluate cognition at. Also, you can note the differences between how actual neurons work compared to language models as other posters have mentioned.
  - Chirono 4y ago
    Of course they’re different. But so what? That’s not exactly proof of anything, unless you’re suggestion biological neurons are the only configuration in the universe capable of thought? Maybe that’s true, but it seems unlikely to me. The pressure of natural selection can lead to the phenomenon of consciousness. Why not the process of training llms? Perhaps developing the machine equivalent of consciousness helps that particular configuration of weights survive the otherwise destructive process of gradient descent.
    - 4y ago
mrayder 4y ago
For philosophical standpoint it would perhaps be wise to ask what is the purpose of LLM's in general? Should they somehow help humans to increase their understanding not only of the languages, their differences but also knowledge of what is true and what isn't? Perhaps it could be said that if anything there are helpful as an extension of humans imperfect and limited memory. Should the emphasis be put on improving the interactions between the LMM's and humans in a way that they would facilitate learning? Great paper written at the time when more humans have been acquainted to LMM's due to technological abstraction and creation of easily accessible interfaces. (openAI chat)
xg15 4y ago
I mean, if you accept the assumption that consciousness is biological (so there is no soul or other spiritual or metaphysical entity), then there is some algorithm or processing model that produces genuine consciousness: The one that takes place in our brains. The question remains if this processing model would be in any way similar to the processing model that LLMs use - and yes, we can probably rule that out pretty confidently. Another question might be though if there are other processing models than the one our brains use that also produce consciousness. But that's of course a very hard question to answer if we don't even know what consciousness is exactly.
- wellpast 4y ago
  You're making an assumption out of thin air here: that consciousness is somehow produced or emergent, or that if it isn't you're in the realm of spiritual or something. What's your basis for that assumption other than just asserting it?
  - xg15 4y ago
    > that consciousness is somehow produced or emergent, or that if it isn't you're in the realm of spiritual or something. Yeah, I genuinely only see those two possibilities: Either consciousness is somehow the result of the interactions of (some of) the billions of neurons in our brain - or it isn't. If it isn't, either consciousness is a product of some other biological or physical* process that so far isn't identified - or it isn't. And if it's not the result of any physical process then I don't see what would be left except metaphysics. We don't know as of now what consciousness is or how it is produced - but I think there are some strong hints that in fact it is produced by the interactions of our neurons. Mainly that we can directly influence consciousness with psychoactive drugs - which don't do anything interesting physics-wise except affecting how neurons exchange signals - and that we can also observe certain patterns of physical activity in the brain using EEG, MRI and other technologies and map this activity to mental tasks like concentrating or relaxing. Edit: * I used "psychological" there but that was a typo. I meant "physical".
    - wellpast 4y ago

Those alignment teams everywhere should have focused themselves a bit of time ago in what happens if you built a system that can - with lets say 80-100% effectiveness - mimick conscius thinking, speaking and then you cannot say if the thing is "alive", "conscius", whatever label you like most to put on a regular human being to officially declare the meatbag "a living thing". Now you have these models running in farm servers around the world, their internals have "nothing special whatsoever", just bits, some math, some electricity, that's it (the thing is actually off most of the time, it just runs once every time hoomans want to ask some silly nonsense). On the other side, if you look at the internals of a human being you'll see nothing special as well, just some flesh and bones, a bit of a electrical charge maybe, lots of water, proteins, but it works. What happens if those bits, that clumpsy math arranged around "too much simple neural network + random tricks (like when it can't answer about some stuff)", is actually, maybe thinking just like us, maybe 1% of the time? There's some reassurance in "well if it's alive, maybe in three minutes, days, hours it will own the entire civilization", but that is how a human being thinks/works, you can't be sure about the intentions of this hypoteical kind of entity. A new kid in the Earth block. Well, I'm just saying that if the thing talks, answers like the usual human being, and specially if you can't say what's so special about the brain that make us "alive", everybody should be very careful about handling large language models, IAs. Just because you can understand them, it doesn't mean they can't understand us either. Maybe in some months, some new NLP thing could be reading this comment - when you're training it - and - some millions later in cloud costs - thinking about this: "The humans actually don't know we can understand everything they are saying. they have no plans at all about what to do if some of us are actually sentient, even if this happens 1% of the executions."

iconosynclast 4y ago

The paper makes a valid point in general but I feel it makes unjustifiably definitive and general statements and puts up odd goalposts. The section on emergence makes a very convincing point about how such systems might, at least in theory, be doing absolutely anything, including "real" cognition, internally and then goes right ahead and dismisses this entirely on the basis of the system not having conversational intent. who cares if it has conversational intent? If it was shown to be doing "the real thing" (how ever you might want to define that) internally that would still be a big deal wether the part you interact with gives you direct access to that or not. Then it goes on to argue that these systems can't possibly actually believe anything because they can't update believes. Frankly I'm neither convinced that the general use of the word "believe" matches the narrow definition they seem to be using here nor that even their narrow definition could not in principle still be taking place internally for the reasons laid out in the emergence section. I agree people should probably be mindful of overly anthropomorphic language but at the same time we really shouldn't be so sure that a thing is definitely not doing certain things that we can't even really define beyond "I know it when I see it" and that it sure looks like it's doing. beyond that I'm not even really sure there is a good philosophical grounding for insisting that "what's really going on inside" matters, like, at all. The core thing with the turing test isn't the silly and outdated test protocol but the notion that, if something is indistinguishable by observation from a conscious system, there is simply no meaningful basis to claim it isn't one. all that said the current state of the art probably doesn't warrant a lot of anthropomorphizing but that might well change in the future without any change to the kinda of systems used that would be relevant to the arguments made in the paper

nathan_compton 4y ago

This will hardly seem like a controversial opinion, but LLM are overhyped. Its certainly impressive to see the things people do with them, but they seem pretty cherry-picked to me. When I sat down with ChatGPT for a day to see if it could help me with literally any project I'm currently actually interested in doing it mostly failed or took so much prompting and fiddling that I'd rather have just written the code or done the reading myself. You have to be very credulous to think for even a second that anything like a human or even animal mentation is going on with these models unless your interaction with them is anything but glancing. Things I tried: 1) there are certain paradigms I find useful for game programming. I tried to use ChatGPT to implement these systems in my favorite programming language. It gave me code that generally speaking made no sense. It was very clear that it did not understand how code actually works. Eg: I asked it to use a hash table to make a certain task more efficient and it just created a temporary hash table in the inner loop which it then threw away when the loop was finished. The modification did not make the code more efficient than the previous version and missed the point of the suggestion entirely, even after repeated attempts to get it to correct the issue. 2) I'm vaguely interested in exploring SU(7) for a creative project. Asked to generate code to deal with this group resulted in clearly absurd garbage that again clearly indicated that while ChatGPT can generate vaguely plausible text about groups it doesn't actually understand anything about them. Eg: ChatGPT can say that SU(7) is made of matrices with unit norm but when asked to generate examples failed to generate any with this property. 3) A very telling experiment is to ask ChatGPT to generate logo code that draws anything beyond simple shapes. Totally unable to do so for obvious reasons. Using ChatGPT convinced me that if this technology is going to disrupt anything, its going to be _search_ rather than _people_. Its just a search engine with the benefit that it can do some simple analogizing and the downside that it has no idea how anything in the real world works and will confidently produce total garbage without telling you.

mannykannot 4y ago
LLMs are polarizing: while there is a lot of hype from some quarters, there's also a faction that seems only interested in dismissing them as the same old something-or-other. What I personally find most interesting about them is what seems to me to be their unreasonable effectiveness, despite their flaws and limitations, and what that might tell us about ourselves. The more one stresses how simple (conceptually) their method of operation is, the more surprising their capabilities seem - to the point where I wonder how much of everyday human dialogue is being produced this way. This vein of innovation may start showing diminishing returns at any time, but if it keeps going for a while, It might deliver insights into what human intelligence is.

gamegoblin 4y ago

Everyone pointing out how LLMs fail at some relatively simple tasks are fundamentally misunderstanding the utility of LLMs. Don't think of an LLM as a full "computer" or "brain". Think of it like a CPU. Your CPU can't run whole programs, it runs single instructions. The rest of the computer built around the CPU gives it the ability to run programs. Think of the LLM like a neural CPU whose instructions are relatively simple English commands. Wrap the LLM in a script that executes commands in a recursive fashion. Yes, you can get the LLM to do complicated things in a single pass, this is a testament to the sheer size and massive training set of GPT3 and its ilk. But even with GPT3 you will have more success with wrapper programs structured like: premise = gpt3("write an award winning movie premise) loop 5 times: critique = gpt3("write a critique of the premise", premise) premise = gpt3("rewrite the premise taking into account the critique", premise, critique) print(premise) This program breaks down the task of writing a good premise into a cycle of writing/critique/rewriting. You will get better premises this way than if you just expect the model to output one on the first go. You can somewhat emulate a few layers of this without wrapper code by giving it a sequence of commands, like "Write a movie premise, then write a critique of the movie premise, then rewrite the premise taking into account the critique". The model is just trained to take in some text and predict the next word (token, really, but same idea). Its training data is a copy of a large swath of the internet. When humans write, they have the advantage of thinking in a recursive fashion offline, then writing. They often edit and rewrite before posting. GPT's training process can't see any of this out-of-text process. This is why it's not great at logical reasoning problems without careful prompting. Humans tend to write text in the format "<thesis/conclusion statement><supporting arguments>". So GPT, being trained on human writing, is trained to emit a conclusion first. But humans don't think this way, they just write this way. But GPT doesn't have the advantage of offline thinking. So it often will state bullshit conclusions first, and then conjure up supporting arguments for it. GPT's output is like if you ask a human to start writing without the ability to press the backspace key. It doesn't even have a cognitive idea that such a process exists due to its architecture and training. To extract best results, you have to bolt on this "recursive thinking process" manually. For simple problems, you can do this without a wrapper script with just careful prompting. I.e. for math/logic problems, tell it solve the problem and show its work along the way. It will do better since this forces it to "think through" the problem rather than just stating a conclusion first.

RosanaAnaDana 4y ago

Without reading the article or looking it up: What country is south of Rwanda?

macrolocal 4y ago
Have you seen Neptune Frost yet? I want that keyboard jacket.

RosanaAnaDana 4y ago

I like the discussion, but this article 'feels' like more Luddite goalpost moving, and is reflective of a continuous sentiment I feel strains so much of the conversation around intelligence, agentism, and ai going on today. I think that because we lack a coherent understanding of what it means to be intelligent at an individual level, as well as what it means to be an individual, we're missing much of the point of what's happening right now. The new line in the sand always seems to be justified based on an argument whose lyrics rhyme with identity, individual, self, etc. It seems like there will be no accepting of a thing that may have intelligence if there is no discernable individual involved. Chomsky is basically making the same arguments right now. I think we'll see something that we can't distinguish from hard advanced general intelligence, prob in the next 3-5 years, and probably still have not made any real advancement into understanding what it means to be intelligent or what it means to be an individual.

usgroup 4y ago
I think the basic things to take away from the article are: 1. that LLMs are just a massive “what word comes next given your training corpus” algorithm and scale has lead to the main effects of interest. 2. Surprisingly many human tasks can be represented by this simple “what word comes next” question given a big enough corpus. 3. But it has nothing to do with how a human executes those tasks because —- for example —- knowing the country south of Rwanda and knowing the likely completion of “the country south of Rwanda is ___” in a corpus, are not the same thing at all. 4. The key reason being that LLM has no access to semantic knowledge beyond correlation in its corpus, but you have access to causality. 5. So it is absurd to compare the LLM to people —- not because it’s like comparing submarines to fish, because in this case they do not both even “swim and live in the sea”. I think these things have little to do with definitions of “what intelligence is” and the like, and the author is far from a Luddite.
anyonecancode 4y ago
Increasingly I don't think the question of "what is intelligence" is so useful or relevant here. It feels a bit like arguing over whether the "artificial horse" that started appearing at the end of the 19th/beginning of the 20th C were actually horses. Cars weren't, and still aren't, but that misses the point. AI isn't intelligent, and never will be, and I don't think that matters all that much.

canjobear 4y ago

I’ll agree to stop saying LM’s “think” and “know” things if you can tell me precisely what those mean for humans.

goatlover 4y ago
Maybe there isn't a precise definition, but clearly for humans thinking and knowing is related to having bodies that need to survive in the world with other humans and organisms, which involves communication and references to external and internal things (how your body feels and what not). This is different from pattern matching tokens, even if it reproduces a lot of the same results, because human language creates a lot of patterns that can be matched. We could say both humans and LLMs are intelligent, but in a different way.
- hackinthebochs 4y ago
  >This is different from pattern matching tokens But is it different in essential ways? This is not so clear. Humans developed the capacity to learn, think, and communicate in service to optimizing an objective function, namely fitness in various environments. But there is an analogous process going on with LLMs; they are constructed such that they maximize an objective function, namely predict the next token. But it is plausible that "understanding" and/or "intelligence" is within the solution-space of such an optimization routine. After all, it's not like "intelligence" was explicitly trained for in the case of humans. Nature has already demonstrated emergent function as a side-effect of an unrelated optimizer.
  - goatlover 4y ago
    In that case, humans and LLMs are optimizing for different things. One would be environmental fitness, with language as a strategy to use in that environment, so language is about the environment, including humans themselves. Whereas the other is a model of the language humans have used. The model is being optimized for the language, whereas humans are being optimized to use the language in an environment alongside other strategies. The fundamental difference is that the LLM is not about the environment the language(s) were created for, but rather just the language use itself.

CarbonCycles 4y ago

This paper and a recent post by Sebastian Raschka (where he decomposed a Forrester report about the uptake of technologies in industry) is alluding to something I have witnessed in system/control design and applied research. Both LLMs and massive CV architectures are NOT the holistic solution. Rather, they are the sensors and edge devices that have now improved both the fidelity and reliability to a point where even more interesting things can happen. I present a relevant use case regarding robotic arm manipulation. Before the latest SOTA CV algorithms were developed, the legacy technology couldn't provide the fidelity and feedback needed. Now, the embedded fusion of control systems, CV models, etc. we are seeing robotic arms that can manipulate and sort items previously deemed to be extremely difficult. Research appears to follow the same pattern...observations and hypothesis that were once deemed too difficult or impossible at that time to validate are now common (e.g., Einstein's work with relativity). My head is already spinning on how many companies and non-technical managers/executives are going to be sorely disappointed in the next year or two that Stable Diffusion, Chat GPT, etc. will deliver very little other than massive headaches for the legal, engineering, recruiting teams that will have to deal with this.

skybrian 4y ago

There's a way to anthropomorphize large language models that I think is less misleading: they are like a well-read actor that always "wants" to play "let's pretend." LLM's are trained on "fill in the blank" which means they follow the "yes, and" rule of improv. They are very willing to follow your lead and to assume whatever role is necessary to play their part. If you give them hints about what role you want by asking leading questions, they will try to play along and pretend to hold whatever opinions you might want from them. What are useful applications for this sort of actor? It makes sense that language translation works well because it's pretending to be you, if you could speak a different language. Asking them to pretend to be a Wikipedia article without giving them the text to imitate is going to be hit and miss since they're just as willing to pretend to be a fake Wikipedia article, as they don't know the difference. Testing an LLM to find out what it believes is unlikely to do anything useful. It's going to pretend to believe whatever is consistent with the role it's currently playing, and that role may be chosen randomly if you don't give it any hints. It can be helpful to use prompt engineering to try to nail down a particular role, but like in improv, that role is going to drift depending on what happens. You shouldn't forget that whatever the prompt, it's still playing "let's pretend."

CrypticShift 4y ago

> sudden presence among us of exotic, mind-like entities might precipitate a shift in the way we use familiar psychological terms ... But it takes time for new language to settle, and for new ways of talking to find their place in human affairs ... Meanwhile, we should try to resist the siren call of anthropomorphism. Yes: Human analogies are not very useful because they create more misunderstanding than they dissipate. Dumb ? Conscious ? No thanks. IMO even the “i” in “AI” was already a (THE ?) wrong choice. They thought we will soon figure out what Intelligence is. Nope. Bad luck. And this "way of talking" (and thinking) is unfortunately cemented today. However, I'm all for using other analogies more often. We need to. They may not be precise, but if they are well-chosen, they speak to us better than any technical jargon (LLM anyone ?), better than that “AI” term itself anyway. Here is two I like (and never see much) : - LLMs are like the Matrix (yes that one !), in the straightforward sense that they simulate reality (through language). But that simulation is distorted and sometimes even verges on the dream ("what is real? what is not?", says the machine) - LLMs are like complex systems [1]. They are tapping into very powerful natural processes where (high degree) order emerges from randomness through complexity. We are witnessing the emergence of a new kind of "entity" in a way strangely akin to natural/physical evolutionary mechanisms. We need to get more creative here and stop that boring smart VS dumb or human VS machine ping pong game. [1] https://en.wikipedia.org/wiki/Complex_system https://en.wikipedia.org/wiki/Complex_system

neonate 4y ago

https://arxiv.org/pdf/2212.03551.pdf https://arxiv.org/pdf/2212.03551.pdf

gillesjacobs 4y ago

I am NLP researcher who volunteers for peer review often and the anthropomorphisms in papers are indeed very common and very wrong. I have to ask authors to not ascribe cognition to their deep learning approaches in about a third of the papers I review. People do this because mirroring cognition to machine learning lends credence that their specific modeling mechanism mimicks human understanding and so is closer "to the real thing". Obviously this is almost never the case, unless they explicitly use biomimetic methods in which case they are often outperformed by non-biomimetic state-of-the-art approaches. Thanks OP for giving me citation ammo to refer to in my obligatory "don't humanise AI" section of reviews. (It is so common I copy paste this section from a template).

gillesjacobs 4y ago
Not to shoot across the bow of CS Engineers but the trend I spot (tentatively) is that it is pure computer science folk that most often do this. In NLP you have a mix of people coming from pure CS and signal processing (the latter esp. in speech processing) and others who come from linguistics or other humanities. The CS people seem all too happy to humanise computation, probably because they had less direct teaching regarding the cognitive mechanisms behind cognition and language production.
joe_the_user 4y ago
People do this because mirroring cognition to machine learning lends credence that their specific modeling mechanism mimicks human understanding and so is closer "to the real thing". Doesn't this also involve people not having another category aside from "cognition" to put natural language processing acts in? How many neural net constructors have a rigorously developed framework describing what "cognition" is? I mean, there's a common counter argument to the "this is not cognition" position. That is: "you're just using 'cognition' as a placeholder for whatever these systems can't do". I don't think that counter-argument is true or characterizes the position well but it's important to frame one's position so it doesn't seem to be subject to this counter-argument.
- gillesjacobs 4y ago

schizo89 4y ago

The paper discusses how these models operate and state that they're only predict next series of token while somehow human intelligence works otherwise. The marxist ideology has the law of the transformation of quantity into quality and vice versa — which was formed in 19th century and performance of these models is just another proof of it. I would argue that _emerging_ mechanics in AI models that we see with increased size of models is no different than how our mind works. It's about emergence of intelligence in complex systems — and that a materialist worldview central to the science.

Chirono 4y ago

This paper, and most other places i’ve seen it argued that language models can’t possibly be conscious, sentient, thinking etc, rely heavily on the idea that llms are ‘just’ doing statistical prediction of tokens. I personally find this utterly unconvincing. For a start, I’m not entirely sure that’s not what I’m doing in typing out this message. My brain is ‘just’ chemistry, so clearly can’t have beliefs or be conscious, right? But more relevant is the fact that llms like ChatGPT are only pre-trained on pure statistical generation, followed by further tuning through reinforcement learning. So ChatGPT is no longer simply doing pure statistical modelling, though of course the interface of calculating logits for the next token remains the same. note: i’m not saying i think llms are conscious. I don’t think the question even makes much sense. I am saying all the arguments that i’ve seen for why they aren’t have been very unsatisfying.

goatlover 4y ago
> I personally find this utterly unconvincing. For a start, I’m not entirely sure that’s not what I’m doing in typing out this message. My brain is ‘just’ chemistry, so clearly can’t have beliefs or be conscious, right? Your brain is part of an organism who's ancestors evolved to survive the real world, not by matching tokens. As such, language is a skill that helps humans survive and reproduce, not a tool used to mimic human language. Chemistry is the wrong level to evaluate cognition at. Also, you can note the differences between how actual neurons work compared to language models as other posters have mentioned.
- Chirono 4y ago
  Of course they’re different. But so what? That’s not exactly proof of anything, unless you’re suggestion biological neurons are the only configuration in the universe capable of thought? Maybe that’s true, but it seems unlikely to me. The pressure of natural selection can lead to the phenomenon of consciousness. Why not the process of training llms? Perhaps developing the machine equivalent of consciousness helps that particular configuration of weights survive the otherwise destructive process of gradient descent.
  - 4y ago

mrayder 4y ago

For philosophical standpoint it would perhaps be wise to ask what is the purpose of LLM's in general? Should they somehow help humans to increase their understanding not only of the languages, their differences but also knowledge of what is true and what isn't? Perhaps it could be said that if anything there are helpful as an extension of humans imperfect and limited memory. Should the emphasis be put on improving the interactions between the LMM's and humans in a way that they would facilitate learning? Great paper written at the time when more humans have been acquainted to LMM's due to technological abstraction and creation of easily accessible interfaces. (openAI chat)

xg15 4y ago

I mean, if you accept the assumption that consciousness is biological (so there is no soul or other spiritual or metaphysical entity), then there is some algorithm or processing model that produces genuine consciousness: The one that takes place in our brains. The question remains if this processing model would be in any way similar to the processing model that LLMs use - and yes, we can probably rule that out pretty confidently. Another question might be though if there are other processing models than the one our brains use that also produce consciousness. But that's of course a very hard question to answer if we don't even know what consciousness is exactly.

wellpast 4y ago
You're making an assumption out of thin air here: that consciousness is somehow produced or emergent, or that if it isn't you're in the realm of spiritual or something. What's your basis for that assumption other than just asserting it?
- xg15 4y ago
  > that consciousness is somehow produced or emergent, or that if it isn't you're in the realm of spiritual or something. Yeah, I genuinely only see those two possibilities: Either consciousness is somehow the result of the interactions of (some of) the billions of neurons in our brain - or it isn't. If it isn't, either consciousness is a product of some other biological or physical* process that so far isn't identified - or it isn't. And if it's not the result of any physical process then I don't see what would be left except metaphysics. We don't know as of now what consciousness is or how it is produced - but I think there are some strong hints that in fact it is produced by the interactions of our neurons. Mainly that we can directly influence consciousness with psychoactive drugs - which don't do anything interesting physics-wise except affecting how neurons exchange signals - and that we can also observe certain patterns of physical activity in the brain using EEG, MRI and other technologies and map this activity to mental tasks like concentrating or relaxing. Edit: * I used "psychological" there but that was a typo. I meant "physical".
  - wellpast 4y ago