5 ms·

The dumb criticize the blind. What an absurd situation! How did we get here? Here are the steps: 1. Large Language Models have been presented as "AI", which p

The dumb criticize the blind. What an absurd situation! How did we get here? Here are the steps: 1. Large Language Models have been presented as "AI", which personifies them instead of describing how they work. 2. Goals for LLM development were set for the personified attributes, and not the actual functionally of the real thing. OpenAI brags about how GPT4 scores at human tests: as if that has any bearing on the model itself, and not simply its content. 3. The success of an LLM at a personified goal is determined by magic. Does the black box output what I am hoping it will? How do I improve it? Weights? Better training corpus? 4. An LLM is understood to be a person in a black box, and we could not possibly understand what that person will do next. Danger!

permo-w 4y ago
I'm not concerned about how they work or whether they're attaining personhood or not. I'm concerned at what people can and will use them for to the detriment of our civilisation
- thomastjeffery 4y ago
  How they work is the most critical piece of that puzzle. It isn't a mystery at all. LLMs implicitly model every pattern they can find in the text they are given. If the pattern aligns to language, humans categorize it as a feature. If it aligns to desired behavior, humans categorize it as a feature. If it aligns to undesired behavior, humans categorize it as a limitation. What humans categorize as features and limitations do not align with what the model is. A truth and a lie are structured with identical text patterns. The LLM can't categorize the two separately. The only risk here is that people continue to personify LLMs, and place trust in their relatively limited ability to fulfill a human role. It's the same category of risk as bad security practices. The potential to make a mistake itself is not a danger: making the mistake is.
  - permo-w 4y ago
    you certainly make an interesting point about fact and falsehood and how they interpret language; however, I disagree that this is relevant and completely disagree that personification has any relevance to the most immediate dangers the danger here is that we make extremely powerful tools extremely available and people use those tools to hurt people, possibly a lot of people, possibly even by accident the scary part is that hacker news is full of the exact people that have the knowledge and capability to do this, and look at the reaction to this post: a resounding fuck off
    - thomastjeffery 4y ago
      I just explained explicitly how and why an LLM is not, and never will be, an "extremely powerful tool". A program that is actually a person in a black box would require the level of concern you are showing. Thankfully, that is not even remotely what we have here in reality.
      - permo-w 4y ago
        have you tried gpt-4? how can you possibly say it's not an extremely powerful tool? what happens when someone determined links it - or thousands of its - or thousands of next year's 3x as powerful its -- up to the internet with a code interpreter and a hard-coded poor intention?
        tome 4y ago
        I don't know; what happens? I'd really like to know what people are imagining it could do, especially if they've got some evidence for it!
        permo-w 4y ago
        it's a rhetorical question. think about it yourself. try it yourself
        qlm 4y ago
        What could it do that isn't already being done? I ask this sincerely. We already have bots constantly attacking every server they find 24/7. We already have botnets Is the concern that GPT will be able to find and exploit e.g. a novel RCE exploit in SSH?
- cthalupa 4y ago
  > I'm concerned at what people can and will use them for to the detriment of our civilisation Maybe? But there's already all sorts of untapped resources for that - and we mostly just don't choose to use them. A lot of the misinformation and "fake news" that came out during prior election cycles in the US came from former Soviet bloc countries, and there is already significant untapped human capital in those places where it could be done for cheap, perhaps even cheaper than the cost of doing inference on premiere LLMs. Meanwhile we see all sorts of efficiency improvements occurring for generally positive use cases from these LLMs, and it is hardly unique for technology to have some risks to it as well as positive benefits. Things like GPT and Midjourney have been a qualitative leap over their predecessors from two decades ago, but they fundamentally are not doing something new - and even another qualitative leap wouldn't bring them to the level of existential threat that this letter seems to be afraid of. I think there are significant and real concerns about AGI and similar, but I can't seem to find any reason to share these same concern around the types of "AI" we are seeing today.
adamsmith143 4y ago
At no point in your rambling, incoherent comment were you even close to anything that could be considered a rational thought. Everyone on this site is now dumber for having read it. May God have mercy on your soul.
- thomastjeffery 4y ago
  My point is that this fear is irrational: based on a misunderstanding about what LLMs can do, which itself is based on a misunderstanding of what LLMs are. Is there a way I could have been more clear? Are you disinterested in the content of my comment, or the presentation? The only thing I can learn from your comment is that mine didn't live up to your mysterious standards.
- dmarchand90 4y ago
  https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- dang 4y ago
  Yikes! Please don't do this on HN. I'm not going to ban you because we haven't had to warn you before, and also because I sense an overtone of good humor in the scorching remarks, but we can't have people posting like this—it leads to flames. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
marcosdumay 4y ago
> Goals for LLM development were set for the personified attributes, and not the actual functionally of the real thing. Well, this is for honest reasons. The goal of a chatbox is to beat the Turing test. It has always been. Those chatboxes didn't actually beat it, but it's clear that it's due to a technicality (they are easy to spot). They can do empty chats on the same level as a human. (And so it turns up that the test wasn't as useful as it was believed to be at first.)
- thomastjeffery 4y ago
  The problem is when you loop that logic around: it becomes circular reasoning. What is the true source of an improved SAT score? If it's a person we are talking about, then it's an understanding of the subjects being tested. If it's an LLM, then it's...complicated. It might be because the training corpus provided more matching text. It might be because the training corpus provided text patterns that aligned better to the patterns in the SAT's text. The structure of phrases is just as important as the context they contain. It might be because the training corpus had fewer text patterns that result in "a wrong answer". Improving any of these means degrading the others. Logic is never involved. Symbolic reference, like defining words or "plugging numbers in" in to mathematical formula, is never involved. Doing well on one test does not mean doing well on a slightly rephrased version of that test.
sebzim4500 4y ago
This is the first time I have heard Yoshua Bengio described as "dumb". According to the Oxford English Dictionary, the definition of AI is: >the theory and development of computer systems able to perform tasks normally requiring human intelligence, such as visual perception, speech recognition, decision-making, and translation between languages. Clearly GPT-3 fits that. You are welcome to have your own definition, but you can't just claim that everyone else is wrong just because they are instead using the standard one. >2. Goals for LLM development were set for the personified attributes, and not the actual functionally of the real thing. OpenAI brags about how GPT4 scores at human tests: as if that has any bearing on the model itself, and not simply its content. Yeah, evaluating language models is extremely difficult. Hence OpenAI Evals, etc. >3. The success of an LLM at a personified goal is determined by magic. Does the black box output what I am hoping it will? How do I improve it? Weights? Better training corpus? Not sure what you mean by magic, but it is certainly hard to predict what they can and can't achieve. I suspect if you asked a panel of experts in 2020 when we would see something that can achieve what GPT-4 was almost all of them would have been wrong, but I am open to evidence to the contrary. >4. An LLM is understood to be a person in a black box, and we could not possibly understand what that person will do next. Danger! No serious person believes this.
- thomastjeffery 4y ago
  > This is the first time I have heard Yoshua Bengio described as "dumb". I was being cheeky: I meant it to mean "unable to speak". By feeding the personification narrative into itself, nothing is truly being said about what the model itself is and does: only the results of using it. Anyone who has based their understanding on that personified narrative is effectively "blind" to the true nature of the tech itself. I admit I was also being rude, and that was intentional. To spend so much effort studying a thing, and base all of your conclusions on its output...I think it's absurd to then call oneself an expert. Expert of what? A black box you never even opened? The LLM itself can only perform inference, but we aren't held back by that limitation! > the theory and development of computer systems able to perform tasks normally requiring human intelligence, such as visual perception, speech recognition, decision-making, and translation between languages. That is an excellent definition for the field of science that is AI. "An AI technology" "Working in AI" These statements use "AI" as a category of pursuit. It's not a great definition for an AI. Those words already define themselves. Artificial Intelligence: That means it was created intentionally, and it thinks for itself. No one needs to search a dictionary to know what "an AI" is. If "Intelligence" is an adjective, the definition you provided fits. If "Intelligence" is the noun, then it better be "an intelligence". > Yeah, evaluating language models is extremely difficult. Hence OpenAI Evals, etc. I wasn't trying to say that it's wrong to evaluate what an LLM does when prompted with the text from a test. I think that can provide valuable data and insights. The score we would give to a human taking the test is not valuable data. The behavior being measured does not align to any part of the LLM's process. We don't have to approach this blindly. We can have an explicit understanding to base research on. That doesn't mean making assumptions about the results, but it does mean better understanding the process that leads to them. >> 4. An LLM is understood to be a person in a black box, and we could not possibly understand what that person will do next. Danger! > No serious person believes this. Not literally, but narratively many do.
lanstin 4y ago
I feel more sympathy for that Google engineer that feel in love with their LLM. I am sure more people will become attached, at least once the rate limits are relaxed.
- int_19h 4y ago
  The rea trap is that LLMs can simulate empathy surprisingly well. If you have many problems to rant about but nobody willing to listen, the LLM is always there, it will never get bored or tell you to shut up, and it will always respond in encouraging and "positive" ways. Given how many people today do not have anyone like that in their lives, it's no wonder that they form an emotional bond.
  - lanstin 4y ago
    I said thanks to each answer :)
  - thomastjeffery 4y ago
    Empathy is what language is for. It's language itself that is doing the empathy: and that isn't simulated at all. It was all written by people. All an LLM can do is follow the structural patterns, and use them to shuffle.
    - int_19h 4y ago
      This doesn't make any sense. Empathy itself is qualia; the language is merely a medium to communicate it, and far from the only one (e.g. facial expressions are generally better at it). As for LLMs "following the structural patterns" of empathetic language - sure, that's exactly what simulating empathy is.
      - thomastjeffery 4y ago
        > the language is merely a medium to communicate it Precisely. What else is communication if not empathy accomplished via language? Failed communication? Language is not the only tool for communication, and communication is not the only tool for empathy. > As for LLMs "following the structural patterns" of empathetic language - sure, that's exactly what simulating empathy is. It's a matter of perspective: a character in a story follows the patterns of emphatic language. Do they empathize directly with the reader? An LLM cannot break the fourth wall. It can only present one that was already broken. An LLM doesn't look at the patterns in its model objectively. It only uses them in the process of modeling more patterns. An LLM itself cannot reason, it can only model the patterns of reason that humans have already encoded into language into text.

permo-w 4y ago

I'm not concerned about how they work or whether they're attaining personhood or not. I'm concerned at what people can and will use them for to the detriment of our civilisation

thomastjeffery 4y ago
How they work is the most critical piece of that puzzle. It isn't a mystery at all. LLMs implicitly model every pattern they can find in the text they are given. If the pattern aligns to language, humans categorize it as a feature. If it aligns to desired behavior, humans categorize it as a feature. If it aligns to undesired behavior, humans categorize it as a limitation. What humans categorize as features and limitations do not align with what the model is. A truth and a lie are structured with identical text patterns. The LLM can't categorize the two separately. The only risk here is that people continue to personify LLMs, and place trust in their relatively limited ability to fulfill a human role. It's the same category of risk as bad security practices. The potential to make a mistake itself is not a danger: making the mistake is.
- permo-w 4y ago
  you certainly make an interesting point about fact and falsehood and how they interpret language; however, I disagree that this is relevant and completely disagree that personification has any relevance to the most immediate dangers the danger here is that we make extremely powerful tools extremely available and people use those tools to hurt people, possibly a lot of people, possibly even by accident the scary part is that hacker news is full of the exact people that have the knowledge and capability to do this, and look at the reaction to this post: a resounding fuck off
  - thomastjeffery 4y ago
    I just explained explicitly how and why an LLM is not, and never will be, an "extremely powerful tool". A program that is actually a person in a black box would require the level of concern you are showing. Thankfully, that is not even remotely what we have here in reality.
    - permo-w 4y ago
      have you tried gpt-4? how can you possibly say it's not an extremely powerful tool? what happens when someone determined links it - or thousands of its - or thousands of next year's 3x as powerful its -- up to the internet with a code interpreter and a hard-coded poor intention?
      - tome 4y ago
        I don't know; what happens? I'd really like to know what people are imagining it could do, especially if they've got some evidence for it!
        permo-w 4y ago
        it's a rhetorical question. think about it yourself. try it yourself
      - qlm 4y ago
        What could it do that isn't already being done? I ask this sincerely. We already have bots constantly attacking every server they find 24/7. We already have botnets Is the concern that GPT will be able to find and exploit e.g. a novel RCE exploit in SSH?
cthalupa 4y ago
> I'm concerned at what people can and will use them for to the detriment of our civilisation Maybe? But there's already all sorts of untapped resources for that - and we mostly just don't choose to use them. A lot of the misinformation and "fake news" that came out during prior election cycles in the US came from former Soviet bloc countries, and there is already significant untapped human capital in those places where it could be done for cheap, perhaps even cheaper than the cost of doing inference on premiere LLMs. Meanwhile we see all sorts of efficiency improvements occurring for generally positive use cases from these LLMs, and it is hardly unique for technology to have some risks to it as well as positive benefits. Things like GPT and Midjourney have been a qualitative leap over their predecessors from two decades ago, but they fundamentally are not doing something new - and even another qualitative leap wouldn't bring them to the level of existential threat that this letter seems to be afraid of. I think there are significant and real concerns about AGI and similar, but I can't seem to find any reason to share these same concern around the types of "AI" we are seeing today.

adamsmith143 4y ago

At no point in your rambling, incoherent comment were you even close to anything that could be considered a rational thought. Everyone on this site is now dumber for having read it. May God have mercy on your soul.

thomastjeffery 4y ago
My point is that this fear is irrational: based on a misunderstanding about what LLMs can do, which itself is based on a misunderstanding of what LLMs are. Is there a way I could have been more clear? Are you disinterested in the content of my comment, or the presentation? The only thing I can learn from your comment is that mine didn't live up to your mysterious standards.
dmarchand90 4y ago
https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
dang 4y ago
Yikes! Please don't do this on HN. I'm not going to ban you because we haven't had to warn you before, and also because I sense an overtone of good humor in the scorching remarks, but we can't have people posting like this—it leads to flames. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.

marcosdumay 4y ago

> Goals for LLM development were set for the personified attributes, and not the actual functionally of the real thing. Well, this is for honest reasons. The goal of a chatbox is to beat the Turing test. It has always been. Those chatboxes didn't actually beat it, but it's clear that it's due to a technicality (they are easy to spot). They can do empty chats on the same level as a human. (And so it turns up that the test wasn't as useful as it was believed to be at first.)

thomastjeffery 4y ago
The problem is when you loop that logic around: it becomes circular reasoning. What is the true source of an improved SAT score? If it's a person we are talking about, then it's an understanding of the subjects being tested. If it's an LLM, then it's...complicated. It might be because the training corpus provided more matching text. It might be because the training corpus provided text patterns that aligned better to the patterns in the SAT's text. The structure of phrases is just as important as the context they contain. It might be because the training corpus had fewer text patterns that result in "a wrong answer". Improving any of these means degrading the others. Logic is never involved. Symbolic reference, like defining words or "plugging numbers in" in to mathematical formula, is never involved. Doing well on one test does not mean doing well on a slightly rephrased version of that test.

sebzim4500 4y ago

This is the first time I have heard Yoshua Bengio described as "dumb". According to the Oxford English Dictionary, the definition of AI is: >the theory and development of computer systems able to perform tasks normally requiring human intelligence, such as visual perception, speech recognition, decision-making, and translation between languages. Clearly GPT-3 fits that. You are welcome to have your own definition, but you can't just claim that everyone else is wrong just because they are instead using the standard one. >2. Goals for LLM development were set for the personified attributes, and not the actual functionally of the real thing. OpenAI brags about how GPT4 scores at human tests: as if that has any bearing on the model itself, and not simply its content. Yeah, evaluating language models is extremely difficult. Hence OpenAI Evals, etc. >3. The success of an LLM at a personified goal is determined by magic. Does the black box output what I am hoping it will? How do I improve it? Weights? Better training corpus? Not sure what you mean by magic, but it is certainly hard to predict what they can and can't achieve. I suspect if you asked a panel of experts in 2020 when we would see something that can achieve what GPT-4 was almost all of them would have been wrong, but I am open to evidence to the contrary. >4. An LLM is understood to be a person in a black box, and we could not possibly understand what that person will do next. Danger! No serious person believes this.

thomastjeffery 4y ago
> This is the first time I have heard Yoshua Bengio described as "dumb". I was being cheeky: I meant it to mean "unable to speak". By feeding the personification narrative into itself, nothing is truly being said about what the model itself is and does: only the results of using it. Anyone who has based their understanding on that personified narrative is effectively "blind" to the true nature of the tech itself. I admit I was also being rude, and that was intentional. To spend so much effort studying a thing, and base all of your conclusions on its output...I think it's absurd to then call oneself an expert. Expert of what? A black box you never even opened? The LLM itself can only perform inference, but we aren't held back by that limitation! > the theory and development of computer systems able to perform tasks normally requiring human intelligence, such as visual perception, speech recognition, decision-making, and translation between languages. That is an excellent definition for the field of science that is AI. "An AI technology" "Working in AI" These statements use "AI" as a category of pursuit. It's not a great definition for an AI. Those words already define themselves. Artificial Intelligence: That means it was created intentionally, and it thinks for itself. No one needs to search a dictionary to know what "an AI" is. If "Intelligence" is an adjective, the definition you provided fits. If "Intelligence" is the noun, then it better be "an intelligence". > Yeah, evaluating language models is extremely difficult. Hence OpenAI Evals, etc. I wasn't trying to say that it's wrong to evaluate what an LLM does when prompted with the text from a test. I think that can provide valuable data and insights. The score we would give to a human taking the test is not valuable data. The behavior being measured does not align to any part of the LLM's process. We don't have to approach this blindly. We can have an explicit understanding to base research on. That doesn't mean making assumptions about the results, but it does mean better understanding the process that leads to them. >> 4. An LLM is understood to be a person in a black box, and we could not possibly understand what that person will do next. Danger! > No serious person believes this. Not literally, but narratively many do.

lanstin 4y ago

I feel more sympathy for that Google engineer that feel in love with their LLM. I am sure more people will become attached, at least once the rate limits are relaxed.

int_19h 4y ago
The rea trap is that LLMs can simulate empathy surprisingly well. If you have many problems to rant about but nobody willing to listen, the LLM is always there, it will never get bored or tell you to shut up, and it will always respond in encouraging and "positive" ways. Given how many people today do not have anyone like that in their lives, it's no wonder that they form an emotional bond.
- lanstin 4y ago
  I said thanks to each answer :)
- thomastjeffery 4y ago
  Empathy is what language is for. It's language itself that is doing the empathy: and that isn't simulated at all. It was all written by people. All an LLM can do is follow the structural patterns, and use them to shuffle.
  - int_19h 4y ago
    This doesn't make any sense. Empathy itself is qualia; the language is merely a medium to communicate it, and far from the only one (e.g. facial expressions are generally better at it). As for LLMs "following the structural patterns" of empathetic language - sure, that's exactly what simulating empathy is.
    - thomastjeffery 4y ago
      > the language is merely a medium to communicate it Precisely. What else is communication if not empathy accomplished via language? Failed communication? Language is not the only tool for communication, and communication is not the only tool for empathy. > As for LLMs "following the structural patterns" of empathetic language - sure, that's exactly what simulating empathy is. It's a matter of perspective: a character in a story follows the patterns of emphatic language. Do they empathize directly with the reader? An LLM cannot break the fourth wall. It can only present one that was already broken. An LLM doesn't look at the patterns in its model objectively. It only uses them in the process of modeling more patterns. An LLM itself cannot reason, it can only model the patterns of reason that humans have already encoded into language into text.