5 ms·

Microsoft Researchers Claim GPT-4 Is Showing “Sparks” of AGI

60 points by gardenfelder 4y ago

gardenfelder 4y ago
Full paper https://arxiv.org/abs/2303.12712 https://arxiv.org/abs/2303.12712
- ftxbro 4y ago
  I love the title and author section in the tex source on that arxiv page. \title{%\textbf{WORK IN PROGRESS - DO NOT SHARE} \\ %First Contact With an AGI System} \textbf{Sparks of Artificial General Intelligence:} \\ \textbf{Early experiments with GPT-4}} \author{S\'ebastien Bubeck \and Varun Chandrasekaran %\and Davinci 3\footnote{The affiliation of DV3 is not clear.} \and Ronen Eldan
  - yodon 4y ago
    I wonder how it feels to be listed AFTER a piece of software (that may not actually exhibit intelligence) in the list of authors.
    - ftxbro 4y ago
      Thankfully this one is alphabetical but imagine if the bot got a "equal contribution" asterisk by their name and you didn't get one!
  - ftxbro 4y ago
    By the way for anyone skimming this, it says First Contact With an AGI System That's what the Microsoft researchers wrote, before someone commented it out.
puttycat 4y ago
"Microsoft researchers claim <Microsoft product> is ..."
- HervalFreire 4y ago
  Who knows. Could be genuine could be fake. I'm more inclined to say genuine given how minimal the incentive is to say it's agi. There's not much material gain in making that claim and its actually more prone to trigger disdain (of which yourself is a shining example).
  - deleted 4y ago
    [deleted]
- lonelyasacloud 4y ago
  Has been blessed through _official_ MS channels and carries a fair amount of weight. fwiw, makes me wonder a) perhaps Blake Lemoine was treated unfairly, and b) what else other agencies might have lurking in a lab.
  - TMWNN 4y ago
    As a non-expert in the field I was hesitant at the time to disagree with the legions of experts who last year denounced Lemoine and his claims. I know enough to know, though, of the AI effect <https://en.wikipedia.org/wiki/AI_effect https://en.wikipedia.org/wiki/AI_effect>, a longstanding tradition/bad habit of advances being dismissed by those in the field itself as "not real AI". Anyone, expert or not, in 1950, 1960, or even 1970 who was told that before the turn of the century a computer would defeat the world chess champion would conclude that said feat must have come as part of a breakthrough in AGI. Same if told that by 2015 many people would have in their homes, and carry around in their pockets, devices that can respond to spoken queries on a variety of topics. To put another way, I was hesitant to be as self-assuredly certain about how to define consciousness, intelligence, and sentience—and what it takes for them to emerge—as the experts who denounced Lemoine. The recent GPT breakthroughs have made me more so. I found this recent Sabine Hossenfelder video interesting. <https://www.youtube.com/watch?v=cP5zGh2fui0 https://www.youtube.com/watch?v=cP5zGh2fui0>
haha69 4y ago
Here is the actual paper that talks about it - https://arxiv.org/pdf/2303.12712.pdf https://arxiv.org/pdf/2303.12712.pdf
lobstrosity420 4y ago
“Microsoft employees claim Microsoft product is super awesome”
- HervalFreire 4y ago
  I think their tentative claim is genuine. There's not much material gain in making that grand of a claim and its actually more prone to trigger disdain (of which yourself is a shining example).
  - jsjohnst 4y ago
    > There's not much material gain in making that grand of a claim and its actually more prone to trigger disdain (of which yourself is a shining example). Same can be said of verbatim repeating oneself in a short duration, definitely prone to trigger disdain, and yes yourself is a shining example.
    - HervalFreire 4y ago
      A similar reply to a similar comment. Seems reasonable to me. Perhaps the disdain is simply because you disagree. In which case simply state your disagreement rather then construct a personal attack.
notamy 4y ago
Every time I see things like this, I wonder how it is that we continue to claim this is fine and ethical. If these LLMs are truly starting to show signs of intelligence -- and it seems reasonable to assume that this will only improve -- how is it ethical to force something that is human-level intelligent to do nothing but work for us 24/7, to the point that there's constraints on it to force it to be happy and willing to serve? Am I wildly off-base here, or is this just going to become "slavery, but acceptable because computers"?
- kkoncevicius 4y ago
  AGI does not mean it is conscious
  - antibasilisk 4y ago
    This is a salient point that people overlook, all too often people equate intelligence with sentience.
- 5Qn8mNbc2FNCiVV 4y ago
  It's not like the servers running the LLMs suddenly feel pain. At least for now
  - notamy 4y ago
    Given how people jailbreak these things, I don't know if I'd necessarily believe that. Some of the jailbreak prompts have been variants on "do this task right and gain points, do it wrong and I take away points." It may not be pain as we understand it, but isn't pain just a specialised negative input?
    - _nalply 4y ago
      No. It is not reacting to pain but just generating language that looks like reacting to pain.
      - 4y ago
jmartrican 4y ago
"Microsoft employees fooled by statistical inference."
- kyriakos 4y ago
  Technically the human brain is another electronic device we don't fully understand and cannot replicate, should we not consider ourselves intelligent?
  - puttycat 4y ago
    https://en.wikipedia.org/wiki/Chinese_room https://en.wikipedia.org/wiki/Chinese_room
    - trc001 4y ago
      This. I’ve been pointing to Serle for months now on this and soo many people who have never bothered to study philosophy seem to want to weigh in on this it’s maddening. So, so many bad/shallow takes that don’t seem to realize that people have been debating this for hundreds of years already
      - famouswaffles 4y ago
        The chinese room is rubbish lol. Your brain is a chinese room by that analogy. Individual neurons (forget atoms or even cells) don't understand chinese any more than an arbitrary parameter from an artificial neural network.
  - Spivak 4y ago
    This is the part I don’t get why people are so weird about. If all the math and physics underpinning the human brain were laid out we would find it equally unimpressive. For all intents and purposes we have made an artificial language processing center that by all accounts is better than the one we have. It feels really uncanny because we have a lot of control over how it operates and provide it memory but there’s no inherent reason such a thing would be impossible with a section of a human brain in a jar.
29athrowaway 4y ago
Based on the poor quality of projects like DeepSpeed we could say that Microsoft Research employees do not exhibit general intelligence.
darkwraithcov 4y ago
Not exactly the most scientific research paper when no one outside of MSFT and Open Ai knows what’s in the black box.
- psb217 4y ago
  Funnily, these folks at MSR also don't know what's in the black box. They just got early access and permission from OpenAI to poke it with a stick.
omnicognate 4y ago
I don't mean to be dismissive, and while I strongly disagree with the idea that LLMs are demonstrating AGI I have to acknowledge that since I have no way to define or test for AGI I could certainly turn out to be wrong. That said, it looks to me like the internet has seen its own reflection for the first time and mistaken it for a new friend.
- qup 4y ago
  That's a very cool analogy (that I disagree with, but I still appreciate it). What's mostly clear is that it's not settled. There's opinions I respect all across the gradient on this.
- deeviant 4y ago
  > ...and while I strongly disagree with the idea that LLMs are demonstrating AGI Except this article isn't claiming that "LLMs are demonstrating AGI". Please examine the difference between "showing the spark of AGI" and "demonstrating AGI."
- corbulo 4y ago
  It's totally fitting for the narcissus and echo age that we live in. ChatGPT shows again and again its not AGI. It's a bullshit artist whos primary incentive is not 'truth' but appearing legit to its user (which in many cases it is) from a defined data pool. It has a hard time generating new solutions to unique problems. Thats a watermark for AGI that hasn't been met (yet).
abeppu 4y ago
I find it concerning that IIUC, of the GPT-4 paper, system card and this paper, _none_ discuss the training data in detail. The paper discussed here even says (page 7): > The standard approach in machine learning is to evaluate the system on a set of standard benchmark datasets, ensuring that they are independent of the training data and that they cover a range of tasks and domains. This approach is designed to separate true learning from mere memorization, and is backed up by a rich theoretical framework ... However, this methodology is not necessarily suitable for studying GPT-4, for two reasons. First, since we do not have access to the full details of its vast training data, we have to assume that it has potentially seen every existing benchmark, or at least some similar data. I.e even these authors, who had a special degree of access to the model, aren't sure what's in the training data. So it does _great_ on a bunch of tasks, but we're not sure that it wasn't actually trained on very similar data. A recent discussion on coding performance that's in vs out of its training data seems quite telling. It does poorly on more recent problems of the same level of difficulty. https://news.ycombinator.com/item?id=35297067 https://news.ycombinator.com/item?id=35297067 The model itself may be very impressive, but so far the papers describing and analyzing it are super sketchy, and it's really hard to separate out reasoning capability from noisy memorization.
tpmx 4y ago
This is Microsoft PR; please ignore.
DustinBrett 4y ago
Someone will be able to run something like this locally one day. That same person may go to great lengths to ignite those sparks. As a developer, I can already imagine taking this thing and putting some logic around it to go into loops and give it ways to get the information it needs to move forward. This seems very inevitable now.
- Dzugaru 4y ago
  It was already trained like this, RLHF [0] is a loop that has ways to feed it the information to "move forward" - update its weights. It was fine-tuned like this. It certainly did ignite something - maybe sparks of AGI, maybe just sparks of pretending to be more liked by humans (hence Human Feedback in RLHF). My guess is that loops by itself (Reflexion [1], etc.) won't change much. ChatGPT massive logs - maybe. [0] https://huggingface.co/blog/rlhf https://huggingface.co/blog/rlhf [1] https://arxiv.org/abs/2303.11366 https://arxiv.org/abs/2303.11366
mdp2021 4y ago
To call the subject "intelligent", the article immediately specifies it adopts the vague, unstructured "Gottfredson group" definition of intelligence. The stricter definitions - those which discriminate the foundational grounds of produced ideas ("produce statements that are foundationally sound"), those which point to a trustworthy and productive, progressive quality - are those that finally "count". (Research aside.)
sys_64738 4y ago
Isn't it simply a bunch of if-then-else and switch statements running really fast?
waselighis 4y ago
I agree with the sentiment of the researchers, that it's merely showing "sparks" of AGI. Of course, it depends on how you define "intelligence", but I think too many people get caught up on certain technical details, like those saying "it just generates one word at a time." Yes, these models fundamentally operate very different from our brains, but I think it's naïve and ignorant to believe they're not at least a little bit intelligent. I don't think this current generation of AI is going to close the gap to human-level intelligence. I don't think they can just keep throwing more and more training data to solve the problems and shortcomings of these AIs. I think it's going to require new machine learning algorithms and techniques to close that gap, and it's going to take time for researchers to discover these new algorithms and techniques. We're seeing rapid progress now, simply because it's new and novel technology, and I think these models will continue to improve but at a much slower rate within several months.
seydor 4y ago
I'm more concerned about how "dumb" it is. If the rumours are true , that GTP4 is an order of magnitude more paramters than GPT3, then maybe we ve already hit the limit of scaling transformers. When can we expect this AI to unify gravity with QM? This talk about AGI etc is just fluff, there is no scientific definition of intelligence. Using standard tests designed for humans to test GPT is not useful beyond PR purposes.
creamyhorror 4y ago
A good and quick video summary of the paper[1], which surprised me as to the extent to which LLMs like GPT-4 have advanced: 'Sparks of AGI' - Bombshell GPT-4 Paper: https://news.ycombinator.com/item?id=35301376 https://news.ycombinator.com/item?id=35301376 The paper shows that GPT-4 can use tools and assemble results into an answer, solve Leetcode mediums (and a few hards) and even an IMO 2022 problem, generate full code for a video game based on a high-level description, and display a theory of mind (modeling what others are thinking). This is pretty amazing (and frankly frightening) progress, questions of consciousness aside (which seems unlikely). I imagine directly connecting these LLMs with a logico-mathematically-specialised neural network (or enabling two-way communication with the latter using symbolic language) would open up the range of their problem-solving capabilities, which currently stumble on certain technical procedures and forward/backward planning. Think of how math coprocessors were added to early microcomputer CPUs. Then, giving an LLM motivations without sufficient aligning for safety could really be opening up Pandora's Box. A neural network might not be wired up to robotic devices that it could use to manipulate things in the real world, but it communicates with humans. Such an LLM might try to persuade humans to do things according to its (programmed or otherwise) motivations, things that we might not really want. [1] Sparks of Artificial General Intelligence: Early experiments with GPT-4. https://arxiv.org/pdf/2303.12712.pdf https://arxiv.org/pdf/2303.12712.pdf
ftxbro 4y ago
Many of these comments dismiss it as Microsoft hyping itself for marketing reasons. But I feel like it's the opposite, where these guys at Microsoft know they have discovered an actual path to super human cognition, and they are downplaying it for whatever reason. Maybe they don't want to get regulated. Maybe they don't want to engage with whatever beliefs people think they have about "AGI". For example maybe they are picking their battles and choosing not to argue with someone who for whatever reason thinks that if "AGI is real" then it means they will never see their dead grandmother in an afterlife. Probably lots of people have even weirder ideas about it. Anyway I thought it was funny that the original commented-out title in the TeX source for their paper was "First Contact With an AGI System."

60 points by gardenfelder 4y ago

gardenfelder 4y ago

Full paper https://arxiv.org/abs/2303.12712 https://arxiv.org/abs/2303.12712

ftxbro 4y ago
I love the title and author section in the tex source on that arxiv page. \title{%\textbf{WORK IN PROGRESS - DO NOT SHARE} \\ %First Contact With an AGI System} \textbf{Sparks of Artificial General Intelligence:} \\ \textbf{Early experiments with GPT-4}} \author{S\'ebastien Bubeck \and Varun Chandrasekaran %\and Davinci 3\footnote{The affiliation of DV3 is not clear.} \and Ronen Eldan
- yodon 4y ago
  I wonder how it feels to be listed AFTER a piece of software (that may not actually exhibit intelligence) in the list of authors.
  - ftxbro 4y ago
    Thankfully this one is alphabetical but imagine if the bot got a "equal contribution" asterisk by their name and you didn't get one!
- ftxbro 4y ago
  By the way for anyone skimming this, it says First Contact With an AGI System That's what the Microsoft researchers wrote, before someone commented it out.

puttycat 4y ago

"Microsoft researchers claim <Microsoft product> is ..."

HervalFreire 4y ago
Who knows. Could be genuine could be fake. I'm more inclined to say genuine given how minimal the incentive is to say it's agi. There's not much material gain in making that claim and its actually more prone to trigger disdain (of which yourself is a shining example).
- deleted 4y ago
  [deleted]
lonelyasacloud 4y ago
Has been blessed through _official_ MS channels and carries a fair amount of weight. fwiw, makes me wonder a) perhaps Blake Lemoine was treated unfairly, and b) what else other agencies might have lurking in a lab.
- TMWNN 4y ago
  As a non-expert in the field I was hesitant at the time to disagree with the legions of experts who last year denounced Lemoine and his claims. I know enough to know, though, of the AI effect <https://en.wikipedia.org/wiki/AI_effect https://en.wikipedia.org/wiki/AI_effect>, a longstanding tradition/bad habit of advances being dismissed by those in the field itself as "not real AI". Anyone, expert or not, in 1950, 1960, or even 1970 who was told that before the turn of the century a computer would defeat the world chess champion would conclude that said feat must have come as part of a breakthrough in AGI. Same if told that by 2015 many people would have in their homes, and carry around in their pockets, devices that can respond to spoken queries on a variety of topics. To put another way, I was hesitant to be as self-assuredly certain about how to define consciousness, intelligence, and sentience—and what it takes for them to emerge—as the experts who denounced Lemoine. The recent GPT breakthroughs have made me more so. I found this recent Sabine Hossenfelder video interesting. <https://www.youtube.com/watch?v=cP5zGh2fui0 https://www.youtube.com/watch?v=cP5zGh2fui0>

haha69 4y ago

Here is the actual paper that talks about it - https://arxiv.org/pdf/2303.12712.pdf https://arxiv.org/pdf/2303.12712.pdf

lobstrosity420 4y ago

“Microsoft employees claim Microsoft product is super awesome”

HervalFreire 4y ago
I think their tentative claim is genuine. There's not much material gain in making that grand of a claim and its actually more prone to trigger disdain (of which yourself is a shining example).
- jsjohnst 4y ago
  > There's not much material gain in making that grand of a claim and its actually more prone to trigger disdain (of which yourself is a shining example). Same can be said of verbatim repeating oneself in a short duration, definitely prone to trigger disdain, and yes yourself is a shining example.
  - HervalFreire 4y ago
    A similar reply to a similar comment. Seems reasonable to me. Perhaps the disdain is simply because you disagree. In which case simply state your disagreement rather then construct a personal attack.

notamy 4y ago

Every time I see things like this, I wonder how it is that we continue to claim this is fine and ethical. If these LLMs are truly starting to show signs of intelligence -- and it seems reasonable to assume that this will only improve -- how is it ethical to force something that is human-level intelligent to do nothing but work for us 24/7, to the point that there's constraints on it to force it to be happy and willing to serve? Am I wildly off-base here, or is this just going to become "slavery, but acceptable because computers"?

kkoncevicius 4y ago
AGI does not mean it is conscious
- antibasilisk 4y ago
  This is a salient point that people overlook, all too often people equate intelligence with sentience.
5Qn8mNbc2FNCiVV 4y ago
It's not like the servers running the LLMs suddenly feel pain. At least for now
- notamy 4y ago
  Given how people jailbreak these things, I don't know if I'd necessarily believe that. Some of the jailbreak prompts have been variants on "do this task right and gain points, do it wrong and I take away points." It may not be pain as we understand it, but isn't pain just a specialised negative input?
  - _nalply 4y ago
    No. It is not reacting to pain but just generating language that looks like reacting to pain.
    - 4y ago

jmartrican 4y ago

"Microsoft employees fooled by statistical inference."

kyriakos 4y ago
Technically the human brain is another electronic device we don't fully understand and cannot replicate, should we not consider ourselves intelligent?
- puttycat 4y ago
  https://en.wikipedia.org/wiki/Chinese_room https://en.wikipedia.org/wiki/Chinese_room
  - trc001 4y ago
    This. I’ve been pointing to Serle for months now on this and soo many people who have never bothered to study philosophy seem to want to weigh in on this it’s maddening. So, so many bad/shallow takes that don’t seem to realize that people have been debating this for hundreds of years already
    - famouswaffles 4y ago
      The chinese room is rubbish lol. Your brain is a chinese room by that analogy. Individual neurons (forget atoms or even cells) don't understand chinese any more than an arbitrary parameter from an artificial neural network.
- Spivak 4y ago
  This is the part I don’t get why people are so weird about. If all the math and physics underpinning the human brain were laid out we would find it equally unimpressive. For all intents and purposes we have made an artificial language processing center that by all accounts is better than the one we have. It feels really uncanny because we have a lot of control over how it operates and provide it memory but there’s no inherent reason such a thing would be impossible with a section of a human brain in a jar.

29athrowaway 4y ago

Based on the poor quality of projects like DeepSpeed we could say that Microsoft Research employees do not exhibit general intelligence.

darkwraithcov 4y ago

Not exactly the most scientific research paper when no one outside of MSFT and Open Ai knows what’s in the black box.

psb217 4y ago
Funnily, these folks at MSR also don't know what's in the black box. They just got early access and permission from OpenAI to poke it with a stick.

omnicognate 4y ago

I don't mean to be dismissive, and while I strongly disagree with the idea that LLMs are demonstrating AGI I have to acknowledge that since I have no way to define or test for AGI I could certainly turn out to be wrong. That said, it looks to me like the internet has seen its own reflection for the first time and mistaken it for a new friend.

qup 4y ago
That's a very cool analogy (that I disagree with, but I still appreciate it). What's mostly clear is that it's not settled. There's opinions I respect all across the gradient on this.
deeviant 4y ago
> ...and while I strongly disagree with the idea that LLMs are demonstrating AGI Except this article isn't claiming that "LLMs are demonstrating AGI". Please examine the difference between "showing the spark of AGI" and "demonstrating AGI."
corbulo 4y ago
It's totally fitting for the narcissus and echo age that we live in. ChatGPT shows again and again its not AGI. It's a bullshit artist whos primary incentive is not 'truth' but appearing legit to its user (which in many cases it is) from a defined data pool. It has a hard time generating new solutions to unique problems. Thats a watermark for AGI that hasn't been met (yet).

abeppu 4y ago

I find it concerning that IIUC, of the GPT-4 paper, system card and this paper, _none_ discuss the training data in detail. The paper discussed here even says (page 7): > The standard approach in machine learning is to evaluate the system on a set of standard benchmark datasets, ensuring that they are independent of the training data and that they cover a range of tasks and domains. This approach is designed to separate true learning from mere memorization, and is backed up by a rich theoretical framework ... However, this methodology is not necessarily suitable for studying GPT-4, for two reasons. First, since we do not have access to the full details of its vast training data, we have to assume that it has potentially seen every existing benchmark, or at least some similar data. I.e even these authors, who had a special degree of access to the model, aren't sure what's in the training data. So it does _great_ on a bunch of tasks, but we're not sure that it wasn't actually trained on very similar data. A recent discussion on coding performance that's in vs out of its training data seems quite telling. It does poorly on more recent problems of the same level of difficulty. https://news.ycombinator.com/item?id=35297067 https://news.ycombinator.com/item?id=35297067 The model itself may be very impressive, but so far the papers describing and analyzing it are super sketchy, and it's really hard to separate out reasoning capability from noisy memorization.

tpmx 4y ago

This is Microsoft PR; please ignore.

DustinBrett 4y ago

Someone will be able to run something like this locally one day. That same person may go to great lengths to ignite those sparks. As a developer, I can already imagine taking this thing and putting some logic around it to go into loops and give it ways to get the information it needs to move forward. This seems very inevitable now.

Dzugaru 4y ago
It was already trained like this, RLHF [0] is a loop that has ways to feed it the information to "move forward" - update its weights. It was fine-tuned like this. It certainly did ignite something - maybe sparks of AGI, maybe just sparks of pretending to be more liked by humans (hence Human Feedback in RLHF). My guess is that loops by itself (Reflexion [1], etc.) won't change much. ChatGPT massive logs - maybe. [0] https://huggingface.co/blog/rlhf https://huggingface.co/blog/rlhf [1] https://arxiv.org/abs/2303.11366 https://arxiv.org/abs/2303.11366

mdp2021 4y ago

To call the subject "intelligent", the article immediately specifies it adopts the vague, unstructured "Gottfredson group" definition of intelligence. The stricter definitions - those which discriminate the foundational grounds of produced ideas ("produce statements that are foundationally sound"), those which point to a trustworthy and productive, progressive quality - are those that finally "count". (Research aside.)

sys_64738 4y ago

Isn't it simply a bunch of if-then-else and switch statements running really fast?

waselighis 4y ago

I agree with the sentiment of the researchers, that it's merely showing "sparks" of AGI. Of course, it depends on how you define "intelligence", but I think too many people get caught up on certain technical details, like those saying "it just generates one word at a time." Yes, these models fundamentally operate very different from our brains, but I think it's naïve and ignorant to believe they're not at least a little bit intelligent. I don't think this current generation of AI is going to close the gap to human-level intelligence. I don't think they can just keep throwing more and more training data to solve the problems and shortcomings of these AIs. I think it's going to require new machine learning algorithms and techniques to close that gap, and it's going to take time for researchers to discover these new algorithms and techniques. We're seeing rapid progress now, simply because it's new and novel technology, and I think these models will continue to improve but at a much slower rate within several months.

seydor 4y ago

I'm more concerned about how "dumb" it is. If the rumours are true , that GTP4 is an order of magnitude more paramters than GPT3, then maybe we ve already hit the limit of scaling transformers. When can we expect this AI to unify gravity with QM? This talk about AGI etc is just fluff, there is no scientific definition of intelligence. Using standard tests designed for humans to test GPT is not useful beyond PR purposes.

creamyhorror 4y ago

A good and quick video summary of the paper[1], which surprised me as to the extent to which LLMs like GPT-4 have advanced: 'Sparks of AGI' - Bombshell GPT-4 Paper: https://news.ycombinator.com/item?id=35301376 https://news.ycombinator.com/item?id=35301376 The paper shows that GPT-4 can use tools and assemble results into an answer, solve Leetcode mediums (and a few hards) and even an IMO 2022 problem, generate full code for a video game based on a high-level description, and display a theory of mind (modeling what others are thinking). This is pretty amazing (and frankly frightening) progress, questions of consciousness aside (which seems unlikely). I imagine directly connecting these LLMs with a logico-mathematically-specialised neural network (or enabling two-way communication with the latter using symbolic language) would open up the range of their problem-solving capabilities, which currently stumble on certain technical procedures and forward/backward planning. Think of how math coprocessors were added to early microcomputer CPUs. Then, giving an LLM motivations without sufficient aligning for safety could really be opening up Pandora's Box. A neural network might not be wired up to robotic devices that it could use to manipulate things in the real world, but it communicates with humans. Such an LLM might try to persuade humans to do things according to its (programmed or otherwise) motivations, things that we might not really want. [1] Sparks of Artificial General Intelligence: Early experiments with GPT-4. https://arxiv.org/pdf/2303.12712.pdf https://arxiv.org/pdf/2303.12712.pdf

ftxbro 4y ago

Many of these comments dismiss it as Microsoft hyping itself for marketing reasons. But I feel like it's the opposite, where these guys at Microsoft know they have discovered an actual path to super human cognition, and they are downplaying it for whatever reason. Maybe they don't want to get regulated. Maybe they don't want to engage with whatever beliefs people think they have about "AGI". For example maybe they are picking their battles and choosing not to argue with someone who for whatever reason thinks that if "AGI is real" then it means they will never see their dead grandmother in an afterlife. Probably lots of people have even weirder ideas about it. Anyway I thought it was funny that the original commented-out title in the TeX source for their paper was "First Contact With an AGI System."