5 ms·

The problem with this article starts at the very begining. Where is says “The last few years have seen a wave of hysteria about LLMs becoming conscious and then

The problem with this article starts at the very begining. Where is says “The last few years have seen a wave of hysteria about LLMs becoming conscious and then suddenly attempting to kill humanity.” That is not a fair description of the fears of AGI. The idea is not that LLMs themselves become a threat. It is not like someone will ask Llama one too many times to talk like a pirate and it will climb out of the GPU and strangle humanity. It is more likely that an AGI will resemble some form of reinforcement learning agent applied by humans on the real world. There are multiple entities on Earth whose stated goal is to make an AGI. Deep mind, OpenAI, readily comes to mind. There could be others who keep secret about their projects for strategical reasons. (Militaries and secret services of the world.) They can use the success of LLMs to get more funding for their projects, but the AGI need not be otherwise a descendant of LLMs. This misunderstanding then goes through the whole article. Under the subtitle “Human writing is full of lies that are difficult to disprove theoretically” which only matter if you think that the AGI needs to learn from text. As opposed to conducting its own experiments, or gaining insight into raw sensor data from experiments.

lukan 2y ago
"The idea is not that LLMs themselves become a threat." Not my idea either, but I did hear and see that idea expressed many times already. Usually more dramatic, the less the persons knows about the actual tech, but even here I saw that point of view.
- ImHereToVote 2y ago
  Can you cite examples. I haven't seen any personally.
  - esperent 2y ago
    The paperclip maximizer, AKA instrumental convergence theory states not that an artificial intelligence would decide to kill humanity, but rather if it has sufficient power it might inadvertently destroy humanity by e.g. using up all resources for computational power. https://en.m.wikipedia.org/wiki/Instrumental_convergence https://en.m.wikipedia.org/wiki/Instrumental_convergence
    - ImHereToVote 2y ago
      What does that have to do with the current LLM's?
      - ben_w 2y ago
        I also don't think instrumental convergence is a risk from LLMs. But: using up all resources for computational power might well kill a lot of — not all — humans. Why? Imagine that some future version can produce human quality output at a human speed for an electrical power draw of 1 kW. At current prices this would cost about the same to run continuously as the UN abject poverty threshold, but it's also four times the current global electricity supply which means that electricity prices would have to go up until human and AI labour were equally priced. But I think that happens at a level where most people stop being able to afford electricity for things like "keeping food refrigerated", let alone "keep the AC or heat pumps running so it's possible to survive summer heatstroke or winter freezes". Devil's in the details though; if some future AI only gets that good around 2030 or so, renewables are likely to at that level all by themselves and exceed it shortly after, and then this particular conflict doesn't happen. Hopefully at that point, AI driven tractors and harvesters etc. get us our food, UBI etc., because that's only a good future if you don't need to work, because if you do still need to work, you're uncompetitive and out of luck.
    - immibis 2y ago
      We already have these - they're called corporations.
      - TeMPOraL 2y ago
        Now imagine one working 1000x faster and not being as dumb as a mind running on bureaucracy is.
  - lukan 2y ago
    I don't usually bookmark or otherwise save weird opinions, but will do so next time (most of the threads here, when ChatGPT became big and hyped contained many of them as far as I remember, maybe I take a nostalgic look)
- kristiandupont 2y ago
  I don't think so either, but I think that LLM's are closer to AGI than what most people seem to think. I think if you let an LLM, even with the models we have today, run in a loop and you get something that comes very close to consciousness. Consciousness, mind you, similar to that of a deaf, blind and otherwise sensory deprived person, which in itself is a bit abstract already. Give it senses and ways to interact with the world, and I would argue that what you have is a type of AGI. By that I do not mean "equivalent to humans", which I don't think is a very good definition of intelligence, but a different branch of evolution. One which I can easily see surpassing human intelligence in a near future.
  - sethammons 2y ago
    A feedback loop, sensors, and a goal; boom, AGI. So many people are stuck on AI hallucinations. How many times have you seen something that, for the briefest of instants, you thought was something else? "Orange circle on kitchen counter, brain says basketball, but, wait, more context, not a real circle, still orange, furry, oh, a cat; it is my orange cat" and all that takes place faster than you blink. You didn't hallucinate, your brain ran a feedback loop and based on past experiences it filled in details and as those details were validated they stayed (still looking at the kitchen counter, orange thing unexpected) and then some details deviated from brain expectations (the orange ball is not behaving like a ball) and so new context is immediately fed back in and understanding restored. For AGI you have to have a loop with feedback. An LLM is one leg of the stool, now add a way to realize when something it generates is failing to stand up to prior experience or existing need and be able to gather more context so it can test and validate its models. Really, that's how anything learns; how could it be anything else?
  - JSteph22 2y ago
    >Give it senses and ways to interact with the world Much easier said than done. You can take just one example, energy storage and usage, to illustrate how far off we are from giving AI real-world physical capabilities. Human biological-based energy management capabilities are many orders of magnitude more advanced than current battery technology. For any AGI to have seriously scary consequences, it needs to have a physical real-world presence that is at least comparable to humans. We're very far from that. Decades at least, if not centuries. The more realistic near-term threat is a non physical presence AGI being used by bad actor humans for malevolent ends.
- xanderlewis 2y ago
  Yes. The whole of Reddit seemed to be signed up to that idea for about six months of 2023.
- SebFender 2y ago
  Yes and can we please reinforce the following key idea "the less the person knows about the actual tech". Far from being an expert, but in tech for 3 decades, I wonder at most comments I hear about AI around restaurants, terraces, bars, family gatherings... the stuff I hear, oh boy. It's easy to become a follower when you've never seen the origin...
roenxi 2y ago
Furthermore, the underlying point of AGI risk is that at no point in the history of forever have we had entities that were more intelligent in all dimensions than a human to contend with. Anything that threatened us we could just push to the limits of their ability to plan then go a bit further to destroy them. And we have humans in every system we've ever built because something has to do the executive decision making and putting a human in was the most cost effective option - which means democratic institutions tend to have an easier time controlling everything. The threat here is we have no idea at all what happens economically when machines are stronger, faster and just generally better than people. In the long term we might plausibly find an economic equilibrium where humans are not worth feeding. We might find (imo probably will find) that a human-free military is more effective. There is a much greater level of disruption possible here than in the industrial revolution that led ultimately to WWII.
- 1992spacemovie 2y ago
  Human free military? Thanks but I’ve already had my potent weed for the day. Cmon have a filter for common sense.
  - pjc50 2y ago
    A field of antipersonnel mines is already a human-free military.
  - crngefest 2y ago
    You don’t need to get high to come up with this. Human free military is literally the wet dream of all generals and military higher ups. As we speak there are dozens of mil-tech startups and huge enterprises that work on exactly that
    - cronin101 2y ago
      The day we have a “human free military” is the day that every human on earth wakes up to find they just got enlisted without notice. Game theory only works when you have skin in the game. You think the other nations’ machines are just going to wreck your machines and then go home?
jfoster 2y ago
Articles like this one make me think we are more likely doomed than previously thought. If others who are skeptical about doom-like scenarios are making similar fundamental errors whilst acting as though they know for certain that the concern is unjustified, that just seems too consistent with the idea of arrogant humans wiping themselves out, unfortunately.
- mistermann 2y ago
  Someone with deep knowledge of both human nature and AI could do a lot of damage.
TeMPOraL 2y ago
> Under the subtitle “Human writing is full of lies that are difficult to disprove theoretically” which only matter if you think that the AGI needs to learn from text. As opposed to conducting its own experiments, or gaining insight into raw sensor data from experiments. Worse than that. The author has not been paying attention at all, because at this point LLMs clearly demonstrated that text encodes a lot of information about reality indirectly, and a model can learn those patterns. The amount learning LLMs already extract from text is maybe not as big as Eliezer would have you believe Bayes theorem allows for, but way further in that direction than the author believes. There are good arguments made against the AI X-risk fears. This article does not make any of them. Rather, it just reiterates the usual basic misconceptions, mixed with a hefty dose of contempt towards people the author thinks are wrong.
mike_hearn 2y ago
> if you think that the AGI needs to learn from text. As opposed to conducting its own experiments The 2010s saw two basic approaches to building AI. In one corner there was the Google Brain approach of building NNs that worked with language motivated by use cases like search and Google Translate. They trained models to predict the next words in sequences extracted from the internet. In the other corner there was reinforcement DL mostly championed by DeepMind, where they trained agents to achieve goals in simplified "real world" environments like video games, based on "sensor data". The former approach won completely. Who is still training RL agents to play games? Last I heard, DeepMind have been almost entirely re-allocated to implementing LLM integrations into Google products. It's been years since we heard boasts of beating some new video game benchmark, and the papers coming out on that topic now are mostly looking for ways to combine jointly trained vision/LLM models to reason through physical world scenarios. We've run this experiment for over a decade now. All the results suggest that useful artificial intelligence comes from training on the internet, not birthing some blank slate that rediscovers fire from first principles. So this really isn't an unreasonable perspective for the author to take. Nor is the first statement wrong either. Have we already forgotten the one-sentence open letter by hundreds of assorted experts which said "mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war". https://thebulletin.org/2023/12/policy-makers-should-plan-for-superintelligent-ai-even-if-it-never-happens/ https://thebulletin.org/2023/12/policy-makers-should-plan-fo... The only type of human level AI we currently have are LLMs and related models, yet they claimed that mitigating extinction risk should be a "global priority". This is some sort of reverse motte-and-bailey strategy in which existing tech is used to create interest, and then the the imaginary risks of a hypothetical technology is used to demand power.
grantcas 2y ago
[dead]

lukan 2y ago

"The idea is not that LLMs themselves become a threat." Not my idea either, but I did hear and see that idea expressed many times already. Usually more dramatic, the less the persons knows about the actual tech, but even here I saw that point of view.

ImHereToVote 2y ago
Can you cite examples. I haven't seen any personally.
- esperent 2y ago
  The paperclip maximizer, AKA instrumental convergence theory states not that an artificial intelligence would decide to kill humanity, but rather if it has sufficient power it might inadvertently destroy humanity by e.g. using up all resources for computational power. https://en.m.wikipedia.org/wiki/Instrumental_convergence https://en.m.wikipedia.org/wiki/Instrumental_convergence
  - ImHereToVote 2y ago
    What does that have to do with the current LLM's?
    - ben_w 2y ago
      I also don't think instrumental convergence is a risk from LLMs. But: using up all resources for computational power might well kill a lot of — not all — humans. Why? Imagine that some future version can produce human quality output at a human speed for an electrical power draw of 1 kW. At current prices this would cost about the same to run continuously as the UN abject poverty threshold, but it's also four times the current global electricity supply which means that electricity prices would have to go up until human and AI labour were equally priced. But I think that happens at a level where most people stop being able to afford electricity for things like "keeping food refrigerated", let alone "keep the AC or heat pumps running so it's possible to survive summer heatstroke or winter freezes". Devil's in the details though; if some future AI only gets that good around 2030 or so, renewables are likely to at that level all by themselves and exceed it shortly after, and then this particular conflict doesn't happen. Hopefully at that point, AI driven tractors and harvesters etc. get us our food, UBI etc., because that's only a good future if you don't need to work, because if you do still need to work, you're uncompetitive and out of luck.
  - immibis 2y ago
    We already have these - they're called corporations.
    - TeMPOraL 2y ago
      Now imagine one working 1000x faster and not being as dumb as a mind running on bureaucracy is.
- lukan 2y ago
  I don't usually bookmark or otherwise save weird opinions, but will do so next time (most of the threads here, when ChatGPT became big and hyped contained many of them as far as I remember, maybe I take a nostalgic look)
kristiandupont 2y ago
I don't think so either, but I think that LLM's are closer to AGI than what most people seem to think. I think if you let an LLM, even with the models we have today, run in a loop and you get something that comes very close to consciousness. Consciousness, mind you, similar to that of a deaf, blind and otherwise sensory deprived person, which in itself is a bit abstract already. Give it senses and ways to interact with the world, and I would argue that what you have is a type of AGI. By that I do not mean "equivalent to humans", which I don't think is a very good definition of intelligence, but a different branch of evolution. One which I can easily see surpassing human intelligence in a near future.
- sethammons 2y ago
  A feedback loop, sensors, and a goal; boom, AGI. So many people are stuck on AI hallucinations. How many times have you seen something that, for the briefest of instants, you thought was something else? "Orange circle on kitchen counter, brain says basketball, but, wait, more context, not a real circle, still orange, furry, oh, a cat; it is my orange cat" and all that takes place faster than you blink. You didn't hallucinate, your brain ran a feedback loop and based on past experiences it filled in details and as those details were validated they stayed (still looking at the kitchen counter, orange thing unexpected) and then some details deviated from brain expectations (the orange ball is not behaving like a ball) and so new context is immediately fed back in and understanding restored. For AGI you have to have a loop with feedback. An LLM is one leg of the stool, now add a way to realize when something it generates is failing to stand up to prior experience or existing need and be able to gather more context so it can test and validate its models. Really, that's how anything learns; how could it be anything else?
- JSteph22 2y ago
  >Give it senses and ways to interact with the world Much easier said than done. You can take just one example, energy storage and usage, to illustrate how far off we are from giving AI real-world physical capabilities. Human biological-based energy management capabilities are many orders of magnitude more advanced than current battery technology. For any AGI to have seriously scary consequences, it needs to have a physical real-world presence that is at least comparable to humans. We're very far from that. Decades at least, if not centuries. The more realistic near-term threat is a non physical presence AGI being used by bad actor humans for malevolent ends.
xanderlewis 2y ago
Yes. The whole of Reddit seemed to be signed up to that idea for about six months of 2023.
SebFender 2y ago
Yes and can we please reinforce the following key idea "the less the person knows about the actual tech". Far from being an expert, but in tech for 3 decades, I wonder at most comments I hear about AI around restaurants, terraces, bars, family gatherings... the stuff I hear, oh boy. It's easy to become a follower when you've never seen the origin...

roenxi 2y ago

Furthermore, the underlying point of AGI risk is that at no point in the history of forever have we had entities that were more intelligent in all dimensions than a human to contend with. Anything that threatened us we could just push to the limits of their ability to plan then go a bit further to destroy them. And we have humans in every system we've ever built because something has to do the executive decision making and putting a human in was the most cost effective option - which means democratic institutions tend to have an easier time controlling everything. The threat here is we have no idea at all what happens economically when machines are stronger, faster and just generally better than people. In the long term we might plausibly find an economic equilibrium where humans are not worth feeding. We might find (imo probably will find) that a human-free military is more effective. There is a much greater level of disruption possible here than in the industrial revolution that led ultimately to WWII.

1992spacemovie 2y ago
Human free military? Thanks but I’ve already had my potent weed for the day. Cmon have a filter for common sense.
- pjc50 2y ago
  A field of antipersonnel mines is already a human-free military.
- crngefest 2y ago
  You don’t need to get high to come up with this. Human free military is literally the wet dream of all generals and military higher ups. As we speak there are dozens of mil-tech startups and huge enterprises that work on exactly that
  - cronin101 2y ago
    The day we have a “human free military” is the day that every human on earth wakes up to find they just got enlisted without notice. Game theory only works when you have skin in the game. You think the other nations’ machines are just going to wreck your machines and then go home?

jfoster 2y ago

Articles like this one make me think we are more likely doomed than previously thought. If others who are skeptical about doom-like scenarios are making similar fundamental errors whilst acting as though they know for certain that the concern is unjustified, that just seems too consistent with the idea of arrogant humans wiping themselves out, unfortunately.

mistermann 2y ago
Someone with deep knowledge of both human nature and AI could do a lot of damage.

TeMPOraL 2y ago

> Under the subtitle “Human writing is full of lies that are difficult to disprove theoretically” which only matter if you think that the AGI needs to learn from text. As opposed to conducting its own experiments, or gaining insight into raw sensor data from experiments. Worse than that. The author has not been paying attention at all, because at this point LLMs clearly demonstrated that text encodes a lot of information about reality indirectly, and a model can learn those patterns. The amount learning LLMs already extract from text is maybe not as big as Eliezer would have you believe Bayes theorem allows for, but way further in that direction than the author believes. There are good arguments made against the AI X-risk fears. This article does not make any of them. Rather, it just reiterates the usual basic misconceptions, mixed with a hefty dose of contempt towards people the author thinks are wrong.

mike_hearn 2y ago

> if you think that the AGI needs to learn from text. As opposed to conducting its own experiments The 2010s saw two basic approaches to building AI. In one corner there was the Google Brain approach of building NNs that worked with language motivated by use cases like search and Google Translate. They trained models to predict the next words in sequences extracted from the internet. In the other corner there was reinforcement DL mostly championed by DeepMind, where they trained agents to achieve goals in simplified "real world" environments like video games, based on "sensor data". The former approach won completely. Who is still training RL agents to play games? Last I heard, DeepMind have been almost entirely re-allocated to implementing LLM integrations into Google products. It's been years since we heard boasts of beating some new video game benchmark, and the papers coming out on that topic now are mostly looking for ways to combine jointly trained vision/LLM models to reason through physical world scenarios. We've run this experiment for over a decade now. All the results suggest that useful artificial intelligence comes from training on the internet, not birthing some blank slate that rediscovers fire from first principles. So this really isn't an unreasonable perspective for the author to take. Nor is the first statement wrong either. Have we already forgotten the one-sentence open letter by hundreds of assorted experts which said "mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war". https://thebulletin.org/2023/12/policy-makers-should-plan-for-superintelligent-ai-even-if-it-never-happens/ https://thebulletin.org/2023/12/policy-makers-should-plan-fo... The only type of human level AI we currently have are LLMs and related models, yet they claimed that mitigating extinction risk should be a "global priority". This is some sort of reverse motte-and-bailey strategy in which existing tech is used to create interest, and then the the imaginary risks of a hypothetical technology is used to demand power.

grantcas 2y ago

[dead]