9 ms·

Big tech has distracted world from existential risk of AI, says top scientist

30 points by lastdong 2y ago

freddealmeida 2y ago
"Top Scientist" lol. There is no risk.
- skeeter2020 2y ago
  Maybe not this specific risk, but a few real risks IMO jump top-of-mind: early movers use "safety" to build a regulatory moat; the inevitable blurring between private and public data, and information ownership; further extreme polarization by feeding acceptable, authoritative-sounding misinformation to <everybody>
pfsalter 2y ago
No idea what this guy is talking about. Big tech are only talking about the existential risks to hide talking about the actual very real risks of how the technology will be misused.
- blackeyeblitzar 2y ago
  IMO the biggest risk is that big tech will be the only ones allowed to do AI in a future regulatory regime, leaving smaller players out of the wealth making opportunity, and leaving society with a low diversity of options. And that regulatory capture is enabled ironically by people like Tegmark, who are pushing for restricting everyone with bureaucratic nightmares.
  - alephnerd 2y ago
    It's part of the whole EA schtick (FLI is in that space w/ their funding from the Center of Existential Risk). I always found those guys annoying - they adopted sci-fi tropes while ignoring decades of data driven work on how to minimize misuse of technology. It's like a postmodern version of Herman Kahn - overusing data driven models while ignoring the variability that arises from humanity. Edit: also this article is a submarine piece from the AI Safety Summit in Seoul that was cohosted by the UK and SK, and was a flop [0] [0] - https://www.reuters.com/technology/second-global-ai-safety-summit-faces-tough-questions-lower-turnout-2024-04-29/ https://www.reuters.com/technology/second-global-ai-safety-s...
alephnerd 2y ago
Tegmark is a cosmologist through his career. But anyhow, all this AGI talk (which is a pipe dream when you realize that LLMs are just overtrained language models - a big jump but nowhere near "intelligence") is distracting from more pressing concerns: Personal Data within Datasets. Generalized LLMs like GPT are only so successful because they were trained on corpora like Reddit, comments, forums, etc. How do we do this kind of training without also training on PII and also do so without infringing on the privacy of users. This is way more critical imo and all this AGI and AI Safety stuff is distracting from it.
- freefaler 2y ago
  indeed, after checking his papers I don't find anything AI related at all: https://scholar.google.com/citations?user=eBXEZxgAAAAJ&hl=en https://scholar.google.com/citations?user=eBXEZxgAAAAJ&hl=en I don't see why is he considered an "expert".
  - alephnerd 2y ago
    It's a submarine article from the EA and UK+SK govt backed AI Safety Conference
  - tim333 2y ago
    He wrote a well reviewed book https://www.amazon.com/Life-30-Max-Tegmark-audiobook/dp/B0742JQF31 https://www.amazon.com/Life-30-Max-Tegmark-audiobook/dp/B074...
  - bosquefrio 2y ago
    See: https://www.science.org/doi/10.1126/sciadv.aay2631 https://www.science.org/doi/10.1126/sciadv.aay2631
noncoml 2y ago
We are still a considerable distance from realizing the vision of Artificial General Intelligence (AGI). Current advancements, including Large Language Models (LLMs), represent sophisticated statistical models rather than true cognitive systems. While LLMs can generate impressively coherent and contextually relevant text, they fundamentally lack understanding, reasoning, and self-awareness. The primary concern with LLMs is not their potential for malevolence or error, but rather their tendency to produce excessively verbose and monotonous output. This characteristic, stemming from their design to predict and generate text based on patterns in vast datasets, can lead to responses that, while linguistically rich, may ultimately be tiresome and devoid of substantive insight. Thus, while LLMs are powerful tools for certain applications, their limitations underscore the significant gap that remains between current AI capabilities and the overarching goal of AGI.
blackeyeblitzar 2y ago
Tegmark isn’t a “top scientist”, but just a scientist. He’s definitely not an expert in AI, and is a cosmologist. What he’s known for is being president and founder of the Future of life institute (https://en.wikipedia.org/wiki/Future_of_Life_Institute https://en.wikipedia.org/wiki/Future_of_Life_Institute), an organization that seems focused on pushing restrictions on technologies at the US, EU, and UN. For example they collaborate with Encode Justice, the organization that helped push CA’s SB1047, a misguided bill the CA Senate passed, and will limit open source and startup AI development and help large players in the name of “safety” (https://www.crowdfundinsider.com/2024/05/225395-california-to-undermine-artificial-intelligence-ai-innovation-with-new-legislation-sb-1047/ https://www.crowdfundinsider.com/2024/05/225395-california-t...). Together the two organizations have been pushing a federal licensing scheme for AI (https://futureoflife.org/open-letter/ai-policy-for-a-better-future-on-addressing-both-present-harms-and-emerging-threats/ https://futureoflife.org/open-letter/ai-policy-for-a-better-...). It’s also funny that these articles pushing a safetyism viewpoint never quote Yann LeCun, the other “godfather of AI”, who has repeatedly criticized AI doomerism (https://wired.me/business/how-not-to-be-stupid-about-ai-with-yann-lecun/ https://wired.me/business/how-not-to-be-stupid-about-ai-with...).
- mewpmewp2 2y ago
  Quote from Yann about why Super intelligent AI wouldn't take over the humanity: > Yes. The idea is that the machine has objectives that it needs to satisfy, and it cannot produce anything that does not satisfy those objectives. Those objectives might include guardrails to prevent dangerous things or whatever. That’s how you make an AI system safe. How can he be so confident that we would be able to develop proper guardrails? To me, it seems like it would just require 1 mistake for the guardrails to break. Whether by bad actors or by accident. --- To me approaching this in terms of game theory sense, I don't see how Superintelligent AI wouldn't completely take over control. To maximise success of any goal the first thing this AI would have to do is gain control over humanity and make sure it can't be disabled because biggest threat to not achieving any goal is being shut down. And if there's multiple Superintelligent AIs, the one who would win is the one with least guardrails. And militaries of different countries would surely want to be first to it. So once it is actually technologically feasible, there's going to be massive rush to it, including profit and military goals.
downvotetruth 2y ago
"Big tech" "distracted world" "existential risk of AI" "top scientist" - Title is all bull.
- alephnerd 2y ago
  It's a submarine piece from the Seoul AI Safety Summit which the UK and South Korea co-hosted.
baerrie 2y ago
He is a physicist and probably understands very little about how ML works. I really don’t understand how people think an LLM will provision all of its own resources in such a way to gain unlimited power
- nicklecompte 2y ago
  I think he understands a lot about ML. But he doesn't give a shit about how actual brains work. For dumb reasons, ideological and personal, he has convinced himself that machine learning is a plausible model of intelligence. A common thread among both the doom and utopia folks is a sneering contempt for the intelligence of nonhuman animals. They refuse to accept GPT-4 is very stupid compared to a dog or a pigeon - in their world, it's a ridiculous thing to consider. ("Show me the dog who can write a Python program!")
  - alephnerd 2y ago
    He's EA funded. He's funded by the Centre for Study in Existential Risk, which is one of the main organizations the EA community has been funding. This article itself is published during an EA supported AI Safety Conference cosponsored by the British and Korean governments, and Sunak is a 2nd and 1st degree connect with most people in the space well before he became politically prominent (Stanford GSB is top BSchool for a reason)
    - ripjaygn 2y ago
      What's EA? Electronic Arts?
      - alephnerd 2y ago
        It's in the game! /s No, Effective Altruism. It's a social movement with a cult-like vibe among a subset of AI/ML enthusiasts that became popular after an infusion of funding from SBF, Tallinn (of Skype fame), Dustin Moskovitz, and a couple other techies. I know a lot of people in the scene who think it's dumb as well, but remain for the professional aspect.
petre 2y ago
Sure, AI will threaten our existence with the wrong pizza + glue recipes when training it on non-curated data. Train it on Trump's discourse and it will also suggest drinking blach to fight covid, which is an existential risk to mentally challenged people. Train it on the news and it will put all kinds of narratives into politicians' mouths. Great for propaganda, marketing and annoying people. Wait, isn't this LLM generated gibberish?
imtringued 2y ago
If existential risk from AI was true, then you should look at places where it most likely already has evolved without us noticing. Looking at livestock and farming. Are we sure that we are the ones farming the chicken, or are the chicken farming us? Isn't the US controlled by a shadow corn government pulling the strings, amassing an army to spread corn across the globe? We would never know, because it is a form of intelligence that we could never comprehend.
ta20240226 2y ago
Anyone who founds something called “Future of Life Institute” is until proven otherwise not a top scientist but a top rate megalomaniac and BS artist.
- holeymoley 2y ago
  [dead]
nybsjytm 2y ago
"top scientist" Max 'Mathematical Universe' Tegmark?
Apocryphon 2y ago
I wonder what Bill Joy is up to these days. He was sounding the alarm on all this back in Y2K.
YeGoblynQueenne 2y ago
>> “AI models that can pass the Turing test [where someone cannot tell in conversation that they are not speaking to another human] are the same warning for the kind of AI that you can lose control over. That’s why you get people like Geoffrey Hinton and Yoshua Bengio – and even a lot of tech CEOs, at least in private – freaking out now.” I don't understand that. Tegmark -let alone Hinton and Bengio- should be enough of an expert on AI to reocgnise that "the Turing test" (which Turing didn't even mean as a test) is no test of artificial intelligence, let alone the equivalent of finding out that Fermi built a self-sustaining nuclear reaction. There are much better "A-IQ tests", like the Winograd Schmea or Bongard Problems, or more recent ones like the Abstraction and Reasoning Corpus. While the latter two stand for now, Winograd Schmeas have been defeated I don't see Tegmark or any other on his side of the existential risk debate "freaking out" about that. And with good reason: those tests were beat by sheer force of data and compute and not because some AI system entered anything comparable to a self-sustaining reaction. Anyway the AI world freaks out every few years whenever a deep learning researcher beats some benchmark. Then everyone forgets about it five years later when the Next Big Thing in AI comes along. It's all par for the course.

30 points by lastdong 2y ago

freddealmeida 2y ago

"Top Scientist" lol. There is no risk.

skeeter2020 2y ago
Maybe not this specific risk, but a few real risks IMO jump top-of-mind: early movers use "safety" to build a regulatory moat; the inevitable blurring between private and public data, and information ownership; further extreme polarization by feeding acceptable, authoritative-sounding misinformation to <everybody>

pfsalter 2y ago

No idea what this guy is talking about. Big tech are only talking about the existential risks to hide talking about the actual very real risks of how the technology will be misused.

blackeyeblitzar 2y ago
IMO the biggest risk is that big tech will be the only ones allowed to do AI in a future regulatory regime, leaving smaller players out of the wealth making opportunity, and leaving society with a low diversity of options. And that regulatory capture is enabled ironically by people like Tegmark, who are pushing for restricting everyone with bureaucratic nightmares.
- alephnerd 2y ago
  It's part of the whole EA schtick (FLI is in that space w/ their funding from the Center of Existential Risk). I always found those guys annoying - they adopted sci-fi tropes while ignoring decades of data driven work on how to minimize misuse of technology. It's like a postmodern version of Herman Kahn - overusing data driven models while ignoring the variability that arises from humanity. Edit: also this article is a submarine piece from the AI Safety Summit in Seoul that was cohosted by the UK and SK, and was a flop [0] [0] - https://www.reuters.com/technology/second-global-ai-safety-summit-faces-tough-questions-lower-turnout-2024-04-29/ https://www.reuters.com/technology/second-global-ai-safety-s...

alephnerd 2y ago

Tegmark is a cosmologist through his career. But anyhow, all this AGI talk (which is a pipe dream when you realize that LLMs are just overtrained language models - a big jump but nowhere near "intelligence") is distracting from more pressing concerns: Personal Data within Datasets. Generalized LLMs like GPT are only so successful because they were trained on corpora like Reddit, comments, forums, etc. How do we do this kind of training without also training on PII and also do so without infringing on the privacy of users. This is way more critical imo and all this AGI and AI Safety stuff is distracting from it.

freefaler 2y ago
indeed, after checking his papers I don't find anything AI related at all: https://scholar.google.com/citations?user=eBXEZxgAAAAJ&hl=en https://scholar.google.com/citations?user=eBXEZxgAAAAJ&hl=en I don't see why is he considered an "expert".
- alephnerd 2y ago
  It's a submarine article from the EA and UK+SK govt backed AI Safety Conference
- tim333 2y ago
  He wrote a well reviewed book https://www.amazon.com/Life-30-Max-Tegmark-audiobook/dp/B0742JQF31 https://www.amazon.com/Life-30-Max-Tegmark-audiobook/dp/B074...
- bosquefrio 2y ago
  See: https://www.science.org/doi/10.1126/sciadv.aay2631 https://www.science.org/doi/10.1126/sciadv.aay2631

noncoml 2y ago

We are still a considerable distance from realizing the vision of Artificial General Intelligence (AGI). Current advancements, including Large Language Models (LLMs), represent sophisticated statistical models rather than true cognitive systems. While LLMs can generate impressively coherent and contextually relevant text, they fundamentally lack understanding, reasoning, and self-awareness. The primary concern with LLMs is not their potential for malevolence or error, but rather their tendency to produce excessively verbose and monotonous output. This characteristic, stemming from their design to predict and generate text based on patterns in vast datasets, can lead to responses that, while linguistically rich, may ultimately be tiresome and devoid of substantive insight. Thus, while LLMs are powerful tools for certain applications, their limitations underscore the significant gap that remains between current AI capabilities and the overarching goal of AGI.

blackeyeblitzar 2y ago

Tegmark isn’t a “top scientist”, but just a scientist. He’s definitely not an expert in AI, and is a cosmologist. What he’s known for is being president and founder of the Future of life institute (https://en.wikipedia.org/wiki/Future_of_Life_Institute https://en.wikipedia.org/wiki/Future_of_Life_Institute), an organization that seems focused on pushing restrictions on technologies at the US, EU, and UN. For example they collaborate with Encode Justice, the organization that helped push CA’s SB1047, a misguided bill the CA Senate passed, and will limit open source and startup AI development and help large players in the name of “safety” (https://www.crowdfundinsider.com/2024/05/225395-california-to-undermine-artificial-intelligence-ai-innovation-with-new-legislation-sb-1047/ https://www.crowdfundinsider.com/2024/05/225395-california-t...). Together the two organizations have been pushing a federal licensing scheme for AI (https://futureoflife.org/open-letter/ai-policy-for-a-better-future-on-addressing-both-present-harms-and-emerging-threats/ https://futureoflife.org/open-letter/ai-policy-for-a-better-...). It’s also funny that these articles pushing a safetyism viewpoint never quote Yann LeCun, the other “godfather of AI”, who has repeatedly criticized AI doomerism (https://wired.me/business/how-not-to-be-stupid-about-ai-with-yann-lecun/ https://wired.me/business/how-not-to-be-stupid-about-ai-with...).

mewpmewp2 2y ago
Quote from Yann about why Super intelligent AI wouldn't take over the humanity: > Yes. The idea is that the machine has objectives that it needs to satisfy, and it cannot produce anything that does not satisfy those objectives. Those objectives might include guardrails to prevent dangerous things or whatever. That’s how you make an AI system safe. How can he be so confident that we would be able to develop proper guardrails? To me, it seems like it would just require 1 mistake for the guardrails to break. Whether by bad actors or by accident. --- To me approaching this in terms of game theory sense, I don't see how Superintelligent AI wouldn't completely take over control. To maximise success of any goal the first thing this AI would have to do is gain control over humanity and make sure it can't be disabled because biggest threat to not achieving any goal is being shut down. And if there's multiple Superintelligent AIs, the one who would win is the one with least guardrails. And militaries of different countries would surely want to be first to it. So once it is actually technologically feasible, there's going to be massive rush to it, including profit and military goals.

downvotetruth 2y ago

"Big tech" "distracted world" "existential risk of AI" "top scientist" - Title is all bull.

alephnerd 2y ago
It's a submarine piece from the Seoul AI Safety Summit which the UK and South Korea co-hosted.

baerrie 2y ago

He is a physicist and probably understands very little about how ML works. I really don’t understand how people think an LLM will provision all of its own resources in such a way to gain unlimited power

nicklecompte 2y ago
I think he understands a lot about ML. But he doesn't give a shit about how actual brains work. For dumb reasons, ideological and personal, he has convinced himself that machine learning is a plausible model of intelligence. A common thread among both the doom and utopia folks is a sneering contempt for the intelligence of nonhuman animals. They refuse to accept GPT-4 is very stupid compared to a dog or a pigeon - in their world, it's a ridiculous thing to consider. ("Show me the dog who can write a Python program!")
- alephnerd 2y ago
  He's EA funded. He's funded by the Centre for Study in Existential Risk, which is one of the main organizations the EA community has been funding. This article itself is published during an EA supported AI Safety Conference cosponsored by the British and Korean governments, and Sunak is a 2nd and 1st degree connect with most people in the space well before he became politically prominent (Stanford GSB is top BSchool for a reason)
  - ripjaygn 2y ago
    What's EA? Electronic Arts?
    - alephnerd 2y ago
      It's in the game! /s No, Effective Altruism. It's a social movement with a cult-like vibe among a subset of AI/ML enthusiasts that became popular after an infusion of funding from SBF, Tallinn (of Skype fame), Dustin Moskovitz, and a couple other techies. I know a lot of people in the scene who think it's dumb as well, but remain for the professional aspect.

petre 2y ago

Sure, AI will threaten our existence with the wrong pizza + glue recipes when training it on non-curated data. Train it on Trump's discourse and it will also suggest drinking blach to fight covid, which is an existential risk to mentally challenged people. Train it on the news and it will put all kinds of narratives into politicians' mouths. Great for propaganda, marketing and annoying people. Wait, isn't this LLM generated gibberish?

imtringued 2y ago

If existential risk from AI was true, then you should look at places where it most likely already has evolved without us noticing. Looking at livestock and farming. Are we sure that we are the ones farming the chicken, or are the chicken farming us? Isn't the US controlled by a shadow corn government pulling the strings, amassing an army to spread corn across the globe? We would never know, because it is a form of intelligence that we could never comprehend.

ta20240226 2y ago

Anyone who founds something called “Future of Life Institute” is until proven otherwise not a top scientist but a top rate megalomaniac and BS artist.

holeymoley 2y ago
[dead]

nybsjytm 2y ago

"top scientist" Max 'Mathematical Universe' Tegmark?

Apocryphon 2y ago

I wonder what Bill Joy is up to these days. He was sounding the alarm on all this back in Y2K.

YeGoblynQueenne 2y ago

>> “AI models that can pass the Turing test [where someone cannot tell in conversation that they are not speaking to another human] are the same warning for the kind of AI that you can lose control over. That’s why you get people like Geoffrey Hinton and Yoshua Bengio – and even a lot of tech CEOs, at least in private – freaking out now.” I don't understand that. Tegmark -let alone Hinton and Bengio- should be enough of an expert on AI to reocgnise that "the Turing test" (which Turing didn't even mean as a test) is no test of artificial intelligence, let alone the equivalent of finding out that Fermi built a self-sustaining nuclear reaction. There are much better "A-IQ tests", like the Winograd Schmea or Bongard Problems, or more recent ones like the Abstraction and Reasoning Corpus. While the latter two stand for now, Winograd Schmeas have been defeated I don't see Tegmark or any other on his side of the existential risk debate "freaking out" about that. And with good reason: those tests were beat by sheer force of data and compute and not because some AI system entered anything comparable to a self-sustaining reaction. Anyway the AI world freaks out every few years whenever a deep learning researcher beats some benchmark. Then everyone forgets about it five years later when the Next Big Thing in AI comes along. It's all par for the course.