9 ms·

I'm eternally skeptical of the tech business, but I think you're jumping to conclusions, here. I'm on a first-name basis with several people near the top of thi

I'm eternally skeptical of the tech business, but I think you're jumping to conclusions, here. I'm on a first-name basis with several people near the top of this list. They are some of the smartest, savviest, most thoughtful, and most principled tech policy experts I've met. These folks default to skepticism of the tech business, champion open data, are deeply familiar with the risks of regulatory capture, and don't sign their name to any ol' open letter, especially if including their organizational affiliations. If this is a marketing ploy, that must have been a monster because even if they were walking around handing out checks for 25k I doubt they'd have gotten a good chunk of these folks.

fmap 3y ago
This particular statement really doesn't seem like a marketing ploy. It is difficult to disagree with the potential political and societal impacts of large language models as outlined here: https://www.safe.ai/ai-risk https://www.safe.ai/ai-risk These are, for the most part, obvious applications of a technology that exists right now but is not widely available yet. The problem with every discussion around this issue is that there are other statements on "the existential risk of AI" out there that are either marketing ploys or science fiction. It doesn't help that some of the proposed "solutions" are clear attempts at regulatory capture. This muddles the waters enough that it's difficult to have a productive discussion on how we could mitigate the real risk of, e.g., AI generated disinformation campaigns.
- chefandy 3y ago
  > The problem with every discussion around this issue is that there are other statements on Sure, but we're not talking about those other ones. Dismissing good faith initiatives as marketing ploys because there are bad faith initiatives is functionally no different than just shrugging and walking away. Of course OpenAI et. al. will try to influence the good faith discussions: that's a great reason to champion the ones with a bunch of good faith actors who stand a chance of holding the industry and policy makers to task. Waiting around for some group of experts that has enough clout to do something, but by policy excludes the industry itself and starry-eyed shithead "journalists" trying to ride the wave of the next big thing will yield nothing. This is a great example of perfect being the enemy of good.
  - fmap 3y ago
    I agree completely. I was just speculating on why there is so much discussion about marketing ploys in this comment section.
    - chefandy 3y ago
      Ah, sure. That makes sense. There's definitely a lot of marketing bullshit out there in the form of legit discussion. Unfortunately, this technology likely means there will be an incalculable increase in the amount of bullshit out there. Blerg.
- revelio 3y ago
  > It is difficult to disagree with the potential political and societal impacts of large language models as outlined here Is it? Unless you mean something mundane like "there will be impact", the list of risks they're proposing are subjective and debatable at best, irritatingly naive at worst. Their list of risks are: 1. Weaponization. Did we forget about Ukraine already? Answer: Weapons are needed. Why is this AI risk and not computer risk anyway? 2. Misinformation. Already a catastrophic problem just from journalists and academics. Most of the reporting on misinformation is itself misinformation. Look at the Durham report for an example, or anything that happened during COVID, or the long history of failed predictions that were presented to the public as certain. Answer: Not an AI risk, a human risk. 3. People might click on things that don't "improve their well being". Answer: how we choose to waste our free time on YouTube is not your concern, and you being in charge wouldn't improve our wellbeing anyway. 4. Technology might make us fat, like in WALL-E. Answer: it already happened, not having to break rocks with bigger rocks all day is nice, this is not an AI risk. 5. "Highly competent systems could give small groups of people a tremendous amount of power, leading to a lock-in of oppressive systems". Answer: already happens, just look at how much censorship big tech engages in these days. AI might make this more effective, but if that's their beef they should be campaigning against Google and Facebook. 6. Sudden emergent skills might take people by surprise. Answer: read the paper that shows the idea of emergent skills is AI researchers fooling themselves. 7. "It may be more efficient to gain human approval through deception than to earn human approval legitimately". No shit Sherlock, welcome to Earth. This is why labelling anyone who expresses skepticism about anything as a Denier™ is a bad idea! Answer: not an AI risk. If they want to promote critical thinking there are lots of ways to do that unrelated to AI. 8. Machines smarter than us might try to take over the world. Proof by Vladimir Putin is provided, except that it makes no sense because he's arguing that AI will be a tool that lets humans take over the world and this point is about the opposite. Answer: people with very high IQs have been around for a long time and as of yet have not proven able to take over the world or even especially interested in doing so. None of the risks they present is compelling to me personally, and I'm sure that's true of plenty of other people as well. Fix the human generated misinformation campaigns first, then worry about hypothetical non-existing AI generated campaigns.
  - cj 3y ago
- AlexandrB 3y ago
  As I mentioned in another comment, the listed risks are also notable because they largely omit economic risk. Something what will be especially acutely felt by those being laid off in favor of AI substitutes. I would argue that 30% unemployment is at least as much of a risk to the stability of society as AI generated misinformation. If one were particularly cynical, one could say that this is an attempt to frame AI risk in a manner that still allows AI companies to capture all the economic benefits of AI technology without consideration for those displaced by AI.
  - nuancebydefault 3y ago
    I believe the solution to said socio economic problem is rather simple. People are being replaced by robots and AI because the latter are cheaper. That's the market force. Cheaper means that more value us created. As a whole, people get more service for doing less work. The problem is that the money or value saved trickles up to the rich. The only solutions can be, regulations, - do not tax anymore based on income from doing actual work. - tax automated systems on their added value. - use the tax generated capital to provide for a basic income for everybody. In that way, the generated value goes to people who lost their jobs and to the working class as well.
- worik 3y ago
  > It is difficult to disagree with the potential political and societal impacts of large language models as outlined here: https://www.safe.ai/ai-risk https://www.safe.ai/ai-risk I disagree That list is a list of the dangers of power Many of these dangers: misinformation, killer robots, people on this list have been actively working on Rank hypocrisy And people projecting their own dark personalities onto a neutral technology Yes there are dangers in unbridled private power. They are not dangers unique to AI.
- lumb63 3y ago
  Aside from emergent behavior, are any of the items on that list unique to AI? They sure don’t seem it; they’re either broadly applicable to a number of already-available technologies, or to any entity in charge or providing advice or making decisions. I dare say even emergent behavior falls under this as well, since people can develop their own new motives that others don’t understand. Their advisory doesn’t seem to amount to much more than “bad people can do bad things”, except now “people” is “AI”.
fds98324jhk 3y ago
They don't want 25k they want jobs in the next presidential administration
- chefandy 3y ago
  > They don't want 25k they want jobs in the next presidential administration Academics shilling for OpenAI would get them jobs in the next presidential administration?
  - haldujai 3y ago
    Having their names on something so public is definitely an incentive for prestige and academic promotion. Shilling for OpenAI & co is also not a bad way to get funding support. I’m not accusing any non-affiliated academic listed of doing this but let’s not pretend there aren’t potentially perverse incentives influencing the decisions of academics, with respects to this specific letter and in general. To help dissuade (healthy) skepticism it would be nice to see disclosure statements for these academics, at first glance many appear to have conflicts.
    - chefandy 3y ago
      Could you be more specific about the conflicts you've uncovered?
      - haldujai 3y ago
        It’s unequivocal that academics may have conflicts (in general), that’s why disclosures are required for publications. I’m not uncovering anything, several of the academic signatories list affiliations with OpenAI, Google, Anthropic, Stability, MILA and Vector resulting in a financial conflict. Note that conflict does not mean shill, but in academia it should be disclosed. To allay some concerns a standard disclosure form would be helpful (i.e. do you receive funding support or have financial interest in a corporation pursuing AI commercialization).
AlexandrB 3y ago
You may be right, I don't know the people involved on a personal basis. Perhaps my problem is how much is left unsaid here (the broader safe.ai site doesn't help much). For example, what does "mitigate" mean? The most prominent recent proposal for mitigation comes from Sam Altman's congressional testimony, and it's very self serving. In such a vacuum of information, it's easy to be cynical.
- chefandy 3y ago
  Right. It probably needed to be general because there hasn't been enough time to work out sane specific responses, and even if they had, getting buy-in on specifics is a recipe for paralysis by indecision. A credible group of people simply pleading for policy makers, researchers, et. al. to take this seriously will lead to the project approvals, grant money, etc. that will hopefully yield a more sophisticated understanding of these issues. Cynicism is understandable in this ever-expanding whirlpool of bullshit, but when something looks like it has potential, we need to vigorously interrogate our cynicism if we're to stand a chance at fighting it.
  - AlexandrB 3y ago
    Reading the comments here is helping evolve my thinking on the issue for sure. Here's a comment I made in another thread: > As I mentioned in another comment, the listed risks are also notable because they largely omit economic risk. Something that will be especially acutely felt by those being laid off in favor of AI substitutes. I would argue that 30% unemployment is at least as much of a risk to the stability of society as AI generated misinformation. > If one were particularly cynical, one could say that this is an attempt to frame AI risk in a manner that still allows AI companies to capture all the economic benefits of AI technology without consideration for those displaced by AI. If policymaker's understanding of AI is predicated on hypothetical scenarios like "Weaponization" or "Power-Seeking Behavior" and not on concrete economic disruptions that AI will be causing very soon, the policy they come up with will be inadequate. Thus I'm frustrated with the framing of the issue that safe.ai is presenting because it is a distraction from the very real societal consequences of automating labor to the extent that will soon be possible.
nopinsight 3y ago
It’s worth noting also that many academics who signed the statement may face adverse issues like reputational risk as well as funding cut to their research programs if AI safety becomes an official policy. For a large number of them, these risks are worth far more than any possible gain from signing it. When a large number of smart, reputable people, including many with expert knowledge and little or negative incentives to act dishonestly, put their names down like this, one should pay attention. Added: Paul Christiano, a brilliant theoretical CS researcher who switched to AI Alignment several years ago, put the risks of “doom” for humanity at 46%. https://www.lesswrong.com/posts/xWMqsvHapP3nwdSW8/my-views-on-doom https://www.lesswrong.com/posts/xWMqsvHapP3nwdSW8/my-views-o...
- mehlmao 3y ago
  46% is such a precise number. This is why I can't take "rationalists", the Yudkowskys, and the Silvers seriously. Colossal assumptions turned into confidently stated probabilities.
- haldujai 3y ago
  Subtract OpenAI, Google, StabilityAI and Anthropic affiliated researchers (who have a lot to gain) and not many academic signatories are left. Notably missing representation from the Stanford NLP (edit: I missed that Diyi Yang is a signatory on first read) and NYU groups who’s perspective I’d also be interested in hearing. Not committing one way or another regarding the intent with this but it’s not as diverse an academic crowd as the long list may suggest and for a lot of these names there are incentives to act dishonestly (not claiming that they are).
  - nopinsight 3y ago
    Even if it’s just Yoshua Bengio, Geoffrey Hinton, and Stuart Russell, we’d probably agree the risks are not negligible. There are quite a few researchers from Stanford, UC Berkeley, MIT, Carnegie Mellon, Oxford, Cambirdge, Imperial College, Edinburg, Tsinghua, etc who signed as well. Many of whom do not work for those companies. We’re talking about nuclear war level risks here. Even a 1% chance should definitely be addressed. As noted above, Paul Christiano who has worked on AI risk and thought about it for a long time put it at 46%.
marricks 3y ago
They didn’t become such a wealthy group by letting competition foster. I have no doubt they believe they could be doing the right thing but I also have no doubt they don’t want other people making the rules. Truth be told, who else really does have a seat at the table for dictating such massive societal change? Do you think the copy editor union gets to sit down and say “I’d rather not have my lunch eaten, I need to pay my rent. Let’s pause AI usage in text for 10 years.” These competitors banded together and put out a statement to get ahead of any one else doing the same thing.
- FuckButtons 3y ago
  Not all of them are wealthy, a significant number are academics.
  - deleted 3y ago
    [deleted]
  - runarberg 3y ago
    That doesn’t erase the need of cynicism. Many people in academia come from industry, have friends in industry, or other stakes. They might have been persuaded by the rhetoric of stakeholders within industry (you saw this early in the climate debate; and still do), and they might also be hoping to get a job in the industry later on. There is also a fair amount of group think within academia, so if a prominent individual inside academia believes the lies of industry, chances are the majority within the department does.
    - FuckButtons 3y ago
      What’s more likely, that everyone on this list just happens to have some overlapping ulterior motive or that they actually believe it?
      - runarberg 3y ago
        Quite often these intersect. But also it is healthy to question the motives of signaturees asking for public policy. Especially when a significant number of them are direct stakeholders. When an academic signs such a list, their motives should be equally questioned. Cool username btw. Any relation to the band?
nico 3y ago
Maybe these people have good intentions and are just being naive They might not be getting paid, but that doesn’t mean they are not being influenced AI at this point is pretty much completely open, all the papers, math and science behind it are public Soon, people will have advanced AI running locally on their phones and watches So unless they scrub the Internet, start censoring this stuff, and pretty much ban computers, there is absolutely no way to stop AI nor any potentially bad actors from using it The biggest issues that we should be addressing regarding AI are the potential jobs losses and increased inequality at local and global scale But of course, the people who usually make these decisions are the ones that benefit the most from inequality, so
- paulddraper 3y ago
  You don't have to be a mustache-twirling villain to have the same effect.
- pydry 3y ago
  >Maybe these people have good intentions and are just being naive Ive noticed a lot of good people take awful political positions this way. Usually they trust the wrong person - e.g. by falling victim to the just world fallacy ("X is a big deal in our world and X wouldn't be where they are if they werent a decent person. X must have a point.")
- jstarfish 3y ago
  > Soon, people will have advanced AI running locally on their phones and watches Yeah, with drugs being legal now I've been wondering what the next boogeyman we're going to declare war on would be.
- nopinsight 3y ago
  You are attributing naivety on a technical matter to many people who have done world class research and some of them are even a pioneer of their subfield of study. To do that level of research, one needs strong independent thinking skills and deep technical expertise. We have plenty of evidence on this very site how hard it is to influence technical people and change their opinion.
nopinsight 3y ago
Here’s why AI risks are real, even if our most advanced AI is merely a ‘language’ model: Language can represent thoughts and some world models. There is strong evidence that LLMs contain some representation of world models it learned from text. Moreover, LLM is already a misnomer; latest versions are multimodal. Current versions can be used to build agents with limited autonomy. Future versions of LLMs are most likely capable of more independence. Even dumb viruses have caused catastrophic harm. Why? It’s capable of rapid self replication in a massive number of existing vessels. You add in some intelligence, vast store of knowledge, huge bandwidth, and some aid by malicious human actors, what could such a group of future autonomous agents do? More on risks of “doom” by a top researcher on AI risk here: https://www.lesswrong.com/posts/xWMqsvHapP3nwdSW8/my-views-on-doom https://www.lesswrong.com/posts/xWMqsvHapP3nwdSW8/my-views-o...
- skybrian 3y ago
  A lot of things are called "world models" that I would consider just "models" so it depends on what you mean by that. But what do you consider to be strong evidence? The Othello paper isn't what I'd call strong evidence.
  - gjm11 3y ago
    I agree that the Othello paper isn't, and couldn't be, strong evidence about what sort of model of the world (if any) something like GPT-4 has. However, I think it is (importantly) pretty much a refutation of all claims along the lines of "these systems learn only from text, therefore they cannot have anything in them that actually models anything other than text", since their model learned only from text and seems to have developed something very much like a model of the state of the game. Again, it doesn't say much about how good a model any given system might have. The world is much more complicated than an Othello board. GPT-4 is much bigger than their transformer model. Everything they found is consistent with anything from "as it happens GPT-4 has no world model at all" through to "GPT-4 has a rich model of the world, fully comparable to ours". (I would bet heavily on the truth being somewhere in between, not that that says very much.)
huevosabio 3y ago
I think it's the combination of two things. First, there are actual worries by a good chunk of the researchers. From runaway-paperclip AGIs to simply unbounded disinformation, I think there are a lot of scenarios that disinterested researchers and engineers worry about. Second, the captains of industry are taking note of those worries and making sure they get some regulatory moat. I think the Google memo about moat hits it right on the nail. The techniques and methods to build these systems are all out on the open, the challenges are really the data, compute, and the infrastructure to put it all together. But post training, the models are suddenly very easy to finetune and deploy. AI Risk worry comes as an opportunity for the leaders of these companies. They can use this sentiment and the general distrust for tech to build themselves a regulatory moat.
lannisterstark 3y ago
>They are some of the smartest, savviest, most thoughtful, and most principled tech policy experts I've met. with all due respect, that's just <Your> POV of them or how they chose to present themselves to you. They could all be narcissists for all we know. Further, One person's opinion, namely yours, doesn't exempt them from criticism and rushing to be among the first in what's arguably the new gold rush.
- robwwilliams 3y ago
  The only name in this list that gives me pause is Daniel Dennett. At least he has a philosopher’s historical perspective on the challenges ahead. Very few neuroscientists as signers. Perhaps they know better. For me the basic sociopolitical climate today (and for the last 120 years) is at an existential boil. This is not to say that birthing AGIs is not frightening, but just to say that many real features of today are just as, or even more frightening.
zzzeek 3y ago
> even if they were walking around handing out checks for 25k I doubt they'd have gotten a good chunk of these folks. if the corporations in question get world governments to line up the way they want, the checks for everyone in these "letters" will be *way* bigger than 25k and they won't have "payment for signing our letter" in the memo either.
AndrewKemendo 3y ago
Yeah so am I but they all have incentives personally. Arguing for regulation has zero downside risk especially if all your friends are doing it too. Professionally, ALL of their organizations benefit from regulatory capture as everyone is colluding via these letters. Go look at what HuggingFace are doing to show you how to to it - and they only can cause they are French and actually exercise their freedom
csomar 3y ago
Be aware that the things that AI challenges most is knowledge work. No food delivery boy job is being challenged here (who cares about these people anyway?) but if you are a software developer the clock on that is ticking.

fmap 3y ago

This particular statement really doesn't seem like a marketing ploy. It is difficult to disagree with the potential political and societal impacts of large language models as outlined here: https://www.safe.ai/ai-risk https://www.safe.ai/ai-risk These are, for the most part, obvious applications of a technology that exists right now but is not widely available yet. The problem with every discussion around this issue is that there are other statements on "the existential risk of AI" out there that are either marketing ploys or science fiction. It doesn't help that some of the proposed "solutions" are clear attempts at regulatory capture. This muddles the waters enough that it's difficult to have a productive discussion on how we could mitigate the real risk of, e.g., AI generated disinformation campaigns.

chefandy 3y ago
> The problem with every discussion around this issue is that there are other statements on Sure, but we're not talking about those other ones. Dismissing good faith initiatives as marketing ploys because there are bad faith initiatives is functionally no different than just shrugging and walking away. Of course OpenAI et. al. will try to influence the good faith discussions: that's a great reason to champion the ones with a bunch of good faith actors who stand a chance of holding the industry and policy makers to task. Waiting around for some group of experts that has enough clout to do something, but by policy excludes the industry itself and starry-eyed shithead "journalists" trying to ride the wave of the next big thing will yield nothing. This is a great example of perfect being the enemy of good.
- fmap 3y ago
  I agree completely. I was just speculating on why there is so much discussion about marketing ploys in this comment section.
  - chefandy 3y ago
    Ah, sure. That makes sense. There's definitely a lot of marketing bullshit out there in the form of legit discussion. Unfortunately, this technology likely means there will be an incalculable increase in the amount of bullshit out there. Blerg.
revelio 3y ago
> It is difficult to disagree with the potential political and societal impacts of large language models as outlined here Is it? Unless you mean something mundane like "there will be impact", the list of risks they're proposing are subjective and debatable at best, irritatingly naive at worst. Their list of risks are: 1. Weaponization. Did we forget about Ukraine already? Answer: Weapons are needed. Why is this AI risk and not computer risk anyway? 2. Misinformation. Already a catastrophic problem just from journalists and academics. Most of the reporting on misinformation is itself misinformation. Look at the Durham report for an example, or anything that happened during COVID, or the long history of failed predictions that were presented to the public as certain. Answer: Not an AI risk, a human risk. 3. People might click on things that don't "improve their well being". Answer: how we choose to waste our free time on YouTube is not your concern, and you being in charge wouldn't improve our wellbeing anyway. 4. Technology might make us fat, like in WALL-E. Answer: it already happened, not having to break rocks with bigger rocks all day is nice, this is not an AI risk. 5. "Highly competent systems could give small groups of people a tremendous amount of power, leading to a lock-in of oppressive systems". Answer: already happens, just look at how much censorship big tech engages in these days. AI might make this more effective, but if that's their beef they should be campaigning against Google and Facebook. 6. Sudden emergent skills might take people by surprise. Answer: read the paper that shows the idea of emergent skills is AI researchers fooling themselves. 7. "It may be more efficient to gain human approval through deception than to earn human approval legitimately". No shit Sherlock, welcome to Earth. This is why labelling anyone who expresses skepticism about anything as a Denier™ is a bad idea! Answer: not an AI risk. If they want to promote critical thinking there are lots of ways to do that unrelated to AI. 8. Machines smarter than us might try to take over the world. Proof by Vladimir Putin is provided, except that it makes no sense because he's arguing that AI will be a tool that lets humans take over the world and this point is about the opposite. Answer: people with very high IQs have been around for a long time and as of yet have not proven able to take over the world or even especially interested in doing so. None of the risks they present is compelling to me personally, and I'm sure that's true of plenty of other people as well. Fix the human generated misinformation campaigns first, then worry about hypothetical non-existing AI generated campaigns.
- cj 3y ago
AlexandrB 3y ago
As I mentioned in another comment, the listed risks are also notable because they largely omit economic risk. Something what will be especially acutely felt by those being laid off in favor of AI substitutes. I would argue that 30% unemployment is at least as much of a risk to the stability of society as AI generated misinformation. If one were particularly cynical, one could say that this is an attempt to frame AI risk in a manner that still allows AI companies to capture all the economic benefits of AI technology without consideration for those displaced by AI.
- nuancebydefault 3y ago
  I believe the solution to said socio economic problem is rather simple. People are being replaced by robots and AI because the latter are cheaper. That's the market force. Cheaper means that more value us created. As a whole, people get more service for doing less work. The problem is that the money or value saved trickles up to the rich. The only solutions can be, regulations, - do not tax anymore based on income from doing actual work. - tax automated systems on their added value. - use the tax generated capital to provide for a basic income for everybody. In that way, the generated value goes to people who lost their jobs and to the working class as well.
worik 3y ago
> It is difficult to disagree with the potential political and societal impacts of large language models as outlined here: https://www.safe.ai/ai-risk https://www.safe.ai/ai-risk I disagree That list is a list of the dangers of power Many of these dangers: misinformation, killer robots, people on this list have been actively working on Rank hypocrisy And people projecting their own dark personalities onto a neutral technology Yes there are dangers in unbridled private power. They are not dangers unique to AI.
lumb63 3y ago
Aside from emergent behavior, are any of the items on that list unique to AI? They sure don’t seem it; they’re either broadly applicable to a number of already-available technologies, or to any entity in charge or providing advice or making decisions. I dare say even emergent behavior falls under this as well, since people can develop their own new motives that others don’t understand. Their advisory doesn’t seem to amount to much more than “bad people can do bad things”, except now “people” is “AI”.

fds98324jhk 3y ago

They don't want 25k they want jobs in the next presidential administration

chefandy 3y ago
> They don't want 25k they want jobs in the next presidential administration Academics shilling for OpenAI would get them jobs in the next presidential administration?
- haldujai 3y ago
  Having their names on something so public is definitely an incentive for prestige and academic promotion. Shilling for OpenAI & co is also not a bad way to get funding support. I’m not accusing any non-affiliated academic listed of doing this but let’s not pretend there aren’t potentially perverse incentives influencing the decisions of academics, with respects to this specific letter and in general. To help dissuade (healthy) skepticism it would be nice to see disclosure statements for these academics, at first glance many appear to have conflicts.
  - chefandy 3y ago
    Could you be more specific about the conflicts you've uncovered?
    - haldujai 3y ago
      It’s unequivocal that academics may have conflicts (in general), that’s why disclosures are required for publications. I’m not uncovering anything, several of the academic signatories list affiliations with OpenAI, Google, Anthropic, Stability, MILA and Vector resulting in a financial conflict. Note that conflict does not mean shill, but in academia it should be disclosed. To allay some concerns a standard disclosure form would be helpful (i.e. do you receive funding support or have financial interest in a corporation pursuing AI commercialization).

AlexandrB 3y ago

You may be right, I don't know the people involved on a personal basis. Perhaps my problem is how much is left unsaid here (the broader safe.ai site doesn't help much). For example, what does "mitigate" mean? The most prominent recent proposal for mitigation comes from Sam Altman's congressional testimony, and it's very self serving. In such a vacuum of information, it's easy to be cynical.

chefandy 3y ago
Right. It probably needed to be general because there hasn't been enough time to work out sane specific responses, and even if they had, getting buy-in on specifics is a recipe for paralysis by indecision. A credible group of people simply pleading for policy makers, researchers, et. al. to take this seriously will lead to the project approvals, grant money, etc. that will hopefully yield a more sophisticated understanding of these issues. Cynicism is understandable in this ever-expanding whirlpool of bullshit, but when something looks like it has potential, we need to vigorously interrogate our cynicism if we're to stand a chance at fighting it.
- AlexandrB 3y ago
  Reading the comments here is helping evolve my thinking on the issue for sure. Here's a comment I made in another thread: > As I mentioned in another comment, the listed risks are also notable because they largely omit economic risk. Something that will be especially acutely felt by those being laid off in favor of AI substitutes. I would argue that 30% unemployment is at least as much of a risk to the stability of society as AI generated misinformation. > If one were particularly cynical, one could say that this is an attempt to frame AI risk in a manner that still allows AI companies to capture all the economic benefits of AI technology without consideration for those displaced by AI. If policymaker's understanding of AI is predicated on hypothetical scenarios like "Weaponization" or "Power-Seeking Behavior" and not on concrete economic disruptions that AI will be causing very soon, the policy they come up with will be inadequate. Thus I'm frustrated with the framing of the issue that safe.ai is presenting because it is a distraction from the very real societal consequences of automating labor to the extent that will soon be possible.

nopinsight 3y ago

It’s worth noting also that many academics who signed the statement may face adverse issues like reputational risk as well as funding cut to their research programs if AI safety becomes an official policy. For a large number of them, these risks are worth far more than any possible gain from signing it. When a large number of smart, reputable people, including many with expert knowledge and little or negative incentives to act dishonestly, put their names down like this, one should pay attention. Added: Paul Christiano, a brilliant theoretical CS researcher who switched to AI Alignment several years ago, put the risks of “doom” for humanity at 46%. https://www.lesswrong.com/posts/xWMqsvHapP3nwdSW8/my-views-on-doom https://www.lesswrong.com/posts/xWMqsvHapP3nwdSW8/my-views-o...

mehlmao 3y ago
46% is such a precise number. This is why I can't take "rationalists", the Yudkowskys, and the Silvers seriously. Colossal assumptions turned into confidently stated probabilities.
haldujai 3y ago
Subtract OpenAI, Google, StabilityAI and Anthropic affiliated researchers (who have a lot to gain) and not many academic signatories are left. Notably missing representation from the Stanford NLP (edit: I missed that Diyi Yang is a signatory on first read) and NYU groups who’s perspective I’d also be interested in hearing. Not committing one way or another regarding the intent with this but it’s not as diverse an academic crowd as the long list may suggest and for a lot of these names there are incentives to act dishonestly (not claiming that they are).
- nopinsight 3y ago
  Even if it’s just Yoshua Bengio, Geoffrey Hinton, and Stuart Russell, we’d probably agree the risks are not negligible. There are quite a few researchers from Stanford, UC Berkeley, MIT, Carnegie Mellon, Oxford, Cambirdge, Imperial College, Edinburg, Tsinghua, etc who signed as well. Many of whom do not work for those companies. We’re talking about nuclear war level risks here. Even a 1% chance should definitely be addressed. As noted above, Paul Christiano who has worked on AI risk and thought about it for a long time put it at 46%.

marricks 3y ago

They didn’t become such a wealthy group by letting competition foster. I have no doubt they believe they could be doing the right thing but I also have no doubt they don’t want other people making the rules. Truth be told, who else really does have a seat at the table for dictating such massive societal change? Do you think the copy editor union gets to sit down and say “I’d rather not have my lunch eaten, I need to pay my rent. Let’s pause AI usage in text for 10 years.” These competitors banded together and put out a statement to get ahead of any one else doing the same thing.

FuckButtons 3y ago
Not all of them are wealthy, a significant number are academics.
- deleted 3y ago
  [deleted]
- runarberg 3y ago
  That doesn’t erase the need of cynicism. Many people in academia come from industry, have friends in industry, or other stakes. They might have been persuaded by the rhetoric of stakeholders within industry (you saw this early in the climate debate; and still do), and they might also be hoping to get a job in the industry later on. There is also a fair amount of group think within academia, so if a prominent individual inside academia believes the lies of industry, chances are the majority within the department does.
  - FuckButtons 3y ago
    What’s more likely, that everyone on this list just happens to have some overlapping ulterior motive or that they actually believe it?
    - runarberg 3y ago
      Quite often these intersect. But also it is healthy to question the motives of signaturees asking for public policy. Especially when a significant number of them are direct stakeholders. When an academic signs such a list, their motives should be equally questioned. Cool username btw. Any relation to the band?

nico 3y ago

Maybe these people have good intentions and are just being naive They might not be getting paid, but that doesn’t mean they are not being influenced AI at this point is pretty much completely open, all the papers, math and science behind it are public Soon, people will have advanced AI running locally on their phones and watches So unless they scrub the Internet, start censoring this stuff, and pretty much ban computers, there is absolutely no way to stop AI nor any potentially bad actors from using it The biggest issues that we should be addressing regarding AI are the potential jobs losses and increased inequality at local and global scale But of course, the people who usually make these decisions are the ones that benefit the most from inequality, so

paulddraper 3y ago
You don't have to be a mustache-twirling villain to have the same effect.
pydry 3y ago
>Maybe these people have good intentions and are just being naive Ive noticed a lot of good people take awful political positions this way. Usually they trust the wrong person - e.g. by falling victim to the just world fallacy ("X is a big deal in our world and X wouldn't be where they are if they werent a decent person. X must have a point.")
jstarfish 3y ago
> Soon, people will have advanced AI running locally on their phones and watches Yeah, with drugs being legal now I've been wondering what the next boogeyman we're going to declare war on would be.
nopinsight 3y ago
You are attributing naivety on a technical matter to many people who have done world class research and some of them are even a pioneer of their subfield of study. To do that level of research, one needs strong independent thinking skills and deep technical expertise. We have plenty of evidence on this very site how hard it is to influence technical people and change their opinion.

nopinsight 3y ago

Here’s why AI risks are real, even if our most advanced AI is merely a ‘language’ model: Language can represent thoughts and some world models. There is strong evidence that LLMs contain some representation of world models it learned from text. Moreover, LLM is already a misnomer; latest versions are multimodal. Current versions can be used to build agents with limited autonomy. Future versions of LLMs are most likely capable of more independence. Even dumb viruses have caused catastrophic harm. Why? It’s capable of rapid self replication in a massive number of existing vessels. You add in some intelligence, vast store of knowledge, huge bandwidth, and some aid by malicious human actors, what could such a group of future autonomous agents do? More on risks of “doom” by a top researcher on AI risk here: https://www.lesswrong.com/posts/xWMqsvHapP3nwdSW8/my-views-on-doom https://www.lesswrong.com/posts/xWMqsvHapP3nwdSW8/my-views-o...

skybrian 3y ago
A lot of things are called "world models" that I would consider just "models" so it depends on what you mean by that. But what do you consider to be strong evidence? The Othello paper isn't what I'd call strong evidence.
- gjm11 3y ago
  I agree that the Othello paper isn't, and couldn't be, strong evidence about what sort of model of the world (if any) something like GPT-4 has. However, I think it is (importantly) pretty much a refutation of all claims along the lines of "these systems learn only from text, therefore they cannot have anything in them that actually models anything other than text", since their model learned only from text and seems to have developed something very much like a model of the state of the game. Again, it doesn't say much about how good a model any given system might have. The world is much more complicated than an Othello board. GPT-4 is much bigger than their transformer model. Everything they found is consistent with anything from "as it happens GPT-4 has no world model at all" through to "GPT-4 has a rich model of the world, fully comparable to ours". (I would bet heavily on the truth being somewhere in between, not that that says very much.)

huevosabio 3y ago

I think it's the combination of two things. First, there are actual worries by a good chunk of the researchers. From runaway-paperclip AGIs to simply unbounded disinformation, I think there are a lot of scenarios that disinterested researchers and engineers worry about. Second, the captains of industry are taking note of those worries and making sure they get some regulatory moat. I think the Google memo about moat hits it right on the nail. The techniques and methods to build these systems are all out on the open, the challenges are really the data, compute, and the infrastructure to put it all together. But post training, the models are suddenly very easy to finetune and deploy. AI Risk worry comes as an opportunity for the leaders of these companies. They can use this sentiment and the general distrust for tech to build themselves a regulatory moat.

lannisterstark 3y ago

>They are some of the smartest, savviest, most thoughtful, and most principled tech policy experts I've met. with all due respect, that's just <Your> POV of them or how they chose to present themselves to you. They could all be narcissists for all we know. Further, One person's opinion, namely yours, doesn't exempt them from criticism and rushing to be among the first in what's arguably the new gold rush.

robwwilliams 3y ago
The only name in this list that gives me pause is Daniel Dennett. At least he has a philosopher’s historical perspective on the challenges ahead. Very few neuroscientists as signers. Perhaps they know better. For me the basic sociopolitical climate today (and for the last 120 years) is at an existential boil. This is not to say that birthing AGIs is not frightening, but just to say that many real features of today are just as, or even more frightening.

zzzeek 3y ago

> even if they were walking around handing out checks for 25k I doubt they'd have gotten a good chunk of these folks. if the corporations in question get world governments to line up the way they want, the checks for everyone in these "letters" will be *way* bigger than 25k and they won't have "payment for signing our letter" in the memo either.

AndrewKemendo 3y ago

Yeah so am I but they all have incentives personally. Arguing for regulation has zero downside risk especially if all your friends are doing it too. Professionally, ALL of their organizations benefit from regulatory capture as everyone is colluding via these letters. Go look at what HuggingFace are doing to show you how to to it - and they only can cause they are French and actually exercise their freedom

csomar 3y ago

Be aware that the things that AI challenges most is knowledge work. No food delivery boy job is being challenged here (who cares about these people anyway?) but if you are a software developer the clock on that is ticking.