18 ms·

This is something weird happening around Rationalism/X-Risk/AGI prognostications. The "Great Minds And Great Leaders" types are rushing to warn about the risks

by ke88y 3y ago

This is something weird happening around Rationalism/X-Risk/AGI prognostications. The "Great Minds And Great Leaders" types are rushing to warn about the risks, as are a large number of people who spend a lot of time philosophizing. But the actual scientists on the ground -- the PhDs and engineers I work with every day and who have been in this field, at the bench, doing to work on the latest generation of generative models, and previous generations, in some cases for decades? They almost all roll their eyes aggressively at these sorts of prognostications. I'd say 90+% either laugh or roll their eyes. Why is that? Personally, I'm much more on the side of the silent majority here. I agree with Altman's criticisms of criticisms about regulatory capture, that they are probably unfair or at least inaccurate. What I actually think is going on here is something more about Egos than Greatness or Nefarious Agendas. Ego, not intelligence or experience, is often the largest differentiator between the bench scientist or mid-level manager/professor persona and the CEO/famous professor persona. (The other important thing, of course, is that the former is the group doing the actual work.) I think that most of our Great Minds and Great Leaders -- in all fields, really -- are not actually our best minds and best leaders. They are, instead, simply our Biggest Egos. And that those people need to puff themselves up by making their areas of ownership/responsibility/expertise sound Existentially Important.

Mouthfeel 3y ago
[dead]
scandox 3y ago
I think something weird is happening but I think it's what Hofstadter stated in the interview. The ground under his work has shifted massively and he's disturbed by that and that is affecting his judgement.
zone411 3y ago
That's not what this survey shows: https://wiki.aiimpacts.org/doku.php?id=ai_timelines:predictions_of_human-level_ai_timelines:ai_timeline_surveys:2022_expert_survey_on_progress_in_ai https://wiki.aiimpacts.org/doku.php?id=ai_timelines:predicti...
- echelon 3y ago
  If these people were so concerned, they'd by shouting from the hilltops and throwing their entire life savings into stopping us. They would organize workplace walkouts and strikes. There would be protests and banners. Burning data centers. Eliezer is one of a handful of people putting their reputation on the line, but that's mostly because that was his schtick in the first place. And even so, his response has been rather muted relative to what I'd expect from someone who thinks the imminent extinction of our species is at hand. Blake Lemoine's take at Google has been the singular act of protest in line with my expectations. We haven't seen anything else like it, and that speaks volumes. As it stands, these people are enabling regulatory capture and are doing little to stop The Terminator. Maybe they don't actually feel very threatened. Look at their actions, not their words.
  - zone411 3y ago
    If you read the survey, you'll find that many concerned researchers don't believe we're at a 90% chance of doom, but e.g. 10%. So, this type of response wouldn't be rational if they're thinking that things will go fine most of the time. If these researchers are thinking logically, they would also realize that this kind of reaction has little chance of success, especially if research continues in places like China. It's more likely that such an approach would backfire in the court of public opinion.
    - echelon 3y ago
      > If you read the survey Not a nice way to engage in debate. I've spent more time listening to and refuting these arguments than most. Including debating Eliezer. > many concerned researchers don't believe we're at a 90% chance of doom, but e.g. 10%. A 10% chance of an asteroid hitting the earth would result in every country in the world diverting all of their budgets into building a means to deflect it. > So, this type of response wouldn't be rational. This is the rational response to a 10% chance? These are funny numbers and nobody really has their skin in the game here. If I believed (truly believed) their arguments, I would throw myself at stopping this. Nobody is doing anything except for making armchair prognostications and/or speaking to congress as an AI CEO about how only big companies such as their own should be doing AI. > especially if research continues in places like China. I like how both sides of this argument are using the specter of China as a means to further their argument.
startupsfail 3y ago
Don’t forget that most of these engineers down in the trenches can’t see further than their battlefield. You actually need someone with vision and track record of doing right predictions and placing right technology bets. Ask engineers that had placed their bet on deep learning and generative models back when discriminative models and support vector machines were a rage of dat, a few years before Alexnet (I’m one of such engineers). I’d bet the answer will be different.
- ke88y 3y ago
  > Don’t forget that most of these engineers down in the trenches can’t see further than their battlefield. I don't think this is the case at all. I'm not primarily talking about a junior or even senior engineer with a decade of experience working on product features. On the contrary, many of these people have PhDs, have been leading research agendas in this field for decades, have been in senior leadership roles for a long time, have launched successful products, etc. etc. > Ask engineers that had placed their bet on deep learning and generative models back when discriminative models and support vector machines were a rage of dat, a few years before Alexnet (I’m one of such engineers). I’d bet the answer will be different. And at that time half of the "Great Minds And Great Leaders" prognosticating on X-Risk were doing social web or whatever else was peak hype cycle back then.
  - jrumbut 3y ago
    > On the contrary, many of these people have PhDs, have been leading research agendas in this field for decades, have been in senior leadership roles for a long time, have launched successful products, etc. etc. Society should definitely hear from those whose careers depend on continued AI research (and I fall into this group myself), but we are a hopelessly biased group. Advancements in AI are going to change everyone's lives, so everyone deserves a say in this. For the last 20-30 years tech has raced ahead of our government and civic processes and we're starting to feel the consequences of that. New technologies are being experienced by people as changes that are inflicted upon them instead of new options open to them. They might have very little understanding of the technology, but everyone is the expert on what impact it is having on their lives. That's something that we shouldn't ignore.
hnaccy 3y ago
> Altman's criticisms of criticisms about regulatory capture What is his criticism? If you agree with silent majority who seem to think it's not dangerous why agree with Altman who rants regulation.
- ke88y 3y ago
  > What is his criticism? I heard in some interview, I think with Bloomberg, where he said that claims about regulatory capture were "so disingenuous I'm not sure what to say", or something like that. I think he's probably not lying when he says that his goal isn't regulatory capture (although I do think other people perceiving that to be his intent aren't exactly insane either...) > who seem to think it's not dangerous On the contrary. They think it's dangerous but in a more mundane way, and that the X-Risk stuff is idiotic. I tend to agree. > why agree with Altman who rants regulation IDK. What even are his proposed regulations? They're so high-level atm that they could literally mean anything. In terms of the senate hearing he was part of, and what the government should be doing in the near term, I think the IBM woman was the only adult in the room regarding what should actually be done over the next 3-5 years. But her recommendations were boring and uninteresting recommendations to do basically the exactly sort of mundane shit the wheels of government tend to do when a new technology arrives on the scene, instead of breathless warnings about killer AI, so everyone brushed her off. But I think she's more or less right -- what should we do? The same old boring shit we always do with any new technology.
petermcneeley 3y ago
They are building consensus and finding alignment. The problem is power bends truth. This is all about access to a new powerful tool. They want to concentrate that access in the hands of those that already have control. The end goal here is the destruction of the general purpose computer.
- ke88y 3y ago
  Perhaps, but I doubt it. Never attribute to malice what can instead be attributing to stroking massive egos.
  - petermcneeley 3y ago
    "People of the same trade seldom meet together, even for merriment and diversion, but the conversation ends in a conspiracy against the public"
dist-epoch 3y ago
Nassim Taleb makes a great point: - The best person to judge the risk of playing roulette is not the carpenter who built it. - The best person to judge the risks of a global pandemic is not a virologist working with viruses daily. You can extend that: - The best person to judge the cybersecurity risks of an application is not the programmer implementing it.
- petermcneeley 3y ago
  Either Nassim is missing the point or you are misquoting him. I think this is more about should experts be in charge of their own domain.
  - dist-epoch 3y ago
    I'm not misquoting him. Nassim is arguing that RISK is a separate discipline, separate from the domain where risk applies. That a person building AI is not the correct choice for estimating AI risk. You don't ask gun making companies to make policies regarding risk of gun owning in society.
    - petermcneeley 3y ago
      Perhaps that is Nassim's argument because he is an expert in risk. The actual issue with experts monitoring experts is the bias induced by self interest (see previous sentence). Nassim sort of gets this when it comes to his discussion of "skin in the game". I dont want to malign Nassim too much but he seems like the kind of person that "sorta gets it" when it comes to everything that matters.
      - dist-epoch 3y ago
        AI researchers also have a vested interest in saying their technology is safe.
        petermcneeley 3y ago
arisAlexis 3y ago
Fake news. 350 people including the top scientists from the top labs signed the petition. Your local univ AI researcher rolls his eyes. Not the guys working at OpenAI or anthropic or deep mind.
- ke88y 3y ago
  > Not the guys working at OpenAI or anthropic or deep mind. A lot of them do. A huge percent of people who aren't speaking out are rolling their eyes. But what are you supposed to do? Contradict your boss's boss's boss?
  - wilg 3y ago
    > But what are you supposed to do? Contradict your boss's boss's boss? Yes? Obviously?
    - ke88y 3y ago
      > Yes? Obviously? I mean, I agree, obviously. My point is that most people don't. And I think for two reasons. The first and more important reason is that most people aren't involved in The Discourse and don't want to be involved in The Discourse. That's probably 99.9% of the Silent Majority -- they simply don't want to talk about anything on HN or Twitter. They view it as a waste of time or worse. and they aren't wrong. I don't think I am changing any minds here and meanwhile the personal insults kind of suck my energy a bit. So it's mostly a waste of time. The second reason is that some don't even want to even pseudo-anonymously say something that might get them into deep water at work. I'm obviously not describing myself, of course. I am here, aren't I :) But I am describing the vast majority of scientists. Keep in mind that most people don't dream of being on the proverbial TED stage and that those who do disproportionately end up on the stage and therefore determine what The Discourse will be. Big Egos == "my work is existentially important" == all the yelling about x-risk. It's mostly ego.
      - kalkin 3y ago
        It's a bit cheeky of you to be complaining about personal insults while the substance of your OP was the assertion that x-risk worriers are motivated by ego rather than real thought.
phillipcarter 3y ago
I've come to learn that a lot of the rationalist crowd are really just fanfiction authors. That's fine - people should be able to do that - but I don't like how they're given the limelight on things that they generally have little expertise and hands-on knowledge with. I want to like them and want to have them engaged in discourse, but I find them so insufferable. Not to mention the sub-crowd of rationalists that is weirdly into eugenics. I wish the rest of the rationalist community would disown them loudly.
- deleted 3y ago
  [deleted]
- bobcostas55 3y ago
  [flagged]
  - jmopp 3y ago
    I love how you jumped straight from "people weirdly into eugenics should be disavowed" to "dysgenics is desirable". That is the epitome of black-and-white thinking if I ever saw it.
    - bobcostas55 3y ago
      If a third alternative existed that would be a meaningful objection.
      - LawTalkingGuy 3y ago
        The lines are being drawn such that counseling to abort fetuses with Down syndrome is called eugenics, rhetorically equal to Nazi racial purges. "In a 1985 book titled Should the Baby Live?, Singer and his coauthor warn their audience that “this book contains conclusions which some readers will find disturbing. We think that some infants with severe disabilities should be killed.” Why? In part because of the burden they’d place on society. This is eugenics of the darkest sort"* -- https://www.truthdig.com/articles/nick-bostrom-longtermism-and-the-eternal-return-of-eugenics-2/ https://www.truthdig.com/articles/nick-bostrom-longtermism-a... The funny thing is that these people are pro abortion. They fully support your right to abort a baby for any reason, even if you just want to go drinking, unless you're doing it because the baby is actually unhealthy. And then, oh no, you have to keep it.
furyofantares 3y ago
I'm going to paint three huge demographic swaths here, with the caveat that the merits of any given argument, or any individual's track record, should override anything I say here. I'm only doing this as a reply to a comment that's also talking about trends among three groups of people. 1. The "people in the trenches" are who I'd least trust about an opinion that everything is OK. Too hard to see the forest for the trees, and too much selection bias. 2. People who gained recognition decades ago, but who are in their slowing-down years as the world continues to change, are among those who I would least trust about an opinion that things are going too fast. It gets harder to keep up as we get older, and as we gain different priorities in life, and I expect this is true no matter how smart we are. 3. People who have spent decades philosophizing about AI-doom are also among those who I would least trust about an opinion that hasn't hugely deviated and become more nuanced as the world has changed and new data has become available. I am absolutely interested in opinions from all three groups, but the arguments have to stand on their merits. If they're in one of these groups and express the expected opinion, that's actually a strike AGAINST their authority and means the merits of their argument need to be stronger. I really, really do want to hear opinions from folks in all these groups, I just want to keep this all in mind. I also want to hear opinions from younger philosophers. Folks who are better in-touch with the current world, and rates of progress, and folks who don't have any reputation to uphold. Also, anyone changing their mind is a big deal. Hofstadter may have changed his mind in the expected direction, but it's still a signal. I'd like to hear more of his thoughts. It doesn't sound carefully considered in the clip in OP's link unfortunately, but that doesn't mean it isn't, and I'd like to hear it.
throwway120385 3y ago
In my opinion the weirdness is that everyone is talking about how AI is so much better than humans and we're all going to get replaced. But almost nobody is talking about how tech companies will use this new technology to abdicate responsibility for the decisions they undertake. If you thought redlining was bad, you should see how mortgage companies treated people when they used an AI to accomplish the same goals. No thought was given to the effect of those decisions, only to the fact that this mysterious and "all-knowing" computer system told them it was a good decision. We're seeing the same thing with Facebook and "algorithms": "It's not us! It's the algorithm! We don't know why it does this." Which completely belies the fact that they wrote the algorithm and they should be held responsible for its effects. We are about to enter a couple of decades of people using these pithy model systems to make real decisions that impact lots of people, and if I've learned anything in the past 20 years its that the impacts that technologists and "emininent minds" are predicting are nothing like what will actually happen. But terrible, banal things will be done at the behest of these systems and nobody is talking about it.
- mnky9800n 3y ago
  As well as these models will all be used to launder content. In the near future you will likely be able to ask an AI to create the next 15 sequels to the movie war games starring a young matthew broderick. Episode 12 will have a karate kid cross over. But how much of what is there will be an actually new thing? How much will be scraped from my War Games fan fic website that contains scripts of 27 sequels to War Games?
  - TedDoesntTalk 3y ago
    Here is ChatGPT 4’s summary of a sequel. I don’t think it’s very good. Maybe another AI would enjoy it though. > Create a summary of a sequel to the movie “War Games” Title: "War Games: Firewall" In "War Games: Firewall", now-adult computer genius David Lightman (Matthew Broderick) has been working as a cyber security consultant for the Pentagon. When an advanced, multi-national cyber-terrorist organization emerges, they launch an unprecedented global cyber attack, disabling most nations' defensive and civil infrastructure. The terrorists reveal their ultimate goal: to dismantle the world order and plunge civilization into chaos. When the attacks render modern cybersecurity methods useless, the Pentagon calls upon Lightman. Using a blend of cutting-edge and old-school methods, including reactivating the dormant WOPR (War Operation Plan Response) system, also known as "Joshua," Lightman and a team of young coders race against time to decode the terrorists' inscrutable and constantly evolving virus. However, the situation takes a dire turn when Joshua, still possessing the capability to initiate global thermonuclear war, starts interpreting the cyber attack as an actual war, threatening to launch nuclear missiles once again. Lightman must not only outsmart the terrorists but also reason with Joshua, triggering a game of chess where each move could save or destroy the world. The climax culminates with Lightman teaching Joshua an updated lesson about the futility of war in a digital age, thus averting global disaster and unmasking the villains. In a world increasingly reliant on technology, "War Games: Firewall" underlines the importance of human judgment and intervention.
wilg 3y ago
IMHO, it's simply that predicting the future is difficult, and people disagree on what matters.
kazinator 3y ago
> Why is that? Because their work cheerfully presents statements similar to "the middle letter of 'cat' is Z" as the unvarnished truth. (Would be my guess.)
- ke88y 3y ago
  Yeah, working on these systems and seeing how brittle they are and how they're actually built is sobering. I guess from the outside it looks like a few geniuses geniusing with unbounded upside, but from the inside it looks like a hard hacking and hard work that will definitely hit hard limits.
marginalia_nu 3y ago
There's this thing where prominent scholars in one field sometimes consider themselves experts in others... https://en.m.wikipedia.org/wiki/Nobel_disease https://en.m.wikipedia.org/wiki/Nobel_disease
FollowingTheDao 3y ago
> But the actual scientists on the ground -- the PhDs and engineers I work with every day and who have been in this field, at the bench, doing to work on the latest generation of generative models, and previous generations, in some cases for decades? They almost all roll their eyes aggressively at these sorts of prognostications. I'd say 90+% either laugh or roll their eyes. > Why is that? It seems pretty obvious that one would likely not criticize something that your are actively profiting from. And I know a lot of alcoholics who do not criticize drinking as well.
- ke88y 3y ago
  > It seems pretty obvious that one would likely not criticize something that your are actively profiting from. This doesn't track, since the people criticizing are benefiting even more from the same products/companies.
lo_zamoyski 3y ago
> The "Great Minds And Great Leaders" types are rushing to warn about the risks, as are a large number of people who spend a lot of time philosophizing. Except they're not philosophizing, not in any real sense of the word. They're terrible at it. Most of them are frauds, quacks, and pseudo-intellectual con artists (like Harari) who adore the limelight offered to them by the media and a TED Talks-watching segment of the public who are, frankly, intellectually out of their depth, but enjoy the delusion and feeling of participating in something they think is "intellectual".
- tasuki 3y ago
  > frauds, quacks, and pseudo-intellectual con artists (like Harari) Uh, mind elaborating? Why is Harari that? Do you have any examples of non-frauds and actual intellectuals? > a TED Talks-watching segment of the public who are, frankly, intellectually out of their depth, but enjoy the delusion and feeling of participating in something they think is "intellectual" I'm afraid that would be me.
  - leftcenterright 3y ago
    I believe at least part of the reason for calling Harari pseudo-intellectual is the use of misleading statements in favor of storytelling in the books and also building up on others' work with better storytelling. More on this: https://www.currentaffairs.org/2022/07/the-dangerous-populist-science-of-yuval-noah-harari https://www.currentaffairs.org/2022/07/the-dangerous-populis...
  - leftcenterright 3y ago
    > I'm afraid that would be me. I think a lot of us are that, personally speaking: honest sincere analyses and investing time into critical analysis almost always will bring out more than what we hear in a talk. It does take a big amount of effort though than just watching a talk while munching on some snack.
- v-erne 3y ago
  >> pseudo-intellectual con artists (like Harari) That seems a little bit harsh - he is just master storyteller that happen to write about intellectual stuff. Its a bit weird to read his books as science papers. I just finished sapiens and even if some facts are skewed I still believe that this book is a must read for anyone who dreams about any kind of sucess in science communication (99.9% of population will never read any kind of scientific paper even if their life depends on it)
rapnie 3y ago
I don't know, but it may also be that the bench scientist is so close their work and passion that they are more likely to overlook or diminish the externalities, dangers and possible consequences of their work. While the more generalist types, the visionaries, futurists, and philosophers tend to have a broader perspective that leads them to provide stern warning. Isn't that how things go in so many technology fields? "Move fast and break things" pressure to deliver and money, reputation and fame involved in doing so are equally Ego-related and leading to biases that make on "laugh or roll their eyes".
heresie-dabord 3y ago
> something weird happening The weirdness is in part an information asymmetry that is exploited on a scale never before seen in human history. There are wealthy corporate plunderers building invasive systems of disinformation. There are people who believe everything they read and feed the panic-for-profit system. There certainly are people who understand the algorithms and implementations. There are people who fear how these algorithms and implementations will be used by the enormous network of for-profit (and for-power) influencing systems. > (from the article) these computational systems that have, you know, a million times or a billion times more knowledge than I have and are a billion times faster. It makes me feel extremely inferior. And I don't want to say deserving of being eclipsed, but it almost feels that way, as if we, all we humans, unbeknownst to us, are soon going to be eclipsed, and rightly so [...] I don't know if humans will be eclipsed, but humanity and civilisation need some strong and dedicated backers at this point.
ymck 3y ago
It's simple. All the "Big Brains" missed the real risks of Web 1.0/Web 2.0, focusing only on the positives in a time of hope and economic growth. Now, we have an internet that focuses on how everything is terrible, and a new tech abruptly hits the scene. Of course, the current "Big Brains" meet the clout need to point out how the sky might fall. AI will be transformative, but it's more likely to follow previous transformations. Unintended consequences, sure, but largely an increase in the standard of living, productivity, and economic opportunity.
maizeq 3y ago
I work in a AI research lab for a large tech company and my observations are completely opposed to yours. Almost every researcher I have spoken believes that real risk exists, to some degree or other. Recent surveys of people in industry have largely borne this out - your anecdote sounds more like an anomaly to me.
- ke88y 3y ago
  > real risk exists, to some degree or other Risk of extinction? Or of bad outcomes? I think everyone understands there are near-certain risks of bad outcomes. That's already happening all around us. Totally uncontroversial. My post was about risk of extinction due to AI (x-risk), and risk of extinction due in particular to run-away AGI (as opposed to eg shit software accidentally launching a nuke, which isn't really an AI-specific concern). I think that view is still pretty eccentric. But please lmk if that's what you meant. I've been at several ai labs and large corps. You at deepmind or openai by any chance? Just a guess ;)
- akomtu 3y ago
  It doesn't take a PhD to see where it goes. Once good enough AI is built, and we're close to that, it will be replicated by China, Russia and other big dictatorships. What would Russian dictator give for an AI that can lead his army with surgical precision? He would do anything. Other nations will have to follow. There won't be benevolent committees deciding how to use AI for good, it will be about survival. And it's going to take a miracle to avoid this scenario.
  - ke88y 3y ago
    Nope. All it takes is a sci-fi essayist. /s
  - Borrible 3y ago
    >What would Russian dictator give for an AI that can lead his army with surgical precision Still needs surgical instruments. But more important, patients that do not defend themselves with claws, teeth and AI.
d13 3y ago
Just like ChatGPT.
kalkin 3y ago
Is there a way that you'd recommend somebody outside the field assess your "90%" claim? Elsewhere in the thread you're dismissive of that one survey - which I agree is weak evidence in itself - and you also deny that the statements of leaders at OpenAI, DeepMind and Anthropic are representative of researchers there, which again may be the case. But how is someone who doesn't know you or your network supposed to assess this?
- kalkin 3y ago
  Relatedly, it'd be helpful if you could point to folks with good ML research credentials making a detailed case against x-risk. The prominent example of whom I'm aware is Yann LeCun, and what I've seen of his arguments take place more in the evopsych field (the stuff about an alleged innate human drive for dominance which AIs will lack) than the field in which he's actually qualified.
  - YeGoblynQueenne 3y ago
    (Not the OP) In this article Le Cun argues in very concrete technical terms about the impossibility of achieving AGI with modern techniques (yes, all of them): Meta's AI guru LeCun: Most of today's AI approaches will never lead to true intelligence https://www.zdnet.com/article/metas-ai-guru-lecun-most-of-todays-ai-approaches-will-never-lead-to-true-intelligence/ https://www.zdnet.com/article/metas-ai-guru-lecun-most-of-to... Edit: on second thought, he gets maybe a bit too technical at times, but I think it should be possible to follow most of the article without specialised knowledge.
    - kalkin 3y ago
      I wouldn't describe most of what's in that interview as "very concrete technical" terms - at least when it comes to other people's research programs. More importantly, while it's perfectly reasonable for LeCun to believe in his own research program and not others, "this one lab's plan is the one true general-AI research program and most researchers are pursuing dead ends" doesn't seem like a very sturdy foundation on which to place "nothing to worry about here" - especially since LeCun doesn't give an argument here why his program would produce something safe.
Gibbon1 3y ago
> Why is that? Because of deep silo myopia. Meaning they have no idea what terrible things the pointy haired bosses and grifters are going to use this stuff for.
pygy_ 3y ago
Humanity’s been a great scaffold for capitalism. AI is the keystone.
sanp 3y ago
The warnings are an admission from the “Great Minds and a great Thinkers” that someone other than them has created something previously thought impossible or, at the very least, several years / decades out. So, I am not sure ego is at play here. Perhaps someone close to the problem (your actual scientists on the ground) is not capable / unwilling to accept the issues raised as that has direct (negative) impact on what they have created.
- ke88y 3y ago
  Perhaps. It's also possible that they don't understand how dangerous what they've created is because it feels so "normal" and "pedestrian" to them. See also: nuclear scientists getting radiation poisoning, I suppose. But that's also true for all of the other people being discussed, I would think. But I do think "wants to be in the discourse and on top" is a pretty strong correlate with the degree to which someone characterizes these as "concerns" vs "x-risk".
13years 3y ago
Those with great visibility often are not the greatest minds. However, in a broader sense we would have been served well if philosophizers would have had more input on the development of social media tech. It is an example of engineers knowing a field narrowly and deeply but not understanding the societal consequence. AI may very well fall into the same pattern and it is something I have written out in some detail of thoughts around alignment and the traps of both ego and misunderstanding human nature for which we want to model alignment. https://www.mindprison.cc/p/ai-singularity-the-hubris-trap https://www.mindprison.cc/p/ai-singularity-the-hubris-trap
- ke88y 3y ago
  I think AI definitely will follow a similar path as social media tech. I don't think that will lead to the extinction of humanity via the development of super human intelligence.
  - 13years 3y ago
    Ironically the destruction to social order by primitive AI may be the very obstacle to achieving AGI.
lowbloodsugar 3y ago
In the "scientists complacent about dangers of the systems they are working on" file, I think we are all familiar with https://en.wikipedia.org/wiki/Demon_core https://en.wikipedia.org/wiki/Demon_core. Now, admittedly, this was scientists being complacent about a thing they knew was dangerous, whereas here we are saying scientists don't think their thing is dangerous. But very clearly, AI could be dangerous, so it's more that these scientists don't think their system could be dangerous. Presumably the scientists and engineers behind the https://en.wikipedia.org/wiki/Therac-25 https://en.wikipedia.org/wiki/Therac-25 didn't think it would kill people. So maybe the problem is precisely that when we bring up extinction events from AGI, scientists rolling their eyes is the very reason we should be fucking worried. Their contempt for the possibility of the threat is what will get us killed.
- ke88y 3y ago
  > Presumably the scientists and engineers behind the https://en.wikipedia.org/wiki/Therac-25 https://en.wikipedia.org/wiki/Therac-25 didn't think it would kill people. Therac-25 is an excellent example, but of EXACTLY the opposite point. On the contrary, abstract AGI safety nonsense taking such a strong grip on academic and industrial AI Safety research is what would most frighten me. In the intervening decades, people concerned about software safety provided us with the tools needed to prevent disasters like Therac-25, while sci-fi about killer robots was entirely unhelpful in preventing software bugs. People concerned about software safety provided us with robust and secure nuclear (anti-)launch systems, while Wargames didn't do much except excite the public imagination. Etc. We need scientist's and engineer's efforts and attention focused on real risks and practical solutions, not fanciful philosophizing about sci-fi tropes.
  - lowbloodsugar 3y ago
    >while sci-fi about killer robots was entirely unhelpful in preventing software bugs. Unless we are now talking about actually building AI and killer robots, which we are assuredly doing.
cubefox 3y ago
The fact that such a depressingly ignorant comment gets upvoted makes me reconsider whether HN is really "my tribe". :(
ke88y 3y ago
Note: I really don't like that my comment is the top comment this story. I think this phenomenon wrt x-risk is real and interesting and worth discussing, but I wish I'd picked a different story to share the thought. (This just happened to be the first related story that popped up since I had the thought.) There are other more interesting comments that deserve more discussion WRT this article in particular. If any mods are reading this, please consider this a request to push this comment down so some of the others get more attention. IDK if there's much more to learn from the discussion happening here anyways.
naasking 3y ago
> They almost all roll their eyes aggressively at these sorts of prognostications. I'd say 90+% either laugh or roll their eyes. Why is that? I literally think they haven't even given it any thought. Who seriously tries to extrapolate beyond 5 years in any field? Even to most doomers, 5 years seems safe. What about 10? 20? 30? Remember, one of the very first things people tried to do was create a self-directed, self-optimizing AI with AutoGPT. How far forward can you project through how many attempts like this with progressively more sophisticated systems before this might produce an outcome that could be considered catastrophic? If your answer is anything but an emphatic "infinity", then welcome to the doomers. Now we're just debating when and under circumstances this might take place, and not whether it might happen at all.
pipes 3y ago
My gut feeling is that they get status from these warnings. Or they subconsciously think they will. Like somehow they are as important as the scientists from the Manhatten project.
m463 3y ago
I suspect that nobody accepts risks until they are realized in some way. Nobody really did anything about air pollution until you could see smog, and then catalytic converters were mandated. Lots of people complained about privacy, but because for practical purposes nobody can "see" a loss of privacy, very little progress has been made. Not many self-driving car fatalities, so the loud pundit proclamations have not gotten much traction. Now AI... will this take the path of privacy, where nobody can see it, so nothing happens?

by ke88y 3y ago

Mouthfeel 3y ago

[dead]

scandox 3y ago

I think something weird is happening but I think it's what Hofstadter stated in the interview. The ground under his work has shifted massively and he's disturbed by that and that is affecting his judgement.

zone411 3y ago

That's not what this survey shows: https://wiki.aiimpacts.org/doku.php?id=ai_timelines:predictions_of_human-level_ai_timelines:ai_timeline_surveys:2022_expert_survey_on_progress_in_ai https://wiki.aiimpacts.org/doku.php?id=ai_timelines:predicti...

echelon 3y ago
If these people were so concerned, they'd by shouting from the hilltops and throwing their entire life savings into stopping us. They would organize workplace walkouts and strikes. There would be protests and banners. Burning data centers. Eliezer is one of a handful of people putting their reputation on the line, but that's mostly because that was his schtick in the first place. And even so, his response has been rather muted relative to what I'd expect from someone who thinks the imminent extinction of our species is at hand. Blake Lemoine's take at Google has been the singular act of protest in line with my expectations. We haven't seen anything else like it, and that speaks volumes. As it stands, these people are enabling regulatory capture and are doing little to stop The Terminator. Maybe they don't actually feel very threatened. Look at their actions, not their words.
- zone411 3y ago
  If you read the survey, you'll find that many concerned researchers don't believe we're at a 90% chance of doom, but e.g. 10%. So, this type of response wouldn't be rational if they're thinking that things will go fine most of the time. If these researchers are thinking logically, they would also realize that this kind of reaction has little chance of success, especially if research continues in places like China. It's more likely that such an approach would backfire in the court of public opinion.
  - echelon 3y ago
    > If you read the survey Not a nice way to engage in debate. I've spent more time listening to and refuting these arguments than most. Including debating Eliezer. > many concerned researchers don't believe we're at a 90% chance of doom, but e.g. 10%. A 10% chance of an asteroid hitting the earth would result in every country in the world diverting all of their budgets into building a means to deflect it. > So, this type of response wouldn't be rational. This is the rational response to a 10% chance? These are funny numbers and nobody really has their skin in the game here. If I believed (truly believed) their arguments, I would throw myself at stopping this. Nobody is doing anything except for making armchair prognostications and/or speaking to congress as an AI CEO about how only big companies such as their own should be doing AI. > especially if research continues in places like China. I like how both sides of this argument are using the specter of China as a means to further their argument.

startupsfail 3y ago

Don’t forget that most of these engineers down in the trenches can’t see further than their battlefield. You actually need someone with vision and track record of doing right predictions and placing right technology bets. Ask engineers that had placed their bet on deep learning and generative models back when discriminative models and support vector machines were a rage of dat, a few years before Alexnet (I’m one of such engineers). I’d bet the answer will be different.

ke88y 3y ago
> Don’t forget that most of these engineers down in the trenches can’t see further than their battlefield. I don't think this is the case at all. I'm not primarily talking about a junior or even senior engineer with a decade of experience working on product features. On the contrary, many of these people have PhDs, have been leading research agendas in this field for decades, have been in senior leadership roles for a long time, have launched successful products, etc. etc. > Ask engineers that had placed their bet on deep learning and generative models back when discriminative models and support vector machines were a rage of dat, a few years before Alexnet (I’m one of such engineers). I’d bet the answer will be different. And at that time half of the "Great Minds And Great Leaders" prognosticating on X-Risk were doing social web or whatever else was peak hype cycle back then.
- jrumbut 3y ago
  > On the contrary, many of these people have PhDs, have been leading research agendas in this field for decades, have been in senior leadership roles for a long time, have launched successful products, etc. etc. Society should definitely hear from those whose careers depend on continued AI research (and I fall into this group myself), but we are a hopelessly biased group. Advancements in AI are going to change everyone's lives, so everyone deserves a say in this. For the last 20-30 years tech has raced ahead of our government and civic processes and we're starting to feel the consequences of that. New technologies are being experienced by people as changes that are inflicted upon them instead of new options open to them. They might have very little understanding of the technology, but everyone is the expert on what impact it is having on their lives. That's something that we shouldn't ignore.

hnaccy 3y ago

> Altman's criticisms of criticisms about regulatory capture What is his criticism? If you agree with silent majority who seem to think it's not dangerous why agree with Altman who rants regulation.

ke88y 3y ago
> What is his criticism? I heard in some interview, I think with Bloomberg, where he said that claims about regulatory capture were "so disingenuous I'm not sure what to say", or something like that. I think he's probably not lying when he says that his goal isn't regulatory capture (although I do think other people perceiving that to be his intent aren't exactly insane either...) > who seem to think it's not dangerous On the contrary. They think it's dangerous but in a more mundane way, and that the X-Risk stuff is idiotic. I tend to agree. > why agree with Altman who rants regulation IDK. What even are his proposed regulations? They're so high-level atm that they could literally mean anything. In terms of the senate hearing he was part of, and what the government should be doing in the near term, I think the IBM woman was the only adult in the room regarding what should actually be done over the next 3-5 years. But her recommendations were boring and uninteresting recommendations to do basically the exactly sort of mundane shit the wheels of government tend to do when a new technology arrives on the scene, instead of breathless warnings about killer AI, so everyone brushed her off. But I think she's more or less right -- what should we do? The same old boring shit we always do with any new technology.

petermcneeley 3y ago

They are building consensus and finding alignment. The problem is power bends truth. This is all about access to a new powerful tool. They want to concentrate that access in the hands of those that already have control. The end goal here is the destruction of the general purpose computer.

ke88y 3y ago
Perhaps, but I doubt it. Never attribute to malice what can instead be attributing to stroking massive egos.
- petermcneeley 3y ago
  "People of the same trade seldom meet together, even for merriment and diversion, but the conversation ends in a conspiracy against the public"

dist-epoch 3y ago

Nassim Taleb makes a great point: - The best person to judge the risk of playing roulette is not the carpenter who built it. - The best person to judge the risks of a global pandemic is not a virologist working with viruses daily. You can extend that: - The best person to judge the cybersecurity risks of an application is not the programmer implementing it.

petermcneeley 3y ago
Either Nassim is missing the point or you are misquoting him. I think this is more about should experts be in charge of their own domain.
- dist-epoch 3y ago
  I'm not misquoting him. Nassim is arguing that RISK is a separate discipline, separate from the domain where risk applies. That a person building AI is not the correct choice for estimating AI risk. You don't ask gun making companies to make policies regarding risk of gun owning in society.
  - petermcneeley 3y ago
    Perhaps that is Nassim's argument because he is an expert in risk. The actual issue with experts monitoring experts is the bias induced by self interest (see previous sentence). Nassim sort of gets this when it comes to his discussion of "skin in the game". I dont want to malign Nassim too much but he seems like the kind of person that "sorta gets it" when it comes to everything that matters.
    - dist-epoch 3y ago
      AI researchers also have a vested interest in saying their technology is safe.
      - petermcneeley 3y ago

arisAlexis 3y ago

Fake news. 350 people including the top scientists from the top labs signed the petition. Your local univ AI researcher rolls his eyes. Not the guys working at OpenAI or anthropic or deep mind.

ke88y 3y ago
> Not the guys working at OpenAI or anthropic or deep mind. A lot of them do. A huge percent of people who aren't speaking out are rolling their eyes. But what are you supposed to do? Contradict your boss's boss's boss?
- wilg 3y ago
  > But what are you supposed to do? Contradict your boss's boss's boss? Yes? Obviously?
  - ke88y 3y ago
    > Yes? Obviously? I mean, I agree, obviously. My point is that most people don't. And I think for two reasons. The first and more important reason is that most people aren't involved in The Discourse and don't want to be involved in The Discourse. That's probably 99.9% of the Silent Majority -- they simply don't want to talk about anything on HN or Twitter. They view it as a waste of time or worse. and they aren't wrong. I don't think I am changing any minds here and meanwhile the personal insults kind of suck my energy a bit. So it's mostly a waste of time. The second reason is that some don't even want to even pseudo-anonymously say something that might get them into deep water at work. I'm obviously not describing myself, of course. I am here, aren't I :) But I am describing the vast majority of scientists. Keep in mind that most people don't dream of being on the proverbial TED stage and that those who do disproportionately end up on the stage and therefore determine what The Discourse will be. Big Egos == "my work is existentially important" == all the yelling about x-risk. It's mostly ego.
    - kalkin 3y ago
      It's a bit cheeky of you to be complaining about personal insults while the substance of your OP was the assertion that x-risk worriers are motivated by ego rather than real thought.

phillipcarter 3y ago

I've come to learn that a lot of the rationalist crowd are really just fanfiction authors. That's fine - people should be able to do that - but I don't like how they're given the limelight on things that they generally have little expertise and hands-on knowledge with. I want to like them and want to have them engaged in discourse, but I find them so insufferable. Not to mention the sub-crowd of rationalists that is weirdly into eugenics. I wish the rest of the rationalist community would disown them loudly.

deleted 3y ago
[deleted]
bobcostas55 3y ago
[flagged]
- jmopp 3y ago
  I love how you jumped straight from "people weirdly into eugenics should be disavowed" to "dysgenics is desirable". That is the epitome of black-and-white thinking if I ever saw it.
  - bobcostas55 3y ago
    If a third alternative existed that would be a meaningful objection.
    - LawTalkingGuy 3y ago
      The lines are being drawn such that counseling to abort fetuses with Down syndrome is called eugenics, rhetorically equal to Nazi racial purges. "In a 1985 book titled Should the Baby Live?, Singer and his coauthor warn their audience that “this book contains conclusions which some readers will find disturbing. We think that some infants with severe disabilities should be killed.” Why? In part because of the burden they’d place on society. This is eugenics of the darkest sort"* -- https://www.truthdig.com/articles/nick-bostrom-longtermism-and-the-eternal-return-of-eugenics-2/ https://www.truthdig.com/articles/nick-bostrom-longtermism-a... The funny thing is that these people are pro abortion. They fully support your right to abort a baby for any reason, even if you just want to go drinking, unless you're doing it because the baby is actually unhealthy. And then, oh no, you have to keep it.

furyofantares 3y ago

I'm going to paint three huge demographic swaths here, with the caveat that the merits of any given argument, or any individual's track record, should override anything I say here. I'm only doing this as a reply to a comment that's also talking about trends among three groups of people. 1. The "people in the trenches" are who I'd least trust about an opinion that everything is OK. Too hard to see the forest for the trees, and too much selection bias. 2. People who gained recognition decades ago, but who are in their slowing-down years as the world continues to change, are among those who I would least trust about an opinion that things are going too fast. It gets harder to keep up as we get older, and as we gain different priorities in life, and I expect this is true no matter how smart we are. 3. People who have spent decades philosophizing about AI-doom are also among those who I would least trust about an opinion that hasn't hugely deviated and become more nuanced as the world has changed and new data has become available. I am absolutely interested in opinions from all three groups, but the arguments have to stand on their merits. If they're in one of these groups and express the expected opinion, that's actually a strike AGAINST their authority and means the merits of their argument need to be stronger. I really, really do want to hear opinions from folks in all these groups, I just want to keep this all in mind. I also want to hear opinions from younger philosophers. Folks who are better in-touch with the current world, and rates of progress, and folks who don't have any reputation to uphold. Also, anyone changing their mind is a big deal. Hofstadter may have changed his mind in the expected direction, but it's still a signal. I'd like to hear more of his thoughts. It doesn't sound carefully considered in the clip in OP's link unfortunately, but that doesn't mean it isn't, and I'd like to hear it.

throwway120385 3y ago

In my opinion the weirdness is that everyone is talking about how AI is so much better than humans and we're all going to get replaced. But almost nobody is talking about how tech companies will use this new technology to abdicate responsibility for the decisions they undertake. If you thought redlining was bad, you should see how mortgage companies treated people when they used an AI to accomplish the same goals. No thought was given to the effect of those decisions, only to the fact that this mysterious and "all-knowing" computer system told them it was a good decision. We're seeing the same thing with Facebook and "algorithms": "It's not us! It's the algorithm! We don't know why it does this." Which completely belies the fact that they wrote the algorithm and they should be held responsible for its effects. We are about to enter a couple of decades of people using these pithy model systems to make real decisions that impact lots of people, and if I've learned anything in the past 20 years its that the impacts that technologists and "emininent minds" are predicting are nothing like what will actually happen. But terrible, banal things will be done at the behest of these systems and nobody is talking about it.

mnky9800n 3y ago
As well as these models will all be used to launder content. In the near future you will likely be able to ask an AI to create the next 15 sequels to the movie war games starring a young matthew broderick. Episode 12 will have a karate kid cross over. But how much of what is there will be an actually new thing? How much will be scraped from my War Games fan fic website that contains scripts of 27 sequels to War Games?
- TedDoesntTalk 3y ago
  Here is ChatGPT 4’s summary of a sequel. I don’t think it’s very good. Maybe another AI would enjoy it though. > Create a summary of a sequel to the movie “War Games” Title: "War Games: Firewall" In "War Games: Firewall", now-adult computer genius David Lightman (Matthew Broderick) has been working as a cyber security consultant for the Pentagon. When an advanced, multi-national cyber-terrorist organization emerges, they launch an unprecedented global cyber attack, disabling most nations' defensive and civil infrastructure. The terrorists reveal their ultimate goal: to dismantle the world order and plunge civilization into chaos. When the attacks render modern cybersecurity methods useless, the Pentagon calls upon Lightman. Using a blend of cutting-edge and old-school methods, including reactivating the dormant WOPR (War Operation Plan Response) system, also known as "Joshua," Lightman and a team of young coders race against time to decode the terrorists' inscrutable and constantly evolving virus. However, the situation takes a dire turn when Joshua, still possessing the capability to initiate global thermonuclear war, starts interpreting the cyber attack as an actual war, threatening to launch nuclear missiles once again. Lightman must not only outsmart the terrorists but also reason with Joshua, triggering a game of chess where each move could save or destroy the world. The climax culminates with Lightman teaching Joshua an updated lesson about the futility of war in a digital age, thus averting global disaster and unmasking the villains. In a world increasingly reliant on technology, "War Games: Firewall" underlines the importance of human judgment and intervention.

wilg 3y ago

IMHO, it's simply that predicting the future is difficult, and people disagree on what matters.

kazinator 3y ago

> Why is that? Because their work cheerfully presents statements similar to "the middle letter of 'cat' is Z" as the unvarnished truth. (Would be my guess.)

ke88y 3y ago
Yeah, working on these systems and seeing how brittle they are and how they're actually built is sobering. I guess from the outside it looks like a few geniuses geniusing with unbounded upside, but from the inside it looks like a hard hacking and hard work that will definitely hit hard limits.

marginalia_nu 3y ago

There's this thing where prominent scholars in one field sometimes consider themselves experts in others... https://en.m.wikipedia.org/wiki/Nobel_disease https://en.m.wikipedia.org/wiki/Nobel_disease

FollowingTheDao 3y ago

> But the actual scientists on the ground -- the PhDs and engineers I work with every day and who have been in this field, at the bench, doing to work on the latest generation of generative models, and previous generations, in some cases for decades? They almost all roll their eyes aggressively at these sorts of prognostications. I'd say 90+% either laugh or roll their eyes. > Why is that? It seems pretty obvious that one would likely not criticize something that your are actively profiting from. And I know a lot of alcoholics who do not criticize drinking as well.

ke88y 3y ago
> It seems pretty obvious that one would likely not criticize something that your are actively profiting from. This doesn't track, since the people criticizing are benefiting even more from the same products/companies.

lo_zamoyski 3y ago

> The "Great Minds And Great Leaders" types are rushing to warn about the risks, as are a large number of people who spend a lot of time philosophizing. Except they're not philosophizing, not in any real sense of the word. They're terrible at it. Most of them are frauds, quacks, and pseudo-intellectual con artists (like Harari) who adore the limelight offered to them by the media and a TED Talks-watching segment of the public who are, frankly, intellectually out of their depth, but enjoy the delusion and feeling of participating in something they think is "intellectual".

tasuki 3y ago
> frauds, quacks, and pseudo-intellectual con artists (like Harari) Uh, mind elaborating? Why is Harari that? Do you have any examples of non-frauds and actual intellectuals? > a TED Talks-watching segment of the public who are, frankly, intellectually out of their depth, but enjoy the delusion and feeling of participating in something they think is "intellectual" I'm afraid that would be me.
- leftcenterright 3y ago
  I believe at least part of the reason for calling Harari pseudo-intellectual is the use of misleading statements in favor of storytelling in the books and also building up on others' work with better storytelling. More on this: https://www.currentaffairs.org/2022/07/the-dangerous-populist-science-of-yuval-noah-harari https://www.currentaffairs.org/2022/07/the-dangerous-populis...
- leftcenterright 3y ago
  > I'm afraid that would be me. I think a lot of us are that, personally speaking: honest sincere analyses and investing time into critical analysis almost always will bring out more than what we hear in a talk. It does take a big amount of effort though than just watching a talk while munching on some snack.
v-erne 3y ago
>> pseudo-intellectual con artists (like Harari) That seems a little bit harsh - he is just master storyteller that happen to write about intellectual stuff. Its a bit weird to read his books as science papers. I just finished sapiens and even if some facts are skewed I still believe that this book is a must read for anyone who dreams about any kind of sucess in science communication (99.9% of population will never read any kind of scientific paper even if their life depends on it)

rapnie 3y ago

I don't know, but it may also be that the bench scientist is so close their work and passion that they are more likely to overlook or diminish the externalities, dangers and possible consequences of their work. While the more generalist types, the visionaries, futurists, and philosophers tend to have a broader perspective that leads them to provide stern warning. Isn't that how things go in so many technology fields? "Move fast and break things" pressure to deliver and money, reputation and fame involved in doing so are equally Ego-related and leading to biases that make on "laugh or roll their eyes".

heresie-dabord 3y ago

> something weird happening The weirdness is in part an information asymmetry that is exploited on a scale never before seen in human history. There are wealthy corporate plunderers building invasive systems of disinformation. There are people who believe everything they read and feed the panic-for-profit system. There certainly are people who understand the algorithms and implementations. There are people who fear how these algorithms and implementations will be used by the enormous network of for-profit (and for-power) influencing systems. > (from the article) these computational systems that have, you know, a million times or a billion times more knowledge than I have and are a billion times faster. It makes me feel extremely inferior. And I don't want to say deserving of being eclipsed, but it almost feels that way, as if we, all we humans, unbeknownst to us, are soon going to be eclipsed, and rightly so [...] I don't know if humans will be eclipsed, but humanity and civilisation need some strong and dedicated backers at this point.

ymck 3y ago

It's simple. All the "Big Brains" missed the real risks of Web 1.0/Web 2.0, focusing only on the positives in a time of hope and economic growth. Now, we have an internet that focuses on how everything is terrible, and a new tech abruptly hits the scene. Of course, the current "Big Brains" meet the clout need to point out how the sky might fall. AI will be transformative, but it's more likely to follow previous transformations. Unintended consequences, sure, but largely an increase in the standard of living, productivity, and economic opportunity.

maizeq 3y ago

I work in a AI research lab for a large tech company and my observations are completely opposed to yours. Almost every researcher I have spoken believes that real risk exists, to some degree or other. Recent surveys of people in industry have largely borne this out - your anecdote sounds more like an anomaly to me.

ke88y 3y ago
> real risk exists, to some degree or other Risk of extinction? Or of bad outcomes? I think everyone understands there are near-certain risks of bad outcomes. That's already happening all around us. Totally uncontroversial. My post was about risk of extinction due to AI (x-risk), and risk of extinction due in particular to run-away AGI (as opposed to eg shit software accidentally launching a nuke, which isn't really an AI-specific concern). I think that view is still pretty eccentric. But please lmk if that's what you meant. I've been at several ai labs and large corps. You at deepmind or openai by any chance? Just a guess ;)
akomtu 3y ago
It doesn't take a PhD to see where it goes. Once good enough AI is built, and we're close to that, it will be replicated by China, Russia and other big dictatorships. What would Russian dictator give for an AI that can lead his army with surgical precision? He would do anything. Other nations will have to follow. There won't be benevolent committees deciding how to use AI for good, it will be about survival. And it's going to take a miracle to avoid this scenario.
- ke88y 3y ago
  Nope. All it takes is a sci-fi essayist. /s
- Borrible 3y ago
  >What would Russian dictator give for an AI that can lead his army with surgical precision Still needs surgical instruments. But more important, patients that do not defend themselves with claws, teeth and AI.

d13 3y ago

Just like ChatGPT.

kalkin 3y ago

Is there a way that you'd recommend somebody outside the field assess your "90%" claim? Elsewhere in the thread you're dismissive of that one survey - which I agree is weak evidence in itself - and you also deny that the statements of leaders at OpenAI, DeepMind and Anthropic are representative of researchers there, which again may be the case. But how is someone who doesn't know you or your network supposed to assess this?

kalkin 3y ago
Relatedly, it'd be helpful if you could point to folks with good ML research credentials making a detailed case against x-risk. The prominent example of whom I'm aware is Yann LeCun, and what I've seen of his arguments take place more in the evopsych field (the stuff about an alleged innate human drive for dominance which AIs will lack) than the field in which he's actually qualified.
- YeGoblynQueenne 3y ago
  (Not the OP) In this article Le Cun argues in very concrete technical terms about the impossibility of achieving AGI with modern techniques (yes, all of them): Meta's AI guru LeCun: Most of today's AI approaches will never lead to true intelligence https://www.zdnet.com/article/metas-ai-guru-lecun-most-of-todays-ai-approaches-will-never-lead-to-true-intelligence/ https://www.zdnet.com/article/metas-ai-guru-lecun-most-of-to... Edit: on second thought, he gets maybe a bit too technical at times, but I think it should be possible to follow most of the article without specialised knowledge.
  - kalkin 3y ago
    I wouldn't describe most of what's in that interview as "very concrete technical" terms - at least when it comes to other people's research programs. More importantly, while it's perfectly reasonable for LeCun to believe in his own research program and not others, "this one lab's plan is the one true general-AI research program and most researchers are pursuing dead ends" doesn't seem like a very sturdy foundation on which to place "nothing to worry about here" - especially since LeCun doesn't give an argument here why his program would produce something safe.

Gibbon1 3y ago

> Why is that? Because of deep silo myopia. Meaning they have no idea what terrible things the pointy haired bosses and grifters are going to use this stuff for.

pygy_ 3y ago

Humanity’s been a great scaffold for capitalism. AI is the keystone.

sanp 3y ago

The warnings are an admission from the “Great Minds and a great Thinkers” that someone other than them has created something previously thought impossible or, at the very least, several years / decades out. So, I am not sure ego is at play here. Perhaps someone close to the problem (your actual scientists on the ground) is not capable / unwilling to accept the issues raised as that has direct (negative) impact on what they have created.

ke88y 3y ago
Perhaps. It's also possible that they don't understand how dangerous what they've created is because it feels so "normal" and "pedestrian" to them. See also: nuclear scientists getting radiation poisoning, I suppose. But that's also true for all of the other people being discussed, I would think. But I do think "wants to be in the discourse and on top" is a pretty strong correlate with the degree to which someone characterizes these as "concerns" vs "x-risk".

13years 3y ago

Those with great visibility often are not the greatest minds. However, in a broader sense we would have been served well if philosophizers would have had more input on the development of social media tech. It is an example of engineers knowing a field narrowly and deeply but not understanding the societal consequence. AI may very well fall into the same pattern and it is something I have written out in some detail of thoughts around alignment and the traps of both ego and misunderstanding human nature for which we want to model alignment. https://www.mindprison.cc/p/ai-singularity-the-hubris-trap https://www.mindprison.cc/p/ai-singularity-the-hubris-trap

ke88y 3y ago
I think AI definitely will follow a similar path as social media tech. I don't think that will lead to the extinction of humanity via the development of super human intelligence.
- 13years 3y ago
  Ironically the destruction to social order by primitive AI may be the very obstacle to achieving AGI.

lowbloodsugar 3y ago

In the "scientists complacent about dangers of the systems they are working on" file, I think we are all familiar with https://en.wikipedia.org/wiki/Demon_core https://en.wikipedia.org/wiki/Demon_core. Now, admittedly, this was scientists being complacent about a thing they knew was dangerous, whereas here we are saying scientists don't think their thing is dangerous. But very clearly, AI could be dangerous, so it's more that these scientists don't think their system could be dangerous. Presumably the scientists and engineers behind the https://en.wikipedia.org/wiki/Therac-25 https://en.wikipedia.org/wiki/Therac-25 didn't think it would kill people. So maybe the problem is precisely that when we bring up extinction events from AGI, scientists rolling their eyes is the very reason we should be fucking worried. Their contempt for the possibility of the threat is what will get us killed.

ke88y 3y ago
> Presumably the scientists and engineers behind the https://en.wikipedia.org/wiki/Therac-25 https://en.wikipedia.org/wiki/Therac-25 didn't think it would kill people. Therac-25 is an excellent example, but of EXACTLY the opposite point. On the contrary, abstract AGI safety nonsense taking such a strong grip on academic and industrial AI Safety research is what would most frighten me. In the intervening decades, people concerned about software safety provided us with the tools needed to prevent disasters like Therac-25, while sci-fi about killer robots was entirely unhelpful in preventing software bugs. People concerned about software safety provided us with robust and secure nuclear (anti-)launch systems, while Wargames didn't do much except excite the public imagination. Etc. We need scientist's and engineer's efforts and attention focused on real risks and practical solutions, not fanciful philosophizing about sci-fi tropes.
- lowbloodsugar 3y ago
  >while sci-fi about killer robots was entirely unhelpful in preventing software bugs. Unless we are now talking about actually building AI and killer robots, which we are assuredly doing.

cubefox 3y ago

The fact that such a depressingly ignorant comment gets upvoted makes me reconsider whether HN is really "my tribe". :(

ke88y 3y ago

Note: I really don't like that my comment is the top comment this story. I think this phenomenon wrt x-risk is real and interesting and worth discussing, but I wish I'd picked a different story to share the thought. (This just happened to be the first related story that popped up since I had the thought.) There are other more interesting comments that deserve more discussion WRT this article in particular. If any mods are reading this, please consider this a request to push this comment down so some of the others get more attention. IDK if there's much more to learn from the discussion happening here anyways.

naasking 3y ago

> They almost all roll their eyes aggressively at these sorts of prognostications. I'd say 90+% either laugh or roll their eyes. Why is that? I literally think they haven't even given it any thought. Who seriously tries to extrapolate beyond 5 years in any field? Even to most doomers, 5 years seems safe. What about 10? 20? 30? Remember, one of the very first things people tried to do was create a self-directed, self-optimizing AI with AutoGPT. How far forward can you project through how many attempts like this with progressively more sophisticated systems before this might produce an outcome that could be considered catastrophic? If your answer is anything but an emphatic "infinity", then welcome to the doomers. Now we're just debating when and under circumstances this might take place, and not whether it might happen at all.

pipes 3y ago

My gut feeling is that they get status from these warnings. Or they subconsciously think they will. Like somehow they are as important as the scientists from the Manhatten project.

m463 3y ago

I suspect that nobody accepts risks until they are realized in some way. Nobody really did anything about air pollution until you could see smog, and then catalytic converters were mandated. Lots of people complained about privacy, but because for practical purposes nobody can "see" a loss of privacy, very little progress has been made. Not many self-driving car fatalities, so the loud pundit proclamations have not gotten much traction. Now AI... will this take the path of privacy, where nobody can see it, so nothing happens?