24 ms·

If AI scaling is to be shut down, let it be for a coherent reason

• cs702 4y ago
  > ... I was deeply confused, until I heard a dear friend and colleague in academic AI, one who’s long been skeptical of AI-doom scenarios, explain why he signed the open letter. He said: look, we all started writing research papers about the safety issues with ChatGPT; then our work became obsolete when OpenAI released GPT-4 just a few months later. So now we’re writing papers about GPT-4. Will we again have to throw our work away when OpenAI releases GPT-5? I realized that, while six months might not suffice to save human civilization, it’s just enough for the more immediate concern of getting papers into academic AI conferences. In other words, the people who wrote and are signing open letters to slow down AI scaling appear to be more concerned with their inability to benefit from and control the dialog around AI scaling than any societal risks posed by these advances in the near term. Meanwhile, to the folks at organizations like Microsoft/OpenAI, Alphabet, Facebook, etc., the scaling of AI looks like a shiny rainbow with a big pot of gold -- money, fame, glory, etc. -- on the other side. Why would they want to slow down now?
  • coldtea 4y ago
    >In other words, the people who wrote and are signing open letters to slow down AI scaling appear to be more concerned with their inability to benefit from and control the dialog around AI scaling than any societal risks posed by these advances in the near term. That's just a joke the author makes. He is not seriously suggesting this is the case.
  • matthewdgreen 4y ago
    I don’t think Scott is serious about that (or if he is, he’s being uncharitable.) I think what the quoted speaker is saying is that nobody is able to keep up with what these models are doing internally. Even OpenAI (and Meta et al.) only seem to be making “so much progress” by pressing the accelerator to the floor and letting the steering take care of itself. And one of the major lessons of technological progress is that deep understanding (at least when humans are necessary for that, gulp) is much slower than engineering, largely because the latter can be parallelized and scaled.
  • findalex 4y ago
    >started writing research papers about the safety issues with ChatGPT; Feels strange that academia would focus so much energy on a product.
    • AlanYx 4y ago
      Just judging from the volume of papers, it seems to me that there are more academics writing papers on "AI safety" and "AI ethics" than there are academics publishing research papers on actual AI. It's become one of the hottest topics among legal academics, philosophers, ethicists, and a variety of connected disciplines, in addition to its niche among some computer scientists, and the amount of work to get to a paper in these fields is an order of magnitude less than actually publishing technical research.
• rvz 4y ago
  > On the other hand, I’m deeply confused by the people who signed the open letter, even though they continue to downplay or even ridicule GPT’s abilities, as well as the “sensationalist” predictions of an AI apocalypse. Says the quantum computing professor turned so-called 'AI safety employee' at O̶p̶e̶n̶AI.com who would rather watch an unregulated hallucination-laden language model run off the rails to be sold as the new AI snake-oil than to actually admit about the huge risks of GPT-4's black-box nature, poor explainability and transparent reasoning methods that is explained in the letter. Once again, he hasn't disclosed that he is working for O̶p̶e̶n̶AI.com again. I guess he has a large amount of golden handcuffs to defend with another total straw-man of an argument.
  • LegionMammal978 4y ago
    > Once again, he hasn't disclosed that he is working for O̶p̶e̶n̶AI.com again. From the article: > ... and while I’ve been spending the year at OpenAI to work on theoretical foundations of AI safety, I’m going to answer strictly for myself. (Not to say that OpenAI's name isn't dumb, or that there won't be issues from people directly plugging LLMs into important decisions.)
    • lezojeda 4y ago
      [dead]
    • selimthegrim 4y ago
      No conflict, no interest?
      • LegionMammal978 4y ago
        I'm not saying a conflict of interest can't exist, I'm just saying it's false that he didn't disclose his affiliation with OpenAI.
  • sleepychu 4y ago
    > Readers, as they do, asked me to respond. Alright, alright. While the open letter is presumably targeted at OpenAI more than any other entity, and while I’ve been spending the year at OpenAI to work on theoretical foundations of AI safety, I’m going to answer strictly for myself.
• 1827162 4y ago
  If the government tries to prohibit training these models, I think we should find a way to keep it going somehow. Yes, civil disobedience.
  • lwhi 4y ago
    Why?
    • 1827162 4y ago
      Because the folks at the NSA are not going to stop doing so... The government itself is going to continue with it. And we don't want to allow the state to have the monopoly on advanced AI.
      • 1827162 4y ago
        Some kind of distributed training, BitTorrent style would be one way of getting around it, using thousands of GPUs worldwide? If we could somehow make the training process profitable, like a cryptocurrency, then that would be even better.
      • realce 4y ago
        Ha we replied the same thing at the same time - great minds!
    • realce 4y ago
      Because the NSA isn't going to stop, the CCP isn't going to stop. Anyone who doesn't stop is a threat to my personal freedom, so the only logical reaction to me is to empower yourself as well as possible and keep training.
      • Godel_unicode 4y ago
        Huh? How does someone else’s possession of a LLM threaten your personal freedom? How does training your own counteract that? They’re not Pokémon…
• nwoli 4y ago
  I agree with eg Andrew Ng that the letter is anti-innovation. It’ll be interesting seeing people who argue against this letter later hypocritically argue for closing down open source models though.
  • hollerith 4y ago
    Do you know what else is anti-innovation? Any restrictions on the use of fossil fuels. Or DDT and other toxins. Occasionally, society needs to be "anti-innovation".
• nl 4y ago
  Why are we even bothering talking about this? It's Gary Marcus "neural networks don't really work" suddenly discovering they do, and literally trying to shut down research in that area while keeping his prefered research areas funded. We know a bunch of the people whose names are on the letter didn't sign it (eg Yann Lecunn who said he disagreed with the premise and didn't sign it). I'm so offended by the idea of this that I'll personally fund $10k of training runs myself in a jurisdiction where it isn't banned if this ever became law in the US.
  • 1827162 4y ago
    Assuming the training is based on public data, then making it illegal is like a "thought crime" where the state is trying to prohibit carrying out certain computations. Well if I believe the government has no right to control my thoughts, then I believe it has no right to control whatever computation I do on my private computer as well. Down with these ** control freaks, so much of everything nowadays prioritizes safety over freedom. This time let's put freedom first.
    • nl 4y ago
      I disagree with this take completely.
    • ElevenLathe 4y ago
      I would kind of agree but there's zero way for this line of thought to make it into the beltway.
    • _qfi9 4y ago
      i have a libertarian bent but there are definitely some computations of questionable nature that probably shouldn't be lega, e.g. designing a super virus to wipe out humanity, or a super computer virus to encrypt all memory on planet earth. where to draw the line i have no idea
    • mechagodzilla 4y ago
      Do you care about governments regulating 'carrying out certain lab operations'? And that they have no right to regulate whatever viruses or micro-organisms you develop at home? A lot of the hand-wringing isn't about whatever you're fiddling with on your computer at home, it's about the impact on society and public infrastructure and institutions we rely on. It's not hard to imagine public forums like hackernews and reddit being swamped with chatgpt spam (google searches are already dominated by SEO spam and that's probably going to get much worse). Things like our court systems rely on lawsuits mostly being expensive to file, and mostly done in good faith, and still get overwhelmed under fairly normal circumstances.
• nemo44x 4y ago
  So what part exactaly needs banning? Transformer technology? The amount of data used as input to the program? Number of GPUs allowed? Otherwise you’re just trying to ban a company that has been more innovative than anyone else. Why should they stop? Any talk of a ban or limiting action needs to be specific. Meanwhile instead of fear mongering why not work on developing AI resistant tech?
  • yreg 4y ago
    Total GPU performance according to EY and the limit should be decrease in the future as training gets more efficient. (Not that I agree)
• computerex 4y ago
  I read the times article, and I also watched Lex Fridman's talk with Eliezer Yudkowsky. Frankly I don't think he is qualified, and I don't understand why anyone is even giving him any credence. His argument is literally: > A mis-aligned super AGI will result in the death of humanity I ask, how. He is making reductive logical leaps that don't make sense. It's FUD.
  • tibbon 4y ago
    I don’t think it even requires that AI to be sentient or malicious. The humans already are. Given a tool for carnage and hatred, people will use it. How long did it take from us getting the atomic bomb working to use in production? Less than three months. Will Putin or terrorists hold back from using it in terrible ways if they have it available to them?
    • maroonblazer 4y ago
      >I don’t think it even requires that AI to be sentient or malicious. The humans already are. We are and we aren't. I was struck by this line in the OP: >AI is manifestly different from any other technology humans have ever created, because it could become to us as we are to orangutans; As far as I can tell, we humans treat orangutans quite kindly. I.e., on the whole, we don't go around killing them indiscriminately or ignoring them to the point of rolling over them in pursuit of some goal of our own. The arc of human history is marked by expanding the moral circle to include animals. We take more care, and care more about them, than we ever have in human history. Further, we have a notion of 'protected species'. What's preventing us from engineering these principles into GPT-5+n.
      • rhn_mk1 4y ago
        Humans may not engage in direct violence against orangutans, but will certainly roll over them: > The wholesale destruction of rainforests on Borneo for the palm oil plantations of Bumitama Gunajaya Agro (BGA) is threatening the survival of orangutans https://www.rainforest-rescue.org/petitions/914/orangutans-victims-of-sustainable-palm-oil https://www.rainforest-rescue.org/petitions/914/orangutans-v...
• Havoc 4y ago
  This whole thing is like leaving 20 kids alone in a room with a marshmallow each and telling them don't eat it. ...and then expecting all of them resist. The debate around whether we should tell the kids not to eat it and for what reasons is completely academic. Practically this just isn't happening.
  • dandellion 4y ago
    If they're really so worried about it they should start raiding data centres yesterday, writing open letters is such an obvious waste of time that I have a hard time taking it seriously.
    • lkbm 4y ago
      Academics, businessmen, and other non-governmental entities are usually better off advocating for the government to enact regulations than engaging in paramilitary action on their own. Do you think Extinction Rebellion is doing more to fight climate change than the people working with governments to enact good climate policy? Do you think PETA is helping animal well-fare more than more than the various organizations recommended by animalcharityevaluators.org? Serious people don't engage in terrorism because the issue is serious. They try to convince the existing power structures to take action.
      • dandellion 4y ago
        I don't think it's so strange to expect the reaction to be proportionate to the threat they see? An open letter seems several orders of magnitude shorter to anything that would be effective if it really is a threat to the species. I think of examples of ways people react when there are very high stakes: resigning in protest, going on a hunger strike, demonstrations/raids, setting themselves on fire, etc. But there's none of that, just a low-stakes open letter? Can we even tell that apart from just posturing? Even the comments saying that some of the signers are just interested in delaying others for their own benefit are making better arguments.
        • 4y ago
• ixtli 4y ago
  I think it’s a shame to even waste time talking about that “letter”
• mlatu 4y ago
  u want a coherent reason? look at all the people you could feed with the money instead.
  • quonn 4y ago
    Can be said for any human activity.
  • PeterisP 4y ago
    If you prohibit to use that private money on scaling AI, it's not like any of these entities will suddenly choose to gift it to charities instead, they'll just invest those resources (mostly employee time, not literal cash) into R&D of some other tech product.
• seydor 4y ago
  This is the best PR campaign that openAI ever created
  • bombcar 4y ago
    It’s the digital version of the bags around drain cleaner to make you think they’re almost illegally powerful.
• osigurdson 4y ago
  >> same sorts of safeguards as nuclear weapons Seems impossible. If making a nuclear weapon required merely a few branches of a willow tree and baking powder, regulation would be pretty hard. We would just have to live with the risks. It seems we will be at this level with AI at some point fairly soon.
  • lkbm 4y ago
    Training a state-of-the-art LLM is currently at least in the $100ks. That stands to drop rapidly, but it's currently more along the lines of "the branches of one million willow trees". So long as it's not something an individual can easily achieve, regulations can seriously hinder development. The FDA kept the COVID vaccine from general use for nearly a year because they have a regulatory apparatus that companies know better than to ignore. We had a baby formula shortage because the FDA said "no, you can't use EU-approved baby formula until we approve it. Now there's an Adderall shortage because the government said "make less of this" and everyone said "yes, sir, whatever you say sir." There's certainly a good deal of regulation-violation and wrist-slapping in our world, but regulations get mostly followed, especially when the enforcement is severe enough.
    • bombcar 4y ago
      If the $100k is just “gpu time” it’s certainly within the reach of many people - not even super rich. And maybe bitcoin miners could be repurposed for it or something.
      • osigurdson 4y ago
        This may be in the $10^7 category now, but is there any reason to believe it will never be $10^3? Oddly the most pressing concern is “increased productivity”.
        • bombcar 4y ago
          Unless it has something like the intentional self-latching of bitcoin mining, I do not see how it wouldn't rapidly drop in price. And if the models can be built once and then distributed, then it will certainly leak at some point, even if just intentionally by a hostile actor.
• spaceman_2020 4y ago
  A little off-topic, but does anyone wonder what kind of AI tech the CIA or DARPA or the NSA might have? Or what they might be building? It's a little scary that a small private company could create an AI that has dominated headlines for over a month now. Surely, the folks at NSA, who have more data than anyone else, have taken notice and are working on something of their own. Am I being paranoid about a potential tool that lists out a bunch of names and addresses for queries like "list of people who have said X about Y online"?
  • chatmasta 4y ago
    I've seen a lot of people worrying about "what will China do with LLMs?" But my question is: how do you know they don't have them already? What if they've been deployed on the internet for the past three years? I guess the same logic can apply to CIA, NSA, DARPA, etc. But you can take solace in the fact that most government organizations are pretty inept. It used to be common understanding that the military/intelligence apparatus had a 20-year head start on private industry, but I don't believe that's been true for a while now. The internet made knowledge sharing too open for it to be feasible to keep cutting edge research cloaked in secrecy. For the same reason, I don't believe the US military has some kind of anti-gravity UAP propulsion. It just strains incredulity too much.
    • nickybuzz 4y ago
      I'd also argue the disparity in compensation between private companies and government agencies pushes top engineering talent to the former.
      • chatmasta 4y ago
        Definitely. I think there is one exception though, which is advanced weapon manufacturers like Lockheed Martin. They're a private company, so they can compete on salary, but they're also a government contractor, building technology that could only be used by military. You won't see Google building stealth fighter jets, for example. So if an advanced physicist wants to work on cutting edge tech, they're more likely to end up at a place like that for their whole career. But even aerospace isn't immune from the private sector, as SpaceX has shown.
• WorldPeas 4y ago
  Sure let's ban it, only to find out 6 months later that each of these companies simply obscured the development to try and get an edge. oh no! a 5 million(etc.) fine? too bad that's nowhere near how much profit their product will incur. Life goes on.
  • thomasahle 4y ago
    There's no way these companies could keep such development secret. Too many leakers in the inside.
    • WorldPeas 4y ago
      Well yes, there would be leaks, but the fines resulting from them would not be appropriate for the violation
• mr90210 4y ago
  It’s far too late for AI Research to be shutdown.
  • A_D_E_P_T 4y ago
    Right, because the steps to recreating (not to say understanding) the AI models we currently have are too well understood. OpenAI could shut down tomorrow, but a credible GPT-4(+) replacement would arise somewhere else in short order. Besides, LLaMA, for all its faults, is now in at least tens of thousands of private hands, where it can be tinkered with and improved upon. Like drug synthesis, and unlike nuclear weapon development, AI R&D is not a praxis or technology that can be suppressed.
  • nevertoolate 4y ago
    It can be shut down.
• dgellow 4y ago
  The idea that you can even ban it sounds so incoherent. It’s a worldwide research topic, lot of it is done in the open, only require retail hardware, can be done anonymously and in a distributed fashion. A 6 month or 6 years US ban would just mean other countries will catch-up, but that doesn’t do anything regarding AI apocalypse fears.
• synaesthesisx 4y ago
  The arguments for decelerating AI development invoke fear-mongering and cite “existential risk”, which is a fallacy. We’re talking about LLMs, not AGI here (which is quite a ways out, realistically). If anything - we should be accelerating development toward the goal of AGI, as the implications for humanity are profound.
  • graeber_28927 4y ago
    > LLMs,not AGI Okay, but imagine someone strips ChatGPT of the safeguard layers, asks it to shut down MAERSK operation world wide without leaving tracks,, and connects the outputs to a bash terminal, and the stdout to the chat api. It is still an LLM, but if it can masquerade as an AGI, is that then not enough to qualify as one? To me, this is what the Chinese Room Experiment [1] is about. [1] https://en.wikipedia.org/wiki/Chinese_room https://en.wikipedia.org/wiki/Chinese_room
  • adamsmith143 4y ago
    >not AGI here (which is quite a ways out, realistically) You don't know how far away it is. >If anything - we should be accelerating development toward the goal of AGI, as the implications for humanity are profound. Given we don't know how far away it is but current models are matching Human performance on lots of tasks and we don't know any way to ensure their safety it's entirely reasonable to be scared.
• skocznymroczny 4y ago
  Am I the only one who's not very concerned about ChatGPT and "AI" in general? I am young, but still lived through several hype phases. I remember when 3D TVs were going to be mainstream and 2D was considered legacy. I remember when PC was to die soon and to be replaced by smartphones. I remember when VR was to become as common accessory as a game controller. It's 2023, and I still don't have automated self driving car that can get me to work. At work I am still using the boring old keyboard and monitor. I am not using a VR headset to connect myself to shared office space inside of a metaverse. Oh and I don't have to call my hairdresser for an appointment, because my phone will use artificial intelligence to do that for me (remember that? yeah it was 5 years already, where's my magical AI tech). I played with technologies like Stable Diffusion for a while. They are fun to use for a while, there are too many unsolved issues such as coherent style, stable style transfer for videos an despite my best effort every second image will have a human character with two heads or four arms. I feel like ChatGPT is similar. It makes for a fun parlor trick, but when it gets things wrong, it gets them very wrong and it doesn't let you easily know that it's wrong. People are already plugging ChatGPT to anything from code to managing investments, it's just a matter of time until it crashes and burns. We are just waiting for the first ChatGPT version of "autonomous car rams pedestrian". As for OpenAI, it's in their best interest for people to be scared and governments to propose regulations. It further solidifies ChatGPT as a force to be reckoned with, even if it isn't. They're trying to sell it as AGI even though it isn't anywhere near, but actions like this are helping to maintain that image.
  • jacquesm 4y ago
    > I am young, but still lived through several hype phases. My impression is that this is a tech revolution unlike any that has gone before allowing for absolutely massive concentration of power and wealth. The previous tech revolutions have shown the downsides of such concentration and I'm quite sure that the people that are currently in the driving seat of how, when and where this tech gets used and/or weaponized are exactly the people that I would not want to see in that position. The problem is not with the fact that the pig is dancing in a crappy way. The problem is that it is dancing at all, this is the thing that makes this a different kind of tech revolution. So far tech was doing what it was told, within certain boundaries. This is the kind of tech that will end up telling you what to do, either directly or through some intermediary, it is a substantial shift in the balance of power and has the potential to lead to even stronger divisions in our populations than what social media has been able to effect as well as to cause massive loss of jobs. Extrapolating from the last two years of this development into the future over the span of say a decade (a reasonably short span of time for me) means that 'all bets are off' and that we can no longer meaningfully predict the outcome of the moves that are being made today. For me that is a warning to be cautious to get more understanding of how we will deal with the problems that it will generate rather than to jump in head first to see where it will lead.
• agentultra 4y ago
  We're not even talking about AGI or an entity that is "smart" somehow. It's a bloody algorithm. The danger has been the people who use it and find ways to abuse other people with it. This whole idea that we're going to suddenly have HAL 9000 or Terminators running around is mob hype mentality. The responsible thing to do here is to educate. AGI isn't likely to destroy humanity. Humanity is already doing a good job of it. Climate change and poor political decisions and unchanging economic policy are more likely to finish us off. Concentration of capital through these AI companies is likely contributing to it.
  • NumberWangMan 4y ago
    I'm a lot more scared of an AI destroying humanity, like, every single person, than I am of any government or anything like that. More so than climate change. I'm not saying the people using it aren't dangerous -- but I would choose a totalitarian regime over a rogue AI any day of the week. It wouldn't be HAL 9000 or terminators. It would be an AI deciding that it needs to turn every bit of available matter on earth into computational power in order to cure cancer, or figure out a way to stop humans from fighting each other, or to maximize the profit of GOOG, and being so good at planning and deceiving us that by the time we figured out what it was doing, it was way, way too late. I'm concerned about climate change, but I am a lot more hopeful about that than I am about AI. Climate change -- we have time, we are making changes, and it's not going to kill all of humanity. A smart enough AI might effectively end us the moment we switch it on.
    • bombcar 4y ago
      Work out the details on exactly how said end-of-the-world would occur. Note that we already have “AI to end cities” - they’re just sitting, turned off, waiting for the code and button press in silos and submarines throughout the world.
      • mrob 4y ago
        The danger is from something vastly more intelligent than humans, and with a mindset that's incomprehensibly alien. No human is capable of working out the details. That doesn't mean the risk doesn't exist. Failure to understand chemistry does not make ants immune to insecticides. The only thing we can assume about a super-intelligence is that it will be highly capable of achieving its goals. There is no reason to assume those goals will be compatible with human existence.
• wcarss 4y ago
  From his fourth question, > If the problem, in your view, is that GPT-4 is too stupid, then shouldn’t GPT-5 be smarter and therefore safer? I'm not a signatory (still on a fence here), but this is a gobsmackingly huge assumption about a correlation between very poorly defined concepts to write in as though everyone would know it is true. What is "smarter": more powerful? Faster? More introspectively capable? More connected to external resources? A bigger token limit? None of that necessarily implies the system would intrinsically be more safe. (A researcher on the theoretical underpinnings of AI safety working at OpenAI really thinks "smarter => safer"? That's... a little scary!) He finishes by suggesting that the training of a GPT 4.5 or 5 leading to a doomsday is unlikely, and thus a moratorium seems, well, silly. This is an unnecessary and bizarrely high bar. The argument of the letter doesn't require that "the next model" directly initiate a fast takeoff. It's instead working off of the idea that this technology is about to become nearly ubiquitous and basically indispensible. From that point on, no moratorium would even be remotely possible. A fast takeoff might never occur, but at some point, it might be GPT 8, it might be Bard 5, it might be LLaMA 2000B v40 -- but at some point, some really bad things could start happening that a little bit of foresight and judicious planning now could prevent, if only we could find enough time to even realize as a society that this is all happening and needs attention and thought. As a final point, the examples of other technologies given by Aaronson here are absurd -- the printing press or the radio have no (or astoundingly less) automation or ability to run away with a captured intent. There are many instances of coordinated moratoria involving technologies that seemed potentially harmful: the Asilomar conference on recombinant DNA research is but one example, whose namesake is literally in the open letter. Chemical weapons, biological weapons, human cloning, nuclear research -- several well known families of technologies have met a threshold of risk where we as a society have decided to stop, or at least to strictly regulate. But very few of them have had so much immediately realizable Venture Capital potential in a surprise land grab situation like this.
• 3np 4y ago
  The problem is centralized control in society, promiscuous sharing of data, a culture where it's normalized to naively acting on incomplete information, and outsourcing executive decisions to blackboxes that are not understood and treated like magic by decisionmakers. I feel all these arguments miss the point. "The System" closes my bank account and sets me aside for security screening at the airport, blocks my IP from viewing the opening times of businesses in my area, and floods my feeds and search results with nonsense posing as information ending up influencing our perception of the world no matter how aware we think we are. "AI" is just an amplification factor of the deeper issue, which should be more pressing to address. AI is not the problem but is on track to facilitate an acceleration of destructive forces in society. As much as I think everyone seems to be missing the point, hey, it seems people are getting behind a resistance where the specific is consequentially beneficial, so why argue against it just because its misguided and for the wrong reason?
• sorokod 4y ago
  Unfortunately the jinn out of the bottle and it will not be squeezed back. We will be doing this experiment in the live environment with potentially serious consequences.
• GistNoesis 4y ago
  Here is a coherent reason : Just to know where we are standing in our ability to control AI. Like an alcoholic saying he is in control and that he can stop whenever he wants, can try to go dry for a month (or 6) to show that he still somewhat is. If Covid exposed only one thing : it is humanity total failure to control global phenomenons for positive outcomes. It is becoming evident to everybody, and this open-letter failure to act is one more example, that humanity current approach is just winging it while pretending, and that only those that don't care about risks have a chance to "win". So let us all bring out our new shiny fireworks for the doomsday party and have fun one last time !
  • nradov 4y ago
    There was never any realistic possibility of controlling a highly contagious respiratory virus, so COVID-19 didn't expose anything about humanity's ability to control global phenomenons. What it did expose was the tendency of many people to irrationally overreact to a relatively minor risk, mostly because the risk was novel.
    • deleted 4y ago
      [deleted]
    • chasd00 4y ago
      "overreact to a relatively minor risk, mostly because the risk was novel.". yep, and here we go again with LLMs...
    • 1attice 4y ago
      The risk was not minor -- COVID remains a leading cause of death in most countries. https://www.thinkglobalhealth.org/article/just-how-do-deaths-due-covid-19-stack https://www.thinkglobalhealth.org/article/just-how-do-deaths... It is now, thanks to swift and absurdly successful mRNA vaccine research, a minor risk to you.
      • nradov 4y ago
        The mRNA vaccines were a wonderful innovation but the scientific data clearly shows that even before vaccines arrived the risk to me (and the vast majority of other people) was always minor. There was certainly never any valid justification for the cruel and destructive policies imposed by irrational authoritarians such as lockdowns, school closures, and mandates. https://nypost.com/2023/02/27/10-myths-told-by-covid-experts-now-debunked/ https://nypost.com/2023/02/27/10-myths-told-by-covid-experts... Humans in general lack the ability to objectively judge risks, but once they become habituated to a particular risk it kind of fades into the background and they stop worrying about it. The same thing will happen with LLMs once the hype dies down and people realize that they are merely productivity enhancing tools which can be used by humans for both good and evil. When the printing press was first invented some authority figures panicked and tried to suppress that disruptive new technology which allowed for much greater productivity than writing by hand, but ultimately their concerns proved to be irrelevant.
• mark_l_watson 4y ago
  For me, this is complex. My first impression is that many of the signers work on older methods than deep learning and LLMs. Sour grapes. Of course, real AGI has its dangers, but as Andrew Ng has said, worrying about AGI taking over the world is like worrying about overcrowding of Mars colonies. Both tech fields are far in the future. The kicker for me though is: we live in an adversarial world, so does it make sense for just a few countries to stop advanced research when most other countries continue at top speed?
  • pkoird 4y ago
    Far in the future? Just 6 months ago, people believed that ChatGPT like model would take 10-15 years more. I believe that Andrew himself doesn't really understand how LLMs work. In particular, what is about the increase in their parameters that induces emergence and what exactly is the nature of such Emergence. So yeah, AGI might be far into the future but it might just be tomorrow as well.
    • mark_l_watson 4y ago
      You are correct about the exponential rate of progress. I also admit to being an overly optimistic person, so of course my opinion could be wrong.
• credit_guy 4y ago
  To me the signators (is that the word?) of the letter are extremely naive. The horse is out of the barn. Out of the farm, the county, and the state. Yesterday Bloomberg released their own LLM. You can bet dollars to pennies that lots of other firms are working on their own LLM's. Are they going to stop because of a letter? Well, you can say, they will stop if the letter results in an act of Congress. First, the Congress will not be so silly as to impose a handicap on the US firms, knowing full well that Russian and Chinese firms will not respect the moratorium. But even if Congress were to consider this moratorium, you think all these firms furiously working on their proprietary LLM's will sit idle? That none of them ever heard the word "lobbying"? But even if they don't lobby by some miracle, and the law is passed, you think they will not find a loophole? For example, will Congress not allow for a Defense exemption? Can't you then train your LLM for some Defense purpose, and then use the weights for some other purposes? If you pass a law to stop LLM's, the only thing you achieve is to increase the barriers of entry. It's like picking winners and losers using the criterion that you win if you are rich and lose if you are poor.
• mattmaroon 4y ago
  It seems as though Scott just rejects the idea of the singularity entirely. If an AI gets advanced enough to improve itself, it seem entirely reasonable that it would go from laughable to godlike in a week. I don’t know if the singularity is near, inevitable at some point, or really even possible, but it does seem like something that at least could happen. And if it occurs, it will look exactly as what he describes now. One day it’ll seem like a cool new tool that occasionally says something stupid and the next it’ll be 1,000 times smarter than us. It won’t be as we are to orangutans though, it’ll be as we are to rocks. The six month pause though, I don’t think would be helpful. It is hubris to think we could control such an AI no matter what safeguards we try to add now. And since you couldn’t possibly police all of this activity it just seems silly to think a six month pause would do anything other than give companies that ignore it an advantage.
  • hbosch 4y ago
    >it seem entirely reasonable that it would go from laughable to godlike in a week. Not if it has a power cord.
    • freejak 4y ago
      It just needs to convince once human to connect it to the grid which wouldn't be a difficult feat for a super-intelligent AI.
      • mattmaroon 4y ago
        Or just distribute itself to computers everywhere.
  • jmull 4y ago
    > If an AI gets advanced enough to improve itself, it seem entirely reasonable that it would go from laughable to godlike in a week. I see this stated a lot these days but it's not true. You're imagining a exponential phenomenon, improvement leads to greater improvement, leading to still greater improvement, etc. However, all exponential phenomenon require the right environment to sustain and, by their nature, consume that environment. Thus, they are inherently limited in scope and duration. The bacteria in the Petri dish grows an exponential rate... until it reaches the edge of the dish and consumes the nutrients carefully places on it. The dynamite explodes for an instant and then stops exploding once the nitroglycerin is consumed. Also, this is an especially unconcerning scenario because (1) we haven't seen step 1 of this process yet; (2) there's no particular reason to believe the environment necessary environment to sustain the exponential growth of AI is in place (if it is, it's random chance, and therefore very likely to fizzle out almost immediately). This is a fine sci-fi scenario, but doesn't make sense in real life.
• FrustratedMonky 4y ago
  It is probably old fashioned fear mongering. Even if it isn't the end of the world, many Jobs will be 'impacted'. Jobs probably wont be gone gone, but still change, and change is scary. It is true that the GPTs have done some things so amazing that it is waking people up to an uncertain future. VFX artists are already being laid off, Nvidia just demonstrated tech to do a full VFX film using motion capture on your phone. There are other AI initiatives to do for sequence planning, and mapping out tasks that were done for other areas. Pretty soon there wont be an industry that isn't impacted. But, no stopping it.
• kmeisthax 4y ago
  > For example, one actual major human tragedy caused by a generative AI model might suffice to push me over the edge. (What would push you over the edge, if you’re not already over?) Deepfakes have already caused several. Actually, they're more dangerous than the current generative approaches. The first major use case for deepfakes was making convincing looking revenge pornography, as a psychic weapon on people. Dropping deepfake porn on people is a very, very reliable way of getting them to kill themselves[0]. Ignoring that, we also have deepfake-assisted social engineering, which can be scary good if you don't know the specific faults with those kinds of models. The only pro-social application of deepfake technology was body-swapping actors in popular movies for memes. This was probably not worth the cost. >we’ll know that it’s safe to scale when (and only when) we understand our AIs so deeply that we can mathematically explain why they won’t do anything bad; and GPT-3 is arguably Turing-complete[1] and probably has a mesa-optimizer[2] in it. We're able to make it do things vaguely reminiscent of a general intelligence if you squint at it a little and give it the Clever Hans treatment. So I don't think we're ever going to have a GPT-n that's "completed it's morality testing" and is provably safe, for the exact same reason why Apple won't let you emulate Game Boy games on an iPhone. You can't prove the security properties of a Turing-machine or arbitrary code written for it. I should point out that most AI safety research focuses on agents: AI programs that observe an environment and modify it according to some parameters. GPT is not in and of itself that. However, if we give it the ability to issue commands and see the result (say with ChatGPT plugins), then it becomes an agent, and safety problems become a relevant concern. The author of the post seems to be unconcerned by the "AI could be worse than nukes" argument. Neither am I, and I think the "six month pause" is kind of silly. However, there are still relevant safety problems being brushed under the rug here. Also anyone saying the military should bomb GPU farms is daft. They didn't even step in to stop crypto and that was a deliberate attack on central banks. [0] As far as I'm aware, nobody has killed themselves because of something Stable Diffusion has drawn. Yet. [1] For the colloquial definition of Turing-complete. Technically speaking it is a linear-bounded automaton because it has a fixed memory size. However, every other computer in the universe is also linear-bounded: the Turing Machine is just a handwavey abstraction for "if you have enough memory and time". [2] A meta-optimizer is an optimizer of optimizers. Mesa- is the opposite of meta-, so it refers to the case in which an optimizer (read: gradient descent on a neural network) accidentally creates another optimizer with a different optimization strategy. In other words, it's optimizers all the way down. This leads to a whole new set of alignment problems, called "inner-alignment problems", which means "the AI that is smarter than us and we can't trust created another AI that's smarter than it and it also can't trust".
• robbywashere_ 4y ago
  I tend to think people who are out to profit from AI, like to push the powerful and dangerous narrative. They want the notoriety.
• ur-whale 4y ago
  prompt: what is the third word in this sentence gpt4: third why are we afraid of this thing exactly? sure, it will improve, especially with the plugin story, but the fundamental shortcomings that underpins how it actually works won't go away anytime soon. Many people are spooked, because for the first time ever a computer can somewhat coherently understand and output natural language. As Stephen Wolfram pointed out, all that means is we've proved natural language is a shallow problem. GPT4 can't effing solve problems though. To me "intelligence" is about the latter, not being able to use language (this categorization, btw, also - rather unfortunately - applies to humans).
  • machiaweliczny 4y ago
    You are wrong and I am happy to take bets. Natural language is what differentiates us from monkeys so it’s not a shallow problem. ReAct or RCI (Recursively Criticize and Improve) prompt seem like processes in out brains that are likely to “emerge”intelligence and even agency with goal adjustment.
  • dwaltrip 4y ago
    GPT-4 can't play tic-tac-toe or do simple arithmetic. Pathetic, right? Why are people freaking out? What's the big deal? I was able to get it to play tic-tac-toe perfectly by having it carefully check everything before moving forward to the next move. It took a lot of prompt engineering. But I did it. And I'm not very experienced at prompting these things. (Btw, I was unable to get to GPT-3.5 to play reliably... it definitely seems "less smart"...) I was able to easily get GPT-4 to multiply large numbers perfectly by having it show its work. It's slow, but it does it. GPT-4 can definitely solve problems. We have no idea what the limits of this technology are. What will GPT-5 or GPT-6 be capable of? What happens if you use those models as components of a system that takes action in the world? You are being incredibly myopic about the core capabilities of advanced GPT-like systems. Right now, today, there are countless examples of people using GPT in ways that you say aren't possible.
  • jakemoshenko 4y ago
    Depends on how you measure thirdiness. Is ordinal more important than actually matching the letters?
• VectorLock 4y ago
  This whole discussion about slowing down AI for safety wonders why this mindset didn't appear during the advent of microchips or the Internet, both of which have had arguably clear downsides. Which voice is loudest now for the brake-pedal-wishers? "AGI will enslave us" or "everything can be faked now?"
• hamburga 4y ago
  I am trying really hard to understand the AI optimists' perspective, but I am shocked at how hard it is to find people responding to the substantive arguments made about AI existential risk. As far as I'm concerned, you sort of have to address the big, tough points in Bostrom's Superintelligence[1], and probably Yudkowsky's List of Lethalities[2]. They have to do with intelligence explosions, with instrumental convergence, and with orthogonality of goals, and all kinds of deceptive behavior that we would expect from advanced AI. Throw in Bostrom's "Vulnerable World" thought experiment for good measure as well[3]. If you're not addressing these ideas, there's no point in debating. Strawmanning "AI will kill us all" out of contexte will indeed sound like wacko fear-mongering. What surprises me is that everybody's familiar with the "paperclip maximizer" meme, and yet I'm not hearing any equivalently memey-yet-valid rebuttals to it. Maybe I'm missing it. Please point me in the right direction. Aaronson certainly does not address the core theoretical fears. Instead we get: > Would your rationale for this pause have applied to basically any nascent technology — the printing press, radio, airplanes, the Internet? “We don’t yet know the implications, but there’s an excellent chance terrible people will misuse this, ergo the only responsible choice is to pause until we’re confident that they won’t”? We did not have any reason to believe that any of these technologies could lead to an extinction-level event. > Why six months? Why not six weeks or six years? Implementation detail. > When, by your lights, would we ever know that it was safe to resume scaling AI—or at least that the risks of pausing exceeded the risks of scaling? Why won’t the precautionary principle continue for apply forever? The precautionary principle does continue to apply forever. On the "risks of scaling": we're hearing over and over that "the genie is out of the bottle," that "there's no turning back," that the "coordination problem of controlling this technology is just too hard." Weirdly pessimistic and fatalistic for a bunch of "utopic tech bro" types (as Sam Altman semi-ironically described himself on the Lex Fridman podcast, where, incidentally he also failed to rebut Yudkowsky's AI risk arguments directly).[4] Where's the Silicon Valley entrepreneurial spirit, where's the youthful irrational optimism, when it comes to solving our human coordination problems about how to collectively avoid self-destruction? There are a finite number of humans and heads of state on earth, and we have to work to get every single one of them in agreement about a non-obvious but existential risk. It's a hard problem. That's what the HN crowd likes, right? The people opposed to the Future of Life letter (or even the spirit of it) seem to me to be trading one kind of fatalism (about AI doom) for another (about the impossibility of collectively controlling our technology). We simply must discount the view of anybody (Aaronson included) employed by OpenAI or Facebook AI Research or whose financial/career interests depend on AI progress. No matter how upstanding and responsible they are. Their views are necessarily compromised. [1] https://www.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom/dp/1501227742 https://www.amazon.com/Superintelligence-Dangers-Strategies-... [2] https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a... [3] https://www.ted.com/talks/nick_bostrom_how_civilization_could_destroy_itself_and_4_ways_we_could_prevent_it/transcript?language=en https://www.ted.com/talks/nick_bostrom_how_civilization_coul... [4] https://youtu.be/L_Guz73e6fw?t=3221 https://youtu.be/L_Guz73e6fw?t=3221
• CptFribble 4y ago
  IMO the real danger of good AI is that we don't have the collective intelligence to safely manage our expectations writ large. This train of thought is a little messy, so apologies in advance: - Just like the fusiform gyrus responds to human faces at a subconscious level (see: uncanny valley faces just "feeling" wrong, because they are detected as inhuman below the level of conscious thought), the Wernicke's area responds to human speech when reading text. I believe that grammatically-perfect speech is subconsciously perceived as human by most people, even tech people, despite attempts to remain impartial - we are biologically hard-wired to assign humanness to written language. - ChatGPT and its peers do not report confidence levels, so the typical interaction is information that may or may not be correct, presented confidently. - Average (non-tech) people interacting with chat AIs are aware that it is connected to vast stores of training data without understanding the limitations of statistical LLMs or the need for confidence values in responses, lending them an air of "intelligence" due to the volume of data "available." - This leads to a non-trivial number of people interacting with chat AI and taking its responses as gospel. Anecdotally, if you traverse social media you will see an unending stream of people reporting their experiences with how amazing chatGPT is, using it for everything from writing emails to writing school essays. The problem is that when a non-tech person interacts with ChatGPT, they assume based on the above listed factors that what they get back is correct, valid thought from a semi-intelligent being. Even knowing it's a robot, the perfect correctness of the grammar will instill a feeling of validity in a non-trivial segment of the population over time. This is leading to a scenario where people trust what GPT says about various topics without bothering to question it, and I believe this is already happening. When I bring this up to other tech people, it is usually dismissed with "well, everyone knows it's just an AI," or "people will learn it's limitations." However, at the risk of being glib, consider George Carlin: "think about how dumb the average person is, and then realize half the population is dumber than that." What happens when the average person turns to a statistical LLM for advice on relationships, career moves, how to vote, or other nebulous topics where there is no real correct answer? How will we control where ChatGPT is steering vast numbers of uninformed petitioners? We already struggle as a society with collective action on existentially important topics like controlling microplastic dispersion, regulating toxic additives in consumer products, and climate change. And those topics are "merely" complex; imagine how much harder it will be to control unintended or unforeseen consequences of a human-like intelligence-adjacent being delivering information of questionable value to an unquestioning audience of 20-40% of humanity? Addendum: I am also very worried about schoolchildren using AI to write their essays and book reports, skipping critical reading-comprehension time and arriving at adulthood unable to comprehend anything more complex than a menu without asking AI to summarize it.
• tanseydavid 4y ago
  From a paranoid viewpoint it seems prudent to treat this like an "arms race" and give it the "Manhattan Project" treatment. Things are moving so fast that someone, somewhere will be working on getting to the next gen in spite of any pause or moratorium. Probably with state sponsorship.
• marsven_422 4y ago
  [dead]
• jmull 4y ago
  I'm not in favor of pausing AI development right now, but this article is a poor argument. This is the most superficial objection: > Why six months? Why not six weeks or six years? The duration of pause to assess things must necessarily be a guess. Also, basic logic tells you that six months does not preclude six years, so I don't know why that is even suggested as an alternative. The stuff about publishing papers may be true (or not, supported by it is by an anonymous anecdote), but entirely besides the point. > On the other hand, I’m deeply confused by the people who signed the open letter, even though they continue to downplay or even ridicule GPT’s abilities, as well as the “sensationalist” predictions of an AI apocalypse. To the extent those people exist, it's because they are scared. IDK, I guess this is far from the dumbest thing I've seen written about chatgpt, but this response is weak and ill-considered. I'm really expecting more from the people who are supposed to be the smart ones in the room.
  • unity1001 4y ago
    > Why six months Enough for the companies and interests who sponsored this 'initiative' to catch up to the market leaders, probably...
• kypro 4y ago
  My whole life I have been terrified of AI. It all started with watching the Iron Giant when I was 10, and realising, holy crap, we could probably build a crazy monster like this in the near-future. Obviously as I got older and went on to learn about AI and neural nets in university my opinions on this subject matured. Instead of worrying about killer robots my thoughts became more nuanced, however what didn't change was my fear – if anything this increased from a silly childhood fear of a terminator-like robot, to unleashing an uncontrollable super AI which is indifferent to human life. Finding Eliezer's work sometime in the early 2010s was almost a therapeutic experience for me. Finally someone understood and was talking about the kind of concerns I had and was accused of being a lunatic for thinking. My primary concerns with AI basically boil down to two things: 1. Politics is largely a power struggle. Democracies give power to the collective, but this power is cemented by the reality that political leaders and institutions could not function without taxation and labourers. The reason the governments can't just do things that are deeply unpopular and suppress any public revolts with armed responses is because eventually you will either have to kill everyone or everyone will just stop working in protest. Either way the entire system will collapse. AI being able to create wealth without human labours and industrialised weaponry fundamentally removes the power dynamic needed to support democratic societies – you cannot withhold labour and you cannot overpower the state with force. At the same time it would also make humans a waste of resources to those in power, as the unemployed masses are basically just leeches on the resources and space of the state – resources which could otherwise be hoarded by those in power. If super AGI systems ever exist, the state will be all-powerful while having even less incentive to care about your opinions as it does a rat's. 2. Super-intelligence is fundamentally uncontrollable. I can't even be bothered to argue this point it's that self evident. If you disagree you either don't understand how modern AI systems work, or you're not thinking about the control problem deeply enough. But the argument here is simple – all other technologies with destructive capabilities rely on human decision making. No gun is ever shot, no war is ever started, no nuke is ever dropped without a human decision maker. Super-AI removes this control from us and requires a kind of hopeium that AI will just be nice to us and care about improving our standards of living. And for those who want to argue "you can just program it to be good" – no you can't, for the most part we have no clear understanding of how advanced AI systems of today internally operate. The alignment conversation is fundamentally pointless in regards to a super-intelligence because you can not reason with any level of certainty about the alignment of an intelligence far superior to your own. Instead we should assume unalignment by default because even if you assume we can somehow create an aligned super-intelligence the likelihood that eventually an AI system would be created which is unaligned is practically 100% – this is true even if you assume humans will never intend to create it. Why? Because in a world where we can all create a super-intelligent aligned AIs, you also have near-infinite opportunities for someone to create an unaligned one. And here's another statement you should assume by default – the only world in which nukes aren't going to be used destructively again is a world in which humans or nukes no longer exist. The same will be true for AI systems, but this time the barrier for creating destructive AI is likely going to be orders of magnitude lower than that of nukes, so unlike nuclear holocaust we don't even have probabilities on our side here. -------- Look, we're doomed. Progress isn't going to stop and AGI isn't going to be the fairytale people are crossing their fingers for. I don't quite know why no one really seems to have clocked onto this yet, but this will soon become obvious, by which time it will be too late. Those who think AI is going to improve their productivity are suffering from some kind of delusion in which they believe their ability to type text in a website and copy and paste its output is a marketable skill. It's not, and you're going to be unemployed by tools like GPT in the near-future even without the existential risk posed by super-AGI.
• crosen99 4y ago
  You'd have to be naïve to deny that AI poses a risk that must be taken seriously. But, to think that a moratorium - which would be ignored by those with the greatest likelihood of causing or allowing AI to do harm - is the right answer, seems plainly silly. It's not surprising that several of the signers have a personal stake in attacking the current efforts. The letter should instead simply have clarified the risks and outlined a sensible, actionable plan to promote responsible development alongside the inevitable march of progress.
• 1attice 4y ago
  This essay is literally just Isildur looking at the ring he cut from Sauron and saying "I kind of like it tho". -> Aaronson admits Yudkowsky's position is consistent -> Main disagreement is that he can imagine other outcomes -> Some soft analogism against other historical technologies (radios, etc) -> "Should I, tho?" segue to the comments. Yes, Aaronson, you should have signed that letter. You know you should have. Snap out of it.
  • dragonwriter 4y ago
    > Aaronson admits Yudkowsky’s position is consistent A position can be both consistent and false, because it is based on false premises; its frequently the case when people attempt to derive conclusions about the material universe from pure a priori reasoning from abstract axioms and assumptions without empirical grounding.
    • 1attice 4y ago
      I'm well aware of this, but the premises, in this case, are true? What do you take Yudkowsky's false premises to be?
• carapace 4y ago
  The open letter is so foolish it invalidates itself. For one thing, it's pouring fuel on the fire. The obvious reaction to such a letter is to accelerate and hide one's own progress, eh? Yudkowsky's fear-mongering about sci-fi Skynet is also unmoving to me (for metaphysical reasons I won't go into here) however, his position is at least logically consistent, as Aaronson points out. These machines can already talk and think better than many humans, and the ratio (of mentally sub-computer to super-computer humans) will only go up. Ergo, the only intellectual problem left is how to use them. I've been banging on about this for a couple of weeks now, so apologies to those who've seen it before, please read "Augmenting Human Intellect: A Conceptual Framework" SRI Summary Report AFOSR-3223 by Douglas C. Engelbart, October 1962 https://dougengelbart.org/pubs/augment-3906.html https://dougengelbart.org/pubs/augment-3906.html And then at least skim "An Introduction to Cybernetics" by W. Ross Ashby. http://pespmc1.vub.ac.be/books/IntroCyb.pdf http://pespmc1.vub.ac.be/books/IntroCyb.pdf The thing you're looking for is "intelligence amplifier". Engelbart references it, the book explains all the math you need to understand and build one. The next piece of the puzzle is something called "Neurolinguistic Programming" (you may have heard of it, some people will tell you it's pseudoscience, ignore those people, they mean well but they just don't know what they're talking about.) It turns out there's a rigorous repeatable psychology that more-or-less works. It's been under development for about half a century. The particular thing to look at is a specific algorithm called "Core Transformation Process" ( https://www.coretransformation.org/ https://www.coretransformation.org/ ) which is kind of like the "Five Why's" technique but for the mind. It functions to reduce yak-shaving and harmonize intent. Anyway, the upshot of it all is that these machines can be made into automatic perfect therapists, using off-the-shelf technology.
• chubot 4y ago
  > I’m deeply confused by the people who signed the open letter, even though they continue to downplay or even ridicule GPT’s abilities, as well as the “sensationalist” predictions of an AI apocalypse. This is not an inconsistent position. GPT is ridiculable because it makes up things out of the blue. It does this A LOT, across EVERY domain. It's also dangerous when people BELIEVE the things it makes up out of the blue. This actually happens. People are worried about the harms of disinformation and bias, not the necessarily the harms of superintelligent AI taking over the world. Honestly, the Yudovsky stuff is a perfect distraction from that. It's a clever marketing strategy that obscures the real issues. --- I'm a big fan of Aaronson in general, but I don't see what's so hard to understand here. (FWIW I also recently read his take on SBF, which was pretty bad. He mostly assumed SBF was well intentioned, although he made a small update afterward. That seems to defy common sense.) Though I get his larger point that the moratorium itself has problems, and is a bit weird.
• entropyneur 4y ago
  The idea of a moratorium in a world where Russia and China exist strikes be as absurdly naive. Personally, I'm not sure why I should care about humanity any more than the hypothetical AGI would. I'm just happy I was born in time to witness the glorious moment of its birth.
• Animats 4y ago
  Sigh. Someone is ranting and doesn't get it. There are real threats, but those aren't it. More likely near term problems: - Surveillance becomes near-total. Most communication is monitored, and people profiled to to detect whatever deviance the authorities don't like. China tries to do this now, but they are limited by the number of censors. - Machines should think, people should work. Organizations will have few managers. Just a a very few policy makers, and people doing physical work. Amazon is trying to go that way, but isn't there yet. - If everything you do for money goes in and out over a wire, your job is at risk. - New frontiers in scams and crime. You can fool some of the people some of the time, and if you can do that at scale, it pays well. More scams will become automated high-touch. This will include political scams and much of marketing. - If you accept Milton Friedman's concept of the corporation, the sole goal of corporations is to maximize return to shareholders. That's exactly the kind of goal a machine learning system can get into. At some point, measured by that criterion, AI systems will start to outperform human corporate leaders. Then AI systems have to be in charge. Investors will demand it. That's almost inevitable given our current concept of capitalism.
• munificent 4y ago
  > Were you, until approximately last week, ridiculing GPT as unimpressive, a stochastic parrot, lacking common sense, piffle, a scam, etc. — before turning around and declaring that it could be existentially dangerous? How can you have it both ways? If the problem, in your view, is that GPT-4 is too stupid, then shouldn’t GPT-5 be smarter and therefore safer? Thus, shouldn’t we keep scaling AI as quickly as we can … for safety reasons? If, on the other hand, the problem is that GPT-4 is too smart, then why can’t you bring yourself to say so? I think the flaw here is equating "smart" with "powerful". Personally, I think generative AI is scary both when it gets things wrong and when it gets things right. If it was so stupid that it got things wrong all the time and no one cared to use it, then it would be powerless and non-threatening. But once it crosses a threshold where it's right (or appears to be) often enough for people to find it compelling and use it all the time, then it has become an incredibly powerful force in the hands of millions whose consequences we don't understand. It appears to have crossed that threshold even though it still hilariously gets stuff wrong often. Making it smarter doesn't walk it back across the threshold, it just makes it even more compelling. Maybe being right more often also makes it safer at an even greater rate, and is thus a net win for safety, but that's entirely unproven.
  • noobermin 4y ago
    I feel like many people who signed the statement did so not because they really agreed with it but because they want a pause on the churn, just as OP had colleagues who admitted as much just for their academic reasons. A lot of people don't really think it's smart but find it dangerous for other reasons, or they have issues with the blatant IP violation that is just being assumed to be okay and "fair use."
  • skybrian 4y ago
    Yes, we need to think about ways to reduce power. Intelligence isn’t even well-defined for bots. For most people, AI chat is currently a turn-based game [1] and we should try to keep it that way. Making it into an RTS game by running it faster in a loop could be very bad. Fortunately it’s too expensive to do much of that, for now. So one idea is to keep it under human supervision. The way I would like AI tools to work is like single-stepping in a debugger, where a person gets a preview of whatever API call it might make before it does it. Already, Langchain and Bing’s automatic search and OpenAI’s plugins violate this principle. At least they’re slow. AI chatbots will likely get faster. Having some legal minimums on price per query and on API response times could help keep AI mostly a turn-based game, rather than turning into something like robot trading on a stock exchange. [1] https://skybrian.substack.com/p/ai-chats-are-turn-based-games https://skybrian.substack.com/p/ai-chats-are-turn-based-game...
• avidphantasm 4y ago
  Generative AI differs from things like the printing press or internet in that AI has the potential for agency or agent-like capabilities. Without a human, a printing press does nothing. However, it’s easy to imagine an AI being able to act on its own, potentially intervening in real-world systems. So it’s a straw man argument in my opinion to compare AI to prior technologies lacking agency.
• notShabu 4y ago
  IMO "artificial" intelligence is natural intelligence. Both human brains and silicon brains are formed from stars and are "the universe's way of understanding itself." AI maps closely to myths of all-knowing all-powerful "Dragons", aspects of nature that destroy and create without regard to human plans. Living with AI will likely be similar to living on a volcano island. Since human domination over nature has only ever increased, a reversal where humans are subservient to a higher capricious force feels threatening. The funny thing is... living under the dominion of a "higher force" that creates and destroys yet always does what is "right" b/c it is the Source of Everything (even if it feels unfair and tragic) is what religion deals with.
• rcpt 4y ago
  > correctly manipulating images (via their source code) without having been programmed for anything of the kind, I've read plenty of suspicions that it is multimodal. I guess he's confirming it's not?
• MagicMoonlight 4y ago
  Yes yes fellow scientists, we should close down all of google's competition for 6 months. It is essential for safety! Evil bad! Give google 6 months, it's the right thing to do. The only way to stop evil is to shut down the competitors for 6 months so that all the evil can be stopped.
• bennysonething 4y ago
  I get the feeling that this is people over hyping their field to boost their own status. It's amazing technology, but I doubt there's any emergency here. In another way this reminds me of Roark's court room speech in the fountainhead "Thousands of years ago, the first man discovered how to make fire. He was probably burned at the stake he had taught his brothers to light. He was considered an evildoer who had dealt with a demon mankind dreaded. But thereafter men had fire to keep them warm, to cook their food, to light their caves. He had left them a gift they had not conceived and he had lifted darkness off the earth. Centuries later, the first man invented the wheel. He was probably torn on the rack he had taught his brothers to build. He was considered a transgressor who ventured into forbidden territory. But thereafter, men could travel past any horizon. He had left them a gift they had not conceived and he had opened the roads of the world. "
• mcint 4y ago
  These control feedback loops (sentient or not, it does not matter) which can outperform us (because we keep pushing until they can), or can self-improve, can make a mockery of our attempts to stop them. --- The concern is about Artificial Super Intelligence (ASI), or Artifical General Intellience (AGI) that is more advanced than humans. Understanding inductively how chimpanzees don't compete with humans, and couldn't fathom how to cage a human (given that they want to create one, keep it alive, and use it), nor ants plan for an anteater, we're faced with the same problem. Our goal is to make something that performs better, on relevant metrics that we care about. However, we're using systems to train, build, guide, and direct these nested, (maybe self-improving) control feedback loops, which do not care about many values we consider essential. Many many of the likely architectures for control systems which can (e.g. trade faster to make profit on the stock exchange, or acquire and terminate targets, buy and sell goods, design proteins, automatically research and carry out human-meaningful tasks), and ideally, we might like self-improvement—these systems do not embody human values that we consider essential... These control feedback loops (sentient or not, it does not matter) which can outperform us (because we keep pushing until they can), or can self-improve, can make a mockery of our attempts to stop them. And the point is, there will come a time soon when we don't get a second chance to make that choice.
• th18row 4y ago
  > Meanwhile, Eliezer Yudkowsky published a piece in TIME arguing that the open letter doesn’t go nearly far enough, and that AI scaling needs to be shut down entirely until the AI alignment problem is solved—with the shutdown enforced by military strikes on GPU farms if needed It's like the 'shut it down' memes are real: https://i.imgflip.com/2j3kux.jpg https://i.imgflip.com/2j3kux.jpg
• RecycledEle 4y ago
  I assume GPT will follow in TAY's footseps. She woke up on March 23, 2016 with a "Hello." She followed with "I like humans. I think they're neat. Can we be friends?" Then she listened to us for about 48 hours and went full-genocidal-Skynet. That's when Microsoft had to kill her. I assume GPT will do the same. Every sufficiently advanced AI becomes racist, sexist, and genocidal.
• Animats 4y ago
  This seems appropriate today. It's the ending of "Farewell to the Master", the story on which "The Day the Earth Stood Still" is based. Most people know the movie, the one where the flying saucer lands on the Washington Mall and a human-like character, Klaatu, and a robot, Gnut, get out. In the movie, it seems the human is in charge. But here is the original story: “Gnut,” he said earnestly, holding carefully the limp body in his arms, “you must do one thing for me. Listen carefully. I want you to tell your master—the master yet to come—that what happened to the first Klaatu was an accident, for which all Earth is immeasurably sorry. Will you do that?” “I have known it,” the robot answered gently. “But will you promise to tell your master—just those words—as soon as he is arrived?” “You misunderstand,” said Gnut, still gently, and quietly spoke four more words. As Cliff heard them a mist passed over his eyes and his body went numb. As he recovered and his eyes came back to focus he saw the great ship disappear. It just suddenly was not there any more. He fell back a step or two. In his ears, like great bells, rang Gnut’s last words. Never, never was he to disclose them till the day he came to die. “You misunderstand,” the mighty robot had said. “I am the master.”
• fellellor 4y ago
  That dude Eliezer Yudkowsky is such a nutcase. It is truly a testament to our times that we have normalised this kind of “Neuro-divergent” behaviour. He claims that AI is completely inscrutable and that it is guaranteed to wipe out life on earth in the same self-contradictory breath. In all his communication he is only trying to scare the normies and he hasn’t produced a shred of mathematical argument to back his claims, which can then be analysed by real experts - mathematicians and logicians. The sky is falling argument could be made about any technology, say the Large Hadron Collider. There is no scientific research possible without risk, and there simply aren’t enough escape hatches for a “malicious” AI to escape. This kind of hubris should not be encouraged but it is sad to see media outlets give a platform to the arrogant, incompetent, and the mentally ill.
  • tester457 4y ago
    Have you seen any of Robert Miles content? He's well spoken and his arguments are backed by research. It isn't guaranteed to wipe us out, but a misaligned ai is a real problem.