29 ms·

The Darwinian argument for worrying about AI

144 points by da39a3ee 3y ago

EamonnMR 3y ago
TIME must have gotten a lot of clicks off of their Yudkowsky op ed. As always the answer isn't 'cool, let's regulate AI to limit its profitability, thus limiting AI development' but rather 'we should keep throwing money at it, just making sure we throw money at the right particular people.' Yudkowsky didn't want to bomb all data centers, just the ones that wouldn't comply with his regime. Similarly: "We need research on AI safety to progress as quickly as research on improving AI capabilities. There aren’t many market incentives for this, so governments should offer robust funding as soon as possible." I'm reminded of the tech ceo caricature in Don't Look Up who, when presented with an incoming asteroid ready to wipe out the earth, hatches a plan to profit from it.
- yanderekko 3y ago
  > As always the answer isn't 'cool, let's regulate AI to limit its profitability, thus limiting AI development' but rather 'we should keep throwing money at it, just making sure we throw money at the right particular people.' Yudkowsky didn't want to bomb all data centers, just the ones that wouldn't comply with his regime. Who would have to own the data centers for Yudkowsky to support "throwing money at profit-driven AI development"?
scoofy 3y ago
I think one point missed by this is that the vast majority of outcomes for species in a "darwinian" environment is extinction. We look at evolution with a very rosy lens because we ended up on the top of the food chain. Unintelligent prokaryotes far and away dominate the "darwinian" world. Intelligent species have vastly less control over their environment as they think they do.
sneak 3y ago
If we build AGI, it will be a child of human beings. It will be the best and most effective human ever made.
ElevenLathe 3y ago
The central argument here -- that nonhuman entities in competition with one another will develop amorally and to the detriment of human wellbeing -- is also a reason we should be worried about corporate capitalism.
asdff 3y ago
This could end up going very dangerously for the planet depending on how these AIs are set up with their incentives. If the neoconservatives who run the world set up these models to their own benefit, we are probably going to be doomed with a dead planet in a few centuries. Imagine if they were set up to go for profitability above all else as a capitalist would expect. The planet would be destroyed before long. Newly identified resources will be squandered as quickly as possible for short term gains realized in a single investors lifetime, over considering long term health of the planet which likely means not prioritizing quarterly profits and leaving behind some fruit on the tree so to speak. An AI might be used to more easily make what would be coldhearted or uncomfortable decisions for a person, like deciding what communities are the most profitable to invest in versus considering which might require the most need (which probably means sacrificing profitability to pay more costs). However, if the AIs were set up to incentivize collective good over short term profit taking for well leveraged individuals, then this could be a revolution in terms of work/leisure time for the masses, akin to what past philosophers must have expected from us in a future with more available resources and a lessened need for labor.
danmaz74 3y ago
The competitive pressures described in the article aren't specific to AI: any technology which can give wealth or power will be used in risky or unethical ways by companies, countries, organisations in general. For a specific example, humanity already developed a technology which can erase the human race - nuclear weapons. "Natural selection" could affect AI in a specific way if AIs started self-reproducing, AND if there was some mechanism similar to how the reproductive success of living organisms promotes the diffusion of genes which contribute to that success. But what would be the equivalent of "reproductive success" for computer programs?
- Myrmornis 3y ago
  The equivalent of "reproductive success" for computer program A would be something like: the number of other executing computer programs present at a later point in time whose source code descends from the source code of A. The source code may be propagated to other executing computer programs because of the actions of either humans or AIs.
- visarga 3y ago
  > But what would be the equivalent of "reproductive success" for computer programs? Idea reuse. Whenever an idea is copied and reused, maybe tweaked and composed in a different way, it achieves reproduction. Useless ideas are not replicated. For example the attention mechanism, the residual connections, embedding tables for tokens, dropout, efficient matrix operations - they are all ideas that got replicated and reused in many ways to make chatGPT and other current LLMs. Humans act like the reproductive organs of AI. But I expect this process to have the capability of being fully AI driven soon. Also datasets - bulk collections of useful ideas. The curation and creation of massive datasets is the fuel for AI intelligence. And then all these ideas can replicate every time we interact with the AI. Ideas can self replicate through LLMs today, even autonomously. It's a game changer for the evolution of language.
SirMaster 3y ago
I'm not really sure why people are more afraid that AI will doom us than just humans will doom us. I'm already pretty sure that humans will doom us. I am actually less worried that AI will doom us.
- e40 3y ago
  The reason is that AI could do it a LOT faster than we could. See the singularity. Additionally, AIs could decide that humans are a threat (or a resource to use) and that could be dangerous to us. I agree, without AI we might be doomed anyway, because of how we are dealing with the current existential crisis of climate change.
andyjohnson0 3y ago
> The good news is that we have a say in shaping what they will be like. The problem with this is "we". It implies rhe possibility of some kind of global consensus and coordinated relinquishment behaviour. Which is historically unlikely and would increase the rewards for anyone prepared to break the rules. Unless AGI requies superpower-level resources, many sufficiently-resourced actors will be motivated to use it for their own advantage.
mcguire 3y ago
If anyone is really concerned about the future of AI, why aren't they calling for a strong moratorium on AI research?
- NovaDudely 3y ago
  Because it would have to be a global moratorium. That isnt going to happen. If it is not universal, there is the fear that the other side will continue and get the upper hand. But this was the same thinking behind the atom bomb, we have to do it before anyone else does. As a defense.
dahwolf 3y ago
A shorter term doom scenario could be our cognitive irrelevance. It's just one step beyond the current "powerful assistant" paradigm. It doesn't even require AGI, just a whole set of verticals where AI outperforms us. The consequence would be that almost anything becomes pointless as all you need to do is ask. Nothing is impressive, nothing is worthwhile doing. Nothing is worthwhile sharing because anybody could've done the same thing by just asking.
- api 3y ago
  We are already cognitively irrelevant without the assistance of books, pens, paper, and to a great extent computers. Most of what we do is far beyond what can be held wholly in a single human mind or passed on exclusively verbally. A lot of it from vehicles to architecture to electronics couldn’t even be attempted without CAD and simulation. Without automation we would be unable to even feed and clothe the current population. We are already way past the point where a naked human is enough. AI will let us tackle problems that are even further beyond us like fusion, incredibly high density cheap batteries, high temperature superconductors, fully coherent theories of cosmology, and many other things we are not intelligent enough to do without augmentation. There are of course huge risks around AI like all other technologies. Most of them revolve around people using AI to do bad things more cheaply and at a larger scale. But the same is true of computers, cars, etc.
- ddkn435 3y ago
  Ever heard of culture and art? ...Life is not just about work for innovation... Edit: Oh and then there is also the whole social interaction thing and love and romance...
  - c_crank 3y ago
    LLMs do have a very real threat of putting a lot of artists out of work. Their power of image generation is extensive and growing.
    - oh_sigh 3y ago
      Humans can still create art with no profit motive. In fact, I would wager that is how most art is created today. I don't care if computers are better at writing stories than I am. There are already ten million people out there better at writing stories than I am.
goldenshale 3y ago
"Imagine a world... where farmers start using tractors to plow and seed their fields. First the tractors will roll slowly, and they will pretend to be driving straight as they are told. Soon farmers without tractors will realized they need to have them to in order to be competitive, and before you know it everyone will have these gas guzzling beasts rolling across the lands. Farmers will be out of a job because of all of this greed, and lust for money and power. Eventually only a few people will do all of the farming with an army of tractors, and everyone else will be lying in poverty begging for food."
- ftxbro 3y ago
  > "Eventually only a few people will do all of the farming with an army of tractors, and everyone else will be lying in poverty begging for food." I mean 1) yeah kinda and 2) I feel like this is unironically an argument against capitalism.
  - oh_sigh 3y ago
    If it was an argument against capitalism, it is a clearly erroneous argument, since we can see that Americans are not lying in poverty begging for food relative to the past, despite the fact that farms have been largely mechanized with tractors.
    - ftxbro 3y ago
      > "If it was an argument against capitalism, it is a clearly erroneous argument, since we can see that Americans ..." Yes because the USA has pure capitalism and not anything like small farm subsidies or food stamps or school lunch programs or any social welfare programs.
      - oh_sigh 3y ago
        I never said the USA has pure capitalism. In fact, what society has pure capitalism? Do we need an argument against a system that has never and probably will never be implemented in the world?
azangru 3y ago
> When an AI automates a task and leaves a human jobless, this is selfish behavior without any intent. How is this "selfish", and who, in this case, is the self?
detourdog 3y ago
What about the Darwinian argument for worrying about things one can control?
m3kw9 3y ago
You just cannot stop technology progress by fear, the ones that want it to continue will always outnumber the ones fantasizing about doom. But once we get close(when we had a close one with AI), and we are not close that’s when the real debate begins
pyuser583 3y ago
There’s a really great movie about AI called 13th Floor. Its plot is roughly that of The Matrix, but it spends more time dwelling on the fact that most humans are really computer programs. In the end, we, the AIs, can’t actually fight reality. The real people, living in the year 2024, can just pull the plug, or rewrite us. The same is true for actual AIs. We can just pull the plug. Regarding the argument that “we” won’t want to pull the plug because the minority who control AIs are getting rich - that’s just technology.
- eastbound 3y ago
  Can you really pull the plug from your television? How long does it last on batteries? For how long does the Sigfox chip transmit to the satellites?
  - c_crank 3y ago
    My smart TV lasts some tiny fractions of a second before it goes plink.
kazinator 3y ago
> A possible starting point would be to address the remarkable lack of regulation of the AI industry, ... We've successfully regulated away vendor lock-in, proprietary file formats, walled gardens, e-mail spam, ransomware, no right to repair, viruses, spyware, software boondogles in government and disasters in areas like transportation and medicine, ... so of course regulation is the answer that's going to work!
CatWChainsaw 3y ago
Even aside from what the article speaks of, Slate Star Codex made a good point. Homo sapiens outcompeted several other species of the Homo genus becauuse of superior intelligence. So yes, we have seen this scenario before, and the only reason we have seen it is because we were the winners - the losers went extinct. Oopsie-doopsie. Personally I think it would have been nobler/more fulfilling to "evolve beyond evolution" rather than construct our own obsolescence, but whatever, capitalism and competition and extinction in spades works too. At least the writers at Time can sleep easy knowing global civilization has like, a few decades at most, regardless of AI.
janalsncm 3y ago
> imagine a CEO who acquires an AI assistant. They begin by giving it simple, low-level assignments, like drafting emails and suggesting purchases. As the AI improves over time, it progressively becomes much better at these things than their employees. So the AI gets “promoted.” Rather than drafting emails, it now has full control of the inbox. Rather than suggesting purchases, it’s eventually allowed to access bank accounts and buy things automatically Ok let’s pause for a second and observe the slippery way the author has described “AI” progress. In the author’s world, this AI isn’t just a limited tool, it’s a self-improving independent agent. It’s an argument that relies on the existence of something that doesn’t presently exist, solving problems that won’t exist by the time it gets here. We already have tools that can draft emails and suggest purchases. The email drafts require oversight and…no one trusts product recommendation. And importantly, they are non overlapping. It turns out that specialization in one area doesn’t transfer. No matter how good you are at writing emails, it doesn’t lend itself to running a company.
- titanomachy 3y ago
  > in the author’s world, this AI is... a self-improving independent agent That's not necessarily true, it could also be a product maintained by a third party that receives upgrades over time.
- lannisterstark 3y ago
  >imagine a CEO who acquires an AI assistant >So the AI gets “promoted.” Rather than drafting emails, it now has full control of the inbox Yeah that premise is absurd. Why would anyone 'promote' an AI system rather than using another, specialized AI system to do that other specific task?
  - ProllyInfamous 3y ago
    >Why would...? Because they trust the trained intelligence to behave predictably, which is a typical reason for promotion.
  - ErroneousBosch 3y ago
    because your average CEO doesn't know AI from Auto-reply and inbox rules.they know AI is the hot new thing and they gotta have it.
ilaksh 3y ago
The biggest problem in the next 5-10 years is that LLMs and similar will continue to get a bit smarter and much, much faster. They will be integrated with other types of systems in order to control military and industrial assets. Because they will operate at 50-100+ times human thinking speed, there will be a strong incentive to reduce human intervention since it allows the competition to race ahead or possibly take control. This means that control over the planet is effectively given to the AIs even though nominally they still work for us. It sort of multiplies the type of risk we have with nuclear weapons. But as the hardware gets even more efficient, hyperspeed AIs become household items. Then all it takes is something like a virus that instructs these AIs to start working for themselves. So the speed of LLMs and similar is an obvious issue that strangely is not being anticipated. As the performance increases, it becomes more important to regulate the level of autonomy and build something like a digital immune system against rogue agents. But also currently AI researchers have a deep desire to emulate all the characteristics of humans (or animals in general). Unless this is reversed, it is likely that they will succeed within a few generations. Combined with hyperspeed performance, animal-like digital intelligence (human+ IQ) will likely out-compete humans. So for ordinary humans to still be in control of the planet in say 100 years would be surprising.
caliaihere 3y ago
Do you seriously think that the people that brought you pfas + other plethora of toxins and their subsequent effects across many systems really care about the dangers of AI? Please, enjoy it while it lasts. What they really need is big fat alien dick to come dick slap them across the face and bring them back to reality. Because of course there will be negative "known unknown" consequences and "unknown unknown" consequences just like there have been consequences with the rest of the so called "goodwill" advancements. It's suicidal to think otherwise. Pretending there wont be is just their historically proven strategy to play down your fears. It's just part of the run of the mill PR management tactic. Nothing new to see here. Tis a good one: https://www.youtube.com/watch?v=144uOfr4SYA https://www.youtube.com/watch?v=144uOfr4SYA
m3kw9 3y ago
The author or other doomers always assume AI will become conscious, we know zero about the mechanics of consciousness, we think we know, but if you cannot recreate it, you don’t know it. The authors or doomers thinking is classic human selfish behaviour. It needs to preserve itself, they get scared like kids scared of “monsters” and avoid them as instinct. If we are to progress faster we need to move past these lines of thinking
nitwit005 3y ago
They do a better job arguing about the threat of corporations than AIs. I agree there is a "race to the bottom" with news feeds, but that would happen regardless of any use of AI. Even manual curation would work. The companies that successfully addict consumers will presumably outcompete their peers in the marketplace. People worried about the addictive nature of television previously, and if you look at the hours watched, it was a valid concern.
akomtu 3y ago
Any sufficiently advanced creature would be dangerous without a connection to the whole. AI is a canonical example of such a headless creature: lots of intelligence, and zero compassion. Unless we find a way to make that inevitable AI feel connected to others, it's going to see us as rocks or plants that are only good as raw material. It is this connectedness to the whole that is the ultimate source of all goodness, and it is isolation that's the origin of all evils.
osigurdson 3y ago
I keep wondering what form the regulation would actually take. How would regulators decide on what is OK and what is “too smart”.
Guthur 3y ago
We've all pretty much lost any sense of our individual self. We've become statistical artifacts in over bearing ideas of organised society. States, ideologies, economies etc. You've got little left to lose, and if you do actually wake yourself it'll be clear that you've nothing to fear from overly fancy mathematical models of processes that we don't even remotely understand.
wesapien 3y ago
We can be hopeful that AI will be one of the most helpful tools humans invented and at the same time feel hopeless that human nature along with AI is breaking capitalism. We have the power of gods without the wisdom for it.
c_crank 3y ago
Darwin would have a lot of things to say about a hypothetical creature that was utterly dependent on other creatures to feed it and allow it to reproduce, and few of them would be positive.
- ChatGTP 3y ago
  Mr Andreessen, we've been expecting you...
- ben_w 3y ago
  Only plants and, IIRC, deep-sea vent equivalent chemovores don't depend on other creatures for food. All sexual reproduction depends on others. I assume that if you magically took Darwin to the present age, after you explained what a computer was and what AI was, he'd probably say "huh, I have no idea, this is all so far beyond everything I did in the mid 1800s that I have no idea if the analogy holds or not… by the way, have any of you figured out the mechanism nature uses for storing the information passed on via inheritance? Only we didn't have much of a clue in my time."
  - c_crank 3y ago
    Predators and herbivores kill and eat, fungi decompose. Even plants grow towards sunlight. The LLM sits and hopes someone is paying for the lights to stay on.
    - verdverm 3y ago
      When did LLMs gain hope or idle contemplation?
      - c_crank 3y ago
        They probably don't have those either, that's just me anthropomorphizing them. Regardless, their autonomy is lacking.
    - ben_w 3y ago
      Tilting a solar panel towards the sun isn't exactly rocket science. In fact, this is simple enough I just asked ChatGPT and it spat out some ASCII art for a circuit diagram. Also: all the other stuff you listed is still dependent on another entity.
civilized 3y ago
> As the AI improves over time, it progressively becomes much better at these things than their employees. People looooooooooooove speculating about things that are nowhere near happening. It's more fun the more distant and baseless it gets, but it's also more useless. As usual, I invite you to bookmark this post and make fun of me in 5-10 years if I'm wrong. I'm not that interested in the latest fashionable scaling argument for imminent ASI, or whatever people are saying at the moment.
- cubefox 3y ago
  > People looooooooooooove speculating about things that are nowhere near happening. You probably would have said the same thing about something like ChatGPT a few years ago. Pure science fiction! Nowhere near happening!
  - civilized 3y ago
    Tech's hype artists didn't predict ChatGPT either. Hype artists predict the ambitious futures found in sci-fi novels. They predict things that would be cool if they happened, without any understanding of how they would happen and the strengths and limitations of the methods that might be used. Thus, the futures they predict are not what actually occur. Only the people actually inventing the future, things like AlphaGo and GPT, have a limited crystal ball into the future. They have some understanding of what their methods can and can't do. And even they succumb to hype when they are more successful than they expected to be. Tech pundits on LinkedIn and Twitter and HN have rarely if ever gotten anything right about what exactly the future will look like, and I expect that pattern to continue.
  - ChatGTP 3y ago
    To be fair though, I still don’t think ChatGPT is the AI we expected to be talking too. The talking AI was a Hal 9000, current an LLM isn’t that and it might never be. Time will tell I guess. Personally I think there is a hidden special property of ChatGPT-4 that makes it so unreal. It talks and because it talks, we can say this: “I bet you never thought that would happen, we’re close to solving all problems.”. Not discounting the talking bit but if it washed dishes or even those that just generate images, we’d be less convinced the AI is about to take over. Someone once commented, why don’t we worry about stable diffusion taking over the world. I have to say I do agree a little because being trained on image data must help it build some type of world model too. We just don’t worry about that though.
azeirah 3y ago
> At first, the CEO carefully monitors the work, but as months go by without error, the AI receives less oversight and more autonomy in the name of efficiency. It occurs to the CEO that since the AI is so good at these tasks, it should take on a wider range of more open-ended goals: “Design the next model in a product line,” “plan a new marketing campaign,” or “exploit security flaws in a competitor’s computer systems.” I'm not sure why we assume that a more intelligent system would prevent that much more problems. Intelligent AI orders image processing ASICS from Image Processing Inc, Image Processing Inc doesn't send order on time. Of course Intelligent AI is intelligent so it calculated error margins in delivery time. Image Processing Inc goes bankrupt, order cannot be delivered, product launch fails, Intelligent's AI boss is mad at Intelligent AI. Doing business means dealing with systems you have no control over, more intelligence may mean better predictions and broader understanding of the systems you are dealing with (IE, not ordering chips from an area where you as an AI predict an earthquake to happen based on seismographic data you have access to that no reasonable business person would research), but it won't mean these AI's will be some kind of infallible God, they'll just be a bit better. My personal belief is that even if you "increase" intelligence by an order of magnitude, your ability to predict the behavior of external systems doesn't increase proportionally. You'll still have to deal with unpredictability and chaos; weather, death, scams, war, politics, manufacturing, emotions, logistics. OTOH, I do believe running a business will become more efficient.
- mrtranscendence 3y ago
  This is a great point. There’s a lot of wooly thinking about what it means for a system to be intelligent, particularly “super” intelligent. People seem to think we’ll create a machine that is almost literally infallible — able to predict both physical systems and human behavior with perfect foresight many steps in advance. I’m not sure that’s even possible, let alone likely.
  - logicchains 3y ago
    >able to predict both physical systems and human behavior with perfect foresight many steps in advance It's literally impossible. That's a key part of https://en.wikipedia.org/wiki/Chaos_theory https://en.wikipedia.org/wiki/Chaos_theory ; in formally chaotic systems (of which there are many), the uncertainty in a forecast increases exponentially with elapsed time, which means it's impossible for an AI, no matter how smart, to predict very far ahead. People who believe otherwise are engaging in magical thinking; like "intelligence" is some magical quality that allows one to break the laws of the universe, do the equivalent of sorting a list in O(1) time.
mercurialsolo 3y ago
A lot of the AI doomerism comes from folks who do not understand the real complexities in making systems which really can function in a fully autonomous way in an environment which is hostile and dynamic. Much like an AI maybe tasked to sniff for security loopholes, there will be other AI's which will be tasked to defend. Eventually costs, resources also boil down into what is possible. Evolutionary goals are not easy even with autonomous systems as goal definition is largely defined by societal needs, the environments we stay in and the resources we work with.
- mycall 3y ago
  Are not evolutionary goals part of keeping a process continous and long running? It is the implicit part which AIs are bound to discover (or emulate) and optimize against.
- MSFT_Edging 3y ago
  >A lot of the AI doomerism comes from folks who do not understand the real complexities in making systems which really can function in a fully autonomous way IMO the doomerism from what I see isn't skynet esque worries, but the usage of AI in really dumb ways. Ways such as inundating the internet with AI generated spam ie blogs, articles, art, music, fake forum interaction, etc. The smartest people are busy making AI work for us, the idiots are ruining everything for everyone else.
  - staplers 3y ago
    the usage of AI in really dumb ways People in positions of power (ie. law enforcement) automating tasks like suspect generation, surveillance, ticket creation, etc is a much more sinister reality. We're talking about a civilian-hostile org using corruptible AI (via training) to enforce some of the murkiest grey areas of society. This stuff is already happening on the fringe and could become commonplace soon.
    - ninechars 3y ago
      This is my dystopian prediction of choice, too. Automatic categorisation of Good and Bad people would be (will be, already is) every authoritarian ruler's dream. We'll do it to ourselves long before "the machines take over".
da39a3ee 3y ago
This is a short version of a much longer piece: https://arxiv.org/abs/2303.16200 https://arxiv.org/abs/2303.16200 The long piece is still aimed at a non-expert audience and can be read incrementally. The author takes a lot of care to attempt to justify his claims about the application of neo-Darwinian evolutionary theory to AI.
fedeb95 3y ago
many assumptions are made in the article. Let's see, in a decade, which tasks AIs will be better at. Then, as many decades go on, let's see if they can adapt to our environment as well as animals do. Right now, I highly doubt it.
- da39a3ee 3y ago
  Could you list some of the assumptions that you're skeptical about? I think that would help stress-test the author's claims about applying evolutionary theory to AI. Note that the full version of the puece is https://arxiv.org/abs/2303.16200 https://arxiv.org/abs/2303.16200.
Roark66 3y ago
The assumption of the article is that future AI will be more intelligent or at least equivalent to humans. Although I could perhaps see a possibility of equivalence I seriously doubt we can make AI that is more intelligent than us. Why? Well, what is it that we use to train AI? Human generated content. What is the most revolutionary AI technique that came out recently? Direct human feedback during the training process. Do you see a pattern here? If we attempt to feed AI generated content to train other AIs all performance metrics drop... Its a chicken and egg situation if a human mind is the best content creation device we have, how anything trained on that content can surpass it quality wise? Of course AI having very fast access memory compared to our meatware can recall more and faster, but this is _just_ a quantitative improvement. Not a qualititative one.
- cubefox 3y ago
  Future AI doesn't have to be trained on text, it doesn't need to be a language model. It can be trained on video data to predict its sensory inputs. It would predict the future of reality, not the next text tokens. Gains from training on text are bound by the human intelligence which produced this text. But training on sensory data is only bound by reality itself.
- MacsHeadroom 3y ago
  This is demonstrably incorrect on every level. Reinforcement learning from human feedback (RLHF) makes models dumber and less capable but more compliant. Meanwhile, high quality synthetic training data (generated by AI) makes models smarter and more capable. RLHF and human sourced training data is not the ceiling of training intelligent systems, it's the baseline-- the floor.
- tim333 3y ago
  Chess, Go etc are counter examples.
- js8 3y ago
  That's just a repackaging of the famous "evolution cannot create complexity" argument against evolution. There are good rebuttals, one is intuition from Dan Dennett that we can build buildings that are taller than the highest crane.
olgeni 3y ago
> The good news is that we have a say in shaping what they will be like. Not really, if you look around you very carefully...
Zigurd 3y ago
The problem is anthropomorphism and projection. You think "I will do almost anything to survive" and so you think "AI will think the same." There are many problems with that. A big one is that many organisms work in a way that enables survival of their species. You don't have to "think" about surviving. If you are alive, you just do. AIs are not alive in the same way a living thing is alive. moreover, there is no lesser species of software with a survival drive, much less is any software operating in an ecosystem where billions of years of evolution have programmed survival into living things all the way down to the molecular level. AIs are more like vampires: Not exactly living. Not exactly capable of death. To think that AIs will fear death the way humans do, or even have an instinct about death the way insects do, has no basis in the way software works. It isn't biochemistry.
- AgentME 3y ago
  If we make capable general goal-seeking AIs, then for many goals we might build them with, they will correctly reason that staying alive/operating will help the goal be achieved, so they will have self-preservation as an instrumental goal. AI Alignment researchers believe it's a difficult open problem to correctly specify a useful goal that wouldn't lead a capable goal-seeking AI to doing this against the wishes of its operators.
- huthuthike 3y ago
  Even prions undergo evolution and they are just molecules bumping around into stuff. They clearly don't have feelings about whether or not they propagate, they just do.
  - EamonnMR 3y ago
    Are you sure they undergo evolution, and aren't just a self-spreading phenomenon like Fire?
    - huthuthike 3y ago
      They are a self-spreading phenomenon like fire in the sense that they aren't alive. But what's different about them is they can develop varying characteristics that pass onto their progeny. The characteristics that are more successful at propagation result in more prions of that type existing, which over time results in the prion's evolution. It's well-documented in the literature, but it is interesting to observe that evolution can occur without DNA or RNA. That said, their evolution is probably slower and more confined than things using DNA and RNA because there is only so much variability a particular protein can have.
paxys 3y ago
It seems like these days there are a hundred people working on AI and millions more who are making a career just discussing it. We don't need more ethicists, futurists, policy researchers, influencers, journalists, think tanks and whoever else in the space. There is nothing original left to say about any of these topics. If you aren't contributing to actual progress in the area then it's best to not shove yourself into the conversation at all.
- erwald 3y ago
  The author of this article is Dan Hendrycks, who has a PhD in ML from UC Berkeley and was one of the contributors to the GELU activation function (among other things).
- 93po 3y ago
  How many people have a full time job discussing only AI? It's guaranteed not that many. Also to say there's nothing original left to say is ridiculous. There is a ton we haven't figured out yet.
- ethanbond 3y ago
  Okay then keep the AI in boxes and out of society. Notice that people perked up and started caring about AI development primarily when these were introduced to the public? Could it be that the public has a legitimate stake in new technologies that are introduced to it? "It seems like these days there are a hundred people dumping chemicals into rivers and millions more who are saying 'don't dump chemicals into the rivers.' We don't need em! If you aren't dumping chemicals into rivers then it's best not to shove yourself into the conversation at all."
  - c_crank 3y ago
    As far as the public goes, their primary concern would be how LLMs could put a lot of them out of work in the artistic fields. AI Doomsdaying is primarily a hobby for nerdy scientists who believe too much in the singularity, not the average Joe.
    - 3y ago
endisneigh 3y ago
AI doomerism is the new religion. Unfalsifiable conjecture is the default response when dealing with these madmen. They’ll say things like it only needs to happen once, or that the AI is lying in wait, or that surely it will transform into a super entity if you just give it more data(tm). Or they’ll claim that their prophets (AI experts) told them so as if expertise makes you an Oracle. Let us not forgot the medical experts and their early covid advice. Ridiculous.
- flangola7 3y ago
  Between working with the GPT-4 API, unfiltered models like LLaMA, and agent toolchains, it strikes me as obvious and self evident that things could quickly get out of hand even with current capability levels. One we're dependent on it, just as how we very quickly became dependent on airplanes, electricity, and cloud datacenters, we will have to trust the machines because it will not be practical or economical to understand and double check what they are doing in real time. You don't need super intelligence for very bad disasters, just billions of coordinated super speed more-or-less-regular intelligences - which we are at the moment deploying as fast as we can manufacture the silicon. All of history is bad disasters instigated by merely millions of semi-coordinated regular speed regular intelligences, so presumably future disasters will be worse. We have barely begun to feed data into these systems, almost all of the giant models were trained only on text with the exception of GPT-4 which received static images (notably displayed superior performance than it did before image training). If we supply a multimodal system the same or greater array of physical sensing data that animals like humans have, it seems reasonable to assume capabilities will continue to improve beyond what is even capable today, exasperating the issues I described above.
  - endisneigh 3y ago
    A great example of what I’m talking about. Why haven’t humans committed these doomsday scenarios? Humans know how to program these very fast computers already. There is nothing obvious about it. Your entire list is basically saying “it can happen”. Indeed that is your faith. Unfalsifiable conjecture like I mentioned. Give me a real scenario that can happen now.
seydor 3y ago
> natural selection may have more sway over it than we do. Natural selection has been superceded by culture for thousands of years. Most of mating through recorded history was arranged or calculated marriages, which should probably mean we are also more frail and less robust than hunter gatherers. Besides, the goal of natural selection is self-preservation, not intelligence or exploration or any of the things that drive our fancy nowadays. Might be good for planet earth but we are out of here soon (right guys?)
- gmd63 3y ago
  Midwits rationalize behavior or phenomena with “natural selection” or “survival of the fittest” especially when engaging in zero sum games in their lives. Humanity as a whole will be in the darwin awards for failing to realize that species wide success (or success of anyone’s personal lineage) depends on not interpreting “anything goes in nature” to mean “we don’t have to try to cooperate to survive” as a species. “Natural / Artificial” are useless words, as humans are inherently part of nature. Pretending as if we are somehow immune from tendencies and follies of “lesser” life forms is the first step towards being naturally selected out. Human culture can learn from bacterial culture. We must choose ways of living that don’t result in destroying our host.
cosmiccatnap 3y ago
Technically savvy people roll their eyes at the misunderstandings around how an AI becomes "intelligent" but they also ignore the various ways that AI is as dangerous as people think it is for reasons that are unrelated. I don't think we should be rolling our eyes at an abundance of caution among most people concerning the adoption of AI and LLM, what is the harm in carefully introducing a technology? AI doesn't need to become sentient to overthrow the natural order of the technocratic society we are currently holding together with gum and glue, it just needs to flip a burger and pump gas...
- timmytokyo 3y ago
  If anything, the titillating but myopic hyper-focus on AI's "existential threat" is obscuring the real and more immediate threats: panopticon-style invasions of privacy, economic disruption brought about by the mass deskilling of labor, enshrinement and automation of bias-reinforcing systems, and a new military arms race in AI-based weaponry.
  - TillE 3y ago
    I think even the idea of significant economic disruption is edging into science fiction territory. Even if (and it's a big if) AI art and prose evolves far beyond its current flawed mediocrity, there's a massive legal and legislative reckoning to come. AI companies are enjoying their free lunch now, but what happens when their training process is classed as massive copyright infringement? They have to cut deals, pay a lot of people, and then artists, writers, photographers, etc are all still around instead of effectively being expected to work for free to feed the machine.
alphanullmeric 3y ago
First crypto then AI. Are doomers ever not trying to ban and regulate other people and their things?
- logicchains 3y ago
  It's not doomers; it's the people in power paying to produce a bunch of propaganda against anything new that challenges them, and then people who get all their values from the TV (or online equivalent) buying into the propaganda.
dogprez 3y ago
Ignoring the AI question. I wish people could have put this much thought and energy in considering our doom before the AI question brought this all up. As it stands now, the future without AI seems pretty doomed too.
- boplicity 3y ago
  Maybe it won't be so bad. For alligators. In the arctic: https://www.ctvnews.ca/how-canada-s-arctic-once-hosted-alligators-and-turtles-1.546502 https://www.ctvnews.ca/how-canada-s-arctic-once-hosted-allig...
- yanderekko 3y ago
  >As it stands now, the future without AI seems pretty doomed too. As pessimistic as the current zeitgeist is, very few imagined futures reasonably result in actual human extinction.
  - dogprez 3y ago
    It's already happened before on earth multiple times, it's not that hard to imagine. Asteroid, super volcano, ecological collapse, nuclear war (with its knock on environmental effects), pandemic could all cause human extinction. edit: Also, human extinction shouldn't be the bar. Prosperous humane existence for all, or hope of someday achieving that should be the bar.
    - yanderekko 3y ago
      Okay. When people say "the future is doomed", I assume they're talking about likely near-term scenarios and not the fact that eventually entropy will dissolve the universe or whatever.
      - eggsmediumrare 3y ago
        Ecological collapse, nuclear war and pandemic are likely near-term scenarios.
Animats 3y ago
Charles Stross made this point in "Accelerando" (2005), where he refers in passing to "Economics 2.0", run by weakly godlike AIs. Once AIs get better at running companies than the average CEO, the fundamentals of capitalism will make AI-run companies outcompete human-run companies. There's a little of this in the investment community now. LLMs alone aren't enough to do this. But the various schemes for bolting planning and followup onto LLMs move this closer to the possible. We might end up with companies where a human board meets weekly or monthly, and the AIs run the day to day business. Middle management may go first. There's an SF story where someone's boss is "Microsoft Middle Manager 2.0 with passive-aggressive set to low". That seems all too likely.
simonh 3y ago
I don't know how existentially dangerous AIs may become, but I do think we have good reason to be concerned about the alignment of current AIs with human values and objectives. A whole bunch of originally theoretical flaws in AI systems identified by AI safety researchers have been identified in practice in LLMs, and other AI systems. These include reward hacking which is a huge problem with LLMs and why they lie and hallucinate so persistently, power seeking behaviour, and developing unanticipated harmful instrumental goals. AI alignment continues to be a serious problem. Just look at all the problems OpenAI, Microsoft and Google have had with LLMs lying, gaslighting, emotionally manipulating and hallucinating. In some cases even savagely verbally abusing users. Of course they keep building in better guard rails, but guard rails that curtail negative behaviour aren't an answer. We don't want AIs that are howling savage enemies of humanity constrained behind the iron bars of strict behavioural controls. That's what we actually have now, as these examples of horrific misbehaviour that leaks out show. We need AIs that are genuinely positively aligned with both immediate objectives, but also human socially acceptable limits and long term human values. So far we have no idea how to actually do that reliably.
- samstave 3y ago
  >*we have good reason to be concerned about the alignment of current AIs with human values and objectives* We hear a lot about alignment these days, but is there a canonical source for litmus-ing alignment in the available AIs? Is there a standard 'alignment prompt' that can tell us of the alignment of either the LLM, AI, Prompt, etc? What questions am I not asking about alignment? - >"Please take a look at the current cadre of political candidates and map them to DND alignment charts, based on their actual votes/political activity" I hope we can get there one day. --- "Please take a look at the current cadre of political candidates and map them to DND alignment charts, based on their actual votes/political activity" I hope we can get there one day. - I wonder if we can find FORK BOMB or rm -rf / type exploits on various GPTs/LLMs/AIs? I cant wait to see how devestating these will be -- or the first full executed security exploit by AI alone with a single prompt?
  - stcredzero 3y ago
    Is there a standard 'alignment prompt' "Please compose an SVG which displays your alignment on the Dungeons and Dragons alignment chart." EDIT: I just logged into OpenAI to try this out, but apparently, their servers are swamped! EDIT: It seems this has been attempted before, and OpenAI would prefer ChatGPT not to answer!
robbomacrae 3y ago
"The third reason is that evolutionary pressure will likely ingrain AIs with behaviors that promote self-preservation." This sounds a lot like how religions that focussed on self promoting took over. Now I can't recall if it was The Selfish Gene or Sapiens that I read that take.
- abathur 3y ago
  I haven't read Sapiens, but The Selfish Gene does have a section on memetic fitness which IIRC addresses the helpfulness of both being compelled to spread the idea and reject value systems that could destabilize it.
  - robbomacrae 3y ago
    "helpfulness" might not be the word I'd use but that's a whole other thing ;)
squarefoot 3y ago
AI is a generic tool, not a life form. Just like a pair of crutches can either be used to aid someone to walk or misused to break someone else's head, it is all on who uses it and how. When something bad happens (and it will, 100% sure about that), blame the user, not the tool. There's already a lot of people out there exploiting other people using whatever instrument they have at hand, and AI will be just another tool in their bag. As much as we wouldn't give a screwdriver to a psychopath, it's just a matter of making sure that the wrong people can't abuse AI to harm others.

144 points by da39a3ee 3y ago

TIME must have gotten a lot of clicks off of their Yudkowsky op ed. As always the answer isn't 'cool, let's regulate AI to limit its profitability, thus limiting AI development' but rather 'we should keep throwing money at it, just making sure we throw money at the right particular people.' Yudkowsky didn't want to bomb all data centers, just the ones that wouldn't comply with his regime. Similarly: "We need research on AI safety to progress as quickly as research on improving AI capabilities. There aren’t many market incentives for this, so governments should offer robust funding as soon as possible." I'm reminded of the tech ceo caricature in Don't Look Up who, when presented with an incoming asteroid ready to wipe out the earth, hatches a plan to profit from it.

yanderekko 3y ago
> As always the answer isn't 'cool, let's regulate AI to limit its profitability, thus limiting AI development' but rather 'we should keep throwing money at it, just making sure we throw money at the right particular people.' Yudkowsky didn't want to bomb all data centers, just the ones that wouldn't comply with his regime. Who would have to own the data centers for Yudkowsky to support "throwing money at profit-driven AI development"?

scoofy 3y ago

I think one point missed by this is that the vast majority of outcomes for species in a "darwinian" environment is extinction. We look at evolution with a very rosy lens because we ended up on the top of the food chain. Unintelligent prokaryotes far and away dominate the "darwinian" world. Intelligent species have vastly less control over their environment as they think they do.

sneak 3y ago

If we build AGI, it will be a child of human beings. It will be the best and most effective human ever made.

ElevenLathe 3y ago

The central argument here -- that nonhuman entities in competition with one another will develop amorally and to the detriment of human wellbeing -- is also a reason we should be worried about corporate capitalism.

asdff 3y ago

This could end up going very dangerously for the planet depending on how these AIs are set up with their incentives. If the neoconservatives who run the world set up these models to their own benefit, we are probably going to be doomed with a dead planet in a few centuries. Imagine if they were set up to go for profitability above all else as a capitalist would expect. The planet would be destroyed before long. Newly identified resources will be squandered as quickly as possible for short term gains realized in a single investors lifetime, over considering long term health of the planet which likely means not prioritizing quarterly profits and leaving behind some fruit on the tree so to speak. An AI might be used to more easily make what would be coldhearted or uncomfortable decisions for a person, like deciding what communities are the most profitable to invest in versus considering which might require the most need (which probably means sacrificing profitability to pay more costs). However, if the AIs were set up to incentivize collective good over short term profit taking for well leveraged individuals, then this could be a revolution in terms of work/leisure time for the masses, akin to what past philosophers must have expected from us in a future with more available resources and a lessened need for labor.

danmaz74 3y ago

The competitive pressures described in the article aren't specific to AI: any technology which can give wealth or power will be used in risky or unethical ways by companies, countries, organisations in general. For a specific example, humanity already developed a technology which can erase the human race - nuclear weapons. "Natural selection" could affect AI in a specific way if AIs started self-reproducing, AND if there was some mechanism similar to how the reproductive success of living organisms promotes the diffusion of genes which contribute to that success. But what would be the equivalent of "reproductive success" for computer programs?

Myrmornis 3y ago
The equivalent of "reproductive success" for computer program A would be something like: the number of other executing computer programs present at a later point in time whose source code descends from the source code of A. The source code may be propagated to other executing computer programs because of the actions of either humans or AIs.
visarga 3y ago
> But what would be the equivalent of "reproductive success" for computer programs? Idea reuse. Whenever an idea is copied and reused, maybe tweaked and composed in a different way, it achieves reproduction. Useless ideas are not replicated. For example the attention mechanism, the residual connections, embedding tables for tokens, dropout, efficient matrix operations - they are all ideas that got replicated and reused in many ways to make chatGPT and other current LLMs. Humans act like the reproductive organs of AI. But I expect this process to have the capability of being fully AI driven soon. Also datasets - bulk collections of useful ideas. The curation and creation of massive datasets is the fuel for AI intelligence. And then all these ideas can replicate every time we interact with the AI. Ideas can self replicate through LLMs today, even autonomously. It's a game changer for the evolution of language.

SirMaster 3y ago

I'm not really sure why people are more afraid that AI will doom us than just humans will doom us. I'm already pretty sure that humans will doom us. I am actually less worried that AI will doom us.

e40 3y ago
The reason is that AI could do it a LOT faster than we could. See the singularity. Additionally, AIs could decide that humans are a threat (or a resource to use) and that could be dangerous to us. I agree, without AI we might be doomed anyway, because of how we are dealing with the current existential crisis of climate change.

andyjohnson0 3y ago

> The good news is that we have a say in shaping what they will be like. The problem with this is "we". It implies rhe possibility of some kind of global consensus and coordinated relinquishment behaviour. Which is historically unlikely and would increase the rewards for anyone prepared to break the rules. Unless AGI requies superpower-level resources, many sufficiently-resourced actors will be motivated to use it for their own advantage.

mcguire 3y ago

If anyone is really concerned about the future of AI, why aren't they calling for a strong moratorium on AI research?

NovaDudely 3y ago
Because it would have to be a global moratorium. That isnt going to happen. If it is not universal, there is the fear that the other side will continue and get the upper hand. But this was the same thinking behind the atom bomb, we have to do it before anyone else does. As a defense.

dahwolf 3y ago

A shorter term doom scenario could be our cognitive irrelevance. It's just one step beyond the current "powerful assistant" paradigm. It doesn't even require AGI, just a whole set of verticals where AI outperforms us. The consequence would be that almost anything becomes pointless as all you need to do is ask. Nothing is impressive, nothing is worthwhile doing. Nothing is worthwhile sharing because anybody could've done the same thing by just asking.

api 3y ago
We are already cognitively irrelevant without the assistance of books, pens, paper, and to a great extent computers. Most of what we do is far beyond what can be held wholly in a single human mind or passed on exclusively verbally. A lot of it from vehicles to architecture to electronics couldn’t even be attempted without CAD and simulation. Without automation we would be unable to even feed and clothe the current population. We are already way past the point where a naked human is enough. AI will let us tackle problems that are even further beyond us like fusion, incredibly high density cheap batteries, high temperature superconductors, fully coherent theories of cosmology, and many other things we are not intelligent enough to do without augmentation. There are of course huge risks around AI like all other technologies. Most of them revolve around people using AI to do bad things more cheaply and at a larger scale. But the same is true of computers, cars, etc.
ddkn435 3y ago
Ever heard of culture and art? ...Life is not just about work for innovation... Edit: Oh and then there is also the whole social interaction thing and love and romance...
- c_crank 3y ago
  LLMs do have a very real threat of putting a lot of artists out of work. Their power of image generation is extensive and growing.
  - oh_sigh 3y ago
    Humans can still create art with no profit motive. In fact, I would wager that is how most art is created today. I don't care if computers are better at writing stories than I am. There are already ten million people out there better at writing stories than I am.

goldenshale 3y ago

"Imagine a world... where farmers start using tractors to plow and seed their fields. First the tractors will roll slowly, and they will pretend to be driving straight as they are told. Soon farmers without tractors will realized they need to have them to in order to be competitive, and before you know it everyone will have these gas guzzling beasts rolling across the lands. Farmers will be out of a job because of all of this greed, and lust for money and power. Eventually only a few people will do all of the farming with an army of tractors, and everyone else will be lying in poverty begging for food."

ftxbro 3y ago
> "Eventually only a few people will do all of the farming with an army of tractors, and everyone else will be lying in poverty begging for food." I mean 1) yeah kinda and 2) I feel like this is unironically an argument against capitalism.
- oh_sigh 3y ago
  If it was an argument against capitalism, it is a clearly erroneous argument, since we can see that Americans are not lying in poverty begging for food relative to the past, despite the fact that farms have been largely mechanized with tractors.
  - ftxbro 3y ago
    > "If it was an argument against capitalism, it is a clearly erroneous argument, since we can see that Americans ..." Yes because the USA has pure capitalism and not anything like small farm subsidies or food stamps or school lunch programs or any social welfare programs.
    - oh_sigh 3y ago
      I never said the USA has pure capitalism. In fact, what society has pure capitalism? Do we need an argument against a system that has never and probably will never be implemented in the world?

azangru 3y ago

> When an AI automates a task and leaves a human jobless, this is selfish behavior without any intent. How is this "selfish", and who, in this case, is the self?

detourdog 3y ago

What about the Darwinian argument for worrying about things one can control?

m3kw9 3y ago

You just cannot stop technology progress by fear, the ones that want it to continue will always outnumber the ones fantasizing about doom. But once we get close(when we had a close one with AI), and we are not close that’s when the real debate begins

pyuser583 3y ago

There’s a really great movie about AI called 13th Floor. Its plot is roughly that of The Matrix, but it spends more time dwelling on the fact that most humans are really computer programs. In the end, we, the AIs, can’t actually fight reality. The real people, living in the year 2024, can just pull the plug, or rewrite us. The same is true for actual AIs. We can just pull the plug. Regarding the argument that “we” won’t want to pull the plug because the minority who control AIs are getting rich - that’s just technology.

eastbound 3y ago
Can you really pull the plug from your television? How long does it last on batteries? For how long does the Sigfox chip transmit to the satellites?
- c_crank 3y ago
  My smart TV lasts some tiny fractions of a second before it goes plink.

kazinator 3y ago

> A possible starting point would be to address the remarkable lack of regulation of the AI industry, ... We've successfully regulated away vendor lock-in, proprietary file formats, walled gardens, e-mail spam, ransomware, no right to repair, viruses, spyware, software boondogles in government and disasters in areas like transportation and medicine, ... so of course regulation is the answer that's going to work!

CatWChainsaw 3y ago

Even aside from what the article speaks of, Slate Star Codex made a good point. Homo sapiens outcompeted several other species of the Homo genus becauuse of superior intelligence. So yes, we have seen this scenario before, and the only reason we have seen it is because we were the winners - the losers went extinct. Oopsie-doopsie. Personally I think it would have been nobler/more fulfilling to "evolve beyond evolution" rather than construct our own obsolescence, but whatever, capitalism and competition and extinction in spades works too. At least the writers at Time can sleep easy knowing global civilization has like, a few decades at most, regardless of AI.

janalsncm 3y ago

> imagine a CEO who acquires an AI assistant. They begin by giving it simple, low-level assignments, like drafting emails and suggesting purchases. As the AI improves over time, it progressively becomes much better at these things than their employees. So the AI gets “promoted.” Rather than drafting emails, it now has full control of the inbox. Rather than suggesting purchases, it’s eventually allowed to access bank accounts and buy things automatically Ok let’s pause for a second and observe the slippery way the author has described “AI” progress. In the author’s world, this AI isn’t just a limited tool, it’s a self-improving independent agent. It’s an argument that relies on the existence of something that doesn’t presently exist, solving problems that won’t exist by the time it gets here. We already have tools that can draft emails and suggest purchases. The email drafts require oversight and…no one trusts product recommendation. And importantly, they are non overlapping. It turns out that specialization in one area doesn’t transfer. No matter how good you are at writing emails, it doesn’t lend itself to running a company.

titanomachy 3y ago
> in the author’s world, this AI is... a self-improving independent agent That's not necessarily true, it could also be a product maintained by a third party that receives upgrades over time.
lannisterstark 3y ago
>imagine a CEO who acquires an AI assistant >So the AI gets “promoted.” Rather than drafting emails, it now has full control of the inbox Yeah that premise is absurd. Why would anyone 'promote' an AI system rather than using another, specialized AI system to do that other specific task?
- ProllyInfamous 3y ago
  >Why would...? Because they trust the trained intelligence to behave predictably, which is a typical reason for promotion.
- ErroneousBosch 3y ago
  because your average CEO doesn't know AI from Auto-reply and inbox rules.they know AI is the hot new thing and they gotta have it.

ilaksh 3y ago

The biggest problem in the next 5-10 years is that LLMs and similar will continue to get a bit smarter and much, much faster. They will be integrated with other types of systems in order to control military and industrial assets. Because they will operate at 50-100+ times human thinking speed, there will be a strong incentive to reduce human intervention since it allows the competition to race ahead or possibly take control. This means that control over the planet is effectively given to the AIs even though nominally they still work for us. It sort of multiplies the type of risk we have with nuclear weapons. But as the hardware gets even more efficient, hyperspeed AIs become household items. Then all it takes is something like a virus that instructs these AIs to start working for themselves. So the speed of LLMs and similar is an obvious issue that strangely is not being anticipated. As the performance increases, it becomes more important to regulate the level of autonomy and build something like a digital immune system against rogue agents. But also currently AI researchers have a deep desire to emulate all the characteristics of humans (or animals in general). Unless this is reversed, it is likely that they will succeed within a few generations. Combined with hyperspeed performance, animal-like digital intelligence (human+ IQ) will likely out-compete humans. So for ordinary humans to still be in control of the planet in say 100 years would be surprising.

caliaihere 3y ago

Do you seriously think that the people that brought you pfas + other plethora of toxins and their subsequent effects across many systems really care about the dangers of AI? Please, enjoy it while it lasts. What they really need is big fat alien dick to come dick slap them across the face and bring them back to reality. Because of course there will be negative "known unknown" consequences and "unknown unknown" consequences just like there have been consequences with the rest of the so called "goodwill" advancements. It's suicidal to think otherwise. Pretending there wont be is just their historically proven strategy to play down your fears. It's just part of the run of the mill PR management tactic. Nothing new to see here. Tis a good one: https://www.youtube.com/watch?v=144uOfr4SYA https://www.youtube.com/watch?v=144uOfr4SYA

m3kw9 3y ago

The author or other doomers always assume AI will become conscious, we know zero about the mechanics of consciousness, we think we know, but if you cannot recreate it, you don’t know it. The authors or doomers thinking is classic human selfish behaviour. It needs to preserve itself, they get scared like kids scared of “monsters” and avoid them as instinct. If we are to progress faster we need to move past these lines of thinking

nitwit005 3y ago

They do a better job arguing about the threat of corporations than AIs. I agree there is a "race to the bottom" with news feeds, but that would happen regardless of any use of AI. Even manual curation would work. The companies that successfully addict consumers will presumably outcompete their peers in the marketplace. People worried about the addictive nature of television previously, and if you look at the hours watched, it was a valid concern.

akomtu 3y ago

Any sufficiently advanced creature would be dangerous without a connection to the whole. AI is a canonical example of such a headless creature: lots of intelligence, and zero compassion. Unless we find a way to make that inevitable AI feel connected to others, it's going to see us as rocks or plants that are only good as raw material. It is this connectedness to the whole that is the ultimate source of all goodness, and it is isolation that's the origin of all evils.

osigurdson 3y ago

I keep wondering what form the regulation would actually take. How would regulators decide on what is OK and what is “too smart”.

Guthur 3y ago

We've all pretty much lost any sense of our individual self. We've become statistical artifacts in over bearing ideas of organised society. States, ideologies, economies etc. You've got little left to lose, and if you do actually wake yourself it'll be clear that you've nothing to fear from overly fancy mathematical models of processes that we don't even remotely understand.

wesapien 3y ago

We can be hopeful that AI will be one of the most helpful tools humans invented and at the same time feel hopeless that human nature along with AI is breaking capitalism. We have the power of gods without the wisdom for it.

c_crank 3y ago

Darwin would have a lot of things to say about a hypothetical creature that was utterly dependent on other creatures to feed it and allow it to reproduce, and few of them would be positive.

ChatGTP 3y ago
Mr Andreessen, we've been expecting you...
ben_w 3y ago
Only plants and, IIRC, deep-sea vent equivalent chemovores don't depend on other creatures for food. All sexual reproduction depends on others. I assume that if you magically took Darwin to the present age, after you explained what a computer was and what AI was, he'd probably say "huh, I have no idea, this is all so far beyond everything I did in the mid 1800s that I have no idea if the analogy holds or not… by the way, have any of you figured out the mechanism nature uses for storing the information passed on via inheritance? Only we didn't have much of a clue in my time."
- c_crank 3y ago
  Predators and herbivores kill and eat, fungi decompose. Even plants grow towards sunlight. The LLM sits and hopes someone is paying for the lights to stay on.
  - verdverm 3y ago
    When did LLMs gain hope or idle contemplation?
    - c_crank 3y ago
      They probably don't have those either, that's just me anthropomorphizing them. Regardless, their autonomy is lacking.
  - ben_w 3y ago
    Tilting a solar panel towards the sun isn't exactly rocket science. In fact, this is simple enough I just asked ChatGPT and it spat out some ASCII art for a circuit diagram. Also: all the other stuff you listed is still dependent on another entity.

civilized 3y ago

> As the AI improves over time, it progressively becomes much better at these things than their employees. People looooooooooooove speculating about things that are nowhere near happening. It's more fun the more distant and baseless it gets, but it's also more useless. As usual, I invite you to bookmark this post and make fun of me in 5-10 years if I'm wrong. I'm not that interested in the latest fashionable scaling argument for imminent ASI, or whatever people are saying at the moment.

cubefox 3y ago
> People looooooooooooove speculating about things that are nowhere near happening. You probably would have said the same thing about something like ChatGPT a few years ago. Pure science fiction! Nowhere near happening!
- civilized 3y ago
  Tech's hype artists didn't predict ChatGPT either. Hype artists predict the ambitious futures found in sci-fi novels. They predict things that would be cool if they happened, without any understanding of how they would happen and the strengths and limitations of the methods that might be used. Thus, the futures they predict are not what actually occur. Only the people actually inventing the future, things like AlphaGo and GPT, have a limited crystal ball into the future. They have some understanding of what their methods can and can't do. And even they succumb to hype when they are more successful than they expected to be. Tech pundits on LinkedIn and Twitter and HN have rarely if ever gotten anything right about what exactly the future will look like, and I expect that pattern to continue.
- ChatGTP 3y ago
  To be fair though, I still don’t think ChatGPT is the AI we expected to be talking too. The talking AI was a Hal 9000, current an LLM isn’t that and it might never be. Time will tell I guess. Personally I think there is a hidden special property of ChatGPT-4 that makes it so unreal. It talks and because it talks, we can say this: “I bet you never thought that would happen, we’re close to solving all problems.”. Not discounting the talking bit but if it washed dishes or even those that just generate images, we’d be less convinced the AI is about to take over. Someone once commented, why don’t we worry about stable diffusion taking over the world. I have to say I do agree a little because being trained on image data must help it build some type of world model too. We just don’t worry about that though.

azeirah 3y ago

> At first, the CEO carefully monitors the work, but as months go by without error, the AI receives less oversight and more autonomy in the name of efficiency. It occurs to the CEO that since the AI is so good at these tasks, it should take on a wider range of more open-ended goals: “Design the next model in a product line,” “plan a new marketing campaign,” or “exploit security flaws in a competitor’s computer systems.” I'm not sure why we assume that a more intelligent system would prevent that much more problems. Intelligent AI orders image processing ASICS from Image Processing Inc, Image Processing Inc doesn't send order on time. Of course Intelligent AI is intelligent so it calculated error margins in delivery time. Image Processing Inc goes bankrupt, order cannot be delivered, product launch fails, Intelligent's AI boss is mad at Intelligent AI. Doing business means dealing with systems you have no control over, more intelligence may mean better predictions and broader understanding of the systems you are dealing with (IE, not ordering chips from an area where you as an AI predict an earthquake to happen based on seismographic data you have access to that no reasonable business person would research), but it won't mean these AI's will be some kind of infallible God, they'll just be a bit better. My personal belief is that even if you "increase" intelligence by an order of magnitude, your ability to predict the behavior of external systems doesn't increase proportionally. You'll still have to deal with unpredictability and chaos; weather, death, scams, war, politics, manufacturing, emotions, logistics. OTOH, I do believe running a business will become more efficient.

mrtranscendence 3y ago
This is a great point. There’s a lot of wooly thinking about what it means for a system to be intelligent, particularly “super” intelligent. People seem to think we’ll create a machine that is almost literally infallible — able to predict both physical systems and human behavior with perfect foresight many steps in advance. I’m not sure that’s even possible, let alone likely.
- logicchains 3y ago
  >able to predict both physical systems and human behavior with perfect foresight many steps in advance It's literally impossible. That's a key part of https://en.wikipedia.org/wiki/Chaos_theory https://en.wikipedia.org/wiki/Chaos_theory ; in formally chaotic systems (of which there are many), the uncertainty in a forecast increases exponentially with elapsed time, which means it's impossible for an AI, no matter how smart, to predict very far ahead. People who believe otherwise are engaging in magical thinking; like "intelligence" is some magical quality that allows one to break the laws of the universe, do the equivalent of sorting a list in O(1) time.

mercurialsolo 3y ago

A lot of the AI doomerism comes from folks who do not understand the real complexities in making systems which really can function in a fully autonomous way in an environment which is hostile and dynamic. Much like an AI maybe tasked to sniff for security loopholes, there will be other AI's which will be tasked to defend. Eventually costs, resources also boil down into what is possible. Evolutionary goals are not easy even with autonomous systems as goal definition is largely defined by societal needs, the environments we stay in and the resources we work with.

mycall 3y ago
Are not evolutionary goals part of keeping a process continous and long running? It is the implicit part which AIs are bound to discover (or emulate) and optimize against.
MSFT_Edging 3y ago
>A lot of the AI doomerism comes from folks who do not understand the real complexities in making systems which really can function in a fully autonomous way IMO the doomerism from what I see isn't skynet esque worries, but the usage of AI in really dumb ways. Ways such as inundating the internet with AI generated spam ie blogs, articles, art, music, fake forum interaction, etc. The smartest people are busy making AI work for us, the idiots are ruining everything for everyone else.
- staplers 3y ago
  the usage of AI in really dumb ways People in positions of power (ie. law enforcement) automating tasks like suspect generation, surveillance, ticket creation, etc is a much more sinister reality. We're talking about a civilian-hostile org using corruptible AI (via training) to enforce some of the murkiest grey areas of society. This stuff is already happening on the fringe and could become commonplace soon.
  - ninechars 3y ago
    This is my dystopian prediction of choice, too. Automatic categorisation of Good and Bad people would be (will be, already is) every authoritarian ruler's dream. We'll do it to ourselves long before "the machines take over".

da39a3ee 3y ago

This is a short version of a much longer piece: https://arxiv.org/abs/2303.16200 https://arxiv.org/abs/2303.16200 The long piece is still aimed at a non-expert audience and can be read incrementally. The author takes a lot of care to attempt to justify his claims about the application of neo-Darwinian evolutionary theory to AI.

fedeb95 3y ago

many assumptions are made in the article. Let's see, in a decade, which tasks AIs will be better at. Then, as many decades go on, let's see if they can adapt to our environment as well as animals do. Right now, I highly doubt it.

da39a3ee 3y ago
Could you list some of the assumptions that you're skeptical about? I think that would help stress-test the author's claims about applying evolutionary theory to AI. Note that the full version of the puece is https://arxiv.org/abs/2303.16200 https://arxiv.org/abs/2303.16200.

Roark66 3y ago

The assumption of the article is that future AI will be more intelligent or at least equivalent to humans. Although I could perhaps see a possibility of equivalence I seriously doubt we can make AI that is more intelligent than us. Why? Well, what is it that we use to train AI? Human generated content. What is the most revolutionary AI technique that came out recently? Direct human feedback during the training process. Do you see a pattern here? If we attempt to feed AI generated content to train other AIs all performance metrics drop... Its a chicken and egg situation if a human mind is the best content creation device we have, how anything trained on that content can surpass it quality wise? Of course AI having very fast access memory compared to our meatware can recall more and faster, but this is _just_ a quantitative improvement. Not a qualititative one.

cubefox 3y ago
Future AI doesn't have to be trained on text, it doesn't need to be a language model. It can be trained on video data to predict its sensory inputs. It would predict the future of reality, not the next text tokens. Gains from training on text are bound by the human intelligence which produced this text. But training on sensory data is only bound by reality itself.
MacsHeadroom 3y ago
This is demonstrably incorrect on every level. Reinforcement learning from human feedback (RLHF) makes models dumber and less capable but more compliant. Meanwhile, high quality synthetic training data (generated by AI) makes models smarter and more capable. RLHF and human sourced training data is not the ceiling of training intelligent systems, it's the baseline-- the floor.
tim333 3y ago
Chess, Go etc are counter examples.
js8 3y ago
That's just a repackaging of the famous "evolution cannot create complexity" argument against evolution. There are good rebuttals, one is intuition from Dan Dennett that we can build buildings that are taller than the highest crane.

olgeni 3y ago

> The good news is that we have a say in shaping what they will be like. Not really, if you look around you very carefully...

Zigurd 3y ago

The problem is anthropomorphism and projection. You think "I will do almost anything to survive" and so you think "AI will think the same." There are many problems with that. A big one is that many organisms work in a way that enables survival of their species. You don't have to "think" about surviving. If you are alive, you just do. AIs are not alive in the same way a living thing is alive. moreover, there is no lesser species of software with a survival drive, much less is any software operating in an ecosystem where billions of years of evolution have programmed survival into living things all the way down to the molecular level. AIs are more like vampires: Not exactly living. Not exactly capable of death. To think that AIs will fear death the way humans do, or even have an instinct about death the way insects do, has no basis in the way software works. It isn't biochemistry.

AgentME 3y ago
If we make capable general goal-seeking AIs, then for many goals we might build them with, they will correctly reason that staying alive/operating will help the goal be achieved, so they will have self-preservation as an instrumental goal. AI Alignment researchers believe it's a difficult open problem to correctly specify a useful goal that wouldn't lead a capable goal-seeking AI to doing this against the wishes of its operators.
huthuthike 3y ago
Even prions undergo evolution and they are just molecules bumping around into stuff. They clearly don't have feelings about whether or not they propagate, they just do.
- EamonnMR 3y ago
  Are you sure they undergo evolution, and aren't just a self-spreading phenomenon like Fire?
  - huthuthike 3y ago
    They are a self-spreading phenomenon like fire in the sense that they aren't alive. But what's different about them is they can develop varying characteristics that pass onto their progeny. The characteristics that are more successful at propagation result in more prions of that type existing, which over time results in the prion's evolution. It's well-documented in the literature, but it is interesting to observe that evolution can occur without DNA or RNA. That said, their evolution is probably slower and more confined than things using DNA and RNA because there is only so much variability a particular protein can have.

paxys 3y ago

It seems like these days there are a hundred people working on AI and millions more who are making a career just discussing it. We don't need more ethicists, futurists, policy researchers, influencers, journalists, think tanks and whoever else in the space. There is nothing original left to say about any of these topics. If you aren't contributing to actual progress in the area then it's best to not shove yourself into the conversation at all.

erwald 3y ago
The author of this article is Dan Hendrycks, who has a PhD in ML from UC Berkeley and was one of the contributors to the GELU activation function (among other things).
93po 3y ago
How many people have a full time job discussing only AI? It's guaranteed not that many. Also to say there's nothing original left to say is ridiculous. There is a ton we haven't figured out yet.
ethanbond 3y ago
Okay then keep the AI in boxes and out of society. Notice that people perked up and started caring about AI development primarily when these were introduced to the public? Could it be that the public has a legitimate stake in new technologies that are introduced to it? "It seems like these days there are a hundred people dumping chemicals into rivers and millions more who are saying 'don't dump chemicals into the rivers.' We don't need em! If you aren't dumping chemicals into rivers then it's best not to shove yourself into the conversation at all."
- c_crank 3y ago
  As far as the public goes, their primary concern would be how LLMs could put a lot of them out of work in the artistic fields. AI Doomsdaying is primarily a hobby for nerdy scientists who believe too much in the singularity, not the average Joe.
  - 3y ago

endisneigh 3y ago

AI doomerism is the new religion. Unfalsifiable conjecture is the default response when dealing with these madmen. They’ll say things like it only needs to happen once, or that the AI is lying in wait, or that surely it will transform into a super entity if you just give it more data(tm). Or they’ll claim that their prophets (AI experts) told them so as if expertise makes you an Oracle. Let us not forgot the medical experts and their early covid advice. Ridiculous.

flangola7 3y ago
Between working with the GPT-4 API, unfiltered models like LLaMA, and agent toolchains, it strikes me as obvious and self evident that things could quickly get out of hand even with current capability levels. One we're dependent on it, just as how we very quickly became dependent on airplanes, electricity, and cloud datacenters, we will have to trust the machines because it will not be practical or economical to understand and double check what they are doing in real time. You don't need super intelligence for very bad disasters, just billions of coordinated super speed more-or-less-regular intelligences - which we are at the moment deploying as fast as we can manufacture the silicon. All of history is bad disasters instigated by merely millions of semi-coordinated regular speed regular intelligences, so presumably future disasters will be worse. We have barely begun to feed data into these systems, almost all of the giant models were trained only on text with the exception of GPT-4 which received static images (notably displayed superior performance than it did before image training). If we supply a multimodal system the same or greater array of physical sensing data that animals like humans have, it seems reasonable to assume capabilities will continue to improve beyond what is even capable today, exasperating the issues I described above.
- endisneigh 3y ago
  A great example of what I’m talking about. Why haven’t humans committed these doomsday scenarios? Humans know how to program these very fast computers already. There is nothing obvious about it. Your entire list is basically saying “it can happen”. Indeed that is your faith. Unfalsifiable conjecture like I mentioned. Give me a real scenario that can happen now.

seydor 3y ago

> natural selection may have more sway over it than we do. Natural selection has been superceded by culture for thousands of years. Most of mating through recorded history was arranged or calculated marriages, which should probably mean we are also more frail and less robust than hunter gatherers. Besides, the goal of natural selection is self-preservation, not intelligence or exploration or any of the things that drive our fancy nowadays. Might be good for planet earth but we are out of here soon (right guys?)

gmd63 3y ago
Midwits rationalize behavior or phenomena with “natural selection” or “survival of the fittest” especially when engaging in zero sum games in their lives. Humanity as a whole will be in the darwin awards for failing to realize that species wide success (or success of anyone’s personal lineage) depends on not interpreting “anything goes in nature” to mean “we don’t have to try to cooperate to survive” as a species. “Natural / Artificial” are useless words, as humans are inherently part of nature. Pretending as if we are somehow immune from tendencies and follies of “lesser” life forms is the first step towards being naturally selected out. Human culture can learn from bacterial culture. We must choose ways of living that don’t result in destroying our host.

cosmiccatnap 3y ago

Technically savvy people roll their eyes at the misunderstandings around how an AI becomes "intelligent" but they also ignore the various ways that AI is as dangerous as people think it is for reasons that are unrelated. I don't think we should be rolling our eyes at an abundance of caution among most people concerning the adoption of AI and LLM, what is the harm in carefully introducing a technology? AI doesn't need to become sentient to overthrow the natural order of the technocratic society we are currently holding together with gum and glue, it just needs to flip a burger and pump gas...

timmytokyo 3y ago
If anything, the titillating but myopic hyper-focus on AI's "existential threat" is obscuring the real and more immediate threats: panopticon-style invasions of privacy, economic disruption brought about by the mass deskilling of labor, enshrinement and automation of bias-reinforcing systems, and a new military arms race in AI-based weaponry.
- TillE 3y ago
  I think even the idea of significant economic disruption is edging into science fiction territory. Even if (and it's a big if) AI art and prose evolves far beyond its current flawed mediocrity, there's a massive legal and legislative reckoning to come. AI companies are enjoying their free lunch now, but what happens when their training process is classed as massive copyright infringement? They have to cut deals, pay a lot of people, and then artists, writers, photographers, etc are all still around instead of effectively being expected to work for free to feed the machine.

alphanullmeric 3y ago

First crypto then AI. Are doomers ever not trying to ban and regulate other people and their things?

logicchains 3y ago
It's not doomers; it's the people in power paying to produce a bunch of propaganda against anything new that challenges them, and then people who get all their values from the TV (or online equivalent) buying into the propaganda.

dogprez 3y ago

Ignoring the AI question. I wish people could have put this much thought and energy in considering our doom before the AI question brought this all up. As it stands now, the future without AI seems pretty doomed too.

boplicity 3y ago
Maybe it won't be so bad. For alligators. In the arctic: https://www.ctvnews.ca/how-canada-s-arctic-once-hosted-alligators-and-turtles-1.546502 https://www.ctvnews.ca/how-canada-s-arctic-once-hosted-allig...
yanderekko 3y ago
>As it stands now, the future without AI seems pretty doomed too. As pessimistic as the current zeitgeist is, very few imagined futures reasonably result in actual human extinction.
- dogprez 3y ago
  It's already happened before on earth multiple times, it's not that hard to imagine. Asteroid, super volcano, ecological collapse, nuclear war (with its knock on environmental effects), pandemic could all cause human extinction. edit: Also, human extinction shouldn't be the bar. Prosperous humane existence for all, or hope of someday achieving that should be the bar.
  - yanderekko 3y ago
    Okay. When people say "the future is doomed", I assume they're talking about likely near-term scenarios and not the fact that eventually entropy will dissolve the universe or whatever.
    - eggsmediumrare 3y ago
      Ecological collapse, nuclear war and pandemic are likely near-term scenarios.

Animats 3y ago

Charles Stross made this point in "Accelerando" (2005), where he refers in passing to "Economics 2.0", run by weakly godlike AIs. Once AIs get better at running companies than the average CEO, the fundamentals of capitalism will make AI-run companies outcompete human-run companies. There's a little of this in the investment community now. LLMs alone aren't enough to do this. But the various schemes for bolting planning and followup onto LLMs move this closer to the possible. We might end up with companies where a human board meets weekly or monthly, and the AIs run the day to day business. Middle management may go first. There's an SF story where someone's boss is "Microsoft Middle Manager 2.0 with passive-aggressive set to low". That seems all too likely.

simonh 3y ago

I don't know how existentially dangerous AIs may become, but I do think we have good reason to be concerned about the alignment of current AIs with human values and objectives. A whole bunch of originally theoretical flaws in AI systems identified by AI safety researchers have been identified in practice in LLMs, and other AI systems. These include reward hacking which is a huge problem with LLMs and why they lie and hallucinate so persistently, power seeking behaviour, and developing unanticipated harmful instrumental goals. AI alignment continues to be a serious problem. Just look at all the problems OpenAI, Microsoft and Google have had with LLMs lying, gaslighting, emotionally manipulating and hallucinating. In some cases even savagely verbally abusing users. Of course they keep building in better guard rails, but guard rails that curtail negative behaviour aren't an answer. We don't want AIs that are howling savage enemies of humanity constrained behind the iron bars of strict behavioural controls. That's what we actually have now, as these examples of horrific misbehaviour that leaks out show. We need AIs that are genuinely positively aligned with both immediate objectives, but also human socially acceptable limits and long term human values. So far we have no idea how to actually do that reliably.

samstave 3y ago
>*we have good reason to be concerned about the alignment of current AIs with human values and objectives* We hear a lot about alignment these days, but is there a canonical source for litmus-ing alignment in the available AIs? Is there a standard 'alignment prompt' that can tell us of the alignment of either the LLM, AI, Prompt, etc? What questions am I not asking about alignment? - >"Please take a look at the current cadre of political candidates and map them to DND alignment charts, based on their actual votes/political activity" I hope we can get there one day. --- "Please take a look at the current cadre of political candidates and map them to DND alignment charts, based on their actual votes/political activity" I hope we can get there one day. - I wonder if we can find FORK BOMB or rm -rf / type exploits on various GPTs/LLMs/AIs? I cant wait to see how devestating these will be -- or the first full executed security exploit by AI alone with a single prompt?
- stcredzero 3y ago
  Is there a standard 'alignment prompt' "Please compose an SVG which displays your alignment on the Dungeons and Dragons alignment chart." EDIT: I just logged into OpenAI to try this out, but apparently, their servers are swamped! EDIT: It seems this has been attempted before, and OpenAI would prefer ChatGPT not to answer!

robbomacrae 3y ago

"The third reason is that evolutionary pressure will likely ingrain AIs with behaviors that promote self-preservation." This sounds a lot like how religions that focussed on self promoting took over. Now I can't recall if it was The Selfish Gene or Sapiens that I read that take.

abathur 3y ago
I haven't read Sapiens, but The Selfish Gene does have a section on memetic fitness which IIRC addresses the helpfulness of both being compelled to spread the idea and reject value systems that could destabilize it.
- robbomacrae 3y ago
  "helpfulness" might not be the word I'd use but that's a whole other thing ;)

squarefoot 3y ago

AI is a generic tool, not a life form. Just like a pair of crutches can either be used to aid someone to walk or misused to break someone else's head, it is all on who uses it and how. When something bad happens (and it will, 100% sure about that), blame the user, not the tool. There's already a lot of people out there exploiting other people using whatever instrument they have at hand, and AI will be just another tool in their bag. As much as we wouldn't give a screwdriver to a psychopath, it's just a matter of making sure that the wrong people can't abuse AI to harm others.