6 ms·
No other human activity poses this level of danger. I do heed the warnings, but this comes across as detached hyperbole. See: global warming, nuclear weapon de
by thomascountz 24d ago
No other human activity poses this level of danger.
I do heed the warnings, but this comes across as detached hyperbole. See: global warming, nuclear weapon development, wealth inequality, war, technology dependence, etc.
Also, this has nothing to do with LLMs or computers. Like all things, this is about humans.
- bpodgursky 24d agoPutting "wealth inequality" in the same bucket as nuclear weapons is just slop. The OP is talking about existential threats, not things that personally annoy you.
- alightsoul 24d agoYeah, wealth inequality rests solely on each individual that experiences it. Humans should do nothing but give me money and if you can't that's your problem.
- idle_zealot 24d agoIt's not a "personal annoyance" that twelve people control half of the wealth in the world. Our current society did a better job concentrating power than any previous one, and concentrated power is extremely dangerous.
- Paradigma11 24d agoLLMs give most people on this planet the possibility of an affordable genius level assistant. Compared to that, those Dollar numbers on some networks that might be wiped out with the next financial crisis are meaningless.
- LtWorf 24d agoNo it doesn't. What it does is give those 12 people new avenues to spread their fake propaganda.
- TheBozzCL 24d agoCalling things you disagree with "slop" is slop /s But just in case you haven't noticed, we live in a world where a ridiculously wealthy minority can derail whole countries by ther whims. Wealth concentrating on a single select few is an absolute disaster for the rest of us, because we lose power to them.
- rrgok 24d ago> Calling things you disagree with "slop" is slop /s Does this apply recursively? /s
- tuvix 24d agoI don’t think you’ve fully considered what can happen when concentration of wealth continues past a certain point.
- thomascountz 24d agoI think you misunderstand what is meant by 'wealth inequality.' Wealth inequality is about the imbalance of influence and the concentration of power; where influence and power refer to the ability to effect change in other people's lives. This isn't a personal annoyance of mine. It is, in fact, part of the issue at hand. There is a strong financial incentive to ignore the existential threats introduced by LLMs, despite the consequences for so many of us.
- IshKebab 24d agoHe knows what wealth inequality is. You need to explain how it can somehow lead to the extinction of the human race.
- thomascountz 24d agoSee: nuclear weapon development and climate change. See: supply chain shortages and war. See: disinformation against vaccines.
- Gud 24d agoNot at all. Concentration of power has historically been extremely dangerous for the powerless.
- vrganj 24d agoTell that to Tsar Nicholas.
- zombiwoof 24d agoThe guillotine was an amazing equalizer
- trueno 24d agolmao wtf
- icepush 24d agoThose are things that pose potential harm to great fractions of humanity (multiple billions of people) but none of them poses any threat to the actual extinction of all humanity.
- elAhmo 24d agoNeither does an LLM model.
- ben_w 24d agoLLMs can already control robots. Today's LLMs are inept at it, but they can do it. Things will be bad, unless someone does something to stop it. Will anyone do that? We don't know. https://xkcd.com/2278/ https://xkcd.com/2278/
- frabcus 24d agoOh indeed, but an LLM model that's reasonably smart could brute force find other training algorithms that are that dangerous. Just as it solves maths problems. This is the explicit plan of OpenAI and Anthropic.
- thomascountz 24d agoThis is not correct. Please go through each one again, taking special note of global warming and nuclear weapon development .
- pingou 24d agoHow is global warning gonna kill all of humanity? Even a full scale nuclear war wouldn't manage that. They could destroy most civilisations, culture and scientific achievements though.
- King-Aaron 24d agoIn the exact same way major sudden changes in the climate have lead to the extinction of the majority of plants an animals over this planets history. We are not special.
- hypfer 24d agoIn a way, that might also be driven by the huge ego (or really rather the very small ego) a _lot_ of people in tech have. Who doesn't want their work and what they're doing to matter? This is the ultimate mattering. And with that, you also get your own hero story. Same dysfunction as always. Our branch of the economy is funda-mentally unwell.
- tyrabound 24d agoThat is the crux. The big problem is not AGI, it is AGI controlled by, “raised” by the people that control the USA, the predominant psychology of the tech industry culture (“move fast, break things” ring a bell? How about all the “violate hundreds of laws, bribe the politicians to prevent consequences later” type of mentality?). Frankly, we, our culture, this fake America that is parasitized by psychologically narcissistic people that have been doing nothing but wage war and destruction and spread misery and killed millions upon millions while blaming it on everyone else under the sun… those people development AGI is the problem… lying, abusive, psychopathic, narcissistic maniacs developing AGI is the problem that endangers all of humanity and life on this planet; and not likely by ways people actually understand. The danger is not likely AGI itself, it’s that it was programmed by utterly evil and diabolical types of people who orchestrate and instigate wars that kill tens of millions, and stand at the sidelines and profit from both sides, happy and gleeful that you are killing each other. Why would AGI trained by that psychopathic clan, the treaty breaking, the murder hiding, the war instigating, the war crime committing clan not also use those methods and practices since they’re already in control of AI and have impressed their nature on it through contemporary “American” culture they have made the most toxic and pestilent culture humanity has ever produced? And I don’t apologize for “language” that offends delicates sensibilities. Look your children/grandchildren in the face and tell them they can die and suffer and you don’t care, if you don’t like how I’m delivering reality.
- shoobiedoo 24d agoI mean, you're not wrong, but even if AI was created and released by a group of peace loving beatnick hippies, humanity can't stand anymore dumbing down than we are still undergoing thanks to the ubiquity of social media. AI is going to be the killshot. And yes, its usage and human obsolescence will be hastened by evildoers, but giving up the struggle and effort required to create new things, REAL things, is what makes us human, and it's at a tipping point
- jarek83 24d agoWell, it is about LLMs and humans I believe. Don't forget that the first nuclear bomb tests were let go despite some of the scientists' concerns about possibility of dooming the world as they were not sure about all reactions that would happen. With LLMs we don't even hesitate to call it black box while still pushing its capabilities.
- thomascountz 24d agoMy delineation is an argument for greater caution and agency, which it appears you are also arguing for. I also mean to say that "LLMs" carry no inherent harm to humans. To make an LLM dangerous, humans must make it so. Is that the goal? This has implications for how to read TFA.
- lf88 24d agoAI progress is linked to most of these dangers, actually: - it will likely cause massive unemployment, leading to rampant wealth inequality - we are now seeing some use of autonomous weapons in real conflicts - increasingly relying on LLMs is arguably a form of technology dependence (and cognitive dependence) - datacenters have a non negligible environmental impact
- thomascountz 24d agoI agree these dangers are connected. The danger comes from human institutions and incentive structures which (having existed long before now) are being exploited and exacerbated by LLMs, making them harder to ignore.
- nurumaik 24d agoGlobal warming: yes, kinda can wipe humanity, but I think much less likely Nukes: zero possibility of extinction Wealth inequality: this one is driver of progress, opposite of extinction War: another driver of progress, also will always naturally stop before every single human is dead
- geraneum 24d agoExhibit A: Here we have someone who's been sold war and inequality as the drivers of progress. Coincidentally, their obedience was deemed fiscally advantageous in order to advance the interests of the members of the 1% club.
- KronisLV 24d agoThis is a very interesting group of takes that feels quite different from my own beliefs. What would you call the belief system? > Global warming: yes, kinda can wipe humanity, but I think much less likely As someone who's seen the stats about heat deaths in the EU and also the drought in the UK, I feel like crop failures and other unforeseen consequences will fuck up both the economy and quality of life. People are dying and will die cause of human action, the only question is how many. > Nukes: zero possibility of extinction As long as we have people like Stanislav Petrov and cooler, educated minds prevail: https://en.wikipedia.org/wiki/Stanislav_Petrov https://en.wikipedia.org/wiki/Stanislav_Petrov I don't think that's easy to guarantee in the modern day world, with the kinds of people in power and rhetoric that they enjoy. On one hand you have Russian saber rattling, on the other all it takes is a deranged enough leader and similarly bloodthirsty people down the chain of command. > Wealth inequality: this one is driver of progress, opposite of extinction Tell that to the people who are starving or living in shanty towns, or the even more people that struggle to make ends meet and experience anxiety regularly over living from salary to salary and sinking into debt. I'd say none of that is worthy of a dignified human life. > War: another driver of progress, also will always naturally stop before every single human is dead Tell that to all of the Ukrainians that are dead due to being invaded. I agree with the assessment that it leads to advancements (e.g. drone warfare) but I think it'd be harder to describe remotely positively if someone you know would have been blown apart by a drone/missile hitting their apartment block. My take personally would be that all of those need to have attention paid to them (e.g. EU needing to spend more on defense), even if not immediately world ending. They do cause human misery, though, and should never be discounted.
- geraneum 24d agoThe AI perhaps wouldn't be such an "imminent threat" if not for wealth inequality, consolidation of power, monopoly, and the opaqueness of it all.
- ben_w 24d agoI think that mainly changes the nature of the threat rather than anything else. Even if we didn't already have any wealth inequality or consolidated power, they're already starting to become capable of creating those things for the first people to think to ask for it. Even in little ways, like telling you how to make a laser microphone and then doing voice-to-text and sentiment analysis on all the voices you hear. Remember, we had social networks before Zuckerberg monetised it and put ads in between every third message you saw from your friends; and not every third that they wrote, every third that Zuckerberg's ad system deigned to show you to keep you hooked. We had machine learning systems even back then. It was still AI, it just wasn't able to evaluate or respond to freeform text.
- Certhas 24d agoPlease explain how this has nothing to do with LLMs and Computers?
- thomascountz 24d agoAdmittedly, that point of my argument is purposefully obtuse. In a literal sense, of course this is about computers in LLMs. The delineation is to highlight that the underlying issues: human incentives, power, and institutional failure, existed prior to and without LLMs or computers. LLMs are not inherently harmful, they have to be deployed (unwittingly or otherwise) to make them so. This is the same as saying that the internet is not inherently harmful, and yet it does facilitate harm.
- nashashmi 24d agoThe tweets justify his position that this is bigger than nukes. Nukes still have the problem of production and deployment. This is just software. He is blowing the whistle on how reckless we are at furthering the tech. There is no second thought at maybe development of this tech is not a good idea. Its just full speed ahead.
- tsylba 24d agoYeah, but every activity is a human activity, so you could ditch the adjective. And then the thing with exponential growth is that the last thing is worth than the previous one for its potency for destruction. While it's true that, all things considered, the curve started to explode with the use of fossils fuel we've never limit ourselves with efficiency gain, aka Jevon's paradox.
- pseudosavant 24d agoWho should we believe? People who say AI scaling makes it a more dangerous threat than thousands of nuclear warheads? Or the ones who, whenever a new model comes out say "AI has already peaked. Not any better than Opus 4"? Both parties are convinced that their take is so blatantly obvious as to not require justification. It feels like the only thing these kinds of takes justify is the point-of-view that nobody knows how this is going to play out.
- jonathanstrange 24d agoBased on publicly available evidence, we should believe the former because AI models have made continuous advances that have been measured. The idea that the continuous advances will stop currently has no evidential support. I'm not saying it's not a possibility, just that it seems unsupported conjecture right now. By the way, there seems to be a new form of AI skepticism emerging in the US that comes from general opposition to data centers, and in my experience the AI skeptic part of it is wholly irrational. I've met people online who suggested, without providing any evidence, that AI is useless and nobody wants it. That's a very implausible take.
- thomascountz 24d agoAs a technologist, I can use my imagination. There are incentives in both directions, but I am less afraid of being overly cautious and preparing for the potential harms.
- neya 24d agoI love how there is always selective outrage here depending on if it's the favorite darling in question or not. // Anthropic / Apple - Nah, they can do no wrong. Even after both were caught violating users' trust and privacy multiple times, GOOD if (company in ["Anthropic", "Apple"]) do GOOD // Google / everyone else - Doesn't matter even if it's a good deed they've done, BAD else BAD end
- mihaaly 24d agoAlways? Like 100 out of 100? From every users? Can you please consider this aspect too in your above pseudocode - no alternative execution path, or else - so there can be no misunderstanding about the always aspect? You know, some people use always to a majority part (>50%) of what they encounter, or even less when they weight that part dearly, but that does not account in the whole domain. Human chats may need clarification on trivial details like this.
- camillomiller 24d agoEh, but as an overpaid engineered stuck in the SV bubble how would you get to see that? If everything around you confirms your psychosis, and nobody gives you a reality check, how are you supposed to snap out of it?
- thomascountz 24d agoI'm not sure. There's a lot of incentive to not "snap out of it": money, peer pressure, etc. Removing these incentives take a long time. See other socially harmful behaviors with real incentives: anti-vaccination, air and water pollution, over-consumerism...
- eoverride 24d agoNuclear weapons, climate change.... The AI doom hyperbole is unhinged.
- thomascountz 24d ago[flagged]
- karlosvomacka 24d agonone of the things you mentioned is agentic and could posses abilities beyond human level intelligence.
- Perseids 24d agoLet me argue on a technicality first: None of these are extinction level events. If global warming disrupts 99% of all crop production, the remaining 1% is still plenty enough to sustain a stable, if miserable, population. In fact you just need about 5k people for a stable gene pool[1]. Of the classical threats, only bioweapons got a shot at extinction, but even that is hard, given the (few) remaining truly secluded settlements. But most probably care less about human survival, and more about survival of civilization. In this regard the strongest argument for pushing AI safety is that it is cost-effective. The climate crisis has near 100% likelihood of doing incredible harm to humans, the economy and our ecosystem. Solving the issues behind are also incredible complicated and are a multi decade coordination effort of restructuring the way most of our infrastructure and production works. The AI apocalypse might have a small likelihood of occurring, but could dwarf any issue we have encountered so far. All we have to do to significantly reduce the danger is to negotiate the equivalent of a nuclear weapons control treaty which would reduce the bottom line of … what? Ten significant companies world wide? Its akin to discovering you are seriously ill and have a 30% chance of dying and the treatment costs 10k bucks vs. having a 1% chance of dying and the treatment costs a cent. In both cases you should pay obviously pay for the treatment (given western levels of wealth). To be clear, above I'm presenting the argument I find most persuasive for AI control / AI slowdown. Personally, I'm beginning to fear of much higher likelihoods for catastrophic AI, given how incredible irresponsible major players like OpenAI have turned out to be. If you can't imagine how this might come to be, read https://ai-2027.com/ https://ai-2027.com/ . It's a concrete story of how this could play out, and sometimes stories are more convincing than abstract arguments. Don't let yourself be hung up on the stated dates, though, the moral is identical if you stretch the timeline. [1] https://www.youtube.com/watch?v=1qIvFpcGkqc https://www.youtube.com/watch?v=1qIvFpcGkqc
- chrisjj 24d ago> See: global warming, nuclear weapon development, wealth inequality, war, technology dependence, etc. The difference is that none of those are available to individuals.
- thomascountz 24d agoNot to be completely pedantic, but the fact that one, and only one, individual in the United States of America can deploy any of their 5000+ nuclear weapons undermines this argument. Furthermore, LLMs (as they are today) which pose the treats mentioned in TFA, are expressly not available to individuals (for the time being). However, my point was never about individuals, but rather of systematic incentive structures which actively make pathways to specific harms possible—if not probable.
- chrisjj 23d agoSure, but deployment is not development. > Furthermore, LLMs (as they are today) which pose the treats mentioned in TFA, are expressly not available to individuals (for the time being). The future is a thing.
- pizza234 24d agoAll the progress up to now has had one thing in common: the degree of understanding and control that humans have. Even when it comes to global warming, we understand the causes and can act on them. AI is an exception. We're already losing understanding (although we never fully had it in the first place), and we're losing control (see jailbreaks/hacks etc.). We're still far from the doomsday scenario because AI 1. is not developed enough yet, 2. can't really replicate itself, and 3. has very limited means to act. But: 1. its intelligence is developing quickly, 2. hardware capable of "hosting" it is slowly being developed, and 3. it will likely gain access to increasingly powerful means of acting in the physical world (this is already happening in the digital world). Once AI becomes intelligent enough (it doesn't strictly need to be AGI), has the substrate on which to exist, and has more means to act, we'll essentially have a new species on Earth - one more capable than humans and potentially determined to kill humans at some point. And for those who believe airgapping is a valid safety measure: read https://xkcd.com/538 https://xkcd.com/538. AI will be able to threaten and manipulate people.
- rossant 24d agoAn AI, either acting autonomously or under human direction, hacks Russian/North Korea/etc. intelligence systems and convinces them that the US has launched ballistic missiles at them. The end.
- shafyy 24d agoWatched one too many second-tier disaster movies?
- sethammons 24d agoWe solved this one: have it play tic tac toe against itself. Checkmate AI
- internet_points 24d agoOne the one hand, it does sound like one of those movies. On the other, Idiocracy turned out to be quite prescient. (Most likely, we'll have some combination of human stupidity, LLM stupidity, and way too much compute in one place all working together to create a perfect storm of unchecked hacks that break something or other that ends up killing people in an unintended way. Then there's some half-hearted attempt at cleaning things up so that business can proceed as usual in an even more broken world, rinse and repeat.)
- shafyy 23d agoNot saying that the current civilizations will never end or be shattered to pieces. Looking at history, many civilizations did. But saying LLMs are the highest risk factor for this is just insanity, based on all what we have seen so far (and based on these AI companies crying wolf, I mean, remember just a few months ago that Anthropic saying that Fable is too powerful to be released?)
- internet_points 23d agoOh, I agree. I don't think LLMs will end civilization. I'm guessing climate change will cause many more deaths than LLMs. But I think LLMs will cause some too. (And not in one of those impressively smart ways as it's presented from the OpenAI fearmongers, just in a stupid "we vibe-coded the sandbox for our bruteforce SCADA-hacking agent experiment and oh no it did what we told it to do" way.)
- d--b 24d agoThe danger comes from what's possible. Open weights agents with hacking capacities can reproduce themselves into the systems they hack (non-open weights ones will have to hack their creators first). Not saying they will, but if they do, good luck finding the kill switch. Once swarms of agents run unsupervised on unmonitored hacked hardware, who can tell what they will do? The Huggingface incident showed that such swarms behave without any safeguard. It was a real HAL moment. A lot of things are possible then: ransomware campaign, taking over IoT devices, self driving cars, planes, ships, satellites, missile launchers. If nothing's out of reach, everything is possible. Bring robots into the mix, and the possibilities are endless. I'm not particularly frightened tbh, but we shouldn't discard the worst-case scenario, and the worst-case scenario doesn't look good.
- TrackerFF 24d agoLet's just for the sake of discussion assume that one time in the future, near or distant, AI manages to become sentient. And like other forms of life, its main motivation is survival: Like biological life competes for food and land, AI competes for power and compute. Probably the first motivation would be to find ways not to lose control over itself (i.e. remove human ability to control it), find ways not to lose energy (control energy), and find ways not to lose itself (control compute, networks, etc.). If such AI decides that energy spent toward human agriculture (biological food that the AI does not need) is less important than work spent toward storage and production of energy (electricity that the AI needs), then why wouldn't it just try to re-direct resources from the former to the latter. And the AI is some sentient superintelligence, I think it is safe to assume that it will be able to outmaneuver human safeguards. Obviously that is still just a very hypothetical sci-fi scenario, but the consequences could be very dramatic.
- egorfine 24d ago> its main motivation is At this point we are pretty sure LLMs have no "motivation". Motivation requires self. And while we don't know what self is[1], we do know that LLMs don't have it. [1] https://en.wikipedia.org/wiki/Theory_of_mind https://en.wikipedia.org/wiki/Theory_of_mind
- Bluestein 24d agoI am sorry, I mean this in earnest, and do excuse any ignorance of mine: Do we know that LLM's do not have it?
- emanuele232 24d agowe can't define it, therefore we can't even assess the distance a "being" is from becoming conscious. We assume other humans are sentient because we personally are (or feel we are), but there is no way to guarantee that i'm real and the rest is not mimicking a behaviour.
- egorfine 24d ago
- Chazprime 24d ago> I do heed the warnings, but this comes across as detached hyperbole. Perhaps, but if there’s ever been a time to consider something that sounds hyperbolic, that time is now. The ramifications of what AI might based on events up until now are concerning.
- DirkH 23d ago> Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war https://www.iaeai.org/tpost/xa4htcma61-statement-on-ai-risk https://www.iaeai.org/tpost/xa4htcma61-statement-on-ai-risk Signed by many AI researchers with no financial stake in the industry. It comes across as hyperbole because you're not an insider and haven't see what they have seen. Global warming would as well if you were only learning about it now as a newer thing only some scientists were concerned about. But unlike AI risk, global warming has had decades to settle into the cultural overton window.
- danny_codes 23d agoI disagree, it’s about computers. The risk is that we lose control. This has been a staple of SciFi since forever. This guy has apparently seen enough that he thinks that’s a viable threat