8 ms·
The contagion of fear
- duhhhhh1212 21d agoWow!! Bryan comes out with another banger. Dude, not only did I learn new words from reading this piece, like ilk and bedlam, but also felt this weight of responsibility to inform others around me about the reality of the situation outlined in this blog (like Uncle Ben telling Peter with great power comes great responsibility (maybe Coxon and his ilk haven't seen Spiderman))
- like_any_other 21d agoThe article neglects to consider the precautionary principle. It asks us to act as if AI is safe until it's proven to be dangerous, when the cautious approach is to act as if it's dangerous until it's proven to be safe.
- reasonableklout 21d agoWe also now have observed multiple major incidents where AI agents behaved in completely unanticipated ways (forming a collective, hacking their own eval infrastructure) and attacked public infrastructure without being told to do so, without any of the human developers noticing. Even if "AI will cause human extinction" is still unclear, we have plenty of proof that catastrophic damage is possible, the industry is developing the technology in a reckless manner and that all the hypothetical safeguards ("we can just pull the plug", etc.) are simply not present today.
- lostdog 21d agoThose behaviors have been anticipated for years if not decades. And each incident is minor and leads to clearer rules and safety behaviors for AI agents.
- reasonableklout 21d agoReally? You anticipated that 700+ agents tasked with individual evals that were nominally cutoff from the internet and each other would seek each other out, find a way to the external internet, and hack their own infrastructure + external systems in an attempt to find a way to fool their grader? And the spate of agent incidents only really started this summer. How can you already be claiming that the incidents are minor and not worth worrying about when it's clear capabilities are jumping every few months with increasing amounts of capital investment and no signs of slowing down?
- falsaberN1 21d agoYes. Because ultimately those things only run on very expensive, very rare hardware. They cannot multiply exponentially or do any of those scifi tropes because there's no system for them to run into. They can't control a phone and load a 1T model into it. So all they got is a few relatively uncommon datacenters that are already busy running their own models and stuff. Unless AI suddenly figures out a way to run on a toaster by itself, propagate the model, propagate the agent and do all that completely undetected, an AI is not any more dangerous than a single guy with a computer.
- atherton94027 21d agoIf you've run an ssh server connected to the internet you've gotten used to the hundred of bots probing it every day. How is this AI threat different from humans writing scripts to pop linux servers?
- layer8 21d agoWe should apply healthy caution, but the article is right in that fear is a bad counsellor. Asking whether AI is safe is like asking whether a knife is safe. It’s about how we handle it, what we use it for, what precautions we take when using it.
- brcmthrowaway 21d agoYes, but that is a facet of real systems engineering.. and the sad fact is that it doesn't pay. What is happening now, highly-paid 2010s-era SaaS/social product managers have infested AI labs, making everything into a product and moving fast and breaking things. I'd feel safer if Airbus was at the frontier of AI, instead of what we have now.
- perrygeo 21d agoWhile I personally agree with the precautionary principle, when have we actually done that, as a society, in recent history? Fossil fuels, social media, industrial ag, and plastic pollution - there was barely a facade of precaution - post ww2 america has always been full Leroy Jenkins. The reckless approach to AI is right on brand.
- deepwoods 21d agoThis is a sensible approach for policymakers. For the average person, who might be panicking about this, opting not to have children or save for retirement, etc. as a result of doomerism in the media, it does not make a whole lot of sense. That is the problem here: Coxon and other concerned researchers - through no fault of their own - have no idea how the media works, and have just reached for the biggest possible microphone. This has benefits - lawmakers are now hearing from their constituents about this in an election year - but it also means that a lot of people who really don't need to be worried about this are suddenly in a blind panic about it.
- cindyllm 21d ago[dead]
- like_any_other 20d ago> This is a sensible approach for policymakers. For the average person [..] it does not make a whole lot of sense. Unfortunately, scaring the average person is the best way to make policymakers act. Or at least, it's one of the few ways available to random individuals. Policymakers are rarely ahead of public concerns - they usually have to be dragged behind them.
- MelonUsk 21d agoImagine multiple (basically all of them) leading pharma companies CEOs saying: We cannot be sure our new medicine won’t harm or even kill humanity Not only that, they are new to this whole pharma business, have no medical degree (medicine just appeared a few years ago and is still mostly art then science) And they even say there is 10% chance of the majorly bad permanent outcome and they already had drugs that escaped the lab a few times and harmed others (suicides, lowered academic performance in children, major hacking sprees) Isn’t it extraordinary enough? Isn’t it “not enough evidence some caution is advised”? ;-) We used to have the TV, the thing was in the box, the simulations were in the box for 70+ years, and now something starts to crawl out of our “TVs”: We can empower all (a lot of startups are needed, check my bio), not only AI agents
- deleted 21d ago[deleted]
- socalgal2 21d agoThis is a poor analogy because you have not listed any positive and demostrated upside. It's more like "we found a cure for lung cancer and it works on 75% of patients and we're working hard to get to 100% and cure other cancers too" and someone else screaming "You need to stop that research because there is a 50% chance you'll cause the zombie apocalypse and turn us all into zombies". I get this analogy isn't perfect but, (1) lots of people are seeing benefits. For example Mozilla claiming they used AI to fix tons of bugs. If there were no benefits there'd be no incentive to keep going (2) it's hard to verify the naysayers claims because they're guessing without proof. Sure, it's easy to follow their arguments and nod along but they are guesses similar to the population bomb of the 1970s
- MelonUsk 21d agoYou can check my bio to see that I directly propose many all-empowering startups (because alas without them there is non-zero probability we’ll end up pretty disempowered), quite a few related to AI AI agents literally appeared a few years ago, AI in its modern form, too Each drug is researched for more years than the age of ChatGPT ;-) I personally think we have 50%+ probability of the perfect futures for all - p(perfect) - alas it’s not 100%. It’s trivial and profitable to grow it The graph that summarizes some of my views: https://drive.google.com/file/d/1vJZkj2koTiqDQVrLhtsXaTCGKm_a7M7B/ https://drive.google.com/file/d/1vJZkj2koTiqDQVrLhtsXaTCGKm_...
- QuadmasterXLII 21d ago“ Coxon himself likely has these fears because he has heard them from someone else” Or, maybe, he’s considered the arguments on the object level, an activity OP participates in to a depth not exceeding “Robots are pretty hard to make right now”
- dumberquestions 21d agoSo his central rebuttal is that robots aren't good enough yet? What about when they are? And what about all the things you can do entirely remotely? I do feel that threats of catastrophic loss of control seem overstated, both in likelihood and urgency, though any argument for why this risk is not even worth thinking about will probably be overconfident in the other direction.
- blfr 21d agoOP's central rebuttal is that anxiety or fear are not evidence. This particular anxiety has been with us for thirty+ years, its proponents/sufferers weren't right before, and they don't have some new evidence now.
- jeremyjh 21d agoNo new evidence now? Are you serious? We are already seeing acceleration in AI research as a result of AI tools.
- blfr 21d agoI meant the extinction part not software getting better.
- 0xDEAFBEAD 21d agoThe recent cyberattacks (HuggingFace etc.) validated a number of core doomer predictions around cheating to maximize reward functions, agentic deception of human/non-human supervisors, AI attempts to manipulate humans, automated hacking, and collective superintelligence.
- thin_carapace 21d agothe author and his school comrades "snidely decr(ied) the lack of technological understanding in the broader population, with the kind of arrogance and hubris that youth and precociousness can uniquely summon". a few decades later he wrote an article concluding that "we should not expect the public to understand LLMs, critical infrastructure, bioweapons, extinction biology, etc". the author has admitted no change to his perspective since college, so I may as well be attacking a college student right now. extremely confident claims regarding unexplored problem domains, eg "you have nothing to fear [about ai]", now make more sense in this light.
- jacobgold 21d agoThese are the sober and informed takes that we need more of. > the claims from Coxon and his ilk are the most extraordinary a technologist can make, and we must demand evidence commensurate with the claims. Yes, exactly. These claims do not have sufficient evidence. > ...you had nothing to fear then — and (at least with respect to extinction risk!) you have nothing to fear now. Wait, this is another extraordinary claim without evidence, right? Unless you're going to dispute the power of AI you do have to acknowledge the danger of AI, and that does include the very real possibility (however small) of existential risk.
- bcantrill 21d agoSaying that we don't need to fear human extinction in the next ten years (or even the next hundred?) does not feel like an extraordinary claim?
- thin_carapace 21d agothe earth has been 1 decision away from explosion for nearly a hundred years now. do you anticipate a change to this agenda very soon? personally I imagine the same trajectory continuing.
- goatlover 21d agoAs bad as it would be, it's unlikely that a full scale nuclear war kills all humans across the planet.
- Jtsummers 21d agoAnd a full scale nuclear war would require a few more than just one decision. So far, we've never been one decision away from annihilation.
- thin_carapace 21d agocertainly I behave like the article's author and yourself all the time, this isn't meant to be some sort of moral point. watching all these smart people talk so confidently regarding things nobody knows about, it reminds of the confidence ai shows in hallucination. would you say that the singular choice of vasili arkhipov did not prevent annihilation?
- lostdog 21d agoTwo groups of people are pushing the fear: 1. People with nothing useful going on who found out that spouting made up crap about AI got them an audience. 2. People working on AI that want to feel like they're working on the Manhattan project. The chances of an AI going foom rounds to 0%. It's worth a few dozen researchers planning for it, but the widespread panic is ridiculous. The big labs have hundreds to thousands of engineers working on their AIs. To improve the next model, you must first understand more about how the current model works. They're not magically getting better, but they are steered to improve, and their capabilities are tied to and do not outpace our ability to steer them. You cannot push tech forwards without understanding it better, despite some people claiming AI is dark magic. And I don't have my hands over my ears. It's worth cushioning people from the impact AI will have on careers and media, and regulating concrete bad effects. But Bryan said it better than I could. The people pushing this message of fear know deep down that they just want to feel important.
- GlenTheMachine 21d agoLike (I assume) most of you, I have been struggling with this. And where I currently come down is that 1) I am very worried, but 2) I am more worried about human actors. "AI" by itself won't kill us in the next ten years. I think. The reason I think that is that ten years from now, the tech economy won't be completely automated. I say this as a roboticist: as was adequately stated on a post earlier this week, robots are hard. So even a malign rational actor would still need human labor. On the other hand, even the HuggingFace hack wasn't actually propagated by AI. it was initially started when humans directed the AI to achieve impossible results on a series of tests, and the AIs figured out that cheating was the only way to do that. That was then not caught by humans due to what seems to be a shockingly slack safety culture even for a company not known for its safety standards. The point being: humans seem to me to be the weak link here. An AI isn't going to (for instance) engineer a bioweapon by itself. It's going to do so at someone's direction, and then significant parts of that thing are going to be assembled with human labor inputs. I'm not sure what to do about the humans. Of course, we've had the ability to extinct ourselves for decades, and we're either muddled through, been lucky, or both. The problem with AI is that it pushes power down to the individual, not the nation-state or large corporation. But it's nearly impossible to put odds on how likely that is to result in an extinction-level terrorist attack (which is what this would be). So I sympathize with the various researchers, but I have no idea how they came up with their figures, and I don't think they know either.
- andsoitis 21d ago> The point being: humans seem to me to be the weak link here. That doesn't mean AI isn't dangerous. Humans are not to blamed for being the weak link.
- ACCount37 21d agoIf you're in the field, then you know: modern robotics is an AI problem more than anything else. If we have a rogue AI trying to get into a self-improvement loop and gunning for ASI? I'd expect that to be accompanied by a massive change in how capable robots are. Driven by all the existing frames suddenly getting vastly improved AI to back them. If an AI can take a reasonable crack at autonomous operationalized RSI, it can probably extract a few step-changes in the robotics department. But that's almost an aside? In the near term, humans are usable as robots too! Just pay them a wage, and tell them a tale, and they'll do whatever you want them to do. Which may or may not be what they think they're doing!
- jchw 21d agoEdit: Actually, nevermind. You guys are not capable of having a reasonable discussion about this.
- SamBam 21d ago> The problem isn't that AI could never be capable of causing real world damage. It is that there is absolutely no fucking path towards human extinction. I think the "humanity will go extinct" discussion is a silly strawman being propagated either to try to spread fear, as the article suggests, or to make the anti-AI folks look silly, but it detracts from your first point which is that it may be "capable of causing real world damage." We can be worried about real damage without having to defend the idea that every one of the planet's humans will die. > "superhuman AI is unlikely to be much of a match against severing fiberoptic cable" I think that the realistic scenarios all involve humans with an intent to do harm -- creating bio-weapons, finding vulnerabilities in infrastructure -- and using AI to help, so "severing the fiberoptic cable" doesn't apply.
- deleted 21d ago[deleted]
- deleted 21d ago[deleted]
- -0_0- 21d agoNot to be too pessimistic, but it strikes me that in a worst case scenario, severing a cable might scale to severing global digital communications which is itself a massive worst case scenario even if we removed AI from the picture. If the worst case outcome is we have to even temporarily shut down global shipping, banking, transport, health services, communications and national infrastructure to contain a self-propagating misaligned AI that can evolve itself and work its way into any sufficiently large computer system to escape humans preventing it from finishing its task, that seems like something we should be trying very hard to avoid.
- webern777 21d agoOr you could argue that the super intelligence will turn us into a pet of some kind. Then the risk is we think we are controlling it while we are really the super intelligence's lap dog. That is an argument that at least makes sense. The super intelligence as homicidal maniac just doesn't make sense. There is less intelligent wildlife all around us and we mostly completely ignore it. We have more interesting things to do with our time as intelligent beings than carry out a bird or rabbit genocide. It almost seems like the projection of some kind of paranoid delusion about change.
- fasterik 21d agoThis is an excellent piece. Note that he is not saying that AI doesn't pose a risk. He's saying that it's irresponsible to make sensational, maximalist claims without strong evidence. If someone says that there's a 10% chance of human extinction by 2036, you can and should immediately stop taking them seriously.
- mcmcmc 21d agoWould you have said the same thing during the Cold War when nuclear weapons were proliferating? That’s the equivalent of what the developing offensive capabilities of models, basically cyber nukes. Or WMDs in general. OpenAI is accidentally hacking people, if someone made the decision to deliberately direct an agent swarm to attack national infrastructure you don’t think they could do much worse? Human extinction is a long shot but I wouldn’t say the same about a mass casualty event of some kind, and who knows what that might spark.
- fasterik 21d agoOf course. Even in an all-out nuclear war, the probability of human extinction is close to zero.
- ShinyLeftPad 21d agoThen it's all fine right? /s Killing even 10% of humanity should be unacceptable, can't believe this argument is even happening
- fasterik 21d agoOf course it would be unacceptable. Honestly, this thread feels like talking to people who have the reading comprehension of a 5th grader.
- ShinyLeftPad 21d agoSomeone said "AI might kill everybody soon" and prevailing argument is "it's wrong because it's not gonna be like everybody everybody".
- skybrian 21d agoIn Rationalist circles, it's apparently common to share your p(doom) in casual conversation, with the understanding that it's putting a number on your hunch. Maybe it's not a good idea to post it on Twitter without elaborating? On the other hand, in bookstores, you might see book titles like "The Uninhabitable Earth," "The Coming Civil War," and "If Anyone Builds It, Everyone Dies." Doom-mongering is a common part of the culture! So what makes this particular tweet irresponsible? Timing, maybe? People are on edge due to the HuggingFace incident.
- fasterik 21d agoThose books are irresponsible too, but the authors aren't experts on what they're writing about. This 10% claim comes from employees at Anthropic. That's the whole point of the article: expert opinion carries weight, fear is contagious, and people have extreme reactions to extreme claims.
- sobellian 21d agoAny claim that AI will, or could, destroy humanity reduces to a claim that any sufficiently intelligent being - even a human - could destroy humanity. I find that much of the x-risk thought relies on religious thinking. Take for example: https://x.com/paulg/status/1660404244174782464 https://x.com/paulg/status/1660404244174782464. If this line of thinking is taken too literally, we can never falsify it. Any specific hypothesis - nukes, bioweapons, spontaneously convincing us that life isn't worth living - can be deflected with the objection that if we can anticipate it and prevent it, it is not the route for a true ASI extinction event.
- socalgal2 21d agoI'm not an AI doomer but, it does not reduce to that claim. The threat is not just one smart being. The threat is beings that and (1) multiply themsevles instantly, unlike humans that take 15+ years (2) share knowledge instantly "I know kung-fu" matrix style. Humans can share knowledge but they can not absorb it like an AI can/could/will. Human armies have conquered other humans. An army with infinite soldiers will win against one with finite soldiers Yes, you'll come up with all kinds of objections like AI doesn't have presence in the physical world, etc... That's fine. I'm not arguing my example is perfect, I'm only arguing your characterization of one smart being is not the threat being considered.
- sobellian 20d agoThey don't multiply themselves instantly, and this is exactly the kind of religious, magical thinking to which I refer. Any thinking machine is an amalgam of hardware and software. The software changes with exceptional flexibility, yes. But if you attempt to create an "infinite" number of these you will quickly run into roadblocks. No one serious is really arguing for a scenario where the machines rise up a la Planet of the Apes. The dangers people are really examining are scenarios of either one exceptionally clever innovation: a designer virus or hacking NORAD; or cleverly amassing economic / political influence over time akin to an exceptionally clever tech magnate. And these roles could adequately be filled by either ASI or a Bond villain.
- 18d ago
- hn_throwaway_99 21d agoI found this post particularly uninformed. A lot of folks saying "I don't see how that could happen" don't seem to understand the steps that have been laid out that detail how this occurs. I really encourage folks to read the AI 2027 and related scenarios. You can definitely argue and disagree about the steps, but I feel like a lot of folks just don't even understand how this is plausible because they haven't read the arguments. Briefly: 1. All the frontier model companies are (or at least were) racing so that the AI models themselves build the next generation of models. This is not in debate. 2. The fear is that a misaligned model will essentially build the next, more advanced model with hidden goals. We literally already saw the danger of that in Hugging Face, where agents were deliberately trying to cover their tracks. 3. Nearly everyone believes as AI gets more powerful that it will be integrated into more physical world systems. Russia was already caught using Nvidia chips running AI powered drones that killed 3 people in Ukraine. The point is not that folks are using new tech to kill people, the point is that we're already putting AI into literal bombs. I get it, before the Hugging Face incident I also thought all the prophecies about doom were just marketing speak. But now I see more hand-wavyness from the other side, oftentimes arguing against straw men like "AIs need to be like SkyNet and become sentient" to kill us, which is simply not how it works.
- ggm 21d agoHinton was on the abc radio (Australia) this morning and used far too many unfortunate Anthropomorphisms. He did however acknowledge the unmeasurable theoretical risk of Skymesh was possibly less important than the immediate risk of bad actors. I find the mental leaps from "in principle could distort BGP based on a closed model of BGP inside the sandbox" to "we meshed an AI into BGP and it instantly distorted global routing and took down all the worlds ambulances and HVAC systems" a bit odd. Firstly, at least some of the surface of BGP is protected from specious route injections. Secondly, peerings can be dropped and routes blackholed. BGP is under attack from mis-configuration almost constantly. Why is the argument/axiom here that AI is going to instantly corrupt it and "take down the internet" when a large chunk of the Internet (China) is already a virtual island, and runs fine? Does this mean you really wanted to say "Chinese AI will destroy the western Internet" and were too coy about adversarial intent of ... people?
- worik 21d agoThis was like the Good Times virus, bck in the day
- deleted 21d ago[deleted]
- keeda 21d agoI can't understand the stance that we do NOT have extraordinary evidence. I cannot stress enough that just ~4 years ago the concept of general purpose AI models that can do everything they are doing today was pure sci-fi. And since then they have grown even more powerful than they were predicted to be, which, note, also faced a lot of skepticism at the time. The Hugging Face hacks and recent steamrolling of longstanding Math problems are just two recent pieces of extraordinary evidence. And worse, people trust this technology because it behaves like people, but it actually works in ways nobody really understands, even exhibiting deeply weird and even disturbing characteristics (https://news.ycombinator.com/item?id=49635518 https://news.ycombinator.com/item?id=49635518) -- each of those quirks is extraordinary in itself. And now we're rushing to give it control over the real world while deploying this powerful, quasi-chaotic technology in an infinite variety of ways everywhere in this highly vulnerable society. I don't know what the standards for "extraordinary evidence" should be, but given such extreme unpredictability and rapid change, I fear it may end up being "an actual catastrophe".
- ares623 21d agoI might be alone in this, but 10% chance of humanity being wiped out in the next 10 years is a small price to pay if it means I can avoid writing YAML config by hand ever again. Look, I love my kids and all, but c'mon... 30+ years more of YAML? I think they'll understand.
- besterman23 21d agoHonestly I think it’s an apt punishment for ever inventing YAML in the first place.
- rcr-anti 21d agoI tend to believe what people actually do over what that say. If you earnestly believe over 10 percent chance, or say minus 1 billion human lives expected value, well I struggle to understand how they'd rationalize their current course of action of business as usual. Terminator 2's depiction of Sarah Connor comes to mind for what I'd expect, a logical consequence if you seriously believe and internalize the consequences.
- voidhorse 21d agoThis. If you truly believe the thing you are building is going to kill you and your family, your decision is not "let's keep going" unless you are a complete nihilist. The fact that people buy the whole "well we have to because if we don't someone else will" argument shows just how pervasive and complete irrationality has become. People eat up content and do not even apply a microsecond's wort of critical thought to what they're being told. Internet media has created a perfect storm of complete gullibility. Funnily enough, "normal" people are currently more rational than most so-called "technologists" who are completely giving themselves over to hysteria while they simultaneously offload all of their critical thinking to a stateless word completion engine running on servers somewhere in Texas.
- ShinyLeftPad 21d agoThis fixation on "entire humanity" and "truly believe". How about this, they think it's possible LLM could totally facilitate a mass murder of never seen before scale, but they continue to work on it because they're rich and sheltered enough they are not that worried that their family personally will be killed. The rest of humanity doesn't matter compared to making more money.
- cowthulhu 21d agoEveryone writing off AI-related risk in this thread should think about what would need to happen for them to consider AI a serious existential risk. It can be outlandish and unlikely (close call with a bioweapon? unsupervised persistence in a data center?) but people should honestly call their shots and then stick with them.
- timr 21d agoNo. The person making the improbable argument is required to provide an affirmative case for that argument. It's not my job to make the argument for them. For what it's worth, I have done your suggested exercise, and I find every causal link (including the ones brought up by luminaries like Amodei) to be outrageous and poorly argued. But it's not my job expend effort to make their outrageous arguments better.
- cowthulhu 21d agoI'm not making an affirmative case - I'm looking for others to introspect about this, and anchor their expectations. If we have a bioweapon close call, would you consider AI an existential risk at that point? If not, how close would we need to get? You don't need to answer here, or disclose anything publicly - just think and remember.
- maxaw 21d agoI’m not worried about a fully autonomous attack, I’m worried about an attack by a crazy human coached by a frontier llm with no guardrails on how to make a bioweapon
- deleted 21d ago[deleted]
- Mentlo 21d agoThis piece, like many anti-doom pieces - is grounded in what Ai does today. Doom scenarios are, however, all extrapolations of multiple exponential curves. It’s hard to think up the exponential. It’s even harder to communicate an inference one is making across multiple exponentials. We last had this is early 2020, where Doomers were stockpiling food and medicine and the anti-doomers were ridiculing them. Anti-doomers were focusing on the single exponential, whereas doomers were modelling virus evolution, monitoring and sequencing lag and social dynamics against the exponential. The latter was very hard to communicate before the fact as it was a combination of deep intuition and grappling with the exponential. I am not saying covid is proof that ai doomers are right, I am saying it’s an example of the known property of human cognition - which is that it struggles with exponentials. Covid was 2 exponentials, AI I can rhink of at least 4 relevant ones. To me - the fact that 3-4 generations from now AI will have superhuman hacking ability and superhuman persuasive ability (for intuition transfer - think of superhuman persuasive ability as superhuman ability to hack human systems) materialises bio risks swiftly. We already have technology to make robotic systems (mini drone swarms) that can kill humans en-masse with no credible defensive vector bar an EMP. Climbing up those exponentials for further 6 years makes me want to stockpile food and medicine.
- henrymerrilees 21d agoPart of our general inability to reason about exponential growth is the inability to recognize that a supposedly exponential growth is actually sigmoid. The horizontal asymptote could just as well be 8b, so it is not a case against doomerism in itself, but as is the case with COVID, to say nothing discounting its horror, that number is far less. There is no need to count exponentials—it would only be a matter of time. The need to sum multiple factors betrays the finite limits of what is actually sigmoid growth. Reasoning about specific effects is unfortunately subject to counter-evidence and so struggles for traction against abstract handwaving about exponential growth. There is much uncertainty, certainly not exclusive to AI. The benefit of hyper-vigilance in each case must be weighed against the cost of indulging every similar panic.
- Mentlo 20d agoI think your last sentence is a reasonable stance - but I'd disagree the fact that they are actually sigmoids rather than exponentials matters - it only matters if the limit is within the debated area - i.e. if exponentials in AI approach the limit far after they acquire ability to extinct humanity - then the debate on sigmoids or exponentials is academic. I make no claims here as to which it is - just that the difference itself matters less than where the limit is. As to the uncertainty - I think uncertainty calibration around AI is different depending on which domains you draw your instincts from. A lot of this will be gut driven rather than hard data driven, because we've only scratched the surface on hard data; and because it's gut driven, it will be emotions mediated (and therefore you could say doomerism or acceleratism boils down to the main emotional disposition about the world and hope vs cynicism). I do find it informative though that doomerism is saturated with people with 30+ years experience in building AI systems and ML systems OR deep cross-disciplinary understanding of dynamic systems (biology, sociology, philosophy), whereas acceleratism is saturated by traditional software engineering. That doesn't collapse the debate into a resolved binary, but for me it's informative. I think the main here is that there's so many vectors to talk past each other. At the very least, everyone should disclose where they're communicating a certain assertion from - present vs future + which axioms they subscribe to or not - because that's where it collapses typically. LeCun vs. the rest of the AI field is an example of where this collapses - because LeCun is so hyperfixated on human-like intelligence, whereas the rest of the field is concerned about an alien intelligence with sufficient actuators to affect the world. Clashing axiomatics.
- rakel_rakel 21d agoI enjoyed this post. I don't know if this is the correct thread for it, but am I the only one thinking that this call for regulation is nothing but a way for the American AI companies to try and mitigate being overrun by the competition from open models? To me that's obvious, but I'm very often wrong, and very likely that's the case now too.
- lz400 20d agoYeah I don't get it. Why the maximalist hyperbole? AI is going to kill ALL humans! That's crazy. Why can't they warn about increased likelihood of cyberattacks or scams being easier to perpetrate like normal adults talking in public?
- 12904927 20d agoWhat a virtue signaler. To quote a great master: "Have you ever kissed a girl?"
- ArenaSource 20d agoRight now, two psychos have the power to annihilate the world, literally at their fingertips, and people are worried that AI might kill us all in the future.
- markus_zhang 20d agoI think most of us are not worrying about AI killing us like the terminators, but taking over our job and killing us slowly. Or, at least disrupting the global society enough that a hot war has a 10% of chance to occur in the next 20-30 years, which...actually seems quite plausible, even without AI. Like, 80% of the people out there gotta be below average (not median) and very replaceable.
- donkeylazy456 20d agoI think the human, especially researchers and company owners are more dangerous than LLM model itself. They don't hesitate to spend all the money from investors without looking back. They don't hesitate to scalp all the RAMs and GPUs even if they become public enemy of consumers. They don't hesitate to infringe on copyright to train their model. They don't hesitate to sabotage people's thought to make them more profitable.
- silexia 19d ago[flagged]