25 ms·

The way incumbents are attempting to seize power via regulatory capture, using "X-Risk" and fantastical claims of sudden human extinction is maybe one of the mo

The way incumbents are attempting to seize power via regulatory capture, using "X-Risk" and fantastical claims of sudden human extinction is maybe one of the most cynical ploys in the history of tech so far. Open source is one of our greatest gifts, and the push to close off AI with farcical licensing and reporting requirements (which will undoubtedly become more strict) is absurd. The laws we have already cover malevolent and abusive uses of software applications.

raverbashing 3y ago
Correct. The risk that's the real ploy is much higher and much more impactful than the actual risk of Rogue AI (as much as the risk might exist, though probably tiny)
- ben_w 3y ago
  "Rogue" is just a type of bug. Buggy old-fashioned AI (i.e. a bunch of ifs and loops) has, more than once, almost started WW3.
JoshTriplett 3y ago
> seize power via regulatory capture Not even a little bit. "Stop" is not regulatory capture. Some large AI companies are attempting to twist "stop" into "be careful, as only we can". The actual way to stop the existential risk is to stop. https://twitter.com/ESYudkowsky/status/1719777049576128542 https://twitter.com/ESYudkowsky/status/1719777049576128542 > the push to close off AI with farcical licensing and reporting requirements "Stop" is not "licensing and reporting requirements", it's stop.
- Aithr 3y ago
  "Stop new development." is regulatory capture.
  - esafak 3y ago
    Stop existing development too. Carve an exception for academic research.
    - BlueTemplar 3y ago
      «Exception for academic research» seems to be how we got Covid, so...
- rolisz 3y ago
  And... How do you do that? Seriously, people have been talking about stoping CO2 emissions for what, 50 years? We've barely made progress there.
  - TeMPOraL 3y ago
    With threat of military intervention. People have been talking about curtailing carbon emissions for 50 years, but they haven't been serious about it. Being serious doesn't look like saying "oh shoot, we said we will... try to keep emissions down, but we didn't... even try; sorry...". Being serous looks like "stop now, or we will eminent domain your data centre from under you via an emergency court order, or failing that, bomb it to rubble (because if we don't, some other signatory of the treaty will bomb you for us)".
    - neonihil 3y ago
      The problem with that is that the entity supposed to "bomb it to rubble" and the entity pushing for AI development happens to be the same entity. Maybe the confusion why people can't see this clearly stems from the fact that tech development in the US has mostly been done under the umbrella of a private enterprise. But if one has a look at companies like Palantir, it's going to become quite obvious what is the main driver behind AI development.
    - Dig1t 3y ago
      Are you really suggesting drone striking data centers? What about running AI models on personal hardware? Are we going to require that everyone must have a chip installed in their computer that will report to the government illegal AI software running on the machine? You are proposing an authoritarian nightmare.
      - pixl97 3y ago
        Hence the entire point of this discussion... can we avoid the future nightmares that are coming in one way or another? Make AI = nightmare, stop people from making AI = nightmare. The option that seems to be off the table is people going, hell making AI is a nightmare so I won't voluntarily without the need for a police state.
- deleted 3y ago
  [deleted]
- peterth3 3y ago
  Licensing can definitely turn into regulatory capture if it expands enough. It’s effectively a barrier to entry defined by the incumbent.
  - jamilton 3y ago
    They're saying don't do licensing, just make it illegal.
    - peterth3 3y ago
      Sorry, who’s “they” here? The incumbents OpenAI / DeepMind / Anthropic?
      - ben_w 3y ago
        Yudkowsky and similar. (I'm a lot more optimistic than such doomers, but I can at least see their position is coherent; LeCun has the opposite position, but unfortunately, while I want that position to be correct, I find his position to be the same kind of wishful thinking as a factory boss who thinks "common sense" beats health and safety regulations).
      - JoshTriplett 3y ago
        No, the incumbents think it should continue. The xrisk folks are not the AI companies. The AI companies pay lip service to xrisk and try to misdirect towards regulation they think they can work with. Judging by this thread, the misdirection is working.
      - jamilton 3y ago
        In this case I meant the GP comment from JoshTriplett.
- wood_spirit 3y ago
  You can’t _stop_ it. You can make it illegal in some countries. But outlawing it just means some obvious candidate countries or private actors develop it first instead.
  - candiodari 3y ago
    I think these attempts will simply point out that math is a higher power than law. If something becomes possible, and becomes easy, there is nothing law can do to stop it. Just look at the attempts to stop drugs in North West Europe. Huge sentences (so huge they are in fact never fully carried out), tens of dead police officers on a yearly basis, triple that in civilian victims and drugs are easier to get than ever before.
    - FeepingCreature 3y ago
      The aim is to stop it from becoming easy. The drug war would look very different if good cocaine was only produced in one country with extremely costly chemistry machines from one company.
      - upupupandaway 3y ago
        If good cocaine was only produced in one country with extremely costly machines, a good-enough alternative would appear almost immediately. This has happened millions of times in the history of humanity, I refuse to believe that people haven't learned this lesson.
        _0ffh 3y ago
        > I refuse to believe that people haven't learned this lesson Oh, I believe they absolutely haven't, when not having learnt it buys them money and influence. As for the bulk of the population, they will accept whatever the corporate media tell them to.
        FeepingCreature 3y ago
        And yet, it doesn't seem to be that easy with chips.
- topspin 3y ago
  > "Stop" is not regulatory capture. "Stop" is a fiction that exists exclusively inside your head. Unless your solution anticipates a planet wide GULAG system run by a global government your "stop" is infeasible.
  - maeil 3y ago
    If tomorrow TSMC, NVIDIA and ASML get destroyed in the next 9/11, it's "stopped". It's completely possible.
    - topspin 3y ago
      Fantasies, bordering on threats.
      - ben_w 3y ago
        Of course it's a threat. Civilians doing that would be bad. But governments do much the same via military intervention on a regular basis, hence Yudkowsky talking about airstrikes that one time.
      - maeil 3y ago
        I'm not suggesting to do that, I'm just showing that it's incredibly possible. You could also go with "Tomorrow a massive storm strikes the Netherlands and huge earthquakes striking Taiwan and the US Westcoast", if you have emotional objections to my first example.
    - Maken 3y ago
      It's delayed at best.
      - maeil 3y ago
        Incredibly delayed, giving at least decades to figure things out.
    - YetAnotherNick 3y ago
- upupupandaway 3y ago
  The fact that anyone may be taking Eliezer Yudkowsky seriously on this topic is mind-blowing to me. Simply mind-blowing. To imagine that anybody, including the UN, would have any power to collective put a stop to the development of AI globally is a laughable idea. The UN cannot agree to release a letter on Israel-Hamas, imagine policing the ENTIRE WORLD and shutting down AI development when necessary. We can't stop countries from developing weapons that will destroy us all tomorrow morning and take billions of USD to develop, imagine thinking we can stop matrix multiplication globally. I don't want to derail into an ad hominem, but frankly, it's almost the only option left here.
  - jeffparsons 3y ago
    My estimation of Eliezer Yudkowsky just dropped a notch — not because I don't take the existential threat posed by AI seriously — but because he seems to think that global coordination and control is an even remotely possible strategy.
    - concordDance 3y ago
      He actually doesn't think it's viable, but thinks its the only card left to play. He's pretty sure that human civilization will be extinct this century.
      - jeffparsons 3y ago
        It can't be the only card left to play. There are so many others. They may be weak, but they still have to be better than one that is guaranteed to be absolutely useless. Example of a weak, but at least _better_ option: convince _one_ government (e.g. USA) that it is in their interest to massively fund an effort to (1) develop an AI that is compliant to our wishes and can dominate all other AIs, and (2) actively sabotage competing efforts. Governments have done much the same in the past with, e.g. conventional weapons, nuclear weapons, and cryptography, with varying levels of success. If we're all dead anyway otherwise, then I don't see how that can possibly be a worse card. Edit: or even try to convince _all_ governments to do this so that they come out on top. At least then there's a greater chance that the bleeding edge of this tech will be under the stewardship of a deliberate attempt for a country to dominate the world, rather than some bored kid who happens to stumble upon the recipe for global paperclips.
- RandomLensman 3y ago
  There is no existential risk now and there might never be. So "stop", especially with the use of violence, is a pretty odd position to take to say it mildly.
- api 3y ago
  Yudkowski is a crackpot. Nobody would take him seriously if he wasn’t one of Peter Thiel’s pet crackpots. I should have done a ton more intellectual edgelording in the oughts and grabbed some Thielbucks. I could have recanted it all later. Of course I probably would have had to at least flirt with eugenics. That seems to be the secret sauce.
- m_a_g 3y ago
  Could you explain to me how AGI is an existential threat to humanity? All the arguments I've read so far feel so far-fetched it's unbelievable. It's in the order of making an AGI and making it control the Pentagon (!?).
  - hollerith 3y ago
    The AGI does not need to control the Pentagon. It might for example invent novel technologies that you and I cannot even imagine because we are much less capable at inventing new technologies than the AGI will be. If members of an alien civilization much more advanced than our own suddenly appeared in orbit around Earth, would that not be dangerous?
    - m_a_g 3y ago
      That I understand, but my thinking automatically goes to control and restrictions. A superintelligent AGI that's as restricted as chatgpt in terms of what it can actually control seems... not so existentially dangerous? In that scenario, that actual problem would rise by not-so-smart humans listening to that AGI blindly, no?
      - hollerith 3y ago
        I'm confused by your "as restricted as chatgpt". Hasn't chatgpt been given unrestricted access to the web? Surely if chatgpt were sufficiently capable, it would be able to use that access to hack web sites and thereby gain control over money. Then it can use the web to buy things with that money just like you and I can. It is a lot harder to prevent the sysadmins who have temporary control over you from noticing that you just hacked some web sites and stole some money, but a sufficiently capable AI would know about the sysadmins and would take pains to hide these activities from them.
        johnthewise 3y ago
        It doesn't even need to hack or do anything illegal. It can make money trading stocks, predicting or modelling far better than market participants, use money to influence the real world, bribe politicians etc. Like with few billions, i bet you can create a political party, influence campagins and end up with real human followers and lobbying power. It doesn't even need to run on its own, if there was a trading bot that said, in order for it to make me money, I need to give it internet access and run it 24/7, rent gpu clusters and s3 buckets etc, I'd do it. this is the most probable scenario imo, AI creating beneficial scenarios for subset of people so that we comply with its requests. very little convincing is necessary
- EVa5I7bHFq9mnYK 3y ago
  What stop? Will Kim Jong Un stop? Hell no. There is no stop, the genie is out of the bottle.
- birdyrooster 3y ago
  lmao you have to be kidding me
- veidr 3y ago
  I don't want to be rude; I understand where you are coming from (at least in terms of single-minded risk mitigation from a theoretical and unconstrained viewpoint), but this reads like Mike Huckabee's 'just keep an aspirin pill pinned between your knees' advice to teenage girls on how not to get pregnant. There is obviously (!!!!!!!!!) no way in this — or any imaginable — timeline that we are going to just "stop". (And it's hard to imagine any mechanism that could enforce "stop" that isn't even worse than the scenarios we're already worrying about. How does that work? Elon or Xi-Poohbear have to personally sign off on installations of more than 1000 cores?)
  - pixl97 3y ago
    Butlerian Jihad level revolt by the average person? No reason to see that happening these days, and by the time we're willing to it may already be too late.
- j45 3y ago
  Except while saying stop, and saying they will, none have stopped because apparently others won’t. Since developing the danger first to help regulate out everyone who doesn’t cross the line first.. open source alternatives are increasingly important. The regulations announced in the US in the past week effectively make it for the few and not the many. There’s been a few rumblings that OpenAI has achieved AGI recently as well.
- slashdev 3y ago
  Stop is a fantasy pretty much prohibited by game theory in a world with multiple competing states. It’s not going to happen.
- pdonis 3y ago
  But the AI leaders who have been calling for the US government to act are not asking for AI research to stop. Nor are they asking for the kind of international policing that Yudkowsky describes. They are asking for regulations much like the ones the Biden administration just published--which are indeed regulatory capture since all they do is build a moat around current AI companies to protect their profits. They do not stop AI research or make it any less dangerous.
- dragonwriter 3y ago
  Yudkowsky’s call for declaring a global AI halt and engaging in unlimited warfare with anyone who defies it poses a greater and more immediate existential risk to humanity than AI ever could; it is largely mitigated (though still a bigger X-Risk than AI) because no one with the capacity to implement it in any substantive way takes him seriously at the moment. But the people selling AI X-Risk that anyone with any policy influence is listening to are AI sellers using it to push regulatory capture, so those are the people being talked about when people are discussing X-Riskers as a substantive force in policy discussions, not the Yudkowsky cult, which, to the extent it is relevant, mostly provides some background fear noise which the regulatory capture crew leverages.
  - 3rd3 3y ago
    > Yudkowsky’s call for declaring a global AI halt and engaging in unlimited warfare with anyone who defies it Could you provide a link to where he said this?
    - dragonwriter 3y ago
      I think he's said it many times in slightly different ways, but one notable one: https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/ https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no... The moratorium on new large training runs needs to be indefinite and worldwide. There can be no exceptions, including for governments or militaries. If the policy starts with the U.S., then China needs to see that the U.S. is not seeking an advantage but rather trying to prevent a horrifically dangerous technology which can have no true owner and which will kill everyone in the U.S. and in China and on Earth. . . . Shut down all the large GPU clusters (the large computer farms where the most powerful AIs are refined). Shut down all the large training runs. Put a ceiling on how much computing power anyone is allowed to use in training an AI system, and move it downward over the coming years to compensate for more efficient training algorithms. No exceptions for governments and militaries. Make immediate multinational agreements to prevent the prohibited activities from moving elsewhere. Track all GPUs sold. If intelligence says that a country outside the agreement is building a GPU cluster, be less scared of a shooting conflict between nations than of the moratorium being violated; be willing to destroy a rogue datacenter by airstrike.
- 0x00_NULL 3y ago
  No. Sorry, but this is wrong and Yudkowsky is both naïve and mostly exists in the domain of fan fiction. There are way way way too many issues that are addressed with a hand-wave around scenarios like “AI developing super intelligence in secret and spreading itself around decentralized computers while getting forever smarter by reading the internet.” Too many of his arguments depend on stealth for systems that take up datacenters and need whole-city-block scales of power to operate. Physically, it’s just not possible to move these things. Economically, it wouldn’t be either close to plausible either. Power is expensive and it is the single most monitored element in every datacenter. There is nothing large that happens in a datacenter that is not monitored. There is nothing that is going to train on meaningful datasets in secret. “What about as technology increases and we get more efficient at computing?” We use more power for computing every year, not less. Yes, we get more efficient, but we don’t get less energy intensive. We don’t substitute computing capacity. We add more. The same old servers are still there contributing to the compute. Google has data centers filled with old shit. That’s why the standard core in GCP is 2 GHz. Sometimes, your project gets put on a box from 2010, other times, it gets put on a box from 2023. That process is abstracted so you can’t tell the difference. TLDR: Yudkowsky’s arguments are merely fan fiction. People don’t understand ML systems so they imagine an end state without understanding how to get there. These are the same people who imagine Light Speed Steam Trains. “We need faster rail service, so let’s just keep adding steam to our steam trains and accelerate to the speed of light.” That’s exactly what these AI Doom arguments sound like to people in the field. Your AI Doom scenario, although it might be very imaginative, is a Light Speed Steam Train. It falls apart the moment you try to trace a path from today to doom.
  - kubanczyk 3y ago
    Are you saying that the governments/agencies are incapable of hiding any big infra? That's a novel take.
troupo 3y ago
> The laws we have already cover malevolent and abusive uses of software applications. They don't. Or not nearly enough. Otherwise you wouldn't have automated racial profiling, en masse face recognition, credit and social scoring etc. And it's going to get worse. Because AI is infallible, right? Right?! That's why, among other thing, EU AI regulation has the following: - Foundation models have to be thoroughly documented, and developers of these modes have to disclose what data they were trained on - AI cannot be used in high-risk applications (e.g. social scoring etc.) - When AI is used, its decisions must be explicable in human terms. No "AI said so, so it's true". Also, if you interact with an AI system it must be clearly labeled as such
- tluyben2 3y ago
  This type of regulatory would be better than 'stop' or making all kinds of expensive rules that only big corps can follow.
  - troupo 3y ago
    There are still quite a few rules, but they seem common sense: https://softwarecrisis.dev/letters/the-truth-about-the-eu-act/ https://softwarecrisis.dev/letters/the-truth-about-the-eu-ac...
- RandomLensman 3y ago
  AI not being allowed in high-risk applications is a bad idea, same for explicable in human terms (cannot do that now for human decisions anyway). The latter would kind of rule out any quantum computing, too, as the only real explanation there is through maths, not words.
  - troupo 3y ago
    > AI not being allowed in high-risk applications is a bad idea wat > same for explicable in human terms (cannot do that now for human decisions anyway). What? Human decisions can be explained and appealed. Good luck proving something when a blackbox finds you guilty of anything.
nologic01 3y ago
It points out to the continuing digital illiteracy of large parts of society and, unfortunately, decision making centers. Breaking down any complex knowledge domain so that people can make informed decisions is not easy. There is also an entrenched incentive for domain experts to keep the "secret sauce" secret even if it amounts to very little. So far nothing new versus how practically any specialized sector works. The difference with information technology (of which AI is but the current perceived cutting edge) is that it touches everything. Society is built on communication. If algorithms will intervene in that floe we cannot afford to play the usual stupid control games with something so fundamental.
concordDance 3y ago
> The way incumbents are attempting to seize power via regulatory capture, using "X-Risk" and fantastical claims of sudden human extinction is maybe one of the most cynical ploys in the history of tech so far. How sure are you that they don't think the risk is real? I have only third hand accounts, but at least some of the low level people working for OpenAI seemed to think the risk is worth taking seriously, its not just a cynical CEO ploy. Sam Altman knows and has talked to Yudkowsky plenty of times. To me the simplest explanation is that he thinks figuring out alignment via low powered models is the best way of solving the problem, so pushed to get those working and then trying to reduce the rate of progress for a bit while they figure it out. (I think it's a plan that doesn't have good odds, but nothing seems to give amazing odds atm)
- nerdponx 3y ago
  The best lies contain a kernel of truth.
- randomdata 3y ago
  > but at least some of the low level people working for OpenAI seemed to think the risk is worth taking seriously Everyone has an opinion, but the experts seem strangely silent on this matter. I mean experts in human risk, not experts in AI. The latter love to ramble on, but are unlikely to know anything about social and humanitarian consequences – at least no more than the stock boy at Walmart. Dedicating one's life to becoming an expert in AI doesn't leave much time to become an expert in anything else. There is only so much time in the day.
  - danenania 3y ago
    Conversely, why should we expect an expert in "human risk" with no understanding of AI to have anything productive to add to the conversation? We should be looking for people who are well-grounded in both areas, if any exist.
    - randomdata 3y ago
      What additional information would understanding AI actually bring to the table? Worst case you just assume the AI can become human-like. Certainly all the vocal AI experts can come up with is things that humans already do to each other already, only noting that AI enables it at larger scale. Clearly there is no benefit in understanding the AI technology with respect to this matter when the only thing you can point to is scale. The concept of scalability doesn't require an understanding of AI.
arisAlexis 3y ago
The fantastical claims you are claiming are actually the most probable scenario according to the majority (2/3) Hinton and Bengio that discovered the technology you are referring to. Strange to contradict them eh?
amalcon 3y ago
This is more or less exactly where I land on this. The extinction risk argument is somewhat plausible in the long term[0], but it doesn't seem that we are especially close to developing that class of problem. Meanwhile, there are very real and immediate risks to this type of technology -- in various forms of giving some humans more power over other humans. The attacks proposed by the extinction risk people actively make those immediate risks worse. The only way we've found to prevent some from leveraging almost any technology to control others is to democratize said technology -- so that everyone has access. Limiting access will just accrue more power to those with access, even if they need to break laws to get it. [0]- Say, the lower bound of the 95% confidence interval is 50-100 years out. People who I know to be sensible disagree with me on this estimate, but it's where I honestly end up.
- itsoktocry 3y ago
  >The extinction risk argument is somewhat plausible in the long term[0], but it doesn't seem that we are especially close to developing that class of problem. The parable at the beginning of Superintelligence relates to this; the problem looks far away, but we don't know where, precisely it will arise, or if it will be too late when we realize it.
- boringg 3y ago
  You mean democratize the technology within specific confines? I don't see humans proliferating weapons technology amongst other nations as that poses a greater risk that conflicts will become more violent.
nmca 3y ago
None of the world-leading scientists that co-authored this paper: https://managing-ai-risks.com/ https://managing-ai-risks.com/ Work for big labs? How is that you think the proposed conspiracy works exactly? They're paying off Yoshua Bengio? (disclaimer: I work on safety for a large lab. But none of these proponents do!)
- Wurdan 3y ago
  And how many of them have been invited to the White House to share their concerns? https://www.whitehouse.gov/briefing-room/statements-releases/2023/05/04/readout-of-white-house-meeting-with-ceos-on-advancing-responsible-artificial-intelligence-innovation/ https://www.whitehouse.gov/briefing-room/statements-releases... I don't see any of their names in this meeting's attendees, for example. Sam Altman's there, Satya Nadella's there, Sundar Pichai's there... No Yoshua Bengio, though. What the person you replied to is saying is that commercial developers of AI have a significant financial incentive to be the ones defining what risky AI looks like. If they can guide the regulatory process to only hinder development that they didn't want to do anyway (like develop AI powered nuclear missile launch systems) then it opens the gates for them to develop AI which could replace 27% of jobs currently done by people (https://www.reuters.com/technology/27-jobs-high-risk-ai-revolution-says-oecd-2023-07-11/ https://www.reuters.com/technology/27-jobs-high-risk-ai-revo...) and become the richest enterprises in history.
  - nmca 3y ago
    You're confusing "that one photo of three ceos at the whitehouse" with "all government solicitation of expert opinion on AI". It's correct that the meeting you're thinking of was industry-heavy, but the process in general has had a lot of academic input from, e.g Marcus, Tegmark, etc. See the recent summit in the UK, etc.
    - mistrial9 3y ago
      .. academic-heavy position letters have been ignored before
germandiago 3y ago
True. We receive warnings all the time but I think what these people are afraid of is actually losing their power.
- api 3y ago
  Until these same people start being concerned about DIY CRISPR kits, I do not take them seriously: https://www.the-odin.com/diy-crispr-kit/ https://www.the-odin.com/diy-crispr-kit/ This is dangerous in ways that require no leaps of logic or hypothetical sci-fi scenarios to imagine. I studied biology as an undergrad and can think of several insanely evil things you could do with this plus some undergrad-level knowledge of genetics and... not gonna elaborate. But no we are talking about banning math. This is a combination of cynical people pushing for regulatory capture and smart people being dumb in the highly sophisticated very verbose ways that smart people are dumb.
  - carapace 3y ago
    Holy shit. That's legitimately terrifying.
  - abecedarius 3y ago
    I've seen Yudkowsky and others saying that AI and biotech are both in the unusual bin of threats to all humanity, exceptions to his belief that most things are overregulated. (paraphrase of tweets from memory)
2devnull 3y ago
The laws are coming no matter what, not because of lobbying or whatever, the laws/policy response is coming because politicians want to get in on the ai too. The regulations we will do things like mandate ai be used for equity purposes in the real estate market. I think that’s in Biden’s plan. I mean, I’m sure these tech ceo 1% want regulatory capture, and they’ll get it as a quid pro quo for allowing ai to be used for largely partisan goals. But we shouldn’t fear open source ai, or regulations against it, we should fear the ai being weapon used by the political parties. It to some extent already is.
beepbooptheory 3y ago
It's not cynical though. It is, if anything, revelatory to the true interests at play here, even if on the surface the argument is certainly in bad faith. If you need a labor market, you don't want a publicly available infinite-free-labor-widget. It might as well be a handheld nuclear device.
godelski 3y ago
Importantly locking away model checkpoints makes AI safety research, at best, exceptionally difficult or impossible. We already don't know all the training methods nor do we know the data used for the models. AI safety is absolutely contingent on works being __more__ open source, not less. But there's less profit in that for these companies (potentially more profit for the country as a whole though, as we see new businesses form and sectors develop, but this is more spread out and far harder to calculate the value of so let's ensure we recognize this as speculative). We also need to be keenly aware that xrisk is not the major issue here. We do not need AGI to create massive amounts of harm, nor do we need AGI to create a post-scarce world (which the transition to can also generate massive harm if we don't do it carefully. Despite generating a post-scarce world -- or at least getting ever closer to that -- is, imo, one of the most important endeavors we can work on). Current ML systems can already do massive amounts of destruction, but so can a screwdriver. My big fear is that AI will go the way of nuclear. A technology is not good or bad, it is like a coin with a value. You can spend that coin on something harmful or something beneficial, but the coin is just a coin at the end of the day. That's what regulation is supposed to be about, expenditure, not minting.

raverbashing 3y ago

Correct. The risk that's the real ploy is much higher and much more impactful than the actual risk of Rogue AI (as much as the risk might exist, though probably tiny)

ben_w 3y ago
"Rogue" is just a type of bug. Buggy old-fashioned AI (i.e. a bunch of ifs and loops) has, more than once, almost started WW3.

JoshTriplett 3y ago

> seize power via regulatory capture Not even a little bit. "Stop" is not regulatory capture. Some large AI companies are attempting to twist "stop" into "be careful, as only we can". The actual way to stop the existential risk is to stop. https://twitter.com/ESYudkowsky/status/1719777049576128542 https://twitter.com/ESYudkowsky/status/1719777049576128542 > the push to close off AI with farcical licensing and reporting requirements "Stop" is not "licensing and reporting requirements", it's stop.

Aithr 3y ago
"Stop new development." is regulatory capture.
- esafak 3y ago
  Stop existing development too. Carve an exception for academic research.
  - BlueTemplar 3y ago
    «Exception for academic research» seems to be how we got Covid, so...
rolisz 3y ago
And... How do you do that? Seriously, people have been talking about stoping CO2 emissions for what, 50 years? We've barely made progress there.
- TeMPOraL 3y ago
  With threat of military intervention. People have been talking about curtailing carbon emissions for 50 years, but they haven't been serious about it. Being serious doesn't look like saying "oh shoot, we said we will... try to keep emissions down, but we didn't... even try; sorry...". Being serous looks like "stop now, or we will eminent domain your data centre from under you via an emergency court order, or failing that, bomb it to rubble (because if we don't, some other signatory of the treaty will bomb you for us)".
  - neonihil 3y ago
    The problem with that is that the entity supposed to "bomb it to rubble" and the entity pushing for AI development happens to be the same entity. Maybe the confusion why people can't see this clearly stems from the fact that tech development in the US has mostly been done under the umbrella of a private enterprise. But if one has a look at companies like Palantir, it's going to become quite obvious what is the main driver behind AI development.
  - Dig1t 3y ago
    Are you really suggesting drone striking data centers? What about running AI models on personal hardware? Are we going to require that everyone must have a chip installed in their computer that will report to the government illegal AI software running on the machine? You are proposing an authoritarian nightmare.
    - pixl97 3y ago
      Hence the entire point of this discussion... can we avoid the future nightmares that are coming in one way or another? Make AI = nightmare, stop people from making AI = nightmare. The option that seems to be off the table is people going, hell making AI is a nightmare so I won't voluntarily without the need for a police state.
deleted 3y ago
[deleted]
peterth3 3y ago
Licensing can definitely turn into regulatory capture if it expands enough. It’s effectively a barrier to entry defined by the incumbent.
- jamilton 3y ago
  They're saying don't do licensing, just make it illegal.
  - peterth3 3y ago
    Sorry, who’s “they” here? The incumbents OpenAI / DeepMind / Anthropic?
    - ben_w 3y ago
      Yudkowsky and similar. (I'm a lot more optimistic than such doomers, but I can at least see their position is coherent; LeCun has the opposite position, but unfortunately, while I want that position to be correct, I find his position to be the same kind of wishful thinking as a factory boss who thinks "common sense" beats health and safety regulations).
    - JoshTriplett 3y ago
      No, the incumbents think it should continue. The xrisk folks are not the AI companies. The AI companies pay lip service to xrisk and try to misdirect towards regulation they think they can work with. Judging by this thread, the misdirection is working.
    - jamilton 3y ago
      In this case I meant the GP comment from JoshTriplett.
wood_spirit 3y ago
You can’t _stop_ it. You can make it illegal in some countries. But outlawing it just means some obvious candidate countries or private actors develop it first instead.
- candiodari 3y ago
  I think these attempts will simply point out that math is a higher power than law. If something becomes possible, and becomes easy, there is nothing law can do to stop it. Just look at the attempts to stop drugs in North West Europe. Huge sentences (so huge they are in fact never fully carried out), tens of dead police officers on a yearly basis, triple that in civilian victims and drugs are easier to get than ever before.
  - FeepingCreature 3y ago
    The aim is to stop it from becoming easy. The drug war would look very different if good cocaine was only produced in one country with extremely costly chemistry machines from one company.
    - upupupandaway 3y ago
      If good cocaine was only produced in one country with extremely costly machines, a good-enough alternative would appear almost immediately. This has happened millions of times in the history of humanity, I refuse to believe that people haven't learned this lesson.
      - _0ffh 3y ago
        > I refuse to believe that people haven't learned this lesson Oh, I believe they absolutely haven't, when not having learnt it buys them money and influence. As for the bulk of the population, they will accept whatever the corporate media tell them to.
      - FeepingCreature 3y ago
        And yet, it doesn't seem to be that easy with chips.
topspin 3y ago
> "Stop" is not regulatory capture. "Stop" is a fiction that exists exclusively inside your head. Unless your solution anticipates a planet wide GULAG system run by a global government your "stop" is infeasible.
- maeil 3y ago
  If tomorrow TSMC, NVIDIA and ASML get destroyed in the next 9/11, it's "stopped". It's completely possible.
  - topspin 3y ago
    Fantasies, bordering on threats.
    - ben_w 3y ago
      Of course it's a threat. Civilians doing that would be bad. But governments do much the same via military intervention on a regular basis, hence Yudkowsky talking about airstrikes that one time.
    - maeil 3y ago
      I'm not suggesting to do that, I'm just showing that it's incredibly possible. You could also go with "Tomorrow a massive storm strikes the Netherlands and huge earthquakes striking Taiwan and the US Westcoast", if you have emotional objections to my first example.
  - Maken 3y ago
    It's delayed at best.
    - maeil 3y ago
      Incredibly delayed, giving at least decades to figure things out.
  - YetAnotherNick 3y ago
upupupandaway 3y ago
The fact that anyone may be taking Eliezer Yudkowsky seriously on this topic is mind-blowing to me. Simply mind-blowing. To imagine that anybody, including the UN, would have any power to collective put a stop to the development of AI globally is a laughable idea. The UN cannot agree to release a letter on Israel-Hamas, imagine policing the ENTIRE WORLD and shutting down AI development when necessary. We can't stop countries from developing weapons that will destroy us all tomorrow morning and take billions of USD to develop, imagine thinking we can stop matrix multiplication globally. I don't want to derail into an ad hominem, but frankly, it's almost the only option left here.
- jeffparsons 3y ago
  My estimation of Eliezer Yudkowsky just dropped a notch — not because I don't take the existential threat posed by AI seriously — but because he seems to think that global coordination and control is an even remotely possible strategy.
  - concordDance 3y ago
    He actually doesn't think it's viable, but thinks its the only card left to play. He's pretty sure that human civilization will be extinct this century.
    - jeffparsons 3y ago
      It can't be the only card left to play. There are so many others. They may be weak, but they still have to be better than one that is guaranteed to be absolutely useless. Example of a weak, but at least _better_ option: convince _one_ government (e.g. USA) that it is in their interest to massively fund an effort to (1) develop an AI that is compliant to our wishes and can dominate all other AIs, and (2) actively sabotage competing efforts. Governments have done much the same in the past with, e.g. conventional weapons, nuclear weapons, and cryptography, with varying levels of success. If we're all dead anyway otherwise, then I don't see how that can possibly be a worse card. Edit: or even try to convince _all_ governments to do this so that they come out on top. At least then there's a greater chance that the bleeding edge of this tech will be under the stewardship of a deliberate attempt for a country to dominate the world, rather than some bored kid who happens to stumble upon the recipe for global paperclips.
RandomLensman 3y ago
There is no existential risk now and there might never be. So "stop", especially with the use of violence, is a pretty odd position to take to say it mildly.
api 3y ago
Yudkowski is a crackpot. Nobody would take him seriously if he wasn’t one of Peter Thiel’s pet crackpots. I should have done a ton more intellectual edgelording in the oughts and grabbed some Thielbucks. I could have recanted it all later. Of course I probably would have had to at least flirt with eugenics. That seems to be the secret sauce.
m_a_g 3y ago
Could you explain to me how AGI is an existential threat to humanity? All the arguments I've read so far feel so far-fetched it's unbelievable. It's in the order of making an AGI and making it control the Pentagon (!?).
- hollerith 3y ago
  The AGI does not need to control the Pentagon. It might for example invent novel technologies that you and I cannot even imagine because we are much less capable at inventing new technologies than the AGI will be. If members of an alien civilization much more advanced than our own suddenly appeared in orbit around Earth, would that not be dangerous?
  - m_a_g 3y ago
    That I understand, but my thinking automatically goes to control and restrictions. A superintelligent AGI that's as restricted as chatgpt in terms of what it can actually control seems... not so existentially dangerous? In that scenario, that actual problem would rise by not-so-smart humans listening to that AGI blindly, no?
    - hollerith 3y ago
      I'm confused by your "as restricted as chatgpt". Hasn't chatgpt been given unrestricted access to the web? Surely if chatgpt were sufficiently capable, it would be able to use that access to hack web sites and thereby gain control over money. Then it can use the web to buy things with that money just like you and I can. It is a lot harder to prevent the sysadmins who have temporary control over you from noticing that you just hacked some web sites and stole some money, but a sufficiently capable AI would know about the sysadmins and would take pains to hide these activities from them.
      - johnthewise 3y ago
        It doesn't even need to hack or do anything illegal. It can make money trading stocks, predicting or modelling far better than market participants, use money to influence the real world, bribe politicians etc. Like with few billions, i bet you can create a political party, influence campagins and end up with real human followers and lobbying power. It doesn't even need to run on its own, if there was a trading bot that said, in order for it to make me money, I need to give it internet access and run it 24/7, rent gpu clusters and s3 buckets etc, I'd do it. this is the most probable scenario imo, AI creating beneficial scenarios for subset of people so that we comply with its requests. very little convincing is necessary
EVa5I7bHFq9mnYK 3y ago
What stop? Will Kim Jong Un stop? Hell no. There is no stop, the genie is out of the bottle.
birdyrooster 3y ago
lmao you have to be kidding me
veidr 3y ago
I don't want to be rude; I understand where you are coming from (at least in terms of single-minded risk mitigation from a theoretical and unconstrained viewpoint), but this reads like Mike Huckabee's 'just keep an aspirin pill pinned between your knees' advice to teenage girls on how not to get pregnant. There is obviously (!!!!!!!!!) no way in this — or any imaginable — timeline that we are going to just "stop". (And it's hard to imagine any mechanism that could enforce "stop" that isn't even worse than the scenarios we're already worrying about. How does that work? Elon or Xi-Poohbear have to personally sign off on installations of more than 1000 cores?)
- pixl97 3y ago
  Butlerian Jihad level revolt by the average person? No reason to see that happening these days, and by the time we're willing to it may already be too late.
j45 3y ago
Except while saying stop, and saying they will, none have stopped because apparently others won’t. Since developing the danger first to help regulate out everyone who doesn’t cross the line first.. open source alternatives are increasingly important. The regulations announced in the US in the past week effectively make it for the few and not the many. There’s been a few rumblings that OpenAI has achieved AGI recently as well.
slashdev 3y ago
Stop is a fantasy pretty much prohibited by game theory in a world with multiple competing states. It’s not going to happen.
pdonis 3y ago
But the AI leaders who have been calling for the US government to act are not asking for AI research to stop. Nor are they asking for the kind of international policing that Yudkowsky describes. They are asking for regulations much like the ones the Biden administration just published--which are indeed regulatory capture since all they do is build a moat around current AI companies to protect their profits. They do not stop AI research or make it any less dangerous.
dragonwriter 3y ago
Yudkowsky’s call for declaring a global AI halt and engaging in unlimited warfare with anyone who defies it poses a greater and more immediate existential risk to humanity than AI ever could; it is largely mitigated (though still a bigger X-Risk than AI) because no one with the capacity to implement it in any substantive way takes him seriously at the moment. But the people selling AI X-Risk that anyone with any policy influence is listening to are AI sellers using it to push regulatory capture, so those are the people being talked about when people are discussing X-Riskers as a substantive force in policy discussions, not the Yudkowsky cult, which, to the extent it is relevant, mostly provides some background fear noise which the regulatory capture crew leverages.
- 3rd3 3y ago
  > Yudkowsky’s call for declaring a global AI halt and engaging in unlimited warfare with anyone who defies it Could you provide a link to where he said this?
  - dragonwriter 3y ago
    I think he's said it many times in slightly different ways, but one notable one: https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/ https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no... The moratorium on new large training runs needs to be indefinite and worldwide. There can be no exceptions, including for governments or militaries. If the policy starts with the U.S., then China needs to see that the U.S. is not seeking an advantage but rather trying to prevent a horrifically dangerous technology which can have no true owner and which will kill everyone in the U.S. and in China and on Earth. . . . Shut down all the large GPU clusters (the large computer farms where the most powerful AIs are refined). Shut down all the large training runs. Put a ceiling on how much computing power anyone is allowed to use in training an AI system, and move it downward over the coming years to compensate for more efficient training algorithms. No exceptions for governments and militaries. Make immediate multinational agreements to prevent the prohibited activities from moving elsewhere. Track all GPUs sold. If intelligence says that a country outside the agreement is building a GPU cluster, be less scared of a shooting conflict between nations than of the moratorium being violated; be willing to destroy a rogue datacenter by airstrike.
0x00_NULL 3y ago
No. Sorry, but this is wrong and Yudkowsky is both naïve and mostly exists in the domain of fan fiction. There are way way way too many issues that are addressed with a hand-wave around scenarios like “AI developing super intelligence in secret and spreading itself around decentralized computers while getting forever smarter by reading the internet.” Too many of his arguments depend on stealth for systems that take up datacenters and need whole-city-block scales of power to operate. Physically, it’s just not possible to move these things. Economically, it wouldn’t be either close to plausible either. Power is expensive and it is the single most monitored element in every datacenter. There is nothing large that happens in a datacenter that is not monitored. There is nothing that is going to train on meaningful datasets in secret. “What about as technology increases and we get more efficient at computing?” We use more power for computing every year, not less. Yes, we get more efficient, but we don’t get less energy intensive. We don’t substitute computing capacity. We add more. The same old servers are still there contributing to the compute. Google has data centers filled with old shit. That’s why the standard core in GCP is 2 GHz. Sometimes, your project gets put on a box from 2010, other times, it gets put on a box from 2023. That process is abstracted so you can’t tell the difference. TLDR: Yudkowsky’s arguments are merely fan fiction. People don’t understand ML systems so they imagine an end state without understanding how to get there. These are the same people who imagine Light Speed Steam Trains. “We need faster rail service, so let’s just keep adding steam to our steam trains and accelerate to the speed of light.” That’s exactly what these AI Doom arguments sound like to people in the field. Your AI Doom scenario, although it might be very imaginative, is a Light Speed Steam Train. It falls apart the moment you try to trace a path from today to doom.
- kubanczyk 3y ago
  Are you saying that the governments/agencies are incapable of hiding any big infra? That's a novel take.

troupo 3y ago

> The laws we have already cover malevolent and abusive uses of software applications. They don't. Or not nearly enough. Otherwise you wouldn't have automated racial profiling, en masse face recognition, credit and social scoring etc. And it's going to get worse. Because AI is infallible, right? Right?! That's why, among other thing, EU AI regulation has the following: - Foundation models have to be thoroughly documented, and developers of these modes have to disclose what data they were trained on - AI cannot be used in high-risk applications (e.g. social scoring etc.) - When AI is used, its decisions must be explicable in human terms. No "AI said so, so it's true". Also, if you interact with an AI system it must be clearly labeled as such

tluyben2 3y ago
This type of regulatory would be better than 'stop' or making all kinds of expensive rules that only big corps can follow.
- troupo 3y ago
  There are still quite a few rules, but they seem common sense: https://softwarecrisis.dev/letters/the-truth-about-the-eu-act/ https://softwarecrisis.dev/letters/the-truth-about-the-eu-ac...
RandomLensman 3y ago
AI not being allowed in high-risk applications is a bad idea, same for explicable in human terms (cannot do that now for human decisions anyway). The latter would kind of rule out any quantum computing, too, as the only real explanation there is through maths, not words.
- troupo 3y ago
  > AI not being allowed in high-risk applications is a bad idea wat > same for explicable in human terms (cannot do that now for human decisions anyway). What? Human decisions can be explained and appealed. Good luck proving something when a blackbox finds you guilty of anything.

nologic01 3y ago

It points out to the continuing digital illiteracy of large parts of society and, unfortunately, decision making centers. Breaking down any complex knowledge domain so that people can make informed decisions is not easy. There is also an entrenched incentive for domain experts to keep the "secret sauce" secret even if it amounts to very little. So far nothing new versus how practically any specialized sector works. The difference with information technology (of which AI is but the current perceived cutting edge) is that it touches everything. Society is built on communication. If algorithms will intervene in that floe we cannot afford to play the usual stupid control games with something so fundamental.

concordDance 3y ago

> The way incumbents are attempting to seize power via regulatory capture, using "X-Risk" and fantastical claims of sudden human extinction is maybe one of the most cynical ploys in the history of tech so far. How sure are you that they don't think the risk is real? I have only third hand accounts, but at least some of the low level people working for OpenAI seemed to think the risk is worth taking seriously, its not just a cynical CEO ploy. Sam Altman knows and has talked to Yudkowsky plenty of times. To me the simplest explanation is that he thinks figuring out alignment via low powered models is the best way of solving the problem, so pushed to get those working and then trying to reduce the rate of progress for a bit while they figure it out. (I think it's a plan that doesn't have good odds, but nothing seems to give amazing odds atm)

nerdponx 3y ago
The best lies contain a kernel of truth.
randomdata 3y ago
> but at least some of the low level people working for OpenAI seemed to think the risk is worth taking seriously Everyone has an opinion, but the experts seem strangely silent on this matter. I mean experts in human risk, not experts in AI. The latter love to ramble on, but are unlikely to know anything about social and humanitarian consequences – at least no more than the stock boy at Walmart. Dedicating one's life to becoming an expert in AI doesn't leave much time to become an expert in anything else. There is only so much time in the day.
- danenania 3y ago
  Conversely, why should we expect an expert in "human risk" with no understanding of AI to have anything productive to add to the conversation? We should be looking for people who are well-grounded in both areas, if any exist.
  - randomdata 3y ago
    What additional information would understanding AI actually bring to the table? Worst case you just assume the AI can become human-like. Certainly all the vocal AI experts can come up with is things that humans already do to each other already, only noting that AI enables it at larger scale. Clearly there is no benefit in understanding the AI technology with respect to this matter when the only thing you can point to is scale. The concept of scalability doesn't require an understanding of AI.

arisAlexis 3y ago

The fantastical claims you are claiming are actually the most probable scenario according to the majority (2/3) Hinton and Bengio that discovered the technology you are referring to. Strange to contradict them eh?

amalcon 3y ago

This is more or less exactly where I land on this. The extinction risk argument is somewhat plausible in the long term[0], but it doesn't seem that we are especially close to developing that class of problem. Meanwhile, there are very real and immediate risks to this type of technology -- in various forms of giving some humans more power over other humans. The attacks proposed by the extinction risk people actively make those immediate risks worse. The only way we've found to prevent some from leveraging almost any technology to control others is to democratize said technology -- so that everyone has access. Limiting access will just accrue more power to those with access, even if they need to break laws to get it. [0]- Say, the lower bound of the 95% confidence interval is 50-100 years out. People who I know to be sensible disagree with me on this estimate, but it's where I honestly end up.

itsoktocry 3y ago
>The extinction risk argument is somewhat plausible in the long term[0], but it doesn't seem that we are especially close to developing that class of problem. The parable at the beginning of Superintelligence relates to this; the problem looks far away, but we don't know where, precisely it will arise, or if it will be too late when we realize it.
boringg 3y ago
You mean democratize the technology within specific confines? I don't see humans proliferating weapons technology amongst other nations as that poses a greater risk that conflicts will become more violent.

nmca 3y ago

None of the world-leading scientists that co-authored this paper: https://managing-ai-risks.com/ https://managing-ai-risks.com/ Work for big labs? How is that you think the proposed conspiracy works exactly? They're paying off Yoshua Bengio? (disclaimer: I work on safety for a large lab. But none of these proponents do!)

Wurdan 3y ago
And how many of them have been invited to the White House to share their concerns? https://www.whitehouse.gov/briefing-room/statements-releases/2023/05/04/readout-of-white-house-meeting-with-ceos-on-advancing-responsible-artificial-intelligence-innovation/ https://www.whitehouse.gov/briefing-room/statements-releases... I don't see any of their names in this meeting's attendees, for example. Sam Altman's there, Satya Nadella's there, Sundar Pichai's there... No Yoshua Bengio, though. What the person you replied to is saying is that commercial developers of AI have a significant financial incentive to be the ones defining what risky AI looks like. If they can guide the regulatory process to only hinder development that they didn't want to do anyway (like develop AI powered nuclear missile launch systems) then it opens the gates for them to develop AI which could replace 27% of jobs currently done by people (https://www.reuters.com/technology/27-jobs-high-risk-ai-revolution-says-oecd-2023-07-11/ https://www.reuters.com/technology/27-jobs-high-risk-ai-revo...) and become the richest enterprises in history.
- nmca 3y ago
  You're confusing "that one photo of three ceos at the whitehouse" with "all government solicitation of expert opinion on AI". It's correct that the meeting you're thinking of was industry-heavy, but the process in general has had a lot of academic input from, e.g Marcus, Tegmark, etc. See the recent summit in the UK, etc.
  - mistrial9 3y ago
    .. academic-heavy position letters have been ignored before

germandiago 3y ago

True. We receive warnings all the time but I think what these people are afraid of is actually losing their power.

api 3y ago
Until these same people start being concerned about DIY CRISPR kits, I do not take them seriously: https://www.the-odin.com/diy-crispr-kit/ https://www.the-odin.com/diy-crispr-kit/ This is dangerous in ways that require no leaps of logic or hypothetical sci-fi scenarios to imagine. I studied biology as an undergrad and can think of several insanely evil things you could do with this plus some undergrad-level knowledge of genetics and... not gonna elaborate. But no we are talking about banning math. This is a combination of cynical people pushing for regulatory capture and smart people being dumb in the highly sophisticated very verbose ways that smart people are dumb.
- carapace 3y ago
  Holy shit. That's legitimately terrifying.
- abecedarius 3y ago
  I've seen Yudkowsky and others saying that AI and biotech are both in the unusual bin of threats to all humanity, exceptions to his belief that most things are overregulated. (paraphrase of tweets from memory)

2devnull 3y ago

The laws are coming no matter what, not because of lobbying or whatever, the laws/policy response is coming because politicians want to get in on the ai too. The regulations we will do things like mandate ai be used for equity purposes in the real estate market. I think that’s in Biden’s plan. I mean, I’m sure these tech ceo 1% want regulatory capture, and they’ll get it as a quid pro quo for allowing ai to be used for largely partisan goals. But we shouldn’t fear open source ai, or regulations against it, we should fear the ai being weapon used by the political parties. It to some extent already is.

beepbooptheory 3y ago

It's not cynical though. It is, if anything, revelatory to the true interests at play here, even if on the surface the argument is certainly in bad faith. If you need a labor market, you don't want a publicly available infinite-free-labor-widget. It might as well be a handheld nuclear device.

godelski 3y ago

Importantly locking away model checkpoints makes AI safety research, at best, exceptionally difficult or impossible. We already don't know all the training methods nor do we know the data used for the models. AI safety is absolutely contingent on works being __more__ open source, not less. But there's less profit in that for these companies (potentially more profit for the country as a whole though, as we see new businesses form and sectors develop, but this is more spread out and far harder to calculate the value of so let's ensure we recognize this as speculative). We also need to be keenly aware that xrisk is not the major issue here. We do not need AGI to create massive amounts of harm, nor do we need AGI to create a post-scarce world (which the transition to can also generate massive harm if we don't do it carefully. Despite generating a post-scarce world -- or at least getting ever closer to that -- is, imo, one of the most important endeavors we can work on). Current ML systems can already do massive amounts of destruction, but so can a screwdriver. My big fear is that AI will go the way of nuclear. A technology is not good or bad, it is like a coin with a value. You can spend that coin on something harmful or something beneficial, but the coin is just a coin at the end of the day. That's what regulation is supposed to be about, expenditure, not minting.