10 ms·

Recursively self-improving AI, of the kind Nick Bostrom outlined in detail way back in his 2014 book Superintelligence and Dr. Omohundro outlined in brief in [1

• StableAlkyne 2y ago
  > Such a program would incur minimal policing costs and could easily scale to handle international threats Who ensures every country enforces the ban? How do you ensure international cooperation against a nation that decides to ignore it, against whom sanctions have no effect? What if that nation is a nuclear power?
• ben_w 2y ago
  > is the only kind which poses a true existential threat You don't accept the possibility that a non-improving tool of an AI system that is fixed at the level of "just got a PhD in everything" by reading all the research papers on arxiv, might possibly be advanced enough for a Jim Jones type figure to design and create a humanity-ending plague because they believe in bringing about the end times?
  • kbenson 2y ago
    Wouldn't there be 100x more of the same capability looking for threats and trying to head them off? A very advanced tool is still just a tool, and subject to countermeasures. I can see why countries would want to regulate it, but personally I think it's a distinctly different category than what the GP comment was talking about. There is no stopping a singularity level event after it's begun, at least not by any process where people play a role.
    • ben_w 2y ago
      > Wouldn't there be 100x more of the same capability looking for threats and trying to head them off? Hard to determine. It's fairly easy to put absolutely everyone under 24/7 surveillance. Not only does almost everyone carry a phone, but also laser microphones are cheap and simple, and WiFi can be used as wall penetrating radar capable of pose detection at sufficient detail for heart rate and breath rate sensing. But people don't like it when they get spied on, it's unconstitutional etc. And we're currently living through a much lower risk arms race of the same general description with automatic code analysis to find vulnerabilities before attackers exploit them, and yet this isn't always a win for the defenders. Biology is not well-engineered code, but we have had to evolve a general purpose anti-virus system, so while I do expect attackers to have huge advantages, I have no idea if I'm right to think that, nor do I know how big an advantage in the event that I am right at all. > There is no stopping a singularity level event after it's begun, at least not by any process where people play a role Mm, though I would caution that singularities in models is a sign the model is wrong: to simplify to IQ (a flawed metric) an AI that makes itself smarter may stop at any point because it can't figure out the next step, and that may be an IQ 85 AI that can only imagine reading more stuff, or an IQ 115 AI that knows it wants more compute so it starts a business that just isn't very successful, or it might be IQ 185 and do all kinds of interesting things but still not know how to make the next step any more than the 80 humans smarter than it, or it might be IQ 250 and beat every human that has ever lived (IQ 250 is 10σ, beating 10σ is p ≈ 7.62e-24, and one way or another when there have been that many humans, they're probably no longer meaningfully human) but still not know what to do next. I prefer to think of it as an event horizon: beyond this point (in time), you can't even make a reasonable attempt at predicting the future. For me, this puts it at around 2030 or so, and has done for the last 15 years. Too many exponentials start to imply weird stuff around then, even if the weird stuff is simply "be revealed as a secret sigmoid all along".
• Simon_ORourke 2y ago
  Try to get that bounty enforced in the courts! All it'll take is one loophole to what would surely be a few vague poorly drafted laws and the floodgates open again.
• mrshadowgoose 2y ago
  In your mind, what will our world look like once AGI is achieved, and that technology will likely be exclusively in the hands of governments and large corporations? What guarantees do we have that it will be used for the benefit of the common person? What will happen to the large swathes of the human population that will not only be permanently economically useless, but worse than useless? They'll need to be fed, clothed, housed and entertained with the only things they provide back being their opinions and complaints. I literally couldn't care less about recursively-improving superintelligence. AGI in the hands of the rich and powerful is already a nightmare scenario.
• johndough 2y ago
  > A simple example would be "if you care caught doing AI research, you are to pay the people who caught you 10 times your yearly total comp in cash." That sounds like a prime example of perverse incentive: https://en.wikipedia.org/wiki/Perverse_incentive https://en.wikipedia.org/wiki/Perverse_incentive > A perverse incentive is an incentive that has an unintended and undesirable result that is contrary to the intentions of its designers. For example, the British government offering a bounty on dead cobra snakes lead to a large number of cobra breeders. Your proposal is conceptionally similar to the Alberta Child, Youth and Family Enhancement Act (fourth bullet point): https://en.wikipedia.org/wiki/Perverse_incentive#Community_safety_and_harm_reduction https://en.wikipedia.org/wiki/Perverse_incentive#Community_s...
  • hiAndrewQuinn 2y ago
    First: Thank you for looking into the idea without dismissing it out of hand. Some thoughts I have: >That sounds like a prime example of perverse The primary perverse incentive I'd be worried about is slowing down research that is seen by people as AI-adjacent, but is not in reality a likely vector to recursively self-improving AI. Sure, whatever. My research into control theory in undergrad might never have happened, etc. Gotta break some eggs to make an omelette. I don't consider the mere existence of a perverse incentive to be a reason not to do something. It's all a matter of scale. All modern bureaucracies are rife with principal-agent problems, for example, but that doesn't mean the effects of centralization and specialization don't still end up good If you buy the that Bostrom-style AI is likely to be eventually developed on our current trajectory, and is likely to kill us all / colonize the lightcone / etc. etc., then it's not hard to argue that we should be willing to spend a lot of resources on making this not happen, even we can't guarantee a perfectly efficient allocation of those resources to stopping it. (That being said, I do feel that my loose policy suggestion gets unusually close to "actually effective" and "not horrendously expensive on a global scale", compared to other heavy-handed approaches I've heard in this space.) >For example, the British government offering a bounty on dead cobra snakes lead to a large number of cobra breeders. Perverse incentives would likely still appear if the British government offered a bounty on cobra breeders themselves -- or, more accurately to my proposal, allowed people to turn in and prove that other people were breeding cobras in order to exact a sizeable portion of the breeders' wealth from them as a bounty -- but they would likely end up as more of the sort "Hey, I don't think this cobra breeder is actually British, I don't think we want to risk this." It probably would still be pretty effective at stopping cobra breeding, within the area of Britain. >In Alberta, under the Child, Youth and Family Enhancement Act, every person must report suspected child abuse to a director or police officer, and failure to do so is punishable by a $10,000 fine plus 6 months of imprisonment,[14] with the fine increased from $2,000 to $10,000 as a politician's response to a little girl who died of a catastrophic head injury after she was placed in kinship care.[15] However, according to criminal law professor Narayan, enforcing it would cause people to overreport, which wastes resources, and it would also create a chilling effect that prevents people from reporting child abuse observed over a period of time, as that would incriminate them for failing to report earlier. I don't see my proposal as much like this at all. For one, the fine is levied on those who suspect but do not report child abuse, rather than the child abusers themselves. For two, the body of the fine goes towards the state itself, not towards private investigators. For three, unlike abusing a child, doing novel research in AI requires a great many rare things to come together in a person - they have to be smart, hard working, probably formally educated in CS or mathematics, and (at least currently) they usually have to buy a lot of compute from someone (which leaves a strong paper trail). Let's leave all that aside, though. What I think you're really getting at, is the key way in which my proposal does run a similar risk: What if the judges do start to rule that merely knowing of someone else doing AI research and not reporting it counts as grounds to be sufficiently involved? I won't deny it is possible, but I see it as unlikely. For one, if word gets out that you as a private detective implicate everyone you meet, indiscriminately, nobody will ever want to talk to you and perhaps give you valuable info that can lead you to the actual ringleaders of covert AI advancement operations (which can be quite lucrative). Indeed one of the most important sources of info a private detective could have in this regard is to negotiate with a "whistleblower" already inside the AI research organization - someone who can make the case totally ironclad, in exchange for being left out of it and/or being cut a slice of the profits at the end. The private transfer of property really does just totally change the incentives at play here, in a way which fines paid to the state can't hope to match. (The ever present threat of a whistleblower blowing the coop in exchange for huge amounts of money is incidentally why we would expect organized crime-style AI research cartels, etc to stay low, slow and local.) For two, most judges probably wouldn't accept this as a valid reason to hand you someone else's 401k or house or used car. Indeed this is a large part of why the Alberta act is ridiculous on the face of it. My understanding is that precedent is very important in Anglo law, so it of course only takes one judge at the start to rule poorly in order to put us back into "perverse incentive" territory - but it equally only takes one halfway reasonable judge to rule decently to prevent that fate from ever happening. The latter seems much more likely to my eyes, and the former would probably quickly get overruled.
• dragonwriter 2y ago
  > Such a program would incur minimal policing costs No, it wouldn’t. Just because the policing costs aren't tax-funded doesn't mean they don't exist. (And I'm not talking just about costs voluntarily incurred by bounty seekers, I'm also talking about the cost that the system imposes involuntarily on others who are neither actually guilty nor bounty seekers, because the financial incentives can motivate pursuits imposing costs on targets who are not factually guilty.) > and could easily scale to handle international threats. No, it couldn’t, especially for things where a major source of the international threat is governmental, where private bounties aren’t going to work at all. And it especially can’t work against developing technology, where the evidence needed in litigation and the litigation would itself be a vector for spreading the knowledge that you are attempting to repress.
  • _cjse 2y ago
    >[D]oesn't mean they don't exist Never said they don't exist, merely that they are "minimal", aka no other policy I could think of seems like it would obviously lead to lower costs while still achieving the desired outcome. >I'm also talking about the cost that the system imposes involuntarily on others who are neither actually guilty nor bounty seekers This is a case from negative externalities. First, consider the simple argument from scale. If you buy the idea that a Bostrom-like AI is both (1) very likely to be created on our current technological trajectory, and (2) will probably kill us all, then it's not hard to argue that the benefits reaped from avoiding that fate justifies a similarly high cost to society, maybe several percentage points of global GDP. After all, you're not just risking deflating present-day GDP, you're risking multiplying all future GDPs by 0. Every country in the First World already imposes tremendous involuntary costs on people for things of much less significance, like 'forcing' you to go through TSA even though you would never in your wildest dreams try to hijack a plane, so the mere existence of involuntary costs doesn't sway me. Alright, but what would the magnitude of those involuntary costs be? If this policy costs everyone a hundred bucks a year in hassle, we've still got a vexation. There is strong reason to believe that, for almost everyone, it really would be very, very low in absolute terms. How much of the population is currently engaged in frontier-pushing AI research right now? 1%? 0.1%? Actually probably a few orders of magnitude lower. OpenAI still employs less than, what, a thousand people, etc. etc. The vast majority of people will never do anything remotely like that in their lives. So one would expect very cheap private insurance policies to appear as an effective way to get out of being pursued and harassed by private bounty hunters. The firms which pop up to provide this service would probably get very, very good at negotiating with private bounty hunters very, very quickly, to leave anyone not directly in the know out ASAP. The cost for almost everyone would be on the order of cents per year of protection, that's how unlikely it is that some random clerical worker in Kansas has any serious involvement in creating the next self-improving AI. In exchange, of course, these insurance policies have a very strong reason not to insure people who actually are involved in such activities, and so they could form a critical source of information for helping the bounty hunters target their own search. >A major source of the international threat is governmental, where private bounties aren’t going to work at all Strongest argument I've heard so far, thank you for raising it. First I'll point out it would already be a dream come true for extending our AI doom timelines if the only people who can actually do AI research without fear of being extradited to a bounty-friendly nation is to work in a government lab. The Department of Defense is very impressive, but they still don't move nearly as quickly as private industry and independent universities do when it comes to work like this. That could be generations more of humanity around to live and love and prosper before we get snuffed out. Don't let the perfect be the enemy of the good! Let's get serious. AI researchers in your home country are the easiest case, because you have unilateral law on your side. AI reseachers in other countries are quite a bit more difficult, because now you're in the messy world of international diplomacy. If the other country adopts a bounty law as well, you both win. If neither of you do, you both lose. But what about the case where one of you does, and one of you doesn't? I posit that here, in the end, you probably have to make it so the bounty-friendly nation is the one that wins, with force - that is, allowing bounty hunters to turn in and extract money from even foreign employees if they get within your borders. And, yes, if the other governments decide to respond by closing their private AI businesses and opening up government labs only ... Well, you've slowed the wave quite a bit, but you're probably going to have to be more careful. No one ever said shifting the Nash equilibrium would be easy. But you do have other options, even here. One hazy possibility in my mind, would be offering US or EU citizenship to any foreign national who is both (1) a credible AI researcher and (2) precommits to stopping their research as soon as they take the offer. Bounty hunters win because (duh) you now have a heavily pre-filtered list of marks you can watch like a hawk for the instant they slip up. And chances are good that someone on that list will, even after getting citizenship, and then you can extract a tidy sum from them for minimal effort. The foreign researchers who take the offer win because living and working in e.g. New York City as an employee of Jane Street is probably much nicer than working on recursviely self-generating AI in a secretive, underpaid, underfunded, underground lab in e.g. Chengdu. (It's important to remember that cutting edge AI researchers are, almost by definition, really really smart and really really good with computers and math. They have a uniquely great set of careers they can switch into easily.) The world wins because the risk of self-caused extinction goes down another iota. China "loses", but it loses in the smallest way possible - it decided to undertake risky research instead of telling its citizens to choose something more straightforwardly good for humanity, and it suffered a bit of brain drain. That's aggravating, but it's hardly worth launching a China v. NATO war over. And hey, if China wants to stop the brain drain, they already have a good model for a very effective law they could implement to get people to stop doing dangerous research - that same bounty law we've been discussing. I freely admit this is the weakest part of my theory, becuase it's the weakest part of anyone's theory. International stuff is always much harder to reason about. Still, whereas most policies I've seen put forward seem to fail instantly and obviously, mine seems only probably destined to fail. That's a big improvement in my eyes.