3 ms·

> A simple example would be "if you care caught doing AI research, you are to pay the people who caught you 10 times your yearly total comp in cash." That soun

> A simple example would be "if you care caught doing AI research, you are to pay the people who caught you 10 times your yearly total comp in cash." That sounds like a prime example of perverse incentive: https://en.wikipedia.org/wiki/Perverse_incentive https://en.wikipedia.org/wiki/Perverse_incentive > A perverse incentive is an incentive that has an unintended and undesirable result that is contrary to the intentions of its designers. For example, the British government offering a bounty on dead cobra snakes lead to a large number of cobra breeders. Your proposal is conceptionally similar to the Alberta Child, Youth and Family Enhancement Act (fourth bullet point): https://en.wikipedia.org/wiki/Perverse_incentive#Community_safety_and_harm_reduction https://en.wikipedia.org/wiki/Perverse_incentive#Community_s...

hiAndrewQuinn 2y ago
First: Thank you for looking into the idea without dismissing it out of hand. Some thoughts I have: >That sounds like a prime example of perverse The primary perverse incentive I'd be worried about is slowing down research that is seen by people as AI-adjacent, but is not in reality a likely vector to recursively self-improving AI. Sure, whatever. My research into control theory in undergrad might never have happened, etc. Gotta break some eggs to make an omelette. I don't consider the mere existence of a perverse incentive to be a reason not to do something. It's all a matter of scale. All modern bureaucracies are rife with principal-agent problems, for example, but that doesn't mean the effects of centralization and specialization don't still end up good If you buy the that Bostrom-style AI is likely to be eventually developed on our current trajectory, and is likely to kill us all / colonize the lightcone / etc. etc., then it's not hard to argue that we should be willing to spend a lot of resources on making this not happen, even we can't guarantee a perfectly efficient allocation of those resources to stopping it. (That being said, I do feel that my loose policy suggestion gets unusually close to "actually effective" and "not horrendously expensive on a global scale", compared to other heavy-handed approaches I've heard in this space.) >For example, the British government offering a bounty on dead cobra snakes lead to a large number of cobra breeders. Perverse incentives would likely still appear if the British government offered a bounty on cobra breeders themselves -- or, more accurately to my proposal, allowed people to turn in and prove that other people were breeding cobras in order to exact a sizeable portion of the breeders' wealth from them as a bounty -- but they would likely end up as more of the sort "Hey, I don't think this cobra breeder is actually British, I don't think we want to risk this." It probably would still be pretty effective at stopping cobra breeding, within the area of Britain. >In Alberta, under the Child, Youth and Family Enhancement Act, every person must report suspected child abuse to a director or police officer, and failure to do so is punishable by a $10,000 fine plus 6 months of imprisonment,[14] with the fine increased from $2,000 to $10,000 as a politician's response to a little girl who died of a catastrophic head injury after she was placed in kinship care.[15] However, according to criminal law professor Narayan, enforcing it would cause people to overreport, which wastes resources, and it would also create a chilling effect that prevents people from reporting child abuse observed over a period of time, as that would incriminate them for failing to report earlier. I don't see my proposal as much like this at all. For one, the fine is levied on those who suspect but do not report child abuse, rather than the child abusers themselves. For two, the body of the fine goes towards the state itself, not towards private investigators. For three, unlike abusing a child, doing novel research in AI requires a great many rare things to come together in a person - they have to be smart, hard working, probably formally educated in CS or mathematics, and (at least currently) they usually have to buy a lot of compute from someone (which leaves a strong paper trail). Let's leave all that aside, though. What I think you're really getting at, is the key way in which my proposal does run a similar risk: What if the judges do start to rule that merely knowing of someone else doing AI research and not reporting it counts as grounds to be sufficiently involved? I won't deny it is possible, but I see it as unlikely. For one, if word gets out that you as a private detective implicate everyone you meet, indiscriminately, nobody will ever want to talk to you and perhaps give you valuable info that can lead you to the actual ringleaders of covert AI advancement operations (which can be quite lucrative). Indeed one of the most important sources of info a private detective could have in this regard is to negotiate with a "whistleblower" already inside the AI research organization - someone who can make the case totally ironclad, in exchange for being left out of it and/or being cut a slice of the profits at the end. The private transfer of property really does just totally change the incentives at play here, in a way which fines paid to the state can't hope to match. (The ever present threat of a whistleblower blowing the coop in exchange for huge amounts of money is incidentally why we would expect organized crime-style AI research cartels, etc to stay low, slow and local.) For two, most judges probably wouldn't accept this as a valid reason to hand you someone else's 401k or house or used car. Indeed this is a large part of why the Alberta act is ridiculous on the face of it. My understanding is that precedent is very important in Anglo law, so it of course only takes one judge at the start to rule poorly in order to put us back into "perverse incentive" territory - but it equally only takes one halfway reasonable judge to rule decently to prevent that fate from ever happening. The latter seems much more likely to my eyes, and the former would probably quickly get overruled.

hiAndrewQuinn 2y ago

First: Thank you for looking into the idea without dismissing it out of hand. Some thoughts I have: >That sounds like a prime example of perverse The primary perverse incentive I'd be worried about is slowing down research that is seen by people as AI-adjacent, but is not in reality a likely vector to recursively self-improving AI. Sure, whatever. My research into control theory in undergrad might never have happened, etc. Gotta break some eggs to make an omelette. I don't consider the mere existence of a perverse incentive to be a reason not to do something. It's all a matter of scale. All modern bureaucracies are rife with principal-agent problems, for example, but that doesn't mean the effects of centralization and specialization don't still end up good If you buy the that Bostrom-style AI is likely to be eventually developed on our current trajectory, and is likely to kill us all / colonize the lightcone / etc. etc., then it's not hard to argue that we should be willing to spend a lot of resources on making this not happen, even we can't guarantee a perfectly efficient allocation of those resources to stopping it. (That being said, I do feel that my loose policy suggestion gets unusually close to "actually effective" and "not horrendously expensive on a global scale", compared to other heavy-handed approaches I've heard in this space.) >For example, the British government offering a bounty on dead cobra snakes lead to a large number of cobra breeders. Perverse incentives would likely still appear if the British government offered a bounty on cobra breeders themselves -- or, more accurately to my proposal, allowed people to turn in and prove that other people were breeding cobras in order to exact a sizeable portion of the breeders' wealth from them as a bounty -- but they would likely end up as more of the sort "Hey, I don't think this cobra breeder is actually British, I don't think we want to risk this." It probably would still be pretty effective at stopping cobra breeding, within the area of Britain. >In Alberta, under the Child, Youth and Family Enhancement Act, every person must report suspected child abuse to a director or police officer, and failure to do so is punishable by a $10,000 fine plus 6 months of imprisonment,[14] with the fine increased from $2,000 to $10,000 as a politician's response to a little girl who died of a catastrophic head injury after she was placed in kinship care.[15] However, according to criminal law professor Narayan, enforcing it would cause people to overreport, which wastes resources, and it would also create a chilling effect that prevents people from reporting child abuse observed over a period of time, as that would incriminate them for failing to report earlier. I don't see my proposal as much like this at all. For one, the fine is levied on those who suspect but do not report child abuse, rather than the child abusers themselves. For two, the body of the fine goes towards the state itself, not towards private investigators. For three, unlike abusing a child, doing novel research in AI requires a great many rare things to come together in a person - they have to be smart, hard working, probably formally educated in CS or mathematics, and (at least currently) they usually have to buy a lot of compute from someone (which leaves a strong paper trail). Let's leave all that aside, though. What I think you're really getting at, is the key way in which my proposal does run a similar risk: What if the judges do start to rule that merely knowing of someone else doing AI research and not reporting it counts as grounds to be sufficiently involved? I won't deny it is possible, but I see it as unlikely. For one, if word gets out that you as a private detective implicate everyone you meet, indiscriminately, nobody will ever want to talk to you and perhaps give you valuable info that can lead you to the actual ringleaders of covert AI advancement operations (which can be quite lucrative). Indeed one of the most important sources of info a private detective could have in this regard is to negotiate with a "whistleblower" already inside the AI research organization - someone who can make the case totally ironclad, in exchange for being left out of it and/or being cut a slice of the profits at the end. The private transfer of property really does just totally change the incentives at play here, in a way which fines paid to the state can't hope to match. (The ever present threat of a whistleblower blowing the coop in exchange for huge amounts of money is incidentally why we would expect organized crime-style AI research cartels, etc to stay low, slow and local.) For two, most judges probably wouldn't accept this as a valid reason to hand you someone else's 401k or house or used car. Indeed this is a large part of why the Alberta act is ridiculous on the face of it. My understanding is that precedent is very important in Anglo law, so it of course only takes one judge at the start to rule poorly in order to put us back into "perverse incentive" territory - but it equally only takes one halfway reasonable judge to rule decently to prevent that fate from ever happening. The latter seems much more likely to my eyes, and the former would probably quickly get overruled.