3 ms·

> But when the median AI researcher believes there's a 10% (!) chance that what they're working on will destroy the world, and the pace of that work is only qui

> But when the median AI researcher believes there's a 10% (!) chance that what they're working on will destroy the world, and the pace of that work is only quickening, that seems like a big issue to worry about. Yet they will not quit their job. What that suggests to me is that they don't really believe it, otherwise they would sacrifice their paycheck to stop it. Seems to me that what they really believe is that they should get more attention and more funding.

dwohnitmok 4y ago
> Yet they will not quit their job. What that suggests to me is that they don't really believe it, otherwise they would sacrifice their paycheck to stop it. I've talked to quite a few researchers. They generally do earnestly believe that AI has huge damage potential. The answers to why they keep developing AI falls in several different buckets (to which I have varying degrees of agreement) 1. We're in a prisoner's dilemma. If everyone could unilaterally agree to slow down AI research I'd be happy to quit, but until that happens if I don't do it someone else will and with even fewer scruples. 2. The only way to develop AGI safely is to push things to the brink. It's impossible to make progress on how to make sure that AGI is safe is to almost make AGI because not until then do you have a clear enough picture of the exact form AGI will take to do productive safety research. 3. Slowing down AI research has its own risks. If you slow down AI research, but other fields of technology continue to progress, you can end up in a scenario where everyone's personal computer is capable not only of running models, but of doing cutting-edge ML training. Right now to coordinate on developing AGI safely, there are maybe ~10 top teams with the infrastructure possible to do the training (not just running the model!) needed to advance the state of the art you have to get cooperation among. Hard, but not impossible. If technology progresses to the point where there tens of thousands of teams all capable of meaningfully advancing the state of the art then coordination is absolutely impossible. 4. If AI can, then AI should replace humans entirely. (the most chilling response I've heard. Luckily few in number, but it shows up even on HN)

dwohnitmok 4y ago

> Yet they will not quit their job. What that suggests to me is that they don't really believe it, otherwise they would sacrifice their paycheck to stop it. I've talked to quite a few researchers. They generally do earnestly believe that AI has huge damage potential. The answers to why they keep developing AI falls in several different buckets (to which I have varying degrees of agreement) 1. We're in a prisoner's dilemma. If everyone could unilaterally agree to slow down AI research I'd be happy to quit, but until that happens if I don't do it someone else will and with even fewer scruples. 2. The only way to develop AGI safely is to push things to the brink. It's impossible to make progress on how to make sure that AGI is safe is to almost make AGI because not until then do you have a clear enough picture of the exact form AGI will take to do productive safety research. 3. Slowing down AI research has its own risks. If you slow down AI research, but other fields of technology continue to progress, you can end up in a scenario where everyone's personal computer is capable not only of running models, but of doing cutting-edge ML training. Right now to coordinate on developing AGI safely, there are maybe ~10 top teams with the infrastructure possible to do the training (not just running the model!) needed to advance the state of the art you have to get cooperation among. Hard, but not impossible. If technology progresses to the point where there tens of thousands of teams all capable of meaningfully advancing the state of the art then coordination is absolutely impossible. 4. If AI can, then AI should replace humans entirely. (the most chilling response I've heard. Luckily few in number, but it shows up even on HN)