3 ms·

> …if it really is a great idea that is going to save civilization, it really ought to have the characteristic that it is still a good idea that is an improveme

> …if it really is a great idea that is going to save civilization, it really ought to have the characteristic that it is still a good idea that is an improvement on the status quo even when applied in the small… Not if we're currently locally optimized. Then it would seem worse before it seemed better. I'm not particularly for or against the idea for my own sake. However, I do think it's going to be inevitable sooner or later. Whether it's a good fit for us right now, I can't determine. In my opinion, experiments are needed not in order to determine whether it should be implemented or not, but how it should be done, and in what form. When the level of unemployment starts leading to riots, we'd better prepare with something other than weapons and harsh measures.

jerf 9y ago
"Not if we're currently locally optimized. Then it would seem worse before it seemed better." I think you don't quite have the point I'm trying to make, which is not surprising as it's the first time I'm trying to express it clearly. As I said, I'm sure there are interventions that are locally worse but globally better. Their existence is not the problem. But given the much, much higher prevalence of solutions which are locally worse, and then also globally worse, and the observable fact that many people believe these are good ideas anyhow (simply look across the political isle; everybody will disagree on what these ideas are but everybody should agree they exist), the challenge becomes one of identifying which are actually globally better even though they are locally worse. And I submit that there is no solution to that. At least not for humans. Maybe for Vulcans, but not for humans. (Anyone who wants to disagree with that, please double-check that your argument doesn't amount to "It would work if everybody would just change their opinions to agree with me.", which I pretty much guarantee it will in the end.) So our only rational alternative is to pursue solutions that are locally better, then to try to scale them up (again, keeping an eye out for if and when they turn negative, because they can), because there is no mechanism for discovering interventions which are locally worse but globally better. If the local experiments being done on BI do not have sufficient local success that all involved want to scale it up, we have no mechanism for rationally deciding "Well, it's not working so great on the small scale, but it'll totally work on the large scale." We could only irrationally leap to it, and we would rationally expect to suffer the usual consequences of such irrational leaps. (We are not scientifically or rationally entitled to simply assume BI will work in any sense.) The other thing I'm trying to point out is that even if none of the interventions to date are "true BI", they are micro-scale BI, and if they aren't working there, it is rational to start considering that evidence against BI. BI also can not work if it must be run "just so". If it is only valid at a particular point in the policy space, it will fail. If it is going to work, it must be robust to perturbation. Policies that work in the real world have to be robust against exceedingly intelligent adversaries, and for all everyone may complain with varying degrees of justification about the current state of the world, the simple truth is that we do indeed have a lot to lose and there is nothing that stops us from losing it if we make wrong decisions. Collecting several samples even from the edge of BI and seeing them come back negative is not a good sign for BI ever being the rational choice.

jerf 9y ago

"Not if we're currently locally optimized. Then it would seem worse before it seemed better." I think you don't quite have the point I'm trying to make, which is not surprising as it's the first time I'm trying to express it clearly. As I said, I'm sure there are interventions that are locally worse but globally better. Their existence is not the problem. But given the much, much higher prevalence of solutions which are locally worse, and then also globally worse, and the observable fact that many people believe these are good ideas anyhow (simply look across the political isle; everybody will disagree on what these ideas are but everybody should agree they exist), the challenge becomes one of identifying which are actually globally better even though they are locally worse. And I submit that there is no solution to that. At least not for humans. Maybe for Vulcans, but not for humans. (Anyone who wants to disagree with that, please double-check that your argument doesn't amount to "It would work if everybody would just change their opinions to agree with me.", which I pretty much guarantee it will in the end.) So our only rational alternative is to pursue solutions that are locally better, then to try to scale them up (again, keeping an eye out for if and when they turn negative, because they can), because there is no mechanism for discovering interventions which are locally worse but globally better. If the local experiments being done on BI do not have sufficient local success that all involved want to scale it up, we have no mechanism for rationally deciding "Well, it's not working so great on the small scale, but it'll totally work on the large scale." We could only irrationally leap to it, and we would rationally expect to suffer the usual consequences of such irrational leaps. (We are not scientifically or rationally entitled to simply assume BI will work in any sense.) The other thing I'm trying to point out is that even if none of the interventions to date are "true BI", they are micro-scale BI, and if they aren't working there, it is rational to start considering that evidence against BI. BI also can not work if it must be run "just so". If it is only valid at a particular point in the policy space, it will fail. If it is going to work, it must be robust to perturbation. Policies that work in the real world have to be robust against exceedingly intelligent adversaries, and for all everyone may complain with varying degrees of justification about the current state of the world, the simple truth is that we do indeed have a lot to lose and there is nothing that stops us from losing it if we make wrong decisions. Collecting several samples even from the edge of BI and seeing them come back negative is not a good sign for BI ever being the rational choice.