3 ms·

I'm honestly offended that you attribute my criticism to some failure of understanding, and then your appeal to authority. Appealing to authority suggests you h

by calf 1y ago

I'm honestly offended that you attribute my criticism to some failure of understanding, and then your appeal to authority. Appealing to authority suggests you have the same bias as "top researchers" school of thought. Ad hominems like those make it impossible for me to respond to your comment fairly. If you want to lead with ad hominems then I studied EECS at elite American universities for undergrad then for my PhD, so I have every right to criticize, within reason, the more prominent leaders of entire field for being philosophically blinkered. The short of it is that your "making the AI want what you want" is called extrinsic internalization. It is highly problematic. It is not surprising that a roomful of CS technocratic researchers (beholden to SV funding) would make this sort of lay psychology mistake--a smuggled premise. It is bad science, and irrational. Their vested interests make the bias seem rational to them. And then people in tech, like yourself, drink their Koolaid. So what's more likely is your knowledge is ignorant, you resort to comparisons about human genetic evolution to make handwavy claims about how important this historic moment will be, which is irrelevant to the validity of alignment as a framing/methodology. In fact a cursory check shows CS AI papers written by dissenting scientists who are very much against the alignment framework. Did you even know that? Hope you don't so hastily accuse them of "not understanding" the issues as you did here.

ben_w 1y ago
> The short of it is that your "making the AI want what you want" is called extrinsic internalization. It is highly problematic. It is not surprising that a roomful of CS technocratic researchers (beholden to SV funding) would make this sort of lay psychology mistake--a smuggled premise. It is bad science, and irrational. Problematic or not, the technical term is "reward function", and you don't get to not have one in an AI, not even when it's e.g. AlphaGo. The hard part is specifying the reward function to match what you actually want, otherwise you get Goodhart's Law. Talking about this in psych terms is a "wet pavements cause rain" error. > If you want to lead with ad hominems then I studied EECS at Berkeley then for my PhD at Princeton, so I have every right to criticize the more prominent leaders of entire field for being philosophically blinkered. The right to, sure. I'm not stopping you from doing it. I'm saying you're making a foundational error in your criticism. > In fact a cursory check shows CS AI papers written by dissenting scientists who are very much against the alignment framework. Naturally. Dissent is normal. As for details of which matters and how hard they might be: inner alignment, outer alignment, Goodharting, specification gaming, principal-agent issues, anything Yudkowsky has ever said about it being impossible because we only get one chance and we're bad at this, etc. But your dismissal, as you wrote it in this thread, is as shallow as you call mine. (Not that I expect you to read this given the other comment, but if you name a scifi setting famously anti-AI, from whence the term "Butlerian Jihad" entered our culture, you should expect to be called out on that. And TBH your replacement wasn't any good either: not only is Measure Of A Man more about the question "Is this AI a person at all?", but fictional references are not evidence of anything beyond the culture in which they were created).

by calf 1y ago

ben_w 1y ago

> The short of it is that your "making the AI want what you want" is called extrinsic internalization. It is highly problematic. It is not surprising that a roomful of CS technocratic researchers (beholden to SV funding) would make this sort of lay psychology mistake--a smuggled premise. It is bad science, and irrational. Problematic or not, the technical term is "reward function", and you don't get to not have one in an AI, not even when it's e.g. AlphaGo. The hard part is specifying the reward function to match what you actually want, otherwise you get Goodhart's Law. Talking about this in psych terms is a "wet pavements cause rain" error. > If you want to lead with ad hominems then I studied EECS at Berkeley then for my PhD at Princeton, so I have every right to criticize the more prominent leaders of entire field for being philosophically blinkered. The right to, sure. I'm not stopping you from doing it. I'm saying you're making a foundational error in your criticism. > In fact a cursory check shows CS AI papers written by dissenting scientists who are very much against the alignment framework. Naturally. Dissent is normal. As for details of which matters and how hard they might be: inner alignment, outer alignment, Goodharting, specification gaming, principal-agent issues, anything Yudkowsky has ever said about it being impossible because we only get one chance and we're bad at this, etc. But your dismissal, as you wrote it in this thread, is as shallow as you call mine. (Not that I expect you to read this given the other comment, but if you name a scifi setting famously anti-AI, from whence the term "Butlerian Jihad" entered our culture, you should expect to be called out on that. And TBH your replacement wasn't any good either: not only is Measure Of A Man more about the question "Is this AI a person at all?", but fictional references are not evidence of anything beyond the culture in which they were created).