> Whether it is or not, the issue remains the same, i.e we are not the ones controlling the learning process I think this is just uninspected, vague intuition.

> Whether it is or not, the issue remains the same, i.e we are not the ones controlling the learning process I think this is just uninspected, vague intuition. What does it mean to control the learning process? No, we control the data, the deterministic learning rule, weights and activations, every step of the way is controllable – it's just it's intractable to control it all. Moreover, if it were tractable, would we know what to do? How would that be different from the problem of programming an AI from scratch? > There isn't really any solace in the other option (surprisingly human) in terms of alignment fears. If there weren't, I suppose major AI doomers (Yudkowsky, Leahy, Besinger etc.) wouldn't have been putting such an emphasis in their Lovecraft-inspired rhetoric on the alienness and inscrutability of those evil matrices of floating point numbers. No, the idea that human mental architecture is safer (and that DL does not approximate it) is very much at the core of alignment fears. E.g. Besinger on why he expects AGI Ruin [1]: "We're building "AI" in the sense of building powerful general search processes (and search processes for search processes), not building "AI" in the sense of building friendly ~humans but in silicon … The key differences between humans and "things that are more easily approximated as random search processes than as humans-plus-a-bit-of-noise" lies in lots of complicated machinery in the human brain. … This doesn't mean the problem is unsolvable; but it means that you either need to reproduce that internal machinery, in a lot of detail, in AI, or you need to build some new kind of machinery that’s safe for reasons other than the specific reasons humans are safe." > Let me put it this way. Unaligned intelligence is dangerous. I would ask you not to condescend but I have learned that this is an impossible request with the AI risk crowd, because you never really encounter pushback in your para-academic hothouse and so come to believe that, indeed, pretty basic intuitions are knockdown arguments only silly people could dismiss. You define intelligence in a certain way that has nothing to do with how you identify it in the wild. The definition is something like "intelligence is an optimization process"; the thing under consideration can be an LLM or a diffusion model that seems really good at its job. You mean to say "processes shaping real-world outcomes that are not optimizing for outcomes people consider good are dangerous". This is true but trivial. The onus is on you to tie this consideration to intelligence in general or to "capabilities" of arbitrarily powerful ML models. 1. https://www.lesswrong.com/posts/eaDCgdkbsfGqpWazi/the-basic-reasons-i-expect-agi-ruin https://www.lesswrong.com/posts/eaDCgdkbsfGqpWazi/the-basic-...