2 ms·

To me, the big dubious part is, how do we know that increasing the capacity of a reasoning machine can always increase the quality of its output by an equal fac

To me, the big dubious part is, how do we know that increasing the capacity of a reasoning machine can always increase the quality of its output by an equal factor, even when its task is 'designing the next iteration of a reasoning machine'? It may very well be that the task of 'designing a reasoning machine' has sharply diminishing returns when you throw more resources at it. (E.g., at least so far, any LLM can only have a very incidental impact on designing the next LLM, it's not going to come up with groundbreaking new approaches.) Yudkowsky builds the whole edifice on top of his very particular conception of intelligence, which insists upon itself, but which I think is far from the only explanation of its nature as observed in humans, other animals, LLMs maybe, and so on.

bbor 2y ago
Well put, and I agree with your general approach/skepticism! That said; 1. I don’t think we need to meet “always can improve itself”; rather, the contention is that there’s a high chance that there’s lots of improvement left to be had. An empirical claim rather than a theoretical one, in other words. The simple fact that humans are evolved creatures backs this up in spades, IMO — we’re still improving our own cognition by leaps and bounds using institutions, tools, and methods, and I don’t see any reason why that same dynamic wouldn’t apply to artificial cognitive systems. 2. I think dodging “intelligence” is exactly what he’s trying to do by listing concrete behavioral/cognitive differences. “Intelligence” is pretty much a useless term in science IMO, as was best expounded by Turing in his seminal 1950 paper, Computing Machinery and Intelligence: https://courses.cs.umbc.edu/471/papers/turing.pdf https://courses.cs.umbc.edu/471/papers/turing.pdf People remember that paper as “you can tell a real AI when it can trick you”, but that’s not what he was trying to say at all; rather, he was trying to highlight that there is no such thing as a “real” AI, or “real” thinking, or “real” intelligence — just behavioral similarities and dissimilarities. If the second bit grabs you/anyone, definitely watch some Chomsky lectures on YouTube about cognition. He centers his analysis on this, pejoratively calling discussions about “person”, “intelligence”, “thinking”, etc. mere terminological disputes, not specific enough to have much scientific value. His old refrain is a great one: does an airplane fly? Does a submarine swim? Kinda, if you want!

bbor 2y ago

Well put, and I agree with your general approach/skepticism! That said; 1. I don’t think we need to meet “always can improve itself”; rather, the contention is that there’s a high chance that there’s lots of improvement left to be had. An empirical claim rather than a theoretical one, in other words. The simple fact that humans are evolved creatures backs this up in spades, IMO — we’re still improving our own cognition by leaps and bounds using institutions, tools, and methods, and I don’t see any reason why that same dynamic wouldn’t apply to artificial cognitive systems. 2. I think dodging “intelligence” is exactly what he’s trying to do by listing concrete behavioral/cognitive differences. “Intelligence” is pretty much a useless term in science IMO, as was best expounded by Turing in his seminal 1950 paper, Computing Machinery and Intelligence: https://courses.cs.umbc.edu/471/papers/turing.pdf https://courses.cs.umbc.edu/471/papers/turing.pdf People remember that paper as “you can tell a real AI when it can trick you”, but that’s not what he was trying to say at all; rather, he was trying to highlight that there is no such thing as a “real” AI, or “real” thinking, or “real” intelligence — just behavioral similarities and dissimilarities. If the second bit grabs you/anyone, definitely watch some Chomsky lectures on YouTube about cognition. He centers his analysis on this, pejoratively calling discussions about “person”, “intelligence”, “thinking”, etc. mere terminological disputes, not specific enough to have much scientific value. His old refrain is a great one: does an airplane fly? Does a submarine swim? Kinda, if you want!