4 ms·

> I sincerely have no idea why people believe so strongly that a human-level AI can build a superhuman AI. ...how on earth would such an AI even know it succeed

by gary_0 2y ago

> I sincerely have no idea why people believe so strongly that a human-level AI can build a superhuman AI. ...how on earth would such an AI even know it succeeded? This touches on one of the few good reasons to be less ardent about AI/AGI: "intelligence" is not very well-defined and we don't have very good ways of measuring it. I don't think this is a total blocker, but it might present difficulties. What if our current approach ends up creating super-autism instead of super-intelligence? There's a long history (starting with Asimov) of drilling down into how vague things get when you start trying to draw clean lines around AI and its implications and those questions are yet to be definitively answered. However, your broader point seems to imply that you can't "bootstrap" intelligence, which I don't find convincing. Humans, after all, could barely master fire a few hundred thousand years ago, and now we have an understanding of the universe that the earliest humans were incapable of comprehending on even a basic level. It's obvious to me that simpler things are capable of building more complex things; blind evolution can do it, so there's nothing in physics preventing intelligence bootstrapping. We also have the ability to use intellectual division of labor to build tools that vastly enhance our abilities as a species. The human brain as hardware is far from some impassable apex; hardware can always be used to build better hardware, much like the earliest CPUs were themselves used to design better CPUs.

AnimalMuppet 2y ago
> However, your broader point seems to imply that you can't "bootstrap" intelligence, which I don't find convincing. How about this weaker statement: "It is not obviously true that humans (or a human-level AI) can bootstrap to a superhuman AI in a small number of years."
- gary_0 2y ago
  That one's a definite possibility, I think. I do think GenAI will prove to be very useful, possibly even world-changing (for better or worse), and the current frenzy of investment and research will probably turn up other useful ANN techniques (eventually). But the success of LLMs is not the Final Portent before the Singularity Arrives. I can think of quite a few major lines of research[0] that might be required before we can achieve superintelligence, and it's far from a given they'll succeed (or even get off the ground) any time soon. Especially if the economy loses the appetite for throwing billions of dollars into the AI furnace. [0] My pet theory is that embodiment might be required rather than solely relying on mostly language-based training data. A general intelligence might need to learn via interaction (initially; then you can just copy-paste the weights), because language is merely the hearsay of actual reality. Also, using attention-based "hacks" for easy parallelism to avoid needing exaflops that our hardware doesn't have might also be an issue (that's just a guess though, as I'm no AI expert).
- ConspiracyFact 2y ago
  This. It’s taken the universe (at least locally) several billion years to achieve human-level intelligence through trial and error. Somehow switching over to silicon is going to get us to superhuman intelligence and beyond within a decade? Why should the intelligence of a system surpass its complexity (if you’ll excuse my playing fast and loose with units)? For all we know we’ll achieve barely-superhuman AI and when we ask it why it doesn’t just make itself exponentially smarter, it will say, “Do you have any idea how complex I am?”

by gary_0 2y ago

AnimalMuppet 2y ago

> However, your broader point seems to imply that you can't "bootstrap" intelligence, which I don't find convincing. How about this weaker statement: "It is not obviously true that humans (or a human-level AI) can bootstrap to a superhuman AI in a small number of years."

gary_0 2y ago
That one's a definite possibility, I think. I do think GenAI will prove to be very useful, possibly even world-changing (for better or worse), and the current frenzy of investment and research will probably turn up other useful ANN techniques (eventually). But the success of LLMs is not the Final Portent before the Singularity Arrives. I can think of quite a few major lines of research[0] that might be required before we can achieve superintelligence, and it's far from a given they'll succeed (or even get off the ground) any time soon. Especially if the economy loses the appetite for throwing billions of dollars into the AI furnace. [0] My pet theory is that embodiment might be required rather than solely relying on mostly language-based training data. A general intelligence might need to learn via interaction (initially; then you can just copy-paste the weights), because language is merely the hearsay of actual reality. Also, using attention-based "hacks" for easy parallelism to avoid needing exaflops that our hardware doesn't have might also be an issue (that's just a guess though, as I'm no AI expert).
ConspiracyFact 2y ago
This. It’s taken the universe (at least locally) several billion years to achieve human-level intelligence through trial and error. Somehow switching over to silicon is going to get us to superhuman intelligence and beyond within a decade? Why should the intelligence of a system surpass its complexity (if you’ll excuse my playing fast and loose with units)? For all we know we’ll achieve barely-superhuman AI and when we ask it why it doesn’t just make itself exponentially smarter, it will say, “Do you have any idea how complex I am?”