5 ms·

Long, and fascinating. I can see why this made the front page. I oscillated between 20% strident disagreement and 60% strident agreement. (The rest -- no strong

Long, and fascinating. I can see why this made the front page. I oscillated between 20% strident disagreement and 60% strident agreement. (The rest -- no strong feelings, or I feel that the questions are too ill-formed to answer.) To pick one point of disagreement, “We do not need as much computational power as the brain has, because our algorithms are (will be) better than that of the brain.” I hope you can see after the descriptions in this blog post that this statement is rather arrogant. Machines are already better than brains at many cognitive tasks of practical interest. Believing that we'll continue to find "tricks" to allow computers to outperform brains on useful cognitive tasks, despite the brains' much greater complexity, seems like a perfectly sober and conservative prediction. If I had to advance my own pet reasons for discounting the likelihood of a technological singularity, here are my top two: 1) It's a more challenging case of the general Fermi paradox. Show me the Hubble images of the computronium Dyson swarms. If it takes less than a century to go from the first transistorized computers to superintelligence, and superintelligence is as prone to run amok as Bostrom/Yudkowsky think, signs should already be visible from Earth. 2) You need experiments to validate scientific models. Even if a machine-intelligence could think a billion times faster than a biological intelligence, it couldn't complete experiments a billion times faster. Technologies that act on the material world will improve sublinearly with respect to thinking/computing power, for at least this reason and probably others as well.

deleted 9y ago
[deleted]
darkmighty 9y ago
I agree with you. Our current results are impressive enough for me to believe we're not that far from AGI (of the kind that would satisfy the author here). This article is already largely outdated because of advances in Reinforcement Learning and other Deep Learning applications that show very strong primordial features of what he claims as epitomes of human capability. For example, trained RL agents will recognize objects without being able to label them, having internal representations for position, object "type", functionality and relevance. You just have to train the agent with a problem that requires this kind of comprehension for efficient performance (kind of comprehension which isn't needed in simple image labeling). AlphaGo and even labeling are indeed milestones in superhuman performance, and I believe Terence Tao's view that AI is a moving target is relevant here. --- 1) Regarding the Fermi paradox, I haven't read too much technical or scientific work on it (it's highly speculative anyway), but I find Isaac Arthur's videos lay the basic ideas pretty well: https://www.youtube.com/watch?v=oIva_60l3ww&t=1016s https://www.youtube.com/watch?v=oIva_60l3ww&t=1016s It becomes pretty convincing that technological intelligent life is an incredible coincidence. There might not be any huge "filters", as they are called (stages which reduce the probability of observing TIL), such as a superintelligence extinction event, but an enormous sequence of minor filters, ranging from low probability of a planet with adequate conditions, to low probability of actual technological development, to low probability of occurrence in our light cone (we can only see fairly young distant galaxies), and limitations to the visibility and spreading velocity of such civilizations. There is a wide range of parameters such that this does not contradict the generalized Mediocrity principle, such that there is probably more than one TIL in the universe, but they're few and far from each other in space and time. 2) I don't follow. This objection is only valid for discovering new laws of physics -- technological development can happen even with fixed knowledge of basic physical laws. At this stage it's not clear that even continued refinement of physical models. For example, the discovery of quarks certainly helped understanding nature, but it hasn't led to any direct technological applications due to quark confinement, and the fact that the particles are just too small. Neutrinos are another conceptually important discovery that doesn't really have applications due to low interactivity with matter.
- philipkglass 9y ago
  This objection is only valid for discovering new laws of physics -- technological development can happen even with fixed knowledge of basic physical laws. No, experimentation is also important for translating fundamental scientific knowledge into working technologies. Consider the problem of building a fusion reactor that generates net electricity. You don't need new laws of physics, but you can't make progress with simulations alone. You have to experiment to find out if devices behave in reality the way they do in your model.
  - darkmighty 9y ago
    Well as simulations improve that tends to be less of a problem. Engineering by now is highly reliant on simulations, building actual prototypes is only done as a last validating step as you mentioned (specially for something as expensive and complicated as a fusion reactor). So validation devices are a O(1) step in the innovation chain, I don't see a scaling issue here. You can also simply make arbitrarily many experimentation/testing devices. I think a greater issue is the general misconception of intelligence as something of a magical attribute, and even more so "superintelligence", and even more so "recursive-self-improving superintelligence" (RSIS for short?). There are not only various limits to hardware, some of which discussed in the article (size of atoms, capacity of communication between parts), but also there are limits to software. For example, while it seems that an RSIS could really solve any problem it wanted, Turing's almost century-old Halting Problem solution already proved that no such computer exists. There's no algorithm that takes an arbitrary conjecture, e.g. the Riemann hypothesis, and outputs either {yes,no,malformed_problem}. In fact the general expectation of what AGIs can do (think creatively, solve arbitrary problems, replace mathematicians, replace programmers) are things that we ourselves can't really do. We solve problems using a small set of heuristics and lots of trial and error, with no success guarantee. Another fact is that there are many asymptotically optimal algorithms that are already know, and the trivial fact that almost all tasks performed by this super-intelligence (many of which are ordinary tasks we already do, analogous to maybe sorting, database queries, and search engines) will just improve constants on near-optimal algorithms. A god-like creature those AIs will be not.
visarga 9y ago
>> “We do not need as much computational power as the brain has, because our algorithms are (will be) better than that of the brain.” > I hope you can see after the descriptions in this blog post that this statement is rather arrogant. The brain does more than computers: the brain builds its internal structure all by itself. When was a computer able to evolve from a bunch of transistors laying on the table? Also, it uses very little energy and is resilient for 80-100 years, compared to the computers of today, its apples and oranges. > Show me the Hubble images of the computronium Dyson swarms. If it takes less than a century to go from the first transistorized computers to superintelligence, and superintelligence is as prone to run amok as Bostrom/Yudkowsky think, signs should already be visible from Earth. What if the AGI will prefer to build virtual worlds and societies of virtual agents instead of grand space domination ? Even humans prefer games to reality nowadays (or a large percentage of us do). If we're part of such a sim, then it would explain the lack of external signals from extraterrestrial aliens or AGIs. The probability that AGI will appear and create amazing simulations is much larger than that of picking up signs of life in the vastness of space. Also, by replacing physical with sim we can do all sorts of things such as use less energy, recover from any accident, hack our own brains/minds, even immortality.
- icc97 9y ago
  The $24M per year (23MW) for the electricity for the Tianhe-2 was quite shocking. This is against the 13W that the brain uses: https://www.scientificamerican.com/article/thinking-hard-calories/ https://www.scientificamerican.com/article/thinking-hard-cal... So the brain has a million times more the processing power and uses a million times less power.
  - JacobiX 9y ago
    >> So the brain has a million times more the processing power and uses a million times less power. A commodity smartphone is millions of times more powerful than all of NASA's combined computing in 1970 and more energy efficient. A factor of 10^6 is maybe achievable with a specialized quantum computer in the mid-term future ?
quickthrower2 9y ago
The super intelligence could be like the Q from Star Trek. I.e. you can only see them when they choose to be seen.
- TeMPOraL 9y ago
  Still, unless that superintelligence is killing all others (at which point we're done), we _should_ be able to spot the ones on their way to Q-hood.
  - quickthrower2 9y ago
    There is a lot of space. Its hard enough to spot colliding black holes. And they're kinda big.

deleted 9y ago

[deleted]

darkmighty 9y ago

I agree with you. Our current results are impressive enough for me to believe we're not that far from AGI (of the kind that would satisfy the author here). This article is already largely outdated because of advances in Reinforcement Learning and other Deep Learning applications that show very strong primordial features of what he claims as epitomes of human capability. For example, trained RL agents will recognize objects without being able to label them, having internal representations for position, object "type", functionality and relevance. You just have to train the agent with a problem that requires this kind of comprehension for efficient performance (kind of comprehension which isn't needed in simple image labeling). AlphaGo and even labeling are indeed milestones in superhuman performance, and I believe Terence Tao's view that AI is a moving target is relevant here. --- 1) Regarding the Fermi paradox, I haven't read too much technical or scientific work on it (it's highly speculative anyway), but I find Isaac Arthur's videos lay the basic ideas pretty well: https://www.youtube.com/watch?v=oIva_60l3ww&t=1016s https://www.youtube.com/watch?v=oIva_60l3ww&t=1016s It becomes pretty convincing that technological intelligent life is an incredible coincidence. There might not be any huge "filters", as they are called (stages which reduce the probability of observing TIL), such as a superintelligence extinction event, but an enormous sequence of minor filters, ranging from low probability of a planet with adequate conditions, to low probability of actual technological development, to low probability of occurrence in our light cone (we can only see fairly young distant galaxies), and limitations to the visibility and spreading velocity of such civilizations. There is a wide range of parameters such that this does not contradict the generalized Mediocrity principle, such that there is probably more than one TIL in the universe, but they're few and far from each other in space and time. 2) I don't follow. This objection is only valid for discovering new laws of physics -- technological development can happen even with fixed knowledge of basic physical laws. At this stage it's not clear that even continued refinement of physical models. For example, the discovery of quarks certainly helped understanding nature, but it hasn't led to any direct technological applications due to quark confinement, and the fact that the particles are just too small. Neutrinos are another conceptually important discovery that doesn't really have applications due to low interactivity with matter.

philipkglass 9y ago
This objection is only valid for discovering new laws of physics -- technological development can happen even with fixed knowledge of basic physical laws. No, experimentation is also important for translating fundamental scientific knowledge into working technologies. Consider the problem of building a fusion reactor that generates net electricity. You don't need new laws of physics, but you can't make progress with simulations alone. You have to experiment to find out if devices behave in reality the way they do in your model.
- darkmighty 9y ago
  Well as simulations improve that tends to be less of a problem. Engineering by now is highly reliant on simulations, building actual prototypes is only done as a last validating step as you mentioned (specially for something as expensive and complicated as a fusion reactor). So validation devices are a O(1) step in the innovation chain, I don't see a scaling issue here. You can also simply make arbitrarily many experimentation/testing devices. I think a greater issue is the general misconception of intelligence as something of a magical attribute, and even more so "superintelligence", and even more so "recursive-self-improving superintelligence" (RSIS for short?). There are not only various limits to hardware, some of which discussed in the article (size of atoms, capacity of communication between parts), but also there are limits to software. For example, while it seems that an RSIS could really solve any problem it wanted, Turing's almost century-old Halting Problem solution already proved that no such computer exists. There's no algorithm that takes an arbitrary conjecture, e.g. the Riemann hypothesis, and outputs either {yes,no,malformed_problem}. In fact the general expectation of what AGIs can do (think creatively, solve arbitrary problems, replace mathematicians, replace programmers) are things that we ourselves can't really do. We solve problems using a small set of heuristics and lots of trial and error, with no success guarantee. Another fact is that there are many asymptotically optimal algorithms that are already know, and the trivial fact that almost all tasks performed by this super-intelligence (many of which are ordinary tasks we already do, analogous to maybe sorting, database queries, and search engines) will just improve constants on near-optimal algorithms. A god-like creature those AIs will be not.

visarga 9y ago

>> “We do not need as much computational power as the brain has, because our algorithms are (will be) better than that of the brain.” > I hope you can see after the descriptions in this blog post that this statement is rather arrogant. The brain does more than computers: the brain builds its internal structure all by itself. When was a computer able to evolve from a bunch of transistors laying on the table? Also, it uses very little energy and is resilient for 80-100 years, compared to the computers of today, its apples and oranges. > Show me the Hubble images of the computronium Dyson swarms. If it takes less than a century to go from the first transistorized computers to superintelligence, and superintelligence is as prone to run amok as Bostrom/Yudkowsky think, signs should already be visible from Earth. What if the AGI will prefer to build virtual worlds and societies of virtual agents instead of grand space domination ? Even humans prefer games to reality nowadays (or a large percentage of us do). If we're part of such a sim, then it would explain the lack of external signals from extraterrestrial aliens or AGIs. The probability that AGI will appear and create amazing simulations is much larger than that of picking up signs of life in the vastness of space. Also, by replacing physical with sim we can do all sorts of things such as use less energy, recover from any accident, hack our own brains/minds, even immortality.

icc97 9y ago
The $24M per year (23MW) for the electricity for the Tianhe-2 was quite shocking. This is against the 13W that the brain uses: https://www.scientificamerican.com/article/thinking-hard-calories/ https://www.scientificamerican.com/article/thinking-hard-cal... So the brain has a million times more the processing power and uses a million times less power.
- JacobiX 9y ago
  >> So the brain has a million times more the processing power and uses a million times less power. A commodity smartphone is millions of times more powerful than all of NASA's combined computing in 1970 and more energy efficient. A factor of 10^6 is maybe achievable with a specialized quantum computer in the mid-term future ?

quickthrower2 9y ago

The super intelligence could be like the Q from Star Trek. I.e. you can only see them when they choose to be seen.

TeMPOraL 9y ago
Still, unless that superintelligence is killing all others (at which point we're done), we _should_ be able to spot the ones on their way to Q-hood.
- quickthrower2 9y ago
  There is a lot of space. Its hard enough to spot colliding black holes. And they're kinda big.