5 ms·

It's an interesting read, but perhaps it only succeeds in pointing out the gaps in our knowledge. When I read their counterargument to the possibility of intel

It's an interesting read, but perhaps it only succeeds in pointing out the gaps in our knowledge. When I read their counterargument to the possibility of intelligence explosion: "Positive feedback loops are common in the world, and very rarely move fast enough and far enough to become a dominant dynamic in the world." the idea that immediately comes to mind is the Harmless Supernova Fallacy, described on this (obnoxiously JavaScript-dependent) site: https://arbital.com/p/harmless_supernova/ https://arbital.com/p/harmless_supernova/ Knowledge of this fallacy is a mental tool I have found quite useful, as it seems to be a type of fallacy that is easy to make by accident. To be fair, the reasoning in the article may not quite reach the level of a fallacy, but the intelligence explosion section ends saying effectively this: "we think the intelligence explosion argument could be strong if strong reason is found to expect an unusually fast and persistent feedback loop [i.e. an intelligence explosion]" which sounds like a classic case of Begging the Question: https://en.wikipedia.org/wiki/Begging_the_question https://en.wikipedia.org/wiki/Begging_the_question

jwellt 9y ago
I think the reasoning in the arguments is left a little loose on purpose simply because it is so difficult make strong arguments about something we know so little about.
- s1dechnl 9y ago
  It is difficult to make arguments in something that has no grounding. I'd expect someone who claims to have a valid argument for an [intelligence explosion] event to have formal education and industry experience designing computational systems such that they could clearly define how exactly it could occur. I have yet to see such an individual w/ such a viewpoint. Instead, I see a ridiculous argument being forwarded by people who are least informed/experienced so as to push fear, uncertainty, doubt either for profit/attention or because it fulfills some sci-fi oriented religious prophecy. Internally, companies require extensive and well reasoned documentation before funding an initiative. Externally, someone w/ no expertise/proof throws their hands up in the air speaking about armageddon and they are able to secure considerable attention and money.
  - red75prime 9y ago
    What is your argument? Why a group of engineers who can think two times faster than average cannot perform the task of designing three times faster engineer quicker than a group of average engineers? They will hit economical and physical limits eventually, sure. But what will stop them at the beginning?
    - pas 9y ago
      I guess the argument is, that so far no one showed good examples of how would that lead to a total extinction. How would it start, etc. So it's kind of a (fallacious) argument against "black box"-ing the whole intelligence explosion problem. If you can't define it, you can't analyze it sort of thing.
      - red75prime 9y ago
        By the same logic there could be no virus which will exterminate humanity, because there was no precedents and we don't know the structure of such virus. Right?
      - s1dechnl 9y ago
        What I stated was that there isn't even a sound/reasoned example as to how a it would escape its bounds unless purposely authored to and even then you run into the scenario of human limits (its creator). Also, having no understanding on how AGI is structured, it is quite foolish to talk about bound leaping in the traditional sense of tech which is mainly associated with computer viruses that are purposely written for that sole purpose. There's nothing to even suggest you can purposely author AGI in such a fashion. So... > How would it start, etc. There has been no credible framing on how it would even start. As such, it's a null point of discussion. > If you can't define it, you can't analyze it sort of thing. With no definition, you can 'attempt' to analyze it and you'll likely be horribly off the mark, wasting tons of resources, and likely produce something that has no bearing on the real thing. Instead of admitting this, it's like people put full faith in these efforts being sound when the reality is the exact opposite. Why engage in this, unless your aim is profit/notoriety, when you could be working on the actual problem? Define intelligence [first]. Time to be honest with oneself. Time to stop projecting ones shortcomings on others. Time to stop using fear/uncertainty/doubt to obscure one's true intent. Time to stop pushing the disinformation cloud. Time for TRUE intelligence.

jwellt 9y ago

I think the reasoning in the arguments is left a little loose on purpose simply because it is so difficult make strong arguments about something we know so little about.

s1dechnl 9y ago
It is difficult to make arguments in something that has no grounding. I'd expect someone who claims to have a valid argument for an [intelligence explosion] event to have formal education and industry experience designing computational systems such that they could clearly define how exactly it could occur. I have yet to see such an individual w/ such a viewpoint. Instead, I see a ridiculous argument being forwarded by people who are least informed/experienced so as to push fear, uncertainty, doubt either for profit/attention or because it fulfills some sci-fi oriented religious prophecy. Internally, companies require extensive and well reasoned documentation before funding an initiative. Externally, someone w/ no expertise/proof throws their hands up in the air speaking about armageddon and they are able to secure considerable attention and money.
- red75prime 9y ago
  What is your argument? Why a group of engineers who can think two times faster than average cannot perform the task of designing three times faster engineer quicker than a group of average engineers? They will hit economical and physical limits eventually, sure. But what will stop them at the beginning?
  - pas 9y ago
    I guess the argument is, that so far no one showed good examples of how would that lead to a total extinction. How would it start, etc. So it's kind of a (fallacious) argument against "black box"-ing the whole intelligence explosion problem. If you can't define it, you can't analyze it sort of thing.
    - red75prime 9y ago
      By the same logic there could be no virus which will exterminate humanity, because there was no precedents and we don't know the structure of such virus. Right?
    - s1dechnl 9y ago
      What I stated was that there isn't even a sound/reasoned example as to how a it would escape its bounds unless purposely authored to and even then you run into the scenario of human limits (its creator). Also, having no understanding on how AGI is structured, it is quite foolish to talk about bound leaping in the traditional sense of tech which is mainly associated with computer viruses that are purposely written for that sole purpose. There's nothing to even suggest you can purposely author AGI in such a fashion. So... > How would it start, etc. There has been no credible framing on how it would even start. As such, it's a null point of discussion. > If you can't define it, you can't analyze it sort of thing. With no definition, you can 'attempt' to analyze it and you'll likely be horribly off the mark, wasting tons of resources, and likely produce something that has no bearing on the real thing. Instead of admitting this, it's like people put full faith in these efforts being sound when the reality is the exact opposite. Why engage in this, unless your aim is profit/notoriety, when you could be working on the actual problem? Define intelligence [first]. Time to be honest with oneself. Time to stop projecting ones shortcomings on others. Time to stop using fear/uncertainty/doubt to obscure one's true intent. Time to stop pushing the disinformation cloud. Time for TRUE intelligence.