9 ms·

"The Dark Forest" is a science fiction novel by Liu Cixin, which introduces a concept called "suspicion."(“猜疑链”) In this concept, the interaction between civili

"The Dark Forest" is a science fiction novel by Liu Cixin, which introduces a concept called "suspicion."(“猜疑链”) In this concept, the interaction between civilizations is viewed as an ongoing game of suspicion, where each civilization is worried that the other civilization may pose a threat to it. This kind of suspicion makes it difficult for civilizations to establish trust, thereby making the universe a dark forest full of war and suspicion. We can apply this concept of suspicion to the phenomenon in the field of modern artificial intelligence research. In AI research, some researchers believe that humans should pause the development of AI technology because of the potential unforeseeable risks it may bring. However, another group of researchers believe that we should not stop because we cannot be certain whether those who are trying to catch up with us will use the advancement of AI technology as a hidden weapon. This suspicion leads to a chain reaction, where people continue to worry about each other's motives and abilities, making it difficult for all parties to establish trust. Therefore, although there are good intentions, such proposals are unlikely to be implemented.

hotpotamus 4y ago
Deleted due to spoiler.
- deleted 4y ago
  [deleted]
kleer001 4y ago
Hmm, sounds like parties not fully grokking the optimal (not thinly logical) solution to the iterated prisoners' dilemma. I saw the same thought process is the otherwise very nice novel "Time's Children". Good for drama, but not thought through.
- mclightning 4y ago
  I have so many questions >> what does it mean to fully grok the optimal? >> what is the iterated prisoners dilemma?
  - Teever 4y ago
    grok means to understand something fully. https://en.wikipedia.org/wiki/Prisoner's_dilemma#The_iterated_prisoner's_dilemma https://en.wikipedia.org/wiki/Prisoner's_dilemma#The_iterate...
    - generalizations 4y ago
      As coined by Heinlein in A Stranger in a Strange World.
  - kleer001 4y ago
    something in there should be grokable best imho, 10 min and done: https://www.youtube.com/watch?v=BOvAbjfJ0x0 https://www.youtube.com/watch?v=BOvAbjfJ0x0 longer and more over produced: https://radiolab.org/episodes/104010-one-good-deed-deserves-another https://radiolab.org/episodes/104010-one-good-deed-deserves-... totally academic : https://onlinelibrary.wiley.com/doi/10.1111/j.1420-9101.2011.02307.x https://onlinelibrary.wiley.com/doi/10.1111/j.1420-9101.2011...
- olddustytrail 4y ago
  If your scenario involves all parties "fully grokking the optimal solution to the iterated prisoners' dilemma" I suspect it's yourself who has not thought things through. It only takes one.
  - pmoriarty 4y ago
    Please elaborate
    - guerrilla 4y ago
      It only takes one player to not fully grok the optimal situation and thus defect.
  - kleer001 4y ago
    No, both parties need to cooperate.
    - wrycoder 4y ago
      If they are playing only one game, the optimum strategy is “defect”. In humans v. AI, there may be only one game, if we don’t figure this out.
echelon 4y ago
Except the concern isn't genuine here. Some of the signatories have AI companies and interests. They want to slow the leaders down so that they can catch up. It's disingenuous.
- mikepalmer 4y ago
  Your idea is also voiced in Forbes magazine. https://www.forbes.com/sites/mattnovak/2023/03/29/the-real-reason-elon-musk-wants-to-pause-ai-development/amp/ https://www.forbes.com/sites/mattnovak/2023/03/29/the-real-r... Search "Graham's hierarchy of disagreement" which is popular on HN. Your current criticism is at the "ad hominem" level. It is entirely beside the point what Elon Musk's motivation is. The question is 1) if the risks referenced in the letter are real, which they certainly are. There has been informed thought on this for more than a decade. Recent LLMs have made the dangers even more clear. 2) is the letter's proposal of a 6 month pause useful. It's the most useful and plausible step I can think of. We need to take stock. It's not up to a handful of researchers to endanger the lives of 6B other people. 3) Is it actually plausible that we could get such a pause. We'll see. I doubt it but also search "death with dignity" by Yudkowsky. Back to Musk as an "authority", the headline is about him but more than 1000 other concerned parties have signed. I will sign. Their signature vetting process is stalled because to the volume of signatures, otherwise there would be many more.
  - echelon 4y ago
    > Your current criticism is at the "ad hominem" level. Allow me to rephrase. I am deeply concerned that there is a possibility that some of the powerful parties backing this may be trying to catch up and using an enforced or agreed to "ceasefire" as a means to catch up in capability. I also worry that some may be able to use political strong arming to accomplish this as a means of unfair competition. > It is entirely beside the point what Elon Musk's motivation is. Is it always beside the point what anyone's motivation is? Motivation matters. > if the risks referenced in the letter are real, which they certainly are. Your opinion. > There has been informed thought on this for more than a decade. Recent LLMs have made the dangers even more clear. > 2) is the letter's proposal of a 6 month pause useful. More opinions. > It's the most useful and plausible step I can think of. We need to take stock. It's not up to a handful of researchers to endanger the lives of 6B other people. 3) Is it actually plausible that we could get such a pause. We'll see. I doubt it but also search "death with dignity" by Yudkowsky. All of this is your opinion. > Back to Musk as an "authority", the headline is about him but more than 1000 other concerned parties have signed. I didn't even mention Musk. I have several other names in mind. Lots of folks with AI companies (including LLMs!) that "may" be experiencing FOMO and sensing a strategy here. Maybe. Hypothetically. In a non-"ad hominem" way. > I will sign. Their signature vetting process is stalled because to the volume of signatures, otherwise there would be many more. People are starting counter petitions, which I'll gladly sign. The one by Suhail posted on Twitter was hilarious af. I'll also go to whatever country doesn't "pause", because this tech is literally the most exciting development of my lifetime. And I want to spend my life doing something that matters instead of gluing distributed systems together to process financial transactions.
JCharante 4y ago
Was this comment generated by a LLM?
- TacticalCoder 4y ago
  The second paragraph definitely was (the first one probably too). 100% crystal clear, it's not even a question. Downvoted.
- xwdv 4y ago
  Yes, it has some of the classic markers of an LLM. Downvoted.
croo 4y ago
Nitpick: dark forest is one possible explanation for the Fermi paradox and not coined in the book(though the book presented neatly). You can read a great article about fermi paradox here if anyone wants to jump into what the dark forest theory is: https://waitbutwhy.com/2014/05/fermi-paradox.html https://waitbutwhy.com/2014/05/fermi-paradox.html
pmoriarty 4y ago
"another group of researchers believe that we should not stop because we cannot be certain whether those who are trying to catch up with us will use the advancement of AI technology as a hidden weapon" It's not just a potential weapon, but a potential source of untold riches and power. That's going to be irresistible to anyone with an interest in either (ie. most politicians, governments, billionaires and corporations... and most ordinary people too, if they only had the means).
- biscottigelato 4y ago
  Its the one ring with the ultimate power. But so powerful that it might, if not almost certainly will, end us all. Perhaps this IS the solution to the Fermi Paradox.
thomastjeffery 4y ago
Suspicion is inference. It fails to provide enough information (on its own) to decide whether something is threatening or desirable. LLMs are inference models.
lanstin 4y ago
This book is the scariest thing I have ever read.
wayfwdmachine 4y ago
Yes. And it's fiction. We might as well discuss Excession, Agency, Diaspora (Banks, Gibson and Egan respectively) all of whom have more interesting and insightful things to say about humanity, game theory and ... seriously, as a life long (and I am OLD) science fiction fan, the fawning over The Remembrance of Earth's past is just silly. The concept of "suspicion". Might I suggest that people read Thomas Schelling's seminal work The Strategy of Conflict to begin with since if you want to understand concepts like brinksmanship and the push and pull of actual inter-civilization (or society or nation state) conflicts then you might want to start there and not with fiction riddled with silly concepts, weird misogyny and stuff that seems to be right out of the discarded parts of a Gregory Benford trilogy. I mean seriously. But ok. Let's dig into this "We can apply this concept of suspicion to the phenomenon in the field of modern artificial intelligence research. In AI research, some researchers believe that humans should pause the development of AI technology because of the potential unforeseeable risks it may bring." An analogy would be that some of the scientists involved with the Manhattan project (see Szilárd petition) attempted to file an official protest against the use of nuclear weapons against civilian population and arguing that the weapons constructed were too dangerous to be a part of an arms race and should be placed under an international control body. "However, another group of researchers believe that we should not stop because we cannot be certain whether those who are trying to catch up with us will use the advancement of AI technology as a hidden weapon." Researchers are not decision makers in this or any other scenario. AI technology as a weapon? Sure but unless you are Bombs Away LeMay circa 1961 this really isn't a winnable or even useful use case. We're still in a blue sky phase. The fun thing about nuclear weapons is that if any nation feels that another is pulling too far ahead they can be used as a way to push for bi/multilateral discussions and eventually treaties limiting the production and/or deployment of weapons that risk of increasing tensions/lethality and so on . (see SALT I/II and START etc) "This suspicion leads to a chain reaction, where people continue to worry about each other's motives and abilities, making it difficult for all parties to establish trust. Therefore, although there are good intentions, such proposals are unlikely to be implemented." Again... read Schelling... this is just nonsense.
- CatWChainsaw 3y ago
  "The only thing stupider than thinking something will happen because it is depicted in science fiction is thinking something will not happen because it is depicted in science fiction." https://philosophybear.substack.com/p/position-statement-on-ai-and-leftwing https://philosophybear.substack.com/p/position-statement-on-...
DonHopkins 4y ago
That might explain why the Vorlons of Babylon 5 were such assholes. https://babylon5.fandom.com/wiki/Vorlon https://babylon5.fandom.com/wiki/Vorlon >Distant Past >The Vorlons were among the first races in the galaxy to attain sentience and become space-faring creatures. In the distant past, the Vorlons attempted a grand experiment into reaching a higher plane of existence where the souls existed and all life was founded. To accomplish this, they poured all their knowledge into creating a gateway to this dimension. After activating the gate, the vortex into Thirdspace was formed and the aliens within that realm came pouring out, intent on destroying all other forms of life. After much hardship, the Vorlons finally managed to push the enemy back, but one of the gateways was lost in Hyperspace where it was hidden from the Vorlons. They would encode some of their servants with an automated message to warn the younger races of the threat from the inhabitants of Thirdspace.

hotpotamus 4y ago

Deleted due to spoiler.

deleted 4y ago
[deleted]

kleer001 4y ago

Hmm, sounds like parties not fully grokking the optimal (not thinly logical) solution to the iterated prisoners' dilemma. I saw the same thought process is the otherwise very nice novel "Time's Children". Good for drama, but not thought through.

mclightning 4y ago
I have so many questions >> what does it mean to fully grok the optimal? >> what is the iterated prisoners dilemma?
- Teever 4y ago
  grok means to understand something fully. https://en.wikipedia.org/wiki/Prisoner's_dilemma#The_iterated_prisoner's_dilemma https://en.wikipedia.org/wiki/Prisoner's_dilemma#The_iterate...
  - generalizations 4y ago
    As coined by Heinlein in A Stranger in a Strange World.
- kleer001 4y ago
  something in there should be grokable best imho, 10 min and done: https://www.youtube.com/watch?v=BOvAbjfJ0x0 https://www.youtube.com/watch?v=BOvAbjfJ0x0 longer and more over produced: https://radiolab.org/episodes/104010-one-good-deed-deserves-another https://radiolab.org/episodes/104010-one-good-deed-deserves-... totally academic : https://onlinelibrary.wiley.com/doi/10.1111/j.1420-9101.2011.02307.x https://onlinelibrary.wiley.com/doi/10.1111/j.1420-9101.2011...
olddustytrail 4y ago
If your scenario involves all parties "fully grokking the optimal solution to the iterated prisoners' dilemma" I suspect it's yourself who has not thought things through. It only takes one.
- pmoriarty 4y ago
  Please elaborate
  - guerrilla 4y ago
    It only takes one player to not fully grok the optimal situation and thus defect.
- kleer001 4y ago
  No, both parties need to cooperate.
  - wrycoder 4y ago
    If they are playing only one game, the optimum strategy is “defect”. In humans v. AI, there may be only one game, if we don’t figure this out.

echelon 4y ago

Except the concern isn't genuine here. Some of the signatories have AI companies and interests. They want to slow the leaders down so that they can catch up. It's disingenuous.

mikepalmer 4y ago
Your idea is also voiced in Forbes magazine. https://www.forbes.com/sites/mattnovak/2023/03/29/the-real-reason-elon-musk-wants-to-pause-ai-development/amp/ https://www.forbes.com/sites/mattnovak/2023/03/29/the-real-r... Search "Graham's hierarchy of disagreement" which is popular on HN. Your current criticism is at the "ad hominem" level. It is entirely beside the point what Elon Musk's motivation is. The question is 1) if the risks referenced in the letter are real, which they certainly are. There has been informed thought on this for more than a decade. Recent LLMs have made the dangers even more clear. 2) is the letter's proposal of a 6 month pause useful. It's the most useful and plausible step I can think of. We need to take stock. It's not up to a handful of researchers to endanger the lives of 6B other people. 3) Is it actually plausible that we could get such a pause. We'll see. I doubt it but also search "death with dignity" by Yudkowsky. Back to Musk as an "authority", the headline is about him but more than 1000 other concerned parties have signed. I will sign. Their signature vetting process is stalled because to the volume of signatures, otherwise there would be many more.
- echelon 4y ago
  > Your current criticism is at the "ad hominem" level. Allow me to rephrase. I am deeply concerned that there is a possibility that some of the powerful parties backing this may be trying to catch up and using an enforced or agreed to "ceasefire" as a means to catch up in capability. I also worry that some may be able to use political strong arming to accomplish this as a means of unfair competition. > It is entirely beside the point what Elon Musk's motivation is. Is it always beside the point what anyone's motivation is? Motivation matters. > if the risks referenced in the letter are real, which they certainly are. Your opinion. > There has been informed thought on this for more than a decade. Recent LLMs have made the dangers even more clear. > 2) is the letter's proposal of a 6 month pause useful. More opinions. > It's the most useful and plausible step I can think of. We need to take stock. It's not up to a handful of researchers to endanger the lives of 6B other people. 3) Is it actually plausible that we could get such a pause. We'll see. I doubt it but also search "death with dignity" by Yudkowsky. All of this is your opinion. > Back to Musk as an "authority", the headline is about him but more than 1000 other concerned parties have signed. I didn't even mention Musk. I have several other names in mind. Lots of folks with AI companies (including LLMs!) that "may" be experiencing FOMO and sensing a strategy here. Maybe. Hypothetically. In a non-"ad hominem" way. > I will sign. Their signature vetting process is stalled because to the volume of signatures, otherwise there would be many more. People are starting counter petitions, which I'll gladly sign. The one by Suhail posted on Twitter was hilarious af. I'll also go to whatever country doesn't "pause", because this tech is literally the most exciting development of my lifetime. And I want to spend my life doing something that matters instead of gluing distributed systems together to process financial transactions.

JCharante 4y ago

Was this comment generated by a LLM?

TacticalCoder 4y ago
The second paragraph definitely was (the first one probably too). 100% crystal clear, it's not even a question. Downvoted.
xwdv 4y ago
Yes, it has some of the classic markers of an LLM. Downvoted.

croo 4y ago

Nitpick: dark forest is one possible explanation for the Fermi paradox and not coined in the book(though the book presented neatly). You can read a great article about fermi paradox here if anyone wants to jump into what the dark forest theory is: https://waitbutwhy.com/2014/05/fermi-paradox.html https://waitbutwhy.com/2014/05/fermi-paradox.html

pmoriarty 4y ago

"another group of researchers believe that we should not stop because we cannot be certain whether those who are trying to catch up with us will use the advancement of AI technology as a hidden weapon" It's not just a potential weapon, but a potential source of untold riches and power. That's going to be irresistible to anyone with an interest in either (ie. most politicians, governments, billionaires and corporations... and most ordinary people too, if they only had the means).

biscottigelato 4y ago
Its the one ring with the ultimate power. But so powerful that it might, if not almost certainly will, end us all. Perhaps this IS the solution to the Fermi Paradox.

thomastjeffery 4y ago

Suspicion is inference. It fails to provide enough information (on its own) to decide whether something is threatening or desirable. LLMs are inference models.

lanstin 4y ago

This book is the scariest thing I have ever read.

wayfwdmachine 4y ago

Yes. And it's fiction. We might as well discuss Excession, Agency, Diaspora (Banks, Gibson and Egan respectively) all of whom have more interesting and insightful things to say about humanity, game theory and ... seriously, as a life long (and I am OLD) science fiction fan, the fawning over The Remembrance of Earth's past is just silly. The concept of "suspicion". Might I suggest that people read Thomas Schelling's seminal work The Strategy of Conflict to begin with since if you want to understand concepts like brinksmanship and the push and pull of actual inter-civilization (or society or nation state) conflicts then you might want to start there and not with fiction riddled with silly concepts, weird misogyny and stuff that seems to be right out of the discarded parts of a Gregory Benford trilogy. I mean seriously. But ok. Let's dig into this "We can apply this concept of suspicion to the phenomenon in the field of modern artificial intelligence research. In AI research, some researchers believe that humans should pause the development of AI technology because of the potential unforeseeable risks it may bring." An analogy would be that some of the scientists involved with the Manhattan project (see Szilárd petition) attempted to file an official protest against the use of nuclear weapons against civilian population and arguing that the weapons constructed were too dangerous to be a part of an arms race and should be placed under an international control body. "However, another group of researchers believe that we should not stop because we cannot be certain whether those who are trying to catch up with us will use the advancement of AI technology as a hidden weapon." Researchers are not decision makers in this or any other scenario. AI technology as a weapon? Sure but unless you are Bombs Away LeMay circa 1961 this really isn't a winnable or even useful use case. We're still in a blue sky phase. The fun thing about nuclear weapons is that if any nation feels that another is pulling too far ahead they can be used as a way to push for bi/multilateral discussions and eventually treaties limiting the production and/or deployment of weapons that risk of increasing tensions/lethality and so on . (see SALT I/II and START etc) "This suspicion leads to a chain reaction, where people continue to worry about each other's motives and abilities, making it difficult for all parties to establish trust. Therefore, although there are good intentions, such proposals are unlikely to be implemented." Again... read Schelling... this is just nonsense.

CatWChainsaw 3y ago
"The only thing stupider than thinking something will happen because it is depicted in science fiction is thinking something will not happen because it is depicted in science fiction." https://philosophybear.substack.com/p/position-statement-on-ai-and-leftwing https://philosophybear.substack.com/p/position-statement-on-...

DonHopkins 4y ago

That might explain why the Vorlons of Babylon 5 were such assholes. https://babylon5.fandom.com/wiki/Vorlon https://babylon5.fandom.com/wiki/Vorlon >Distant Past >The Vorlons were among the first races in the galaxy to attain sentience and become space-faring creatures. In the distant past, the Vorlons attempted a grand experiment into reaching a higher plane of existence where the souls existed and all life was founded. To accomplish this, they poured all their knowledge into creating a gateway to this dimension. After activating the gate, the vortex into Thirdspace was formed and the aliens within that realm came pouring out, intent on destroying all other forms of life. After much hardship, the Vorlons finally managed to push the enemy back, but one of the gateways was lost in Hyperspace where it was hidden from the Vorlons. They would encode some of their servants with an automated message to warn the younger races of the threat from the inhabitants of Thirdspace.