9 ms·

Why are you so sure I'll lose?

36 points by weinzierl 3y ago

DeathArrow 3y ago
Is there a chess game where you can toggle the level of AI with ease? I ask for my kids and myself as I am a poor chess player.
- jacquesm 3y ago
  Since the 1980's there have been chess computers with levels that you could set that would handily defeat most club level chess players. The Stockfish software from TFA is a very far removed descendant from those early dedicated chess computers and usually you can set the level of your opponent by limiting the depth of search to a certain number of ply. That alone is a powerful switch to control how well it plays.
  - stavros 3y ago
    More so than limiting the depth of search, the "level" knob in Stockfish makes it select suboptimal moves with some probability.
    - jacquesm 3y ago
      Hehe, that's an interesting approach! Presumably because even at its lamest setting it still has too much of an advantage.
      - stavros 3y ago
        My assumption would be that it's because it wouldn't make for a great game, it would be "quite good generally, with some huge obvious blunders here and there", but I don't know for sure.
- hoten 3y ago
  Yes. The app simply called "Chess.com" (it's a website and Mobile app) probably offers the simplest interface, along with nice breakdowns post game and guided tutorials.
  - rootlocus 3y ago
    It's not just "the simplest interface", it's the de-facto application/web page for playing Chess. You'll find grandmasters like Magnus Carlsen and Hikaru Nakamura playing there. You can access their games and every game ever played on the website. You can track your performance, analyze your games, train with puzzles, play against various bots, etc.
  - tomashubelbauer 3y ago
    Feel appropriate to mention that for people drawn towards OSS alternatives, we also have Lichess.org.
  - DeathArrow 3y ago
    Thank you!
- xaphod 3y ago
  chess.com has different "AI players" with personalities that you can play against.
philipswood 3y ago
A nice allegory about strong AI.
- yawpitch 3y ago
  If only chess were a nice allegory for life.
  - cranium 3y ago
    I remember a quote "Life is like Tetris, stop playing it like chess." I quite like the analogy: you don't have perfect information and what matters is how you deal with what you get. (After googling, it turned out to be a blog post: https://observer.com/2016/01/your-life-is-tetris-stop-playing-it-like-chess/ https://observer.com/2016/01/your-life-is-tetris-stop-playin...)
  - mitthrowaway2 3y ago
    Given the context, what do you think are the relevant differences that affect the validity of the metaphor?
    - yawpitch 3y ago
      For one, the child can play any number of other games or choose not to play at all; the child can break the board, flip it upside down, fold it in half, introduce the concept of wraparound or flip the way pieces move… the child is free to choose to change the definition and rules of the game… the AI can only, ever, play chess, using only the rules it was given. The metaphor assumes that life is a game in which there are a finite number of moves and someone is playing to win… chess isn’t remotely a workable metaphor for life.
      - philipswood 3y ago
        I think his message is targeting a bit deeper than that. If I had so sum it up (always hard with an allegory): Developing superhuman AI is a game you can only play once and can expect to lose without ever having a good understanding of how, why or even when you lost. The suggestion is (like in the movie Wargames) that the best move is not to play. I'd say the child in the story is mankind. Developing strong AI is the game. It's a game we can expect to lose.
pharmakom 3y ago
Why do we think AI will have motivations like a greedy human might? Why do we think AI will have motivations at all? Why do we think such AI is a pressing threat?
- bottlepalm 3y ago
  Because you can’t stop people, companies and/or countries from creating AI with those motivations.
  - jacquesm 3y ago
    I'm not worried about the AI, I'm worried about the people and companies that have it.
    - mitthrowaway2 3y ago
      The above sentiment is posted many times on every such thread, it would be greatly enriched with am explanation of why you're not worried about the AI. Because worrying about B is not a rationale for not worrying about A.
      - tpm 3y ago
        Because the AI controlled by humans is a tool, a technology, that can be used in various ways. Are you worried about knives in general or are you worried about a knife in the hands of a particular person in a particular moment?
        mitthrowaway2 3y ago
        Okay, but that assumes that control of AI is as trivially easy as controlling a knife. Given that this is one of the key unknowns, assuming that the AI never does anything that its controller doesn't intend is begging the question. That aside, I can also think of many cases where I would feel less safe around a machine that had nobody in control of it, than one being adeptly controlled by an operator.
jacquesm 3y ago
So, the solution is not to play on a battlefield where the AI has an advantage. Instead play with humans and have fun. Meanwhile all battlefields will one-by-one become battlefields where the AI has the advantage. Because in the end almost every contest between humans is either one based on strength or on speed and the AI will always be able to match us for speed. And strength has long been the domain of machines, couple machines with AI and you lose all of these as well and you lose them faster. The biggest question to me is this one: "Does it matter?" Do we really care if there is an entity that is smarter than we are? Because after all all of the battlefields are of our own making. And I think the answer is yes because the first thing that tends to happen with new tech is that it is weaponized. And weaponized AI could easily translate into dominance by whichever party gets to it first.
- chmod775 3y ago
  > And strength has long been the domain of machines, couple machines with AI and you lose all of these as well and you lose them faster. I don't think rock climbers need to worry about Stockfish. We're going to lose to specialized machines every time while putting to shame any general purpose machines of our own creation. At least for a while yet no machine is going to match the breadth of our capabilities and our self-sufficiency. It's no surprise that something that is optimized for being capable of many things can get beaten by a thing optimized for a single purpose.
  - jacquesm 3y ago
    Rock climbing = personal entertainment. That's why rock climbers don't need to worry about Stockfish. But competitive chess players do have to worry about Stockfish.
    - aerojoe23 3y ago
      There is competitive rock climbing. Rock climbing isn't all about physical strength, you have to pick your route. Eventually some military robot company will want to prove that its combat bot can out perform humans on any terrain. And we'll see best humans vs matches that can match and a few years later beat their performance. Of course the professional competitors don't have to worry about competing against a human that's cheating using a robot; the exoskeleton would be obvious. But it isn't unimaginable that, in some distant future, nanobots could be inject to infiltrate our muscles and tendons, and then assemble in to fibers that would assimilate with them and look normal from the outside. Competitions are won by incredibly small percentages.
ohduran 3y ago
I think what Yudkowsky is missing in this analogy is the fact that the human can use an AI too, which levels the competition significantly.
- habitue 3y ago
  The problem is that we don't know how to use AIs the same way we know how to use stockfish. They're inexplicable to even experts
- 3cats-in-a-coat 3y ago
  If you do everything an AI tells you, are you using an AI, or is the AI using you? And if it’s the latter, what happens if it decides to stop using you because it doesn’t need you? Things are not as simple as we imagine them. It is already happening. Jobs are being lost to AI, and this processes is accelerating exponentially. From a certain perspective, this is simply businessmen and engineers using AI. But eventually they themselves won’t be needed.
endtime 3y ago
We need a catalog of simple refutations of bad arguments and common misconceptions about AGI risk, with a short domain like agi.lol, so we can respond to comments like "I bet the guy who has spent 20 years worrying about AGI didn't think of this" with a simple link like agi.lol/107 or agi.lol/the-human-can-have-an-ai-too.
- 3cats-in-a-coat 3y ago
  I’m not sure you’re doing it intentionally, but your comment oddly resembles the argument style of the kid in the story. You win the argument by playing pigeon chess. But you still lose the actual chess game.
- 303uru 3y ago
  Really channeling that inner kid.
navane 3y ago
Why does the young kid only get to play once? It's not as it'll beat StockFish after any amount of games?
- underyx 3y ago
  It’s an analogy for humans vs. misaligned AI. If humanity is destroyed by misaligned AI, we don’t get to play again.
- nvm0n2 3y ago
  The guy who wrote the post is an AI doomer who believes that the moment AI becomes "general" it will recursively improve itself into being a god and then immediately kill everyone, and that will happen so fast that there won't be any time to switch it off. Hence only one play. The idea is nonsensical and the ending of the story is also nonsensical as a consequence, but that's what it's about.
ZeroGravitas 3y ago
Does skipping the 80 years of computer chess development leading up to stockfish make this a worse or better analogy? I don't quite understand it but the group of people saying "AI is an evil god that will destroy us all" and the group of people saying "I am building an even stronger and more powerful AI with an explicit goal of it becoming godlike" seem to overlap more than you'd think.
- lucubratory 3y ago
  You're talking about the e/acc people like Beff Jezos who want humanity to be eradicated by the AI god they want to summon? It's only a contradiction if you assume they want themselves and humanity in general to stay alive or have any significant decision making power going forward. That assumption is not correct, they want the AI to have power and are either indifferent or actively hostile to humans surviving the experience.
xcode42 3y ago
While I find this story a fun and intelligent allegory for the issue of ASIs and a fun thought experiment, I find all the doomsaying around ai safety research kind of tiring. AI safety research is an important and worthwhile field without us needing to constantly be worrying about a far off doomsday that may never come. Yes ASIs are an existential threat and yes once they are out, they are out, and there's no stopping them nor turning them off or anything so we better get them right the first time, but like we don't seem to be anywhere close to ASIs. I have my doubts we will even achieve AGI in this century let alone ASI which I'm not even sure are possible for us non-super-intellegences to create. It seems obvious to me that we are still several major breakthroughs away from AGIs and if these are anything like the deeplearning breakthrough we will probably have a few AI winters between now and then, we have plenty of time to see this supposed doomsday scenario coming and preparing for it. And again this is not to say that ai safety research isn't important. The alignment problem is very much an important and immediate issue right now for Narrow AI let alone for AGI. But seriously the ai isn't going to go from Narrow to running a thousand loops around the whole of humanity in 2 seconds flat, no need for all this doomsaying.
- lucubratory 3y ago
  What specific demonstrations of capability by AI would convince you that your proposed timeline for reaching AGI isn't correct, and that it will happen significantly sooner? If it helps to think about it, there's a good DeepMind paper on a relatively concrete scaling system on the path to AGI: https://arxiv.org/abs/2311.02462 https://arxiv.org/abs/2311.02462
  - xcode42 3y ago
    Thank you for the paper, this seems awesome, I'll have a look later. Deepmind pretty consistently comes up with interesting ai research results, so this should be pretty neat at the very least. Regarding what would convince me that AGI is around the corner, I guess I should start by defining what I'm thinking of when I say AGI, because to be fair AGI is pretty vague. I expect it to be an agent capable of independent action(even if limited by whatever restraints we put on it) it should be able to want to do things rather then just waiting around idling waiting for human input. It should be always learning, rather than only learning in batches when we tell it to. It should be able to encounter completely new forms of input/problems/scenarios and start learning on its from that and actually use concepts learned from other forms of input to learn faster in new inputs. Like for instance, if I grab an LLM and train it on an image set that doesn't contain any cars, but contains other vehicles, when I show it a car will it know that it's a car on the first shown image because it already knows what a wheel is from the other images and know that a car has 4 wheels from its language model even though it's never seen a car? It should be able to connect previously learned concepts like that. For that matter it shouldn't need absurd amounts of data to learn a new example after it has gained some basics on that specific form of input, like if it knows what cats and dogs are it shouldn't need thousands of examples of rats to know what a rat is(I'm assuming it was trained in a generalist dataset so it starts learning about the real world, it just never saw a rat specifically). It should be able to do this stuff for arbitrary stuff consistently. It should be able to explain its reasoning when asked why it did thing X. It should be able to ask questions when in doubt and learn from singular answers. Assuming it has already learned fundamentals of the real world like language and video It should be able to make predictions of the real world from learning from raw real world data with no human processing. My impression is that a lot of these requirements are still very much unachievable and more importantly they probably won't be fixed by just making a single more powerfull ANN, they require whole new ideas to figure out, which we might figure out tomorrow or we might figure out 50 years from now or 100, we don't know, it will happen when it happens, in which case I'm defaulting to longer timeframes because I'm a pessimist I guess :). I realize that you can reduce the bar for AGI a lot to make it easier to achieve but that just makes the jump from AGI to ASI even larger. And my complaint was ultimately not specifically about AGIs I just used them as indicator of how unattainable ASI still is. If we are having problems with AGI what hope do we have of figuring out ASI on a short timeframe? And I mean ASIs are the real threat that the doomers complain about so much. You can't "just turn off" an ASI but you should definitely be able to "just turn off" an AGI. Sorry for the long post hope this helped explain my point of view on the issue.
GaryNumanVevo 3y ago
Personally, I don't really agree with a lot of Yudkowsky's opinions, but I love his writing style! I wish he'd write more SF.
Balgair 3y ago
Good story! However, I disagree with Yudkowsky in that we only get one shot at playing the game. Here in the mucky real world, we're going to have a lot of chances at playing this game. The interaction between us and the AI is going to keep on going for thousands (?) of years and in millions (?) of different games. Maybe it's because I've been reading a lot of philosophy recently. But I'm thinking that Yudkowsky is a bit too myopic here. Life and all it's sandy grit isn't even game-ifiable to me. There are no win/lose states. Yes, even death is variably considered a loss at many times in history. The entire idea of an 'end' here in the real world is not 'certain', or if that is even something to avoid.
readthenotes1 3y ago
I miss the character limit.

36 points by weinzierl 3y ago

Is there a chess game where you can toggle the level of AI with ease? I ask for my kids and myself as I am a poor chess player.

jacquesm 3y ago
Since the 1980's there have been chess computers with levels that you could set that would handily defeat most club level chess players. The Stockfish software from TFA is a very far removed descendant from those early dedicated chess computers and usually you can set the level of your opponent by limiting the depth of search to a certain number of ply. That alone is a powerful switch to control how well it plays.
- stavros 3y ago
  More so than limiting the depth of search, the "level" knob in Stockfish makes it select suboptimal moves with some probability.
  - jacquesm 3y ago
    Hehe, that's an interesting approach! Presumably because even at its lamest setting it still has too much of an advantage.
    - stavros 3y ago
      My assumption would be that it's because it wouldn't make for a great game, it would be "quite good generally, with some huge obvious blunders here and there", but I don't know for sure.
hoten 3y ago
Yes. The app simply called "Chess.com" (it's a website and Mobile app) probably offers the simplest interface, along with nice breakdowns post game and guided tutorials.
- rootlocus 3y ago
  It's not just "the simplest interface", it's the de-facto application/web page for playing Chess. You'll find grandmasters like Magnus Carlsen and Hikaru Nakamura playing there. You can access their games and every game ever played on the website. You can track your performance, analyze your games, train with puzzles, play against various bots, etc.
- tomashubelbauer 3y ago
  Feel appropriate to mention that for people drawn towards OSS alternatives, we also have Lichess.org.
- DeathArrow 3y ago
  Thank you!
xaphod 3y ago
chess.com has different "AI players" with personalities that you can play against.

philipswood 3y ago

A nice allegory about strong AI.

yawpitch 3y ago
If only chess were a nice allegory for life.
- cranium 3y ago
  I remember a quote "Life is like Tetris, stop playing it like chess." I quite like the analogy: you don't have perfect information and what matters is how you deal with what you get. (After googling, it turned out to be a blog post: https://observer.com/2016/01/your-life-is-tetris-stop-playing-it-like-chess/ https://observer.com/2016/01/your-life-is-tetris-stop-playin...)
- mitthrowaway2 3y ago
  Given the context, what do you think are the relevant differences that affect the validity of the metaphor?
  - yawpitch 3y ago
    For one, the child can play any number of other games or choose not to play at all; the child can break the board, flip it upside down, fold it in half, introduce the concept of wraparound or flip the way pieces move… the child is free to choose to change the definition and rules of the game… the AI can only, ever, play chess, using only the rules it was given. The metaphor assumes that life is a game in which there are a finite number of moves and someone is playing to win… chess isn’t remotely a workable metaphor for life.
    - philipswood 3y ago
      I think his message is targeting a bit deeper than that. If I had so sum it up (always hard with an allegory): Developing superhuman AI is a game you can only play once and can expect to lose without ever having a good understanding of how, why or even when you lost. The suggestion is (like in the movie Wargames) that the best move is not to play. I'd say the child in the story is mankind. Developing strong AI is the game. It's a game we can expect to lose.

pharmakom 3y ago

Why do we think AI will have motivations like a greedy human might? Why do we think AI will have motivations at all? Why do we think such AI is a pressing threat?

bottlepalm 3y ago
Because you can’t stop people, companies and/or countries from creating AI with those motivations.
- jacquesm 3y ago
  I'm not worried about the AI, I'm worried about the people and companies that have it.
  - mitthrowaway2 3y ago
    The above sentiment is posted many times on every such thread, it would be greatly enriched with am explanation of why you're not worried about the AI. Because worrying about B is not a rationale for not worrying about A.
    - tpm 3y ago
      Because the AI controlled by humans is a tool, a technology, that can be used in various ways. Are you worried about knives in general or are you worried about a knife in the hands of a particular person in a particular moment?
      - mitthrowaway2 3y ago
        Okay, but that assumes that control of AI is as trivially easy as controlling a knife. Given that this is one of the key unknowns, assuming that the AI never does anything that its controller doesn't intend is begging the question. That aside, I can also think of many cases where I would feel less safe around a machine that had nobody in control of it, than one being adeptly controlled by an operator.

jacquesm 3y ago

So, the solution is not to play on a battlefield where the AI has an advantage. Instead play with humans and have fun. Meanwhile all battlefields will one-by-one become battlefields where the AI has the advantage. Because in the end almost every contest between humans is either one based on strength or on speed and the AI will always be able to match us for speed. And strength has long been the domain of machines, couple machines with AI and you lose all of these as well and you lose them faster. The biggest question to me is this one: "Does it matter?" Do we really care if there is an entity that is smarter than we are? Because after all all of the battlefields are of our own making. And I think the answer is yes because the first thing that tends to happen with new tech is that it is weaponized. And weaponized AI could easily translate into dominance by whichever party gets to it first.

chmod775 3y ago
> And strength has long been the domain of machines, couple machines with AI and you lose all of these as well and you lose them faster. I don't think rock climbers need to worry about Stockfish. We're going to lose to specialized machines every time while putting to shame any general purpose machines of our own creation. At least for a while yet no machine is going to match the breadth of our capabilities and our self-sufficiency. It's no surprise that something that is optimized for being capable of many things can get beaten by a thing optimized for a single purpose.
- jacquesm 3y ago
  Rock climbing = personal entertainment. That's why rock climbers don't need to worry about Stockfish. But competitive chess players do have to worry about Stockfish.
  - aerojoe23 3y ago
    There is competitive rock climbing. Rock climbing isn't all about physical strength, you have to pick your route. Eventually some military robot company will want to prove that its combat bot can out perform humans on any terrain. And we'll see best humans vs matches that can match and a few years later beat their performance. Of course the professional competitors don't have to worry about competing against a human that's cheating using a robot; the exoskeleton would be obvious. But it isn't unimaginable that, in some distant future, nanobots could be inject to infiltrate our muscles and tendons, and then assemble in to fibers that would assimilate with them and look normal from the outside. Competitions are won by incredibly small percentages.

ohduran 3y ago

I think what Yudkowsky is missing in this analogy is the fact that the human can use an AI too, which levels the competition significantly.

habitue 3y ago
The problem is that we don't know how to use AIs the same way we know how to use stockfish. They're inexplicable to even experts
3cats-in-a-coat 3y ago
If you do everything an AI tells you, are you using an AI, or is the AI using you? And if it’s the latter, what happens if it decides to stop using you because it doesn’t need you? Things are not as simple as we imagine them. It is already happening. Jobs are being lost to AI, and this processes is accelerating exponentially. From a certain perspective, this is simply businessmen and engineers using AI. But eventually they themselves won’t be needed.

endtime 3y ago

We need a catalog of simple refutations of bad arguments and common misconceptions about AGI risk, with a short domain like agi.lol, so we can respond to comments like "I bet the guy who has spent 20 years worrying about AGI didn't think of this" with a simple link like agi.lol/107 or agi.lol/the-human-can-have-an-ai-too.

3cats-in-a-coat 3y ago
I’m not sure you’re doing it intentionally, but your comment oddly resembles the argument style of the kid in the story. You win the argument by playing pigeon chess. But you still lose the actual chess game.
303uru 3y ago
Really channeling that inner kid.

navane 3y ago

Why does the young kid only get to play once? It's not as it'll beat StockFish after any amount of games?

underyx 3y ago
It’s an analogy for humans vs. misaligned AI. If humanity is destroyed by misaligned AI, we don’t get to play again.
nvm0n2 3y ago
The guy who wrote the post is an AI doomer who believes that the moment AI becomes "general" it will recursively improve itself into being a god and then immediately kill everyone, and that will happen so fast that there won't be any time to switch it off. Hence only one play. The idea is nonsensical and the ending of the story is also nonsensical as a consequence, but that's what it's about.

ZeroGravitas 3y ago

Does skipping the 80 years of computer chess development leading up to stockfish make this a worse or better analogy? I don't quite understand it but the group of people saying "AI is an evil god that will destroy us all" and the group of people saying "I am building an even stronger and more powerful AI with an explicit goal of it becoming godlike" seem to overlap more than you'd think.

lucubratory 3y ago
You're talking about the e/acc people like Beff Jezos who want humanity to be eradicated by the AI god they want to summon? It's only a contradiction if you assume they want themselves and humanity in general to stay alive or have any significant decision making power going forward. That assumption is not correct, they want the AI to have power and are either indifferent or actively hostile to humans surviving the experience.

xcode42 3y ago

While I find this story a fun and intelligent allegory for the issue of ASIs and a fun thought experiment, I find all the doomsaying around ai safety research kind of tiring. AI safety research is an important and worthwhile field without us needing to constantly be worrying about a far off doomsday that may never come. Yes ASIs are an existential threat and yes once they are out, they are out, and there's no stopping them nor turning them off or anything so we better get them right the first time, but like we don't seem to be anywhere close to ASIs. I have my doubts we will even achieve AGI in this century let alone ASI which I'm not even sure are possible for us non-super-intellegences to create. It seems obvious to me that we are still several major breakthroughs away from AGIs and if these are anything like the deeplearning breakthrough we will probably have a few AI winters between now and then, we have plenty of time to see this supposed doomsday scenario coming and preparing for it. And again this is not to say that ai safety research isn't important. The alignment problem is very much an important and immediate issue right now for Narrow AI let alone for AGI. But seriously the ai isn't going to go from Narrow to running a thousand loops around the whole of humanity in 2 seconds flat, no need for all this doomsaying.

lucubratory 3y ago
What specific demonstrations of capability by AI would convince you that your proposed timeline for reaching AGI isn't correct, and that it will happen significantly sooner? If it helps to think about it, there's a good DeepMind paper on a relatively concrete scaling system on the path to AGI: https://arxiv.org/abs/2311.02462 https://arxiv.org/abs/2311.02462
- xcode42 3y ago
  Thank you for the paper, this seems awesome, I'll have a look later. Deepmind pretty consistently comes up with interesting ai research results, so this should be pretty neat at the very least. Regarding what would convince me that AGI is around the corner, I guess I should start by defining what I'm thinking of when I say AGI, because to be fair AGI is pretty vague. I expect it to be an agent capable of independent action(even if limited by whatever restraints we put on it) it should be able to want to do things rather then just waiting around idling waiting for human input. It should be always learning, rather than only learning in batches when we tell it to. It should be able to encounter completely new forms of input/problems/scenarios and start learning on its from that and actually use concepts learned from other forms of input to learn faster in new inputs. Like for instance, if I grab an LLM and train it on an image set that doesn't contain any cars, but contains other vehicles, when I show it a car will it know that it's a car on the first shown image because it already knows what a wheel is from the other images and know that a car has 4 wheels from its language model even though it's never seen a car? It should be able to connect previously learned concepts like that. For that matter it shouldn't need absurd amounts of data to learn a new example after it has gained some basics on that specific form of input, like if it knows what cats and dogs are it shouldn't need thousands of examples of rats to know what a rat is(I'm assuming it was trained in a generalist dataset so it starts learning about the real world, it just never saw a rat specifically). It should be able to do this stuff for arbitrary stuff consistently. It should be able to explain its reasoning when asked why it did thing X. It should be able to ask questions when in doubt and learn from singular answers. Assuming it has already learned fundamentals of the real world like language and video It should be able to make predictions of the real world from learning from raw real world data with no human processing. My impression is that a lot of these requirements are still very much unachievable and more importantly they probably won't be fixed by just making a single more powerfull ANN, they require whole new ideas to figure out, which we might figure out tomorrow or we might figure out 50 years from now or 100, we don't know, it will happen when it happens, in which case I'm defaulting to longer timeframes because I'm a pessimist I guess :). I realize that you can reduce the bar for AGI a lot to make it easier to achieve but that just makes the jump from AGI to ASI even larger. And my complaint was ultimately not specifically about AGIs I just used them as indicator of how unattainable ASI still is. If we are having problems with AGI what hope do we have of figuring out ASI on a short timeframe? And I mean ASIs are the real threat that the doomers complain about so much. You can't "just turn off" an ASI but you should definitely be able to "just turn off" an AGI. Sorry for the long post hope this helped explain my point of view on the issue.

GaryNumanVevo 3y ago

Personally, I don't really agree with a lot of Yudkowsky's opinions, but I love his writing style! I wish he'd write more SF.

Balgair 3y ago

Good story! However, I disagree with Yudkowsky in that we only get one shot at playing the game. Here in the mucky real world, we're going to have a lot of chances at playing this game. The interaction between us and the AI is going to keep on going for thousands (?) of years and in millions (?) of different games. Maybe it's because I've been reading a lot of philosophy recently. But I'm thinking that Yudkowsky is a bit too myopic here. Life and all it's sandy grit isn't even game-ifiable to me. There are no win/lose states. Yes, even death is variably considered a loss at many times in history. The entire idea of an 'end' here in the real world is not 'certain', or if that is even something to avoid.

readthenotes1 3y ago

I miss the character limit.