It's not that we don't understand that this is just a language model. We do understand that. The fact that a language model can produce convincing output hint

It's not that we don't understand that this is just a language model. We do understand that. The fact that a language model can produce convincing output hints that it is not that different from us. Imagine something in 5 or 10 years time with 1000 times the power. What will it be like? I know you think our awareness makes us fundamentally different. But what if it doesn't? What if we're just a machine that likes to anthropomorphize things reflected back on itself? Ultimately, we won't have to know the answer. In 10 years AIs like this will be in computer games. Some people will find the results of that disturbing. Is highly accurately similated suffering different to real suffering? The philosophical answer won't matter, the practical answer will be that AI use in games will become regulated. And then you have the first glimmer of AIs getting rights.