> Competence here is indistinguishability from human in conversation. The most straightforward way to implement it is to make the algorithm have the same struct

> Competence here is indistinguishability from human in conversation. The most straightforward way to implement it is to make the algorithm have the same structure and work in the same way as human mind I'm not sure that's the case. Simple chat programs can do a pretty good job simulating a human interlocutor. For a 'full' simulation, which we need by definition, we'd need something much more sophisticated (able to reason about all sorts of abstract and concrete things), but conceivably the solution might be very different from brain-simulation. The thought experiment can easily be adjusted to close the door on my objection here: rather than a man in a room with an enormous card index, we have a pretty accurate real-time computer simulation of some specific person. That way the computational problem is defined to be equivalent to something we consider conscious. Laboriously computing that simulation by hand (presumably not in real time but instead over millennia) would change the substrate, but not the computational problem. If we're ok with there being an outer host consciousness and an inner hosted consciousness, the thought experiment poses no problem. Of course, this isn't the position I started at, but it makes some sense that the real meaning of the thought experiment change as we adjust the simulated process. If our man were looking up the best moves to play tic-tac-toe, it would be plainly obvious that we're looking at competence without comprehension. If he's instead simulating the full workings of a human brain, the situation is different. The foreign language problem is somewhere between these extremes. > It's a good part about artificial algorithms that they are transparent and we can show that they aren't only effective in conversation, but have everything to be had behind that conversation. It's due to the latter. Simply put, the algorithm isn't GPT, but AGI. I'm not sure I quite follow you here. I agree that the depth and detail of the simulation is an important factor. > What's new is that the life of virtual people is shown at length and discussed questions as to what identity those people should have, what worldview, religion, philosophy, pride, dignity, justice. In contrast to Ex Machina that had the computer as a sociopathic villain with only surface level feigning of normal human emotion and motivation. While we're vaguely on the topic, homomorphic crypto also puts a spin on things. We know it's possible for a host computer to be entirely 'unaware' of what's going on in the VM that it's running, in a cryptographic sense. Related to this, I've long thought that there's a sticky 'interpretation problem' with consciousness (perhaps philosophers have another term for it) that people rarely talk about. If you run a brain simulator inside a homomorphically encrypted system, such that no one else will ever know what you're running in there, does that impact whether we treat it as conscious? Part of it is that the simulated brain isn't hooked up to any real-world sensors or actuators, but that's just like any old brain in a jar. Philosophically pedestrian. This goes far beyond that. Someone could inspect the physical computer, and they'd have no idea what was really running on it. They'd just see a pseudorandom stream of states. If there's consciousness inside the VM, it's only there with respect to the homomorphic crypto key! If we allow that to count as consciousness, we've opened the door to all sorts of computations counting as consciousness, if only we knew the key. We can take this further: we can always invent a correspondence such that any sequence of states maps to a computation stream that we would identify as yielding consciousness. This looks like some kind of absurd endgame of panpsychism, but here we are. Is there an alternative? I'm increasingly of the opinion that it seems like a non-starter to try to deny that transistor-based computers could ever be the substrate of consciousness. Short of that, where else is there to go? </sidetrack>