That's in fact the real problem, and I don't think it's solvable because it's irrational. Only rational things are "solvable". Here's a similar conundrum: With

That's in fact the real problem, and I don't think it's solvable because it's irrational. Only rational things are "solvable". Here's a similar conundrum: Without using a human as a "living proxy ruler" or actual sales data, come up with an algorithm that uses only empirical data (not touching human behavior around it!) to determine the fair market value of the Mona Lisa. Then, apply that to some artwork it's never seen before and see if it concords with what "meat computers" (humans) believe the value of it is. My strong position is that not only will you not be able to do this now, but you will not be able to do this ever. I think rationality will get us VERY far, though, so we should keep doubling down on it. My money's on it being insufficient to produce an apparent intelligent and unique being, though.