It is the same with questions like what counts as intelligence, what counts as alive? The boundaries shy away. In fact it is probably impossible to find simple

It is the same with questions like what counts as intelligence, what counts as alive? The boundaries shy away. In fact it is probably impossible to find simple definitions for these terms, i.e. definitions that get by without special rules for all kinds of instances which just feel more intelligent, alive or like a computer, or less. The definition is irreducible in the sense that we cannot explain our intuition without tracing back the history of our species and of the memes and neuronal firing patterns inside our brains, information that likely cannot be compressed very well. The same applies to human values. Stephen Wolfram just gave an interesting talk and Q/A session on this problem: https://www.youtube.com/watch?v=uxQwpdB9wbg https://www.youtube.com/watch?v=uxQwpdB9wbg