Yep, this is a big part of it. Intelligence and consciousness are barely understood beyond "I'll know it when I see it", which doesn't work for things you can't

Yep, this is a big part of it. Intelligence and consciousness are barely understood beyond "I'll know it when I see it", which doesn't work for things you can't see - and in the case of consciousness, most definitions are explicitly based on concepts that are not only invisible but ineffable. And then we have no solid idea whether these things we can't really define, detect, or explain are intrinsically linked to each other or have a causal relationship in either direction. Almost any definition you pick is going to lead to some unsatisfying conclusions vis a vis non-human animals or "obviously not intelligent" forms of machine learning. It's a real mess.