To expand on this consider the amount of brains that can directly query the amount of a specific neurotransmitter used between two neurons and interpret what th

To expand on this consider the amount of brains that can directly query the amount of a specific neurotransmitter used between two neurons and interpret what that means. That isn't exactly the same, but a hopefully illustrative analogy. An LLM's output is an emergent property of its structure and all the weights being multiplied together just as an organism's intelligence is an emergent property of its brain structure and communication between neurons. Neither of the emergent parts can currently reliably query the substrate they emerge from for similar reasons. That isn't to say that such an LLM or biological brain couldn't be built, but none presently are.