Neither Geertz nor Doctorow are making an assertion about distinguishing winks vs. twitches, and the assertion they are actually making is the opposite of stupi

Neither Geertz nor Doctorow are making an assertion about distinguishing winks vs. twitches, and the assertion they are actually making is the opposite of stupid. They are pointing out that the system under observation, the system of interest (a person, a conversation, a society) is extremely complex and when it emits a signal it is not possible in the general case to know what that signal “means” without some understanding of the system’s state and processes. We are not capable of directly interrogating, representing or computing over the true state and processes of these systems, so we must use heuristics. Humans develop “theory of mind”, the cognitive ability to model other people's cognitive state and process, including their human values and motivations. People with a better theory of mind, which makes more accurate, more fine-grained inferences in a wider variety of contexts, have a decisive social competitive advantage. This doesn't even touch on “theory of relationship”... when you take a system of humans, what are their shared human values? How are value conflicts negotiated in groups at every point of scale? What is the current state of the negotiated compromises? A dataset and a machine learning algorithm is an observer, inferrer and predictor of human and social systems whose actual state and processes are irreducibly large and invisible, and whose human values (and the domain of possible human values) are mysterious. Even if we gloss over the symbol grounding problem, it is clear that machine learning systems cannot be expected to operate in a way that is aligned with the human values of individual or groups. In the worst case they may operate to extinguish those values, and as an unintended consequence hugely diminish quality of life. And maybe we won't notice, because surely there is an “Overton window” of imaginable possible lives, and ML has the potential to slowly and insidiously shift that window until “being happy and fulfilled” becomes “being in the centre-right of the main distribution” across the major ML models in society. And yes, I do realize that similar normalizing forces have been present in society forever, operating with similar effect. (And I recognize their value in reducing transaction costs of all kinds). But to date they have been based on human-driven mechanisms which lacked the ability to centrally micro-observe and micromanage behaviour on a global scale. Alternative value systems could evade these pressures and develop, compete and co-exist, at every point of social scale. But with the advent of the global Internet and sufficient computational throughput, ML has the potential to micro-observe and micromanage everyone everywhere. It can know more about us than we know about ourselves, while entirely missing the point of being human.