I actually started reading the Book of Why and I recommend it to the HN crowd. Pearl does a really good job of going through the history of causality, including

I actually started reading the Book of Why and I recommend it to the HN crowd. Pearl does a really good job of going through the history of causality, including the quite interesting story behind the now known to be wrong (or at least incomplete) claim that "correlation is not causality". He then places causality on a 3-rung scale. The bottom rung is association in data, which is where he says AI is stuck. Then there's intervention, "what if I do this thing?", and then there's counterfactuals, "what if I had done some other thing?" He then makes a case for what intelligence actually is, and unsurprisingly it's getting up those three rungs. The method revolves around directed graphs, which have certain unintuitive properties. For instance if A and B can both cause C, knowing that A is unlikely make B more likely, given that C happened. There's a few other stock situations in various graphs that he walks through as well. In the end the point seems to be that we could have a causal machine if we'd spend some more time on it. It would take data and try out some potential graphs, and some of the graphs would be ruled out by the data. And then some algo would tell you things like whether a randomized trial is necessary or even whether you'd need one (yes, this is another revelation). I think there's also an argument that this is how people actually think, which makes sense because the graphs are not terribly large and they need to fit in your meat hardware. I haven't finished it but I would guess that you could take it in another direction and say this is why some animals sorta have intelligence in that they learn patterns, but they don't know the higher rungs. Really interesting ideas, and at least clarifies what we mean by causality.