5 ms·
This problem is tightly coupled with issues of correlation vs causation. In real data, correlation is mostly transitive (there are toy exceptions - https://terr
by willis77 11y ago
This problem is tightly coupled with issues of correlation vs causation. In real data, correlation is mostly transitive (there are toy exceptions - https://terrytao.wordpress.com/2014/06/05/when-is-correlation-transitive/ https://terrytao.wordpress.com/2014/06/05/when-is-correlatio..., but they are just that). This means that if you want to predict something, and that something is correlated with some some uncomfortable association, trying to predict the something without the uncomfortable association can leave little residual behind.
For example, if hair length is societally taboo for gender prediction and I make an algorithm that uses a "politically correct" determination using XY chromosomes, I have also made an algorithm that correlates with hair length. Moreover, if I try to statistically correct my algorithm so that it does not correlate with hair length, I end up with an algorithm that works on the tiny leftover residual created by people who buck the trend, i.e. one that's much more likely to be wrong.
Algorithms find both correlational and causal factors. If 9/10 men are from Mars, and you tell me you're from Mars, it is often via correlation that the algorithm labels you a man. You are not allowed jump to the assumption that, say, the drinking water on Mars is turning people into Men.
- seanflyon 11y agoI don't see where causation entered the conversation. Correlation and causation have the same predictive power. If the water on Mars turns 90% of people into men or 90% of the people that drink said water are already men, either way if you tell me you drank the water on Mars I know with a 90% certainty you are a man. Algorithms find both correlation and causal factors which is fine because I want accuracy whether or not it is based on causality.
- gohrt 11y ago> Correlation and causation have the same predictive power. Is only true in a system without feedback. If your response to observations can affect the subjets under study, then acting on the correlation can change the correlation. If you decide to offer everyone in Nairobi a $million to join your super-jumpers space-exploration program, because Nairobi correlates to high jump ability, you will find that low jumpers will flood into Kenya to collect on your offer. Whereas, no matter how much you abuse the fact that gravity make things fall, you aren't going to make gravity stop making things fall.
- seanflyon 11y agoFeedback is just as much of an issue whether the relationship is correlation or causation. All correlations are either based on random chance (thus not statistically significant) or based on some causal relationship, even if we can't identify what it is. Could be A->B, B->A, C->A and C->B, A->C->B... Correlation and causation have the same predictive power.
- gohrt 11y agoThe exceptions are not toys at all! The example he gave is very common: Whenever 2 independent factors contribute (casaully) to an effect, both factors are correlated to each the effect, but are not in generla (well) correlated to each other.