3 ms·

Its fun, but the point should be well known (i know its not). Time series are very prone to spurious correlations - r² is not useful. Its even worse if you jus

Its fun, but the point should be well known (i know its not). Time series are very prone to spurious correlations - r² is not useful. Its even worse if you just eyeball a graph. If something changes over time, you need to use appropriate measures.

feoren 1y ago
> r² is not useful People want so badly to have an "objective" measure of truth that they can just pump data into and get a numeric result that "doesn't lie". r², p < 0.05, χ2, etc. It's too far to say these numbers aren't useful at all -- they are -- but we're just never going to be able to avoid the difficult and subjective task of interpreting experimental results in their broader context and reconciling them with our pre-existing knowledge. I think this is why people are so opposed to anything Bayesian: we don't want to have to do that work, to have those arguments with each other about what we believed before the experiment and how strongly. But the more we try to be objective, the more vulnerable we are to false positives and spurious correlations.
SOLAR_FIELDS 1y ago
I agree - the problem here is probably less that the AI makes the mistake and more that it’s just really easy to make this mistake whether you’re AI or human. It’s probably true that the AI selects more correlations that would be obviously irrelevant to the end user though. I, like others doing SRE work, spend a fair amount of time trawling through these types of graphs and it’s very common to see something that looks like a correlation, look closer, then dismiss it due to noise. Time series are also very much subject to MTUP[1]. Something that looks like a correlation or problem at a certain zoom level becomes totally normal behavior when you zoom out, for instance 1: https://en.m.wikipedia.org/wiki/Modifiable_temporal_unit_problem https://en.m.wikipedia.org/wiki/Modifiable_temporal_unit_pro...

feoren 1y ago

> r² is not useful People want so badly to have an "objective" measure of truth that they can just pump data into and get a numeric result that "doesn't lie". r², p < 0.05, χ2, etc. It's too far to say these numbers aren't useful at all -- they are -- but we're just never going to be able to avoid the difficult and subjective task of interpreting experimental results in their broader context and reconciling them with our pre-existing knowledge. I think this is why people are so opposed to anything Bayesian: we don't want to have to do that work, to have those arguments with each other about what we believed before the experiment and how strongly. But the more we try to be objective, the more vulnerable we are to false positives and spurious correlations.

SOLAR_FIELDS 1y ago

I agree - the problem here is probably less that the AI makes the mistake and more that it’s just really easy to make this mistake whether you’re AI or human. It’s probably true that the AI selects more correlations that would be obviously irrelevant to the end user though. I, like others doing SRE work, spend a fair amount of time trawling through these types of graphs and it’s very common to see something that looks like a correlation, look closer, then dismiss it due to noise. Time series are also very much subject to MTUP[1]. Something that looks like a correlation or problem at a certain zoom level becomes totally normal behavior when you zoom out, for instance 1: https://en.m.wikipedia.org/wiki/Modifiable_temporal_unit_problem https://en.m.wikipedia.org/wiki/Modifiable_temporal_unit_pro...