5 ms·
This is a case where knowing the data and it's inherent structure is just as important, if not more important, then knowing how to apply machine learning. In t
by minimaxir 10y ago
This is a case where knowing the data and it's inherent structure is just as important, if not more important, then knowing how to apply machine learning.
In this case, the formula was (time ~ content length + number of tags). Except, by construction, the content length is positively correlated with the number of tags! And since the parsing tags is the primary purpose of HTML scrapers, it's likely that content length is completely redundant or a poor predictor of time. (which was the conclusion). If you wanted to calculate the maginitude/significance of an explanatory variable, a linear regression is more than sufficient, and R has great tools for that.
The quantile plot revealed there is a skew in time processed, presumably the number of tags has a similar skew.
- thanatropism 10y agoI keep telling people in my profession who didn't really learn important things in school: "regression" should really be called "iid regression". For example: time-series econometrics is a whole field dedicated to try to squeeze iid information from stuff that's not iid. So you can use regression. But they're receptive to my message because they were repetitively warned that curve fitting was not regression analysis. Somehow this is lost in our "data" culture.
- thijsvandien 10y agoi.i.d.: independent and identically distributed
- ekianjo 10y ago> to try to squeeze iid information from stuff that's not iid. So you can use regression. I'm not sure I understand your last sentence here - so you can, or you can't use regression for non iid stuff? It would seem that you cannot.
- pyromine 10y agoThere is more than certainly forms of regressions that can work on heteroskedastic errors, i.e. non-iid (can be clustering, autoregressive, or more), but part of the general ordinary least squares assumptions and interpretations is iid. errors. If you do not have iid. errors, any interpretations of the model will be skewed. There are however other heteroskedasticaly robust interpretations of regressions.
- felixpl 10y agoWeighted least squares should do the trick, i.e. scale your data with the inverse covariance matrix