4 ms·
"Just stir the pile until [the answers] start looking right" is actually a pretty decent description of gradient descent.
by jey 9y ago
"Just stir the pile until [the answers] start looking right" is actually a pretty decent description of gradient descent.
- denzil_correa 9y agoI'm not so sure. Gradient Descent refers to reaching the best solution available. Xkcd seems to be referring to more like the Bonferroni principle where you are looking for patterns without a hypothesis and post facto justifying it.
- ardivekar 9y agoAKA half the work done in Machine Learning nowadays.
- YeGoblynQueenne 9y agoI think the point is about how data manipulation is about 80% of machine learning work. If an algorithm is giving you crap results for some data set, once you've fiddled with its hyperparameters through cv and so on there's not much you can do besides data manipulations like PCA, ICA and the like, to try and get a better result. Most algorithms work pretty badly with raw data. I'm guessing that, in the sciences, that is a big no-no. Imagine if doctors, seeing that a new drug being trialled is failing to cure a disease, simply started chucking out the sick subjects until all the ones that were left where healthy, declaring the sick ones to be "noise" and the trial a success. Somehow, I don't think that would fly...
- denzil_correa 9y ago> I think the point is about how data manipulation is about 80% of machine learning work. If an algorithm is giving you crap results for some data set, once you've fiddled with its hyperparameters through cv and so on there's not much you can do besides data manipulations like PCA, ICA and the like, to try and get a better result. Data manipulation is different from data transformation. Manipulation changes the nature of data, transformation does not change the nature of data. PCA is data transformation, not data manipulation.
- YeGoblynQueenne 9y agoTell me what you mean by "the nature of data" and I'll tell you if I agree with your definition. I don't see why the dimensionality of a dataset is not its "nature" for example.
- randomsearch 9y agoThis is confusing "cherry picking" (selecting some data and discarding other data) with "data transformation". Transformations are applied for perfectly good reasons, such as dimensionality reduction, or to enable the subsequent application of certain statistical tests that rely on assumptions not valid for the data in its original form.
- randomsearch 9y agoTo be precise: the best solution reachable from a given starting point, which may be a local minima rather than a global minima.
- dmitriy_ko 9y agoWith gradient descent you are walking in the direction of negative gradient. Stirring the pile implies that you are walking randomly from point to point.
- andai 9y agoSo more like a genetic algorithm then?
- randomsearch 9y agoGAs also exploit structure in the search space. To see this most clearly, read up on estimation of distribution algorithms (EDAs), which generalise the ways in which GAs work.
- TeMPOraL 9y agoNo, by stirring you're just making the pile to move, and things descend down the gradient of gravity. :).
- steanne 9y agoThe little things moreso. Brazil nut effect!
- yakult 9y agoWhen you stir, you're stirring in a pattern, probably a circular one. Plus you're probably not paying too much attention to the shape of the pile, so it's gradient-free. It's pattern search. Except it's only approximately a circular pattern. There are random variations. Stochastic pattern search.
- james_a_craig 9y agoIt's clearly simulated annealing. You stir less as you get tired. :)
- randomsearch 9y agoSimulated annealing also exploits gradient information, so it is not equivalent to random search, although it is a stochastic algorithm.
- deleted 9y ago[deleted]
- jrdt 9y agoIt actually seems almost like the definition of RANSAC. https://en.wikipedia.org/wiki/Random_sample_consensus https://en.wikipedia.org/wiki/Random_sample_consensus
- nabla9 9y ago(iterative) stochastic gradient descent to be specific. Data is stirred for each pass to prevent cycles.
- netheril96 9y agoIt is more like tuning the hyper-parameters, which is guided by anything but logic.
- randomsearch 9y agoGradient descent is not random search, so this is an inaccurate description. It works by exploiting the gradient in a structured landscape of solutions.