9 ms·

It’s always fascinating to apply data/results like from this paper to help evaluate the hypothesis that machine learning/AI is mostly just a rough “lookup table

It’s always fascinating to apply data/results like from this paper to help evaluate the hypothesis that machine learning/AI is mostly just a rough “lookup table” or memorization.

6gvONxR4sf7o 6y ago
Anything trained by gradient descent is in some sense a weighted combination lookup table (see domingos’s recent paper on everything being kernel machines, I’m on my phone so won’t try to find it). The crucial bit is finding an appropriate weighting. If you look up the nearest training data point and just use it to predict, you assume a notion of “nearest.” Getting that right is the trick. So even if it all is mathematically equivalent to approximate lookup and approximate memorization, that doesn’t mean it’s “just” that.
- antipaul 6y ago
  Thanks for the reference. Here it is: “We show, however, that deep networks learned by the standard gradient descent algorithm are in fact mathematically approximately equivalent to kernel machines, a learning method that simply memorizes the data and uses it directly for prediction via a similarity function (the kernel)“ https://arxiv.org/abs/2012.00152 https://arxiv.org/abs/2012.00152
jacobwilliamroy 6y ago
It kind of is. The first self-learning algorithm I was taught adjusted a single variable in order to aproximate a linear function. Now, being continuous and unbounded, it is mathematically impossible to memoize such a function, however you can memoize the function calls as they happen. The results would be indistinguishable. However real machine learning tries to approximate functions in n-dimensions and that is really, really, really hard to do. Currently no one really has a lookup table. There are some inputs where the error level is acceptably low, and others where the model just isn't optimized enough and the errors are ridiculous. The only question that remains to be answered, is whether any of these high-dimensional functions can actually be found by machine learning, or if we are just stuck with these endless approximations. Also I suppose you could ask if any such functions actually exist; maybe certain phenomena are just pure chaos.

It’s always fascinating to apply data/results like from this paper to help evaluate the hypothesis that machine learning/AI is mostly just a rough “lookup table” or memorization.

6gvONxR4sf7o 6y ago

Anything trained by gradient descent is in some sense a weighted combination lookup table (see domingos’s recent paper on everything being kernel machines, I’m on my phone so won’t try to find it). The crucial bit is finding an appropriate weighting. If you look up the nearest training data point and just use it to predict, you assume a notion of “nearest.” Getting that right is the trick. So even if it all is mathematically equivalent to approximate lookup and approximate memorization, that doesn’t mean it’s “just” that.

antipaul 6y ago
Thanks for the reference. Here it is: “We show, however, that deep networks learned by the standard gradient descent algorithm are in fact mathematically approximately equivalent to kernel machines, a learning method that simply memorizes the data and uses it directly for prediction via a similarity function (the kernel)“ https://arxiv.org/abs/2012.00152 https://arxiv.org/abs/2012.00152

jacobwilliamroy 6y ago

It kind of is. The first self-learning algorithm I was taught adjusted a single variable in order to aproximate a linear function. Now, being continuous and unbounded, it is mathematically impossible to memoize such a function, however you can memoize the function calls as they happen. The results would be indistinguishable. However real machine learning tries to approximate functions in n-dimensions and that is really, really, really hard to do. Currently no one really has a lookup table. There are some inputs where the error level is acceptably low, and others where the model just isn't optimized enough and the errors are ridiculous. The only question that remains to be answered, is whether any of these high-dimensional functions can actually be found by machine learning, or if we are just stuck with these endless approximations. Also I suppose you could ask if any such functions actually exist; maybe certain phenomena are just pure chaos.