4 ms·
It kinda can't be any other way though. What are models? They're a way to describe the data. So this is where a load of philosophical stuff like Occam's Razor
by lordnacho 8y ago
It kinda can't be any other way though.
What are models? They're a way to describe the data. So this is where a load of philosophical stuff like Occam's Razor comes in and favours things like having fewer degrees of freedom, lower errors, etc.
What is data? It's what makes one model a more likely explanation than another.
You can make up any number of ways to describe some phenomenon, but without data there's no way to tell which of them is better. Or rather, you will fall back on some model with fewer specifics, because of those considerations we mentioned earlier.
So getting smarter (ie more complex) with models can't help on its own.
- darawk 8y agoI'm not sure i'm understanding you, or i'm not sure I agree. Humans, for instance, can learn from substantially less data than (most) machine learning models, if the data is presented in a way that we're optimized for (e.g. human facial recognition). If we can devise ML algorithms capable of generalizing from substantially smaller datasets, while data will still be relevant, it will be a less important bottleneck in the process than it is now.
- joe_the_user 8y agoHuman beings can be smart in using a fairly small set of data. Humans have an enormous store of data to work with but when confronted with a bit of new data on a new subject, they can sometimes do very well. This is a result of the ML paradigm essentially starting new with whatever data set it is trying to "solve" but overall, the paradigm doesn't have to be that.
- PeterisP 8y agoIMHO pretty much every example I've seen for how humans can get good results with using a fairly small set of data actually involves successful generalization / transfer of "generic life experience" or "generic audiovisual processing" to that new problem instead of actual learning from limited data. We know how to do that in ML, in general - we can do transfer learning from few examples quite well if the underlying generic data is good enough. However, we currently don't have good enough underlying generic data to match the years of generic life data that any human kid has accumulated. A human learning to play an Atari game that involves an agent jumping over a pit has to learn only the mechanics of that game, and that can be done in minutes. Learning that game from scratch, on the other hand, requires also learning interpreting vision and the whole concept that the world has objects that may move around - which takes months of learning for human brain. So comparing sample efficiency is an apples to elephants comparison if we disregard the ability to reuse/transfer knowledge from related tasks that all humans learn during childhood.