3 ms·

I’ve been wondering about the limits of data-centric approach – there seems to be this implicit notion that more data equals better performing ML or AI. I think

I’ve been wondering about the limits of data-centric approach – there seems to be this implicit notion that more data equals better performing ML or AI. I think it would be interesting to imagine a point of diminishing return on additional data if we consider that our ability to perceive is probably largely based on two parts - sensory input and knowledge. Note that I’m making an explicit distinction here on the difference between data and knowledge. For instance, an English speaker and a non-English speaker may listen to someone speaking English and while the auditory signals received by both are the same, the meaning of the speech will only be perceived by the English speaker. When we’re learning a new language, it’s this ‘knowledge’ aspect that we’re enhancing in our brain, however that is encoded. This knowledge part is what allows us to see what’s not there but should be (e.g. the curious incident of the dog in the night) and when the data is inconsistent (e.g. all the nuclear close calls). I’m really not sure how this ‘knowledge’ part will be approached by the AI community but feel like we’re already close to having squeezed out as much as we can from just the data side of things. Somewhat related, we have a saying in Korean – ‘you see as much as you know’.

Longwelwind 5y ago
Can't you consider that knowledge is a function of previous data? In your example, the 2 individuals actually didn't receive the same amount of data because the English speakers received data previously that allowed him to build some kind of "knowledge" that allows him to solve specific related tasks (understanding a spoken sentence). This would be the equivalent of transfer learning where "knowledge" is a model trained on previous, more general, data.
- mjburgess 5y ago
  Nope, it's never a function of data -- because data is always ambiguous. It is never possible just to infer the conceptual model of the data from the data alone. Animals solve this problem by having bodies and moving around. It is that we take the bent stick out of the water which allows us to impart a theory to the "data" we receive... a theory implicit in our actions. Since we are causally active in the world, sequenced in time, and directly changing it -- our bodies enable us to resolve this problem. The motor system is the heart of intelligence, not the frontal lobe -- which is merely book-keeping and accounting for what our bodies are doing.
  - a_square_peg 5y ago
    Yep. Perhaps what I should have written is that much our knowledge is tacit in nature. We implicitly understand that things we can sense are limited projections of the real world and are able to derive mental model of what that reality might be more or less. Chomsky also talks about this notion of separating mind and body as an incorrect approach to understanding human intelligence which I think aligns with your view. The notion that 'all we need is data and data is all there is' seems to summarize a lot of ML sentiments.
mdp2021 5y ago
> more data equals It does in general, but what is elaborated and how? Structuring patterns is not the same as "knowledge" (there are missing subsystems), and that fed data is not fed efficiently, with ideal efficiency - compare with the realm in which "told one notion you acquire it" (this while CS is one of the disciplines focusing on optimization, so it would be a crucial point).
machiaweliczny 5y ago
I have a feeling that too much knowledge might slow learning process as it's harder to spot/test observe steepest gradient. At least that's how it feels intuitively from human PoC. From computation that would be just little more computation but I guess would mean slower convergence also. Taking math as more extreme example it's hard to understand something complex unless you understand basic algebra. Anyone knows if this might be true mathematically speaking? Does order of data matters?

Longwelwind 5y ago

Can't you consider that knowledge is a function of previous data? In your example, the 2 individuals actually didn't receive the same amount of data because the English speakers received data previously that allowed him to build some kind of "knowledge" that allows him to solve specific related tasks (understanding a spoken sentence). This would be the equivalent of transfer learning where "knowledge" is a model trained on previous, more general, data.

mjburgess 5y ago
Nope, it's never a function of data -- because data is always ambiguous. It is never possible just to infer the conceptual model of the data from the data alone. Animals solve this problem by having bodies and moving around. It is that we take the bent stick out of the water which allows us to impart a theory to the "data" we receive... a theory implicit in our actions. Since we are causally active in the world, sequenced in time, and directly changing it -- our bodies enable us to resolve this problem. The motor system is the heart of intelligence, not the frontal lobe -- which is merely book-keeping and accounting for what our bodies are doing.
- a_square_peg 5y ago
  Yep. Perhaps what I should have written is that much our knowledge is tacit in nature. We implicitly understand that things we can sense are limited projections of the real world and are able to derive mental model of what that reality might be more or less. Chomsky also talks about this notion of separating mind and body as an incorrect approach to understanding human intelligence which I think aligns with your view. The notion that 'all we need is data and data is all there is' seems to summarize a lot of ML sentiments.

mdp2021 5y ago

> more data equals It does in general, but what is elaborated and how? Structuring patterns is not the same as "knowledge" (there are missing subsystems), and that fed data is not fed efficiently, with ideal efficiency - compare with the realm in which "told one notion you acquire it" (this while CS is one of the disciplines focusing on optimization, so it would be a crucial point).

machiaweliczny 5y ago

I have a feeling that too much knowledge might slow learning process as it's harder to spot/test observe steepest gradient. At least that's how it feels intuitively from human PoC. From computation that would be just little more computation but I guess would mean slower convergence also. Taking math as more extreme example it's hard to understand something complex unless you understand basic algebra. Anyone knows if this might be true mathematically speaking? Does order of data matters?