3 ms·

How would you model embodiment and embodied experience?

s1mplicissimus 2y ago
Is there anything except sensory input that you assume part of the embodied experience? What would that be? Apart from that, I'm afraid that at this point research on sensory input apart from audio and visual needs much more advancement. For example, it's not clear to me what kind of data structure would be a good fit for olfactory or sensory training data
- tomohelix 2y ago
  As mentioned above, olfactory data can be just chemical fingerprints. Mass spectrometers already do this and provide very distinct signals for every chemical component. Touch and such can have some approximation done through various sensors like temperature, force, humidity, electromagnetic, etc.
  - s1mplicissimus 2y ago
    Sure you can punch in a chemical fingerprint for say the smell of a specific type of rose. Maybe it doesn't matter for the learning process that in an equivalent human experience it was preceeded by someone making you a compliment a couple minutes earlier, or that it was combined with all the other chemical fingerprints present at the moment, like maybe it just rained shortly before and there's a slight smell of wet earth in the air or someone smoking a cigarette walked by and there's minimal leftovers of that, or the window wasn't opened for a couple hours and everything has a slight tint of "used air" to it, which might add a factor of dampened learning, which might be necessary for the specific learning process to happen slow enough to sink in properly etc... Don't get me wrong I would be curious to see such research done to see whether it would improve anything above the stochastic parrot level - it's just going to take a while to figure out what is even relevant
    - tomohelix 2y ago
      I think those factors you mentioned are important but ultimately additional context to the main data, i.e. "rose smell". They certainly can add additional meanings and alter how the main data is processed, but they are just "context" added on, just like how the word "lie" is very context dependent and by itself it is nigh impossible to know what it means. Is it a verb? noun? Which verb, lying to someone or lie on a couch? But an LLM has no problem at all deciphering and processing and most importantly, responding meaningfully to all the ways we can use or encounter the word "lie". I contend that if a model large enough is trained on enough data, the concepts will automatically blend and explain each other sufficiently, or at least enough to cover your example and those similar to it.
tomohelix 2y ago
Sight, sound are quite obvious. Taste and olfactory are matters of chemical compositions. It will take an incredible effort but something similar to a mass spectrometer can be used to detect every taste and smell we can think of and beyond. How fast and how efficient they can be is probably the main challenge. Touch is difficult. We don't even know fully why or how does an itch "work". But force, temperature, atmospheric, humidity sensors, etc are widely available. They can provide a crude approximation, imo. Just off the top of my head. I am sure smarter people can come up with much more suitable ways to "embody" a machine learning model.
- cratermoon 2y ago
  I suggest, before throwing around words like "obvious" and "crude approximation", reading some Martin Heidegger, Hubert Dreyfus, or Joseph Weizenbaum. Half-baked attempts at mechanistic and reductionist implementations of embodiment are a dime a dozen.
  - selfhoster11 2y ago
    I mean, sight and sound in large language models are obvious by now, in that rendering them into a token-based representation that an LLM can manipulate and learn from (however well it actually succeeds in picking up on the patterns - nothing about that is guaranteed, but the information is there theoretically) is currently a conceptually solved problem that will be gradually improved upon. If reproducing the artifacts and failure modes of human modes of interpretation of this physical data (say, yanny/laurel, or optical illusions, or persistence of vision phenomena) is deemed important, that's another matter. If all that's required is a black-box understanding that is idiosyncratic to LLMs in particular, but where it's functionally good enough to be used as sight and hearing, then I don't see see why it can't be called "solved" for most intents and purposes in six months' time. I guess it boils down to this: do you want "sight" to mean "machine" sight or "human" sight. The latter is a hard problem, but I'd prefer to let machines be machines. It's less work, and gives us a brand-new cognitive lens to analyse what we observe, a truly alien perspective that might prove useful.
    - 2y ago

Is there anything except sensory input that you assume part of the embodied experience? What would that be? Apart from that, I'm afraid that at this point research on sensory input apart from audio and visual needs much more advancement. For example, it's not clear to me what kind of data structure would be a good fit for olfactory or sensory training data

tomohelix 2y ago
As mentioned above, olfactory data can be just chemical fingerprints. Mass spectrometers already do this and provide very distinct signals for every chemical component. Touch and such can have some approximation done through various sensors like temperature, force, humidity, electromagnetic, etc.
- s1mplicissimus 2y ago
  Sure you can punch in a chemical fingerprint for say the smell of a specific type of rose. Maybe it doesn't matter for the learning process that in an equivalent human experience it was preceeded by someone making you a compliment a couple minutes earlier, or that it was combined with all the other chemical fingerprints present at the moment, like maybe it just rained shortly before and there's a slight smell of wet earth in the air or someone smoking a cigarette walked by and there's minimal leftovers of that, or the window wasn't opened for a couple hours and everything has a slight tint of "used air" to it, which might add a factor of dampened learning, which might be necessary for the specific learning process to happen slow enough to sink in properly etc... Don't get me wrong I would be curious to see such research done to see whether it would improve anything above the stochastic parrot level - it's just going to take a while to figure out what is even relevant
  - tomohelix 2y ago
    I think those factors you mentioned are important but ultimately additional context to the main data, i.e. "rose smell". They certainly can add additional meanings and alter how the main data is processed, but they are just "context" added on, just like how the word "lie" is very context dependent and by itself it is nigh impossible to know what it means. Is it a verb? noun? Which verb, lying to someone or lie on a couch? But an LLM has no problem at all deciphering and processing and most importantly, responding meaningfully to all the ways we can use or encounter the word "lie". I contend that if a model large enough is trained on enough data, the concepts will automatically blend and explain each other sufficiently, or at least enough to cover your example and those similar to it.

tomohelix 2y ago

Sight, sound are quite obvious. Taste and olfactory are matters of chemical compositions. It will take an incredible effort but something similar to a mass spectrometer can be used to detect every taste and smell we can think of and beyond. How fast and how efficient they can be is probably the main challenge. Touch is difficult. We don't even know fully why or how does an itch "work". But force, temperature, atmospheric, humidity sensors, etc are widely available. They can provide a crude approximation, imo. Just off the top of my head. I am sure smarter people can come up with much more suitable ways to "embody" a machine learning model.

cratermoon 2y ago
I suggest, before throwing around words like "obvious" and "crude approximation", reading some Martin Heidegger, Hubert Dreyfus, or Joseph Weizenbaum. Half-baked attempts at mechanistic and reductionist implementations of embodiment are a dime a dozen.
- selfhoster11 2y ago
  I mean, sight and sound in large language models are obvious by now, in that rendering them into a token-based representation that an LLM can manipulate and learn from (however well it actually succeeds in picking up on the patterns - nothing about that is guaranteed, but the information is there theoretically) is currently a conceptually solved problem that will be gradually improved upon. If reproducing the artifacts and failure modes of human modes of interpretation of this physical data (say, yanny/laurel, or optical illusions, or persistence of vision phenomena) is deemed important, that's another matter. If all that's required is a black-box understanding that is idiosyncratic to LLMs in particular, but where it's functionally good enough to be used as sight and hearing, then I don't see see why it can't be called "solved" for most intents and purposes in six months' time. I guess it boils down to this: do you want "sight" to mean "machine" sight or "human" sight. The latter is a hard problem, but I'd prefer to let machines be machines. It's less work, and gives us a brand-new cognitive lens to analyse what we observe, a truly alien perspective that might prove useful.
  - 2y ago