4 ms·

Great article! It is very true that NLP is amongst the most lagging divisions within machine learning. Mainly because text content is very unstructured everywhe

Great article! It is very true that NLP is amongst the most lagging divisions within machine learning. Mainly because text content is very unstructured everywhere, let alone working consistently for different languages. It is fascinating to see how things got better over the last couple of years though!

misterman0 8y ago
"Mainly because text content is very unstructured everywhere" Encoded into this somewhat unstructured data is the very thing NLP is after: the meaning. If you cannot teach a machine a language by talking to it with "unstructured" language, how are you supposed to make it both understand it, grok it, and speak it? We have no trouble teaching small kids about history and math using untructured language. Language is maybe unstructured, but the internets has lots of it, so to me, the state of NLP today is a dissapointment. I'm convinced though that general AI will be achieved through NLP. I mean, most of what I know I have either been told or I have read it somewhere. And my parent's didn't use annotated data much.
- GolDDranks 8y ago
  As a (former) linguist, I'm baffled about talking about language as "unstructured". Sure it isn't _rigidly_ structured, but "un"? There's a lot of structure into it, it's just so complex (and so connected to extralinguistic structures) that our ML models don't grok it!
  - killjoywashere 8y ago
    Is a (never) linguist, I think the edge case successes demonstrate how far we really have to go in ML generally. Driving is an entirely engineered phenomenon. Down to the species level of things that might run out in front of it, every atmospheric perturbation, we can define every event that's going to affect a car, and they have been designed over more than 100 years to deal with them. Medical imaging: another area where highly trained humans have thought rigorously (for about the same amount of time) about what this all means. What ML is lacking is the data and labels a baby bootstraps from into the "common world". How many ML models have been training on the taste of breast milk, smell of mother, the smell of dirt, the upward view of everything (think about how much time babies, since the stone age, have spent laid on their backs, looking up. These things have to be segregated in a fairly unsupervised way using little more than reflexes (cry, suck, fencer, grasp, etc) for a while: smell of mom sometimes comes with warm milk, but not always. Associating this warm body with the smell, not just food. Sound of mom doesn't always come with smell of mom.
  - denzil_correa 8y ago
    > I'm baffled about talking about language as "unstructured". It's a (imprecise) term in Computer Science which may not refer to the same thing you are thinking about. Hence, the confusion. https://en.wikipedia.org/wiki/Unstructured_data https://en.wikipedia.org/wiki/Unstructured_data
  - imh 8y ago
    Typically "unstructured" is opposed to tabular data, where rows and columns are a natural representation.
- YeGoblynQueenne 8y ago
  >> Encoded into this somewhat unstructured data is the very thing NLP is after: the meaning. The problem is that we don't know how meaning is encoded into language utterances and we don't know how meaning is represented, once it's decoded from those utterances (i.e. in our minds). It's very unlikely that these elements, the encoding process that turns meaning to language and back again and the reprsentation of meaning, are carried around in language utterances themselves [1]. And yet- we keep trying to figure out both, the encoding process and the representation, just by looking at the encoded utterances. Imagine having a compressed string and trying to figure out a) the compression algorithm and b) the uncompressed string, withouth having ever seen examples of either. That's what natural language understanding from raw, unstructured text is like. ______________ [1] Edit: Is that even possible? Is it possible to send an encoded message including its own encoding procedure, so that the message can be decoded even when the procedure is not known beforehand? Wouldn't that require that the procedure is somehow possible to decode independently of the message? Is there another way?
  - misterman0 8y ago
    You make some good points. >> we keep trying to figure out both, the encoding process and the representation, just by looking at the encoded utterances I think what we're trying to do or what I'm trying to do at least is to find a model that would produce the same interpretations of an utterance as a human would. I don't see why we couldn't find such a model pretty soon, given the vast amount of data out there, however unstructured it might be.
    - YeGoblynQueenne 8y ago
      I don't think what's stopping us from modelling meaning is the lack of structure in textual data. I think it's the fact that text is not meaning. It somehow encodes meaning, but we don't know how and we don't know what a "unit of meaning" is supposed to look like. When you say that we have "vast amounts of data" you mean that we have vast amounts of text- but by modelling text we will not model meaning, we will only model text. We have not observed "meaning" and we have no examples of meaning turning into text and back again. If I may be allowed the simile, training on text to model meaning is a bit like looking at a screen hiding a figure of a person and trying to learn something about the person from the screen they're hiding behind. You can't model phenomena you can't observe.
- visarga 8y ago
  > I'm convinced though that general AI will be achieved through NLP. I think you're wrong. General AI is much more related to reinforcement learning and embodiment. Language without embodiment is ungrounded.
- imh 8y ago
  >We have no trouble teaching small kids about history and math using untructured language. Crucially it's unstructured data, plus interaction. You could use it in a sentence hundreds of times and hope the kid learns about cats, but it's much easier to just let the kid play with a cat.
gok 8y ago
It seems more structured to me than acoustic or visual data...
romaniv 8y ago
I disagree. We have dictionaries, URLs, Wikipedia categories and concept taxonomies. There is a lot of structured data. It's just that "external" structures don't seem to be all that useful when analyzing internal structure of a text.

misterman0 8y ago

"Mainly because text content is very unstructured everywhere" Encoded into this somewhat unstructured data is the very thing NLP is after: the meaning. If you cannot teach a machine a language by talking to it with "unstructured" language, how are you supposed to make it both understand it, grok it, and speak it? We have no trouble teaching small kids about history and math using untructured language. Language is maybe unstructured, but the internets has lots of it, so to me, the state of NLP today is a dissapointment. I'm convinced though that general AI will be achieved through NLP. I mean, most of what I know I have either been told or I have read it somewhere. And my parent's didn't use annotated data much.

GolDDranks 8y ago
As a (former) linguist, I'm baffled about talking about language as "unstructured". Sure it isn't _rigidly_ structured, but "un"? There's a lot of structure into it, it's just so complex (and so connected to extralinguistic structures) that our ML models don't grok it!
- killjoywashere 8y ago
  Is a (never) linguist, I think the edge case successes demonstrate how far we really have to go in ML generally. Driving is an entirely engineered phenomenon. Down to the species level of things that might run out in front of it, every atmospheric perturbation, we can define every event that's going to affect a car, and they have been designed over more than 100 years to deal with them. Medical imaging: another area where highly trained humans have thought rigorously (for about the same amount of time) about what this all means. What ML is lacking is the data and labels a baby bootstraps from into the "common world". How many ML models have been training on the taste of breast milk, smell of mother, the smell of dirt, the upward view of everything (think about how much time babies, since the stone age, have spent laid on their backs, looking up. These things have to be segregated in a fairly unsupervised way using little more than reflexes (cry, suck, fencer, grasp, etc) for a while: smell of mom sometimes comes with warm milk, but not always. Associating this warm body with the smell, not just food. Sound of mom doesn't always come with smell of mom.
- denzil_correa 8y ago
  > I'm baffled about talking about language as "unstructured". It's a (imprecise) term in Computer Science which may not refer to the same thing you are thinking about. Hence, the confusion. https://en.wikipedia.org/wiki/Unstructured_data https://en.wikipedia.org/wiki/Unstructured_data
- imh 8y ago
  Typically "unstructured" is opposed to tabular data, where rows and columns are a natural representation.
YeGoblynQueenne 8y ago
>> Encoded into this somewhat unstructured data is the very thing NLP is after: the meaning. The problem is that we don't know how meaning is encoded into language utterances and we don't know how meaning is represented, once it's decoded from those utterances (i.e. in our minds). It's very unlikely that these elements, the encoding process that turns meaning to language and back again and the reprsentation of meaning, are carried around in language utterances themselves [1]. And yet- we keep trying to figure out both, the encoding process and the representation, just by looking at the encoded utterances. Imagine having a compressed string and trying to figure out a) the compression algorithm and b) the uncompressed string, withouth having ever seen examples of either. That's what natural language understanding from raw, unstructured text is like. ______________ [1] Edit: Is that even possible? Is it possible to send an encoded message including its own encoding procedure, so that the message can be decoded even when the procedure is not known beforehand? Wouldn't that require that the procedure is somehow possible to decode independently of the message? Is there another way?
- misterman0 8y ago
  You make some good points. >> we keep trying to figure out both, the encoding process and the representation, just by looking at the encoded utterances I think what we're trying to do or what I'm trying to do at least is to find a model that would produce the same interpretations of an utterance as a human would. I don't see why we couldn't find such a model pretty soon, given the vast amount of data out there, however unstructured it might be.
  - YeGoblynQueenne 8y ago
    I don't think what's stopping us from modelling meaning is the lack of structure in textual data. I think it's the fact that text is not meaning. It somehow encodes meaning, but we don't know how and we don't know what a "unit of meaning" is supposed to look like. When you say that we have "vast amounts of data" you mean that we have vast amounts of text- but by modelling text we will not model meaning, we will only model text. We have not observed "meaning" and we have no examples of meaning turning into text and back again. If I may be allowed the simile, training on text to model meaning is a bit like looking at a screen hiding a figure of a person and trying to learn something about the person from the screen they're hiding behind. You can't model phenomena you can't observe.
visarga 8y ago
> I'm convinced though that general AI will be achieved through NLP. I think you're wrong. General AI is much more related to reinforcement learning and embodiment. Language without embodiment is ungrounded.
imh 8y ago
>We have no trouble teaching small kids about history and math using untructured language. Crucially it's unstructured data, plus interaction. You could use it in a sentence hundreds of times and hope the kid learns about cats, but it's much easier to just let the kid play with a cat.

gok 8y ago

It seems more structured to me than acoustic or visual data...

romaniv 8y ago

I disagree. We have dictionaries, URLs, Wikipedia categories and concept taxonomies. There is a lot of structured data. It's just that "external" structures don't seem to be all that useful when analyzing internal structure of a text.