4 ms·

I think the point about language being a model of reality was interesting. I have an MA in linguistics including some NLP from about a decade ago and was lookin

by rvense 6y ago

I think the point about language being a model of reality was interesting. I have an MA in linguistics including some NLP from about a decade ago and was looking at a career in academic NLP. I ultimately left to become a programmer because (of life circumstances and the fact that) I didn't see much of a future for the field, precisely because it was ignoring the (to me) obvious issues of written language bias, ignorance of multi-modality and situatedness etc. that are brought up in this post. All of these results are very interesting, but I'm not really feeling like we've been proved wrong yet. There is a big question of scalability here, at least as far as the goal of AGI goes, which the author also admits: > Of course everyday language stands in a woolier relation to sheep, pine cones, desire and quarks than the formal language of chess moves stands in relation to chess moves, and the patterns are far more complex. Modality, uncertainty, vagueness and other complexities enter but the isomorphism between world and language is there, even if inexact. This woolly relation between language and reality is well-known. It has been studied in various ways in linguistics and the philosophy of language, for instance by Frege and not least Foucault and everything after. I also think many modern linguistic schools take a very different view of "uncertainty and vagueness" than I sense in the author here, but they are obviously writing for non-specialist audience and trying not to dwell on this subject here. My point is, when making and evaluating these NLP methods and the tools they are used to construct, it is extremely important to understand that language models social realities rather than any single physical one. It seems to me all too easy, coming from formal grammar or pure stats or computer science, to rush into these things with naive assumptions about what words are or how they mean things to people. I dread to think what will happen if we base our future society on tools made in that way.

skybrian 6y ago
These tools seem to be getting pretty good at fiction. In particular, playing around with AI Dungeon, it doesn't believe anything, or alternately you could say it believes everything at once. It's similar to a library that contains books by different authors, some fact, some fiction. Contradictions don't matter. Only local consistency matters, and even then, not that much. Unfortunately, many people want to believe that they are being understood. But, on the bright side, this stuff seems artistically useful? Entertainment is a big business.
- rvense 6y ago
  > many people want to believe that they are being understood Not only that, they also want to understand. I read some computer poetry a few years ago that worked. The computer did not intend any meaning, but I found it anyway. And of course I did. This is how language works. We assume that people (and by extension, the texts they write) follow a set of rules which structures how we engage with them in a successful manner. Paul Grice formulated this as the cooperative principle in his study of everyday language, but I believe it is the exact thing that is at play when people meet NLP: https://en.wikipedia.org/wiki/Cooperative_principle https://en.wikipedia.org/wiki/Cooperative_principle
joe_the_user 6y ago
My point is, when making and evaluating these NLP methods and the tools they are used to construct, it is extremely important to understand that language models social realities rather than any single physical one. I'd claim actual involves language both a general shared reality and quite concrete and specific discussions of single physical and logic facts/models. Some portion of language certainl looks mostly like a stream of associations. But within it is also are references to physical reality and world model and the two are complexly intertwined (making logical sense is akin to a "parity check" - you can for a while without it but then you have to look at the whole to get it). I believe one can see this in a GPT paragraph, where the first two sentences seem intelligence, well written but the third sentence contradicts the first two sentences sufficiently one's mind isn't sure "what's being said" (and even here, our "need for logic" might be loose enough that we only notice the "senselessness" after the third logical error).
- rvense 6y ago
  In human language I think physical reality is always a few layers out. Language is social, first and foremost, and naming something is not neutral. We can hardly refer to single objects directly, we mostly do it through their class membership, which always, always include a whole range of associations, reductions, metaphor, etc. that are cultural.
  - joe_the_user 6y ago
    In human language I think physical reality is always a few layers out. Yes, the point is you can neglect physical (and logic) reality for a while in a stream of sentences. But not forever and that's where current NLP's output has it's limits. Just a simple level, a stream of glittering adjectives can be added to a thing and just add up to "desirable" unless those adjectives "go over threshold" and contradict each other and then the description can get tagged by the brain as a bit senseless.
    - stosto88 6y ago
      Are you a NLP bot?
fernly 6y ago
> it is extremely important to understand that language models social realities rather than any single physical one... Oh gosh yes. Gorge the ML tool with the right streams of text and you'd have a glib anti-vax, white supremacist, flat-earth generator pouring out endless logorrheic twaddle. Oh wait... could that already... it would explain so much...
hinkley 6y ago
It's probably worse than that. It's a model of reality as filtered through the human brain, with all of its neuroses, most of which we have only begun to model. Aren't we stuck on things like sarcasm? How are you going to model everything from confirmation bias to undisclosed PTSD, in order to have a prayer of filtering the noise from actionable information?

by rvense 6y ago

skybrian 6y ago

These tools seem to be getting pretty good at fiction. In particular, playing around with AI Dungeon, it doesn't believe anything, or alternately you could say it believes everything at once. It's similar to a library that contains books by different authors, some fact, some fiction. Contradictions don't matter. Only local consistency matters, and even then, not that much. Unfortunately, many people want to believe that they are being understood. But, on the bright side, this stuff seems artistically useful? Entertainment is a big business.

rvense 6y ago
> many people want to believe that they are being understood Not only that, they also want to understand. I read some computer poetry a few years ago that worked. The computer did not intend any meaning, but I found it anyway. And of course I did. This is how language works. We assume that people (and by extension, the texts they write) follow a set of rules which structures how we engage with them in a successful manner. Paul Grice formulated this as the cooperative principle in his study of everyday language, but I believe it is the exact thing that is at play when people meet NLP: https://en.wikipedia.org/wiki/Cooperative_principle https://en.wikipedia.org/wiki/Cooperative_principle

joe_the_user 6y ago

My point is, when making and evaluating these NLP methods and the tools they are used to construct, it is extremely important to understand that language models social realities rather than any single physical one. I'd claim actual involves language both a general shared reality and quite concrete and specific discussions of single physical and logic facts/models. Some portion of language certainl looks mostly like a stream of associations. But within it is also are references to physical reality and world model and the two are complexly intertwined (making logical sense is akin to a "parity check" - you can for a while without it but then you have to look at the whole to get it). I believe one can see this in a GPT paragraph, where the first two sentences seem intelligence, well written but the third sentence contradicts the first two sentences sufficiently one's mind isn't sure "what's being said" (and even here, our "need for logic" might be loose enough that we only notice the "senselessness" after the third logical error).

rvense 6y ago
In human language I think physical reality is always a few layers out. Language is social, first and foremost, and naming something is not neutral. We can hardly refer to single objects directly, we mostly do it through their class membership, which always, always include a whole range of associations, reductions, metaphor, etc. that are cultural.
- joe_the_user 6y ago
  In human language I think physical reality is always a few layers out. Yes, the point is you can neglect physical (and logic) reality for a while in a stream of sentences. But not forever and that's where current NLP's output has it's limits. Just a simple level, a stream of glittering adjectives can be added to a thing and just add up to "desirable" unless those adjectives "go over threshold" and contradict each other and then the description can get tagged by the brain as a bit senseless.
  - stosto88 6y ago
    Are you a NLP bot?

fernly 6y ago

> it is extremely important to understand that language models social realities rather than any single physical one... Oh gosh yes. Gorge the ML tool with the right streams of text and you'd have a glib anti-vax, white supremacist, flat-earth generator pouring out endless logorrheic twaddle. Oh wait... could that already... it would explain so much...

hinkley 6y ago

It's probably worse than that. It's a model of reality as filtered through the human brain, with all of its neuroses, most of which we have only begun to model. Aren't we stuck on things like sarcasm? How are you going to model everything from confirmation bias to undisclosed PTSD, in order to have a prayer of filtering the noise from actionable information?