4 ms·

My understanding is that ChatGPT (&co.) was not designed as, and is not intended to be, any sort of expert system, or knowledge representation system. The fact

by tjr 4y ago

My understanding is that ChatGPT (&co.) was not designed as, and is not intended to be, any sort of expert system, or knowledge representation system. The fact that it does as well as it does anyway is pretty amazing. But even so -- as you said, it's still dealing chiefly with the statistical probability of words/tokens, not with facts and truths. I really don't "trust" it in any meaningful way, even if it already has, and will continue to, prove itself useful. Anything it says must be vetted.

ChatGTP 4y ago
So do you find it alarming that people are trying to give such a system “agency” ?
- gwright 4y ago
  I do. I think the anthropomorphic language that people use to describe these systems is inaccurate and misleading. An Australian mayor has claimed that ChatGPT "defamed" him. The title of this article says that we should teach people that text generation tools "lie". Other articles suggest that ChatGPT "knows" things. It is extremely interesting to me how much milage can be gotten out of an LLM by observing patterns in text and generating similar patterns. It is extremely frustrating to me to see how easily people think that this is evidence of intelligence or knowledge or "agency" as you suggest.
  - simonw 4y ago
    I agree with you: convincing people that these systems do not have intelligence or agency is the second most important problem to solve. The most important problem is to ensure people understand that these systems cannot be trusted to tell them the truth. I'm OK with starting out with "these systems will lie to you", then following up with "but you do need to understand that they're not anthropomorphic etc" later on.
    - int_19h 4y ago
      You might have better success convincing people if you stop bundling it together as "intelligence and agency", for starters. I agree with the agency part and strongly disagree with intelligence, for one. In general, "intelligence" is so vaguely and informally defined that it's simply not something that can be reasonably argued objectively either way. (If what you really meant is humanlike intelligence, then I would agree with that claim, but it's a very different one.)
  - olalonde 4y ago
    I also disagree about some of the anthropomorphism (e.g. it doesn't intentionally "lie") but I'd say it passes the "duck test"[0] for knowing things and intelligence, to some degree. I would even go as far to say it has opinions although it seems OpenAI has gone out of their way to limit answers that could be interpreted as opinions. [0] https://en.wikipedia.org/wiki/Duck_test https://en.wikipedia.org/wiki/Duck_test
  - IIAOPSW 4y ago
    I disagree with your confident assertions about its "agency" and "intent" when there's no goal post for what those things mean in humans to begin with. Barring OpenAI's filters, if I asked it to participate as a party in a business negotiation determining a fair selling price for its writing services, I'm sure it could emulate that sort of character long enough to pass. And if it can successfully fake being a self interested actor working towards goals of its own material interest, whats the difference between faking it and actually having agency? At some point you have to acknowledge emergent agency and other properties as a possibility.
    - gwright 4y ago
      > I disagree with your confident assertions about its "agency" and "intent" when there's no goal post for what those things mean in humans to begin with. I'm not sure how confident I am! I think the success of LLMs are serving to challenge our understandings of cognition and intelligence in humans. I tend to agree with you that our understanding of agency, intent, intelligence, cognition, free will, and related ideas in humans is incomplete and so it is a challenge to think about how they might apply (or not) to LLMs. > whats the difference between faking it and actually having agency? Good question. The Chinese Room Argument is very close to this but framed as "understanding" vs "agency": https://plato.stanford.edu/entries/chinese-room/ https://plato.stanford.edu/entries/chinese-room/
- deleted 4y ago
  [deleted]
- te_chris 4y ago
  Yes. It’s very weird seeing the religiosity surrounding this whole thing.
- olalonde 4y ago
  What is alarming about it?
moffkalast 4y ago
Having used GPT 4 for a while now I would say I trust its factual accuracy more than the average human you'd talk to on the street. The sheer volume of things we make up on a daily basis through no malice of our own but bad memory and wrong associations is just astounding. That said, fact checking is still very much needed. Once someone figures out how to streamline and automate that process it'll be on Google's level of general reliability.
- crabmusket 4y ago
  It's interesting to imagine an "average human on the street" being put in the position that an LLM is in. If you asked someone on the street to summarise a page of React Native documentation for you, they'd probably look at you strangely and say "sorry I have a bus to catch, good luck!" But if you had someone in a room, sat them down with a pen and paper and instructions along the lines of: - answer the question on the page without diversion - do not ask for additional clarifying information - provide your answer in 3-5 paragraphs with a summary and didn't let them leave until they'd completed the task, then .. I'm sure the results would be "worse", and much more variable, than GPT output, while being conceptually similar. Of course, you don't ask a random person on the street to do this, probably because of social constraint, inconvenience, and the expectation that they will not do a better or faster job than you will yourself. But if you were stuck with a stranger, began to converse, and asked them e.g. "should I get a divorce?" I imagine the conversation would go very differently. If they weren't using their agency to end the conversation as quickly as possible, I imagine there'd be back and forth asking for more detail, exploring the context, sharing their own experience, etc. I'm rambling, but I just find it very diverting to compare not the output of LLMs versus the output of humans, but the context and situation.
  - moffkalast 4y ago
    Indeed LLMs are currently better at some things and worse at others than people, but I was comparing in terms of accuracy of provided info (assuming they know the answer to whatever question was posed). An equal footing example would be something like seeing a statue of someone known and trying to remember when they were born (or what exactly they did) when the last time you heard about them was in high school years ago. Or like on which address is some restaurant, or to which pressure your bike or car tyres need inflating to. The type of data that both humans and LLMs are kind of bad at replicating perfectly. Also in terms of math I'd also definitely trust GPT 4 quite a bit and 3.5 not at all. For humans, well, it's definitely a range from one to the other.

by tjr 4y ago

ChatGTP 4y ago

So do you find it alarming that people are trying to give such a system “agency” ?

gwright 4y ago
I do. I think the anthropomorphic language that people use to describe these systems is inaccurate and misleading. An Australian mayor has claimed that ChatGPT "defamed" him. The title of this article says that we should teach people that text generation tools "lie". Other articles suggest that ChatGPT "knows" things. It is extremely interesting to me how much milage can be gotten out of an LLM by observing patterns in text and generating similar patterns. It is extremely frustrating to me to see how easily people think that this is evidence of intelligence or knowledge or "agency" as you suggest.
- simonw 4y ago
  I agree with you: convincing people that these systems do not have intelligence or agency is the second most important problem to solve. The most important problem is to ensure people understand that these systems cannot be trusted to tell them the truth. I'm OK with starting out with "these systems will lie to you", then following up with "but you do need to understand that they're not anthropomorphic etc" later on.
  - int_19h 4y ago
    You might have better success convincing people if you stop bundling it together as "intelligence and agency", for starters. I agree with the agency part and strongly disagree with intelligence, for one. In general, "intelligence" is so vaguely and informally defined that it's simply not something that can be reasonably argued objectively either way. (If what you really meant is humanlike intelligence, then I would agree with that claim, but it's a very different one.)
- olalonde 4y ago
  I also disagree about some of the anthropomorphism (e.g. it doesn't intentionally "lie") but I'd say it passes the "duck test"[0] for knowing things and intelligence, to some degree. I would even go as far to say it has opinions although it seems OpenAI has gone out of their way to limit answers that could be interpreted as opinions. [0] https://en.wikipedia.org/wiki/Duck_test https://en.wikipedia.org/wiki/Duck_test
- IIAOPSW 4y ago
  I disagree with your confident assertions about its "agency" and "intent" when there's no goal post for what those things mean in humans to begin with. Barring OpenAI's filters, if I asked it to participate as a party in a business negotiation determining a fair selling price for its writing services, I'm sure it could emulate that sort of character long enough to pass. And if it can successfully fake being a self interested actor working towards goals of its own material interest, whats the difference between faking it and actually having agency? At some point you have to acknowledge emergent agency and other properties as a possibility.
  - gwright 4y ago
    > I disagree with your confident assertions about its "agency" and "intent" when there's no goal post for what those things mean in humans to begin with. I'm not sure how confident I am! I think the success of LLMs are serving to challenge our understandings of cognition and intelligence in humans. I tend to agree with you that our understanding of agency, intent, intelligence, cognition, free will, and related ideas in humans is incomplete and so it is a challenge to think about how they might apply (or not) to LLMs. > whats the difference between faking it and actually having agency? Good question. The Chinese Room Argument is very close to this but framed as "understanding" vs "agency": https://plato.stanford.edu/entries/chinese-room/ https://plato.stanford.edu/entries/chinese-room/
deleted 4y ago
[deleted]
te_chris 4y ago
Yes. It’s very weird seeing the religiosity surrounding this whole thing.
olalonde 4y ago
What is alarming about it?

moffkalast 4y ago

Having used GPT 4 for a while now I would say I trust its factual accuracy more than the average human you'd talk to on the street. The sheer volume of things we make up on a daily basis through no malice of our own but bad memory and wrong associations is just astounding. That said, fact checking is still very much needed. Once someone figures out how to streamline and automate that process it'll be on Google's level of general reliability.

crabmusket 4y ago
It's interesting to imagine an "average human on the street" being put in the position that an LLM is in. If you asked someone on the street to summarise a page of React Native documentation for you, they'd probably look at you strangely and say "sorry I have a bus to catch, good luck!" But if you had someone in a room, sat them down with a pen and paper and instructions along the lines of: - answer the question on the page without diversion - do not ask for additional clarifying information - provide your answer in 3-5 paragraphs with a summary and didn't let them leave until they'd completed the task, then .. I'm sure the results would be "worse", and much more variable, than GPT output, while being conceptually similar. Of course, you don't ask a random person on the street to do this, probably because of social constraint, inconvenience, and the expectation that they will not do a better or faster job than you will yourself. But if you were stuck with a stranger, began to converse, and asked them e.g. "should I get a divorce?" I imagine the conversation would go very differently. If they weren't using their agency to end the conversation as quickly as possible, I imagine there'd be back and forth asking for more detail, exploring the context, sharing their own experience, etc. I'm rambling, but I just find it very diverting to compare not the output of LLMs versus the output of humans, but the context and situation.
- moffkalast 4y ago
  Indeed LLMs are currently better at some things and worse at others than people, but I was comparing in terms of accuracy of provided info (assuming they know the answer to whatever question was posed). An equal footing example would be something like seeing a statue of someone known and trying to remember when they were born (or what exactly they did) when the last time you heard about them was in high school years ago. Or like on which address is some restaurant, or to which pressure your bike or car tyres need inflating to. The type of data that both humans and LLMs are kind of bad at replicating perfectly. Also in terms of math I'd also definitely trust GPT 4 quite a bit and 3.5 not at all. For humans, well, it's definitely a range from one to the other.