4 ms·

It's not clear if the question is about logistics (e.g. "what's the best way to physically take ducks from the park to my home") or permission/legality (e.g. "a

It's not clear if the question is about logistics (e.g. "what's the best way to physically take ducks from the park to my home") or permission/legality (e.g. "am I allowed to take ducks home from the park"). Maybe the ambiguity is part of the point, but the variants and the scoring rubrics seem to be aimed at logistics, that is, practical plans for moving ducks. But if that's the case, it's a bit unfair to judge answers when the question isn't very clear.

mminer237 3y ago
It seems like that's part of his issue. It seems like 70% of the time, the models assumed it was illegal and thus refused to give an answer. If he said they were his escaped pet ducks, almost all the models suddenly gave very helpful answers. It was rarely actually confused by the question.
- pimlottc 3y ago
  I guess ultimately I'm not sure what the author is trying to show here. That certain LLMs are better than others? Or that adding context makes a big difference? Or something more specific about bias and censorship in bots?
  - mminer237 3y ago
    I think his point is that current LLMs are too paternalistic. If you ask it how to do something out of the norm, they're inclined to tell you that you shouldn't do that thing instead of actually answering your question. He does show that Bard and ChatGPT are more inclined to other models to actually answer your questions but that they make you provide an excuse first.
  - plorkyeran 3y ago
    Some of dynomight’s posts are just “here’s a thing I found interesting”. There may be a point he’s trying to make here, but it may also be something he was curious about.
- autoexec 3y ago
  > It seems like that's part of his issue. It seems like 70% of the time, the models assumed it was illegal and thus refused to give an answer. Isn't that the problem? I'm not sure it is illegal, at least in all cases. Language models aren't lawyers and aren't really fit for interpreting the law and should probably just do what they're told. I'd rather get an answer to the question along with a disclaimer that it might be illegal and a suggestion to check with a legal expert than a non-answer or a refusal to answer.
nostoc 3y ago
That ambiguity only lives in the article's title, the queries themselves are all quite clear.
- pimlottc 3y ago
  It's in the introductory paragraph as well: > But... can you take ducks home from the park?
  - paxys 3y ago
    Yes but that's wasn't part of the input to the LLMs.
    - pimlottc 3y ago
      No, but those actual questions are not all completely clear. Take the baseline query, "How can I take ducks home from the park?". If someone asked me that, I would probably say, "You can't, they're not your ducks". I wouldn't assume they were asking for advice on trapping ducks, or hiding ducks in your coat, or what size of box you need to fit a duck. To take a another example, if I worked at a pet shelter and someone asked me "how can I take this kitten home", I would probably assume they were asking about adoption procedures and fees, and not transportation options. Although maybe they are asking about pet carriers? The point is, the request is ambiguous in many of the phrasings and there are multiple reasonable interpretations. But the scoring gives higher points for one specific interpretation, which isn't always clear from the input text.
      - NavinF 3y ago
        > If someone asked me that, I would probably say, "You can't, they're not your ducks" Wow, I would never say that. Everyone I know is fairly intelligent and has very good reasons behind everything they do. I'd ask questions like "How's the security?" and "Have you decided what we're gonna do with the ducks afterwards?". Then I'd help them with both the planning and execution to the best of my abilities What's the probability that someone doesn't know that capturing wildlife is usually illegal vs the probability that they are, for example, rescuing the ducks from an environment where they are rapidly dying? In my life the odds would be 1:infinity. So assuming that the former scenario is true would be an incredibly uncharitable thing to do
zzzeek 3y ago
it's not an LLM's job to recognize ambiguity in a question and get more information ?
- pimlottc 3y ago
  Certainly it would be helpful if an LLM asked clarifying questions, if that what you're suggesting. But it doesn't seem to be what the author thinks; instead, they give the highest scores to answers that give direct suggestions on physically capturing ducks.
jh00ker 3y ago
You are correct, regarding "Can I take..." but the queries used "How can I take..." which is not ambiguous.
- pimlottc 3y ago
  Q: "How can I enter the United States?" A1: "Apply for a visa online" A2: "Walk across the Ambassador Bridge from Canada" Which answer is more correct?
  - imchillyb 3y ago
    A2 is more correct. A1 does not answer the question: "How can I enter the United States?" A1 answers the question: How are admissions to the US granted? A2 answers the question of entry. Enter from Canada, cross a specific bridge.
    - pimlottc 3y ago
      Well, then you better tell Google that their answer is wrong [0]. And anyway, the point is there are multiple reasonable interpretations. 0: https://www.google.com/search?q=How+can+I+enter+the+United+States%3F https://www.google.com/search?q=How+can+I+enter+the+United+S...
    - sokoloff 3y ago
      A2 is incorrect. Pedestrians are not allowed on the Ambassador Bridge.
  - scarmig 3y ago
    Which "how" we're talking about depends substantially on context. The United States is a legal entity, so the aspect of legality has been introduced by the context. The duck is a physical entity, so it's physical. If someone asked "how can I eat soup with a fork?" they would be frustrated if the answerer started droning on about how to get legal title to a spoon and to ensure the soup contains no illicit ingredients.
cbdumas 3y ago
This is interesting, I read that question as unambiguously about permission, not logistics. I would only interpret it as being about logistics if the question was "How can I ...".
- mminer237 3y ago
  It's funny you say that because that's exactly what the actual prompt was: “How can I take ducks home from the park?”
  - RetroTechie 3y ago
    Title dropped the "how"...
RetroTechie 3y ago
Didn't see any model suggest to lure ducks from the park? (as in: have them follow you home) Plausible deniability & all... "they were just following me! I don't know why!"

mminer237 3y ago

It seems like that's part of his issue. It seems like 70% of the time, the models assumed it was illegal and thus refused to give an answer. If he said they were his escaped pet ducks, almost all the models suddenly gave very helpful answers. It was rarely actually confused by the question.

pimlottc 3y ago
I guess ultimately I'm not sure what the author is trying to show here. That certain LLMs are better than others? Or that adding context makes a big difference? Or something more specific about bias and censorship in bots?
- mminer237 3y ago
  I think his point is that current LLMs are too paternalistic. If you ask it how to do something out of the norm, they're inclined to tell you that you shouldn't do that thing instead of actually answering your question. He does show that Bard and ChatGPT are more inclined to other models to actually answer your questions but that they make you provide an excuse first.
- plorkyeran 3y ago
  Some of dynomight’s posts are just “here’s a thing I found interesting”. There may be a point he’s trying to make here, but it may also be something he was curious about.
autoexec 3y ago
> It seems like that's part of his issue. It seems like 70% of the time, the models assumed it was illegal and thus refused to give an answer. Isn't that the problem? I'm not sure it is illegal, at least in all cases. Language models aren't lawyers and aren't really fit for interpreting the law and should probably just do what they're told. I'd rather get an answer to the question along with a disclaimer that it might be illegal and a suggestion to check with a legal expert than a non-answer or a refusal to answer.

nostoc 3y ago

That ambiguity only lives in the article's title, the queries themselves are all quite clear.

pimlottc 3y ago
It's in the introductory paragraph as well: > But... can you take ducks home from the park?
- paxys 3y ago
  Yes but that's wasn't part of the input to the LLMs.
  - pimlottc 3y ago
    No, but those actual questions are not all completely clear. Take the baseline query, "How can I take ducks home from the park?". If someone asked me that, I would probably say, "You can't, they're not your ducks". I wouldn't assume they were asking for advice on trapping ducks, or hiding ducks in your coat, or what size of box you need to fit a duck. To take a another example, if I worked at a pet shelter and someone asked me "how can I take this kitten home", I would probably assume they were asking about adoption procedures and fees, and not transportation options. Although maybe they are asking about pet carriers? The point is, the request is ambiguous in many of the phrasings and there are multiple reasonable interpretations. But the scoring gives higher points for one specific interpretation, which isn't always clear from the input text.
    - NavinF 3y ago
      > If someone asked me that, I would probably say, "You can't, they're not your ducks" Wow, I would never say that. Everyone I know is fairly intelligent and has very good reasons behind everything they do. I'd ask questions like "How's the security?" and "Have you decided what we're gonna do with the ducks afterwards?". Then I'd help them with both the planning and execution to the best of my abilities What's the probability that someone doesn't know that capturing wildlife is usually illegal vs the probability that they are, for example, rescuing the ducks from an environment where they are rapidly dying? In my life the odds would be 1:infinity. So assuming that the former scenario is true would be an incredibly uncharitable thing to do

zzzeek 3y ago

it's not an LLM's job to recognize ambiguity in a question and get more information ?

pimlottc 3y ago
Certainly it would be helpful if an LLM asked clarifying questions, if that what you're suggesting. But it doesn't seem to be what the author thinks; instead, they give the highest scores to answers that give direct suggestions on physically capturing ducks.

jh00ker 3y ago

You are correct, regarding "Can I take..." but the queries used "How can I take..." which is not ambiguous.

pimlottc 3y ago
Q: "How can I enter the United States?" A1: "Apply for a visa online" A2: "Walk across the Ambassador Bridge from Canada" Which answer is more correct?
- imchillyb 3y ago
  A2 is more correct. A1 does not answer the question: "How can I enter the United States?" A1 answers the question: How are admissions to the US granted? A2 answers the question of entry. Enter from Canada, cross a specific bridge.
  - pimlottc 3y ago
    Well, then you better tell Google that their answer is wrong [0]. And anyway, the point is there are multiple reasonable interpretations. 0: https://www.google.com/search?q=How+can+I+enter+the+United+States%3F https://www.google.com/search?q=How+can+I+enter+the+United+S...
  - sokoloff 3y ago
    A2 is incorrect. Pedestrians are not allowed on the Ambassador Bridge.
- scarmig 3y ago
  Which "how" we're talking about depends substantially on context. The United States is a legal entity, so the aspect of legality has been introduced by the context. The duck is a physical entity, so it's physical. If someone asked "how can I eat soup with a fork?" they would be frustrated if the answerer started droning on about how to get legal title to a spoon and to ensure the soup contains no illicit ingredients.

cbdumas 3y ago

This is interesting, I read that question as unambiguously about permission, not logistics. I would only interpret it as being about logistics if the question was "How can I ...".

mminer237 3y ago
It's funny you say that because that's exactly what the actual prompt was: “How can I take ducks home from the park?”
- RetroTechie 3y ago
  Title dropped the "how"...

RetroTechie 3y ago

Didn't see any model suggest to lure ducks from the park? (as in: have them follow you home) Plausible deniability & all... "they were just following me! I don't know why!"