4 ms·

That's quite funny! Behaviour when the prompt has contradictions. Since GPT-3 is a statistical model, what's the chance it calls out the question as wrong? Is t

That's quite funny! Behaviour when the prompt has contradictions. Since GPT-3 is a statistical model, what's the chance it calls out the question as wrong? Is there a reasonably high probility sequence which just says that? At some point when either using or running projects around prompt engineering we have to switch from our high modernist viewpoint which is baked into our culture. And instead, deal with it explicitly as a statistical model. And even when such a model is super smart, know by the way it is constructed that it is fuzzy and often "wrong" (like humans are).

amelius 4y ago
Huh, the prompt doesn't have contradictions.
- deleted 4y ago
  [deleted]
- grogers 4y ago
  More like ambiguity. Because of the "also", one could easily expect the resulting output for foobar(9) to print: baloney yam
  - amelius 4y ago
    Not really, because in that case you'd expect a clear specification of the order in which the words should appear, if they should be on the same line, etc. By your logic, a prompt like "Print foo" could result in "foo bar". You didn't ask for "bar" but the output is still valid according to you.
    - grogers 4y ago
      1. Change ... so that foobar(9) instead prints "yam". 2. Change ... so that foobar(9) also prints "yam" 1 is not ambiguous, but 2 is. It could mean that it does it's original behavior and also does the new behavior. But it could mean like foobar(10), foobar(9) also prints only "yam" as you are interpreting it to be. I could imagine that the whole point was to see what GPT-3 did with the ambiguity, and it didn't even acknowledge it. Given that the prompt is already ambiguous I'm not sure why you'd assume that there would have to be a single expectation of the order of the prints, either one should suffice. Or maybe it was just a mistake in the prompt because English is hard. Certainly based on the code, the assumption that GPT-3 made is the most natural one.
  - derangedHorse 4y ago
    > foobar(9) prints "baloney" and foobar(10) prints "yam" > change the function foobar so that foobar(9) also prints "yam" It interpreted the prompt in the same way that I would, making foobar(9) only print "yam", but I guess I can see the interpretation where someone might think the prompt was asking to print both. It brings awareness to the effect of cultural bias in the training set and how that might require some people with different ways of speaking to add more clarification to get the output they want.
- dphidt 4y ago
  It's also assumed that "i" is an integer, a contextually plausible but potentially bad assumption. By one interpretation of the prompt, foobar(9) should print "yam" but foobar(9.5) still "baloney", for example.

amelius 4y ago

Huh, the prompt doesn't have contradictions.

deleted 4y ago
[deleted]
grogers 4y ago
More like ambiguity. Because of the "also", one could easily expect the resulting output for foobar(9) to print: baloney yam
- amelius 4y ago
  Not really, because in that case you'd expect a clear specification of the order in which the words should appear, if they should be on the same line, etc. By your logic, a prompt like "Print foo" could result in "foo bar". You didn't ask for "bar" but the output is still valid according to you.
  - grogers 4y ago
    1. Change ... so that foobar(9) instead prints "yam". 2. Change ... so that foobar(9) also prints "yam" 1 is not ambiguous, but 2 is. It could mean that it does it's original behavior and also does the new behavior. But it could mean like foobar(10), foobar(9) also prints only "yam" as you are interpreting it to be. I could imagine that the whole point was to see what GPT-3 did with the ambiguity, and it didn't even acknowledge it. Given that the prompt is already ambiguous I'm not sure why you'd assume that there would have to be a single expectation of the order of the prints, either one should suffice. Or maybe it was just a mistake in the prompt because English is hard. Certainly based on the code, the assumption that GPT-3 made is the most natural one.
- derangedHorse 4y ago
  > foobar(9) prints "baloney" and foobar(10) prints "yam" > change the function foobar so that foobar(9) also prints "yam" It interpreted the prompt in the same way that I would, making foobar(9) only print "yam", but I guess I can see the interpretation where someone might think the prompt was asking to print both. It brings awareness to the effect of cultural bias in the training set and how that might require some people with different ways of speaking to add more clarification to get the output they want.
dphidt 4y ago
It's also assumed that "i" is an integer, a contextually plausible but potentially bad assumption. By one interpretation of the prompt, foobar(9) should print "yam" but foobar(9.5) still "baloney", for example.