4 ms·
> given the input question “Sammy wanted to go to where the people are. Where might he go? Answer Choices: (a) populated areas, (b) race track, (c) desert, (d)
by p-e-w 3y ago
> given the input question “Sammy wanted to go to where the people are. Where might he go? Answer Choices: (a) populated areas, (b) race track, (c) desert, (d) apartment, (e) roadblock”, distilling step-by-step provides the correct answer to the question, “(a) populated areas”
Huh? My answer as a human would have been "race track", as that is probably "where the people are" (during a race).
Did I fail? Am I a poor language model? Or is the whole thing just tea leaf reading to begin with?
- smeagull 3y agoA race track or an apartment are likely to be populated areas as well. "A" is a broader answer that encompasses more possibilities.
- Zuiii 3y agoA depending on the size of a "populated area", the density can be far lower than a crowded race track.
- kybernetikos 3y agoYeah "I felt lonely and wanted to be where the people are, so I wandered through the suburbs" really doesn't make sense. I think this is just a bad question.
- tuukkah 3y agoFurther, not all populated areas are known for their nightlife and social events.
- p-e-w 3y ago"Where the people are" doesn't simply mean "where a large number of people live". It also has connotations of "where it's happening", as in "where people are coming together (right now)". But my point is really that speaking of the correct answer with a question as vague and open to interpretation as this one is absurd.
- BoorishBears 3y agoIf you really want to go full logician on this, even the format in which you're asked the question should make (a) being the correct answer clear. You're being asked a multiple choice question: - There's an implication of intended vagueness/indirection and multiple possibly suitable answers. - Textual entailment is a very common type of multiple choice format, which would lean towards a more plain semantic correlation
- kybernetikos 3y agoIndeed, I pattern matched (a) as the correct answer too, but on reflection on the content of the words and what it would all actually mean, I think that (a) is a bad answer. If we aren't careful we'll train our AIs to be good at giving incorrect, pat answers to inadequately thought out questions.
- BoorishBears 3y agoTraining an LLM on a ton of multiple choice questions doesn't "infect it" like you're thinking. The tokens capture the fact it's a multiple choice question, and the LLM eventually captures the nuance of textual entailment as a common form of multiple choice question. In a more natural conversational setting, you'd get a different answer: https://chat.openai.com/share/00fed9d6-e3de-4319-9c76-ae180095a68a https://chat.openai.com/share/00fed9d6-e3de-4319-9c76-ae1800... https://chat.openai.com/share/dc9a796c-870c-44ee-b421-31c24b3ff2cf https://chat.openai.com/share/dc9a796c-870c-44ee-b421-31c24b...
- Reubend 3y ago> Did I fail? Am I a poor language model? Or is the whole thing just tea leaf reading to begin with? No offense to you, but I never would have picked "race track". That answer doesn't make much sense since the original prompt doesn't mention anything about racing, and the definition of a populated area fits well with the question being asked.
- stevepike 3y agoFascinating. I would 100% pick race track. Maybe because populated areas is plural? Or because the question gives a named subject and nobody would ever respond to "where are you going?" with "populated areas"?
- diziet 3y agoA google image search for "race track" has 0 people visible in the 17 images above the fold.
- zarzavat 3y agoBut if I watch F1 on TV there are a lot of people there. A race track is like a stadium, it’s a place constructed for a spectator sport. It may be empty most of the time but when there is a race on then there will be crowds there. “Populated areas” does not mean much. Montana is a populated area. It’s not a highly populated area but it’s populated alright.
- deleted 3y ago[deleted]