3 ms·

a good example of people not understanding what they're asking is where people showcase the whole "which is bigger? 9.11 or 9.9?" and of course a LLM will retur

a good example of people not understanding what they're asking is where people showcase the whole "which is bigger? 9.11 or 9.9?" and of course a LLM will return 9.11 because in most contexts (chapters, versions etc.) it is the bigger number, but if you specify the context is fractional numbers then most models will correctly say 9.9 is bigger. A while ago I kept seeing people tout the conventional "9.11 is bigger" answer as proof that LLM's don't understand numbers and are bad at math. While this is still empirically true, I don't think most people understand these things as well as they believe they do either. Specifically, people don't understand how broad and ill defined their questions often are, and how much of their preconceptions need to be communicated for their questions to mean what they think they mean.

fodkodrasz 2y ago
To most people in most life situations these series of symbols do not represent chapters or versions, but numbers. If you have to always give this context to an AI then the I is not there yet.
- osullip 2y ago
  Or they could be dates. Context is key. Using LLMs has taught me I often assume everyone has the same context as me when I ask a question.
  - Grimblewald 2y ago
    Same for me, it has also made me a much better communicator, because I have learned that while I often assume everyone has the context I have, this is too often not the case and becomes the basis of most of the miscommunication issues / conflicts that I encounter. Another issue is a miss-match in the precise definitions of words I have learned over time vs what others have learned. It is honestly a marvel that LLM's could learn to plausibly reproduce human communication as well as they have given how utterly insane it is that we even manage to communicate effectively among one another in the first place.
    - j45 2y ago
      Clarifying context before communicating or reviewing something is critical. Especially since GPT-4o has become an over-exuberant and eager intern with a bias towards action to solve. It needs reminders every 2-3 messages to not jump to solutions until the problem has been defined and potential approaches beyond the one it wants to spin it's propellor on. Statistically this approach must work for a good chunk of the prompts and queries and I assume my need otherwise may not be the common sie.
- Grimblewald 2y ago
  Well, I for one wouldn't assume unless the conversation up to that point indicates as much. If a random on the street asked me the same question I would ask clarifying questions first. An LLM I fine tuned on my own conversations / dialogues has the same habit and when I asked it this question it did exactly that and then given how short the answer was it gave a breif answer (paraphrasing) "if the context is chapters/versions etc. then 9.11, but if it is decimal numbers then 9.9" which made me realise most LLM's are probably having similar ambiguity issues but are trained to just give an answer without asking for clarification when equally directions for answering are possible. So I tested it, and found that was exactly the case. that's less a problem of the AI missing the I and more a problem of ill defined expectations.
j45 2y ago
Mathematically 9.9 is bigger. Numbering wise in a text document, 9.11 can be higher. There's a chance that LLMs might be trained on more documents than mathematical info, despite the clear understanding that 9.9 is a higher mathematical value. I ask which is a higher mathematical value vs a higher numbering value in a document and it seems to be better when I have come across this. Happy to learn from your and others experiences.

fodkodrasz 2y ago

To most people in most life situations these series of symbols do not represent chapters or versions, but numbers. If you have to always give this context to an AI then the I is not there yet.

osullip 2y ago
Or they could be dates. Context is key. Using LLMs has taught me I often assume everyone has the same context as me when I ask a question.
- Grimblewald 2y ago
  Same for me, it has also made me a much better communicator, because I have learned that while I often assume everyone has the context I have, this is too often not the case and becomes the basis of most of the miscommunication issues / conflicts that I encounter. Another issue is a miss-match in the precise definitions of words I have learned over time vs what others have learned. It is honestly a marvel that LLM's could learn to plausibly reproduce human communication as well as they have given how utterly insane it is that we even manage to communicate effectively among one another in the first place.
  - j45 2y ago
    Clarifying context before communicating or reviewing something is critical. Especially since GPT-4o has become an over-exuberant and eager intern with a bias towards action to solve. It needs reminders every 2-3 messages to not jump to solutions until the problem has been defined and potential approaches beyond the one it wants to spin it's propellor on. Statistically this approach must work for a good chunk of the prompts and queries and I assume my need otherwise may not be the common sie.
Grimblewald 2y ago
Well, I for one wouldn't assume unless the conversation up to that point indicates as much. If a random on the street asked me the same question I would ask clarifying questions first. An LLM I fine tuned on my own conversations / dialogues has the same habit and when I asked it this question it did exactly that and then given how short the answer was it gave a breif answer (paraphrasing) "if the context is chapters/versions etc. then 9.11, but if it is decimal numbers then 9.9" which made me realise most LLM's are probably having similar ambiguity issues but are trained to just give an answer without asking for clarification when equally directions for answering are possible. So I tested it, and found that was exactly the case. that's less a problem of the AI missing the I and more a problem of ill defined expectations.

j45 2y ago

Mathematically 9.9 is bigger. Numbering wise in a text document, 9.11 can be higher. There's a chance that LLMs might be trained on more documents than mathematical info, despite the clear understanding that 9.9 is a higher mathematical value. I ask which is a higher mathematical value vs a higher numbering value in a document and it seems to be better when I have come across this. Happy to learn from your and others experiences.