3 ms·
Tried the 1B model with the "think step by step" prompt. It gets "which is larger: 9.11 or 9.9?" right if it manages to mention that decimals need to be compar
by kgeist 2y ago
Tried the 1B model with the "think step by step" prompt.
It gets "which is larger: 9.11 or 9.9?" right if it manages to mention that decimals need to be compared first in its step-by-step thinking. If it skips mentioning decimals, then it says 9.11 is larger.
It gets the strawberry question wrong even after enumerating all the letters correctly, probably because it can't properly count.
- khafra 2y agoOf course, in many contexts, it is correct to put 9.11 after 9.9--software versioning does it that way, for example.
- KeplerBoy 2y agoThat's why it's an interesting question and why it struggles so hard. A good answer would explain that and state both results if the context is not hundred percent clear.
- khafra 2y agoI'm not sure how useful that question is in exploring capabilities--"ask the user clarifying questions if the answer is ambiguous" is more of a rlhf or fine tune thing than a base model thing.
- vergessenmir 2y agoWhat is the "think step by step" prompt? An example would be great, Is this part of the system prompt?
- potatoman22 2y agoIt's appending "think step-by-step" to the end of the prompt to elicit a chain-of-thought response. See: https://arxiv.org/abs/2205.11916 https://arxiv.org/abs/2205.11916
- bick_nyers 2y agoDoes anyone know of a CoT dataset somewhere for finetuning? I would think exposing it to that type of modality during a finetune/lora would help.
- altruios 2y agoMy understanding is the way the tokenization works prevents the LMM from being able to count occurrences of words or individual characters.