7 ms·
The o1 example is interesting. In the CoT summary it acknowledges that the most recent official information is 1611m, but it then chooses to say 1622 because it
by ascorbic 2y ago
The o1 example is interesting. In the CoT summary it acknowledges that the most recent official information is 1611m, but it then chooses to say 1622 because it's more commonly cited. It's like it over-thinks itself into the wrong answer.
- freehorse 2y agoDoes it search the internet for that? I assume so because else claiming how often something is cited does not make sense, but would be interesting to know surely. Even gpt4o mini with kagi gets it right with search enabled (and wrong without search enabled - tried over a few times to make sure).
- asl2D 2y agoCan the claim about citation frequency be just an answer pattern and not model's exact reasoning?
- freehorse 2y agoYeah there could be parts of the training set with 1611 being explicitly called the official and 1622 being explicitly called the most common answer. But it can also have access to search results directly I think. Is there a way to know if it does or not?
- sd9 2y agoI don’t think the public o1 can search the internet yet, unlike 4o. In principle it could know that something is more commonly cited based on its training data. But it could also just be hallucinating.
- diggan 2y ago> In principle it could know that something is more commonly cited based on its training data Could it? Without explicit training for that, how would it be expected to know it has to be able to count occurrences of something?
- sd9 2y agoI think it would be more vibes based - commonly occurring things would be reinforced more in the weights. Rather than it explicitly counting the number of occurrences.
- diggan 2y agoSo the probabilities would be skewed towards something, but unless the model could somehow count/infer its own weights, I don't see how it could "introspect" to see if something is more common than something else.
- freehorse 2y ago> it could know that something is more commonly cited based on its training data No there is no such concept or way to do something like that. LLMs do not have such kind of meta-knowledge over their training data or weights. But there could be explicit mentions about this on their training data and they could pick on that and that is probably the simplest explanation.
- famouswaffles 2y ago>LLMs do not have such kind of meta-knowledge over their training data or weights. Not sure this is a claim that can be confidently made. https://arxiv.org/abs/2309.00667 https://arxiv.org/abs/2309.00667 https://x.com/flowersslop/status/1873115669568311727?t=eBMbK-FOvEn1ghqkaqZrHg&s=19 https://x.com/flowersslop/status/1873115669568311727?t=eBMbK...
- blueflow 2y agoHow could a language model infer that the official information overrules anything else?
- mistercow 2y agoI’m not sure what kind of response you’re looking for, or if this is a rhetorical question or not. But “how could a language model infer…?” can be asked about a whole lot of things that language models have no problem reliably inferring.
- ben_w 2y agoSame way as we can: learning which sources are more trustworthy. There's limits to how far you can go with this — not only do humans make mistakes with this, but even in the abstract theoretical it can never be perfect: https://en.wikipedia.org/wiki/Münchhausen_trilemma https://en.wikipedia.org/wiki/Münchhausen_trilemma — but it is still the "how".
- fullstackwife 2y agofor the last 25+ years we rather not learned, but trusted the top3 of SERPs. Every ranking algorithm will be gamed eventually
- ben_w 2y agoI would say that we learned to trust the search engines; but otherwise I agree with you: every ranking algorithm will be gamed eventually. (I wonder if giving an LLM content with intent to cause its users to spend money they didn't need to, would count as fraud, hacking, both, something else entirely?)
- patrulek 2y agoI think i had similar case yesterday for Python script. It gave me code for older version of a module, but when i pasted the error i got, it corrected itself and gave me proper solution for version i had installed.