vokhanhan25

3 ms·

1.
by vokhanhan25 1y ago
Models can be biased, but it doesn't seem like it should be a reason to get the answer wrong, right? Humans have biases too, but we don't get those simple questions wrong
2.
by vokhanhan25 1y ago
I see a clear difference. One is objective (only one correct answer), one is subjective (multiple plausible answers)
3.
by vokhanhan25 1y ago
I agree. If it doesn't know the abnormality then how can it control its output
4.
by vokhanhan25 1y ago
You should try with other models besides GPT-4o, because in the paper they also show that GPT4.1 (~GPT-4o) gives 4 legs instead of 2 legs.
5.
by vokhanhan25 1y ago
Agreed. It would be even more dangerous if we were talking about weird edge cases in self-driving cars or medical imaging.
6.
by vokhanhan25 1y ago
Not really. Rather, the model is still overconfident in what it has learned, the question is if it is trained only to do counting without relying on knowledge, can it do this?
7.
by vokhanhan25 1y ago
Please check Table 3 in the paper. Birds (2 legs) have only 1%, while Mammals (4 legs) have 2.5%
8.
by vokhanhan25 1y ago
I think LLMs can solve puzzles pretty well because the thinking ability of current models on text is quite good. Moreover, puzzles are not easy for a 7-year-old like this benchmark.
9.
by vokhanhan25 1y ago
This paper explores a different aspect of the limitations of VLMs compared to the paper VLMs are Blind ( https://vlmsareblind.github.io ). While in VLMs are Blind, o3 achieved 90% accuracy ( https://openai.com/index