Priors from the language play a much bigger role in the answers that are predicted than the image itself. So for example, if you ask 'What color is ...?', irres

Priors from the language play a much bigger role in the answers that are predicted than the image itself. So for example, if you ask 'What color is ...?', irrespective of the image, it is more likely to spit out colors as the answer. The answers are usually well-aligned with the question that is being asked. 'Yes/no' for binary questions, 'red/blue/etc' for 'What color...', 'tennis/baseball/etc' for 'What sport...' and so on.