I often think that LLM issues like this could be solved by a final pass of "is the information in this image the same as this text" (ie generally a verification

I often think that LLM issues like this could be solved by a final pass of "is the information in this image the same as this text" (ie generally a verification pass). It might be that you would want to use a different model {non-generative} for that last pass -- which is like the 'array of experts' type approach. Or comparing to your human analogy, like reading back the list to your partner before you leave for the shops.