I haven't done deep reading on LLM architectures and I don't really know if LLMs have logits in the traditional sense of a CNN or something, but I think the pro

I haven't done deep reading on LLM architectures and I don't really know if LLMs have logits in the traditional sense of a CNN or something, but I think the problem with this is that the LLM's logits would have absolutely no bearing on it's confidence of the location being correct, only on it's confidence that the tokens making up the answer it provides follow from the tokens that were encoded from the provided image, which isn't the same thing.