4 ms·
You just rediscovered that LLMs become much stupider when using images as input. I think that has been already shown for gpt-4 as well.
by nolist_policy 2y ago
You just rediscovered that LLMs become much stupider when using images as input. I think that has been already shown for gpt-4 as well.
- dtquad 2y agoEven when using the web search tool GPT-4 becomes stupider. Do they use a dumber model for tool/vision?
- dwaltrip 2y agoI’m guessing that it’s just a much harder problem. Images often contain more information but it is far less structured and refined than language. The transformation process that occurs when people speak and write is incredibly rich and complex. Compared to images which are essentially just the outputs of cameras or screen captures — there isn’t an “intelligent” transformation process occurring.
- justsid 2y agoI think images also have much higher information density than words, or at least they can. There is a reason a picture is worth 1000 words.
- tbruckner 2y agoThis is news to me. Any good examples of this outside of the above?
- nolist_policy 2y agoVision language models are blind (192 comments) https://news.ycombinator.com/item?id=40926734 https://news.ycombinator.com/item?id=40926734
- fnordpiglet 2y agoOn the other hand if I take pictures of circuits, boards, electronic components, etc GPT4o is pretty reliably able to explain the me the pinouts, the board layouts, reference material in the data sheets, and provide pretty reasonable advice about how to use it (I.e., where to put resistors and why, what pins to use for the component on an esp32, etc). As a beginner in electronics this is fabulously helpful. Its ability to pass vision tests seems like a pretty dumb utility metric when most people judge utility by how useful things are.