4 ms·
Missing multimodal again? It is so valuable in practise to be able to have the models see screenshots - I guess if they aren't in the benchmarks then nobody wi
by zmmmmm 2mo ago
Missing multimodal again?
It is so valuable in practise to be able to have the models see screenshots - I guess if they aren't in the benchmarks then nobody will focus on it. But it completely nixes these for some of my main use cases.
- arcanemachiner 2mo agoI would assume that GLM 6 will be multimodal, but 5.x will be text-only.
- xscott 2mo agoProbably not what you're after, but I've considered having a separate small mm-model act as a seeing-eye dog for the bigger more capable one.
- cosmez 2mo agoBut how do you prompt this smaller model to give back information? I've tried that in the past but didn't go well. What I do is to send written handoff files between models to pass context around, but only had good results with big vision models as well.
- xscott 2mo agoSo many possibilities for how you could glue it all together. However, when I send Gemma 4 12B in llama.cpp an image with no accompanying text, it assumes I want a description and gives me one. I just tried with an audio file, and it transcribed the lyrics as I hoped. Then it made a bunch of suggestions about what do next, which I wasn't after. I could probably fix that by sending some text to narrow the scope.
- bellowsgulch 2mo agoI've tried this, too, and for whatever reason, simple things like having a subagent with vision capabilities hallucinates responses to the blind models. I don't know why, and it's not consistent, but I don't see this happening when the primary agent is a vision model.
- pllbnk 2mo agoI can’t come up with a use case where I couldn’t extract the image details using another, multimodal model and pass it into the GLM’s context with as many details as I need.
- zmmmmm 2mo agoI think you lose a lot by not having the vision capability shared with the text. It is the joint reasoning across them where the power lies (the same model that sees the code and made the changes to produce the visual presentation, sees the image of it and reasons about it).
- cmrdporcupine 2mo agoI mean this is assuming the thing you're working on has a UI? Not all of us work in that space.