Recently played with something similar using Llama 3.2 Vision locally. Worked pretty well, and if the model fit in G-RAM decently fast. Main issue was prompt

Recently played with something similar using Llama 3.2 Vision locally. Worked pretty well, and if the model fit in G-RAM decently fast. Main issue was prompt adherence. In my experience prompt adherence goes down significantly when you reduce the model size. Llama 3.2 Vision seems to be tuned hard to provide a summary at the end, usually with some social commentary, and was difficult to get the 8B model to avoid outputting it. Also adding multiple if-this-then-that clauses, like in your prompt, was often ignored in the 8B and smaller model compared to 90B. I've tried the Gemma 2 model before for assistant tasks, and was very pleased with the 9B performance. It had good prompt adherence and performed well on various tasks. So looking forward to trying this.