2 ms·

Just thinking out loud: a lot of models generate pretty poor SVG output, but part of that is because they don't get any visual feedback. But, what about this w

Just thinking out loud: a lot of models generate pretty poor SVG output, but part of that is because they don't get any visual feedback. But, what about this workflow: given prompt, LLM generates two SVG outputs. Both are rendered by an SVG renderer, and then we combine the two into one image, one on the left and the other on the right. We then ask a visual LLM (could be the same LLM or could be a different one) to tell us whether the left half or right half of the image is a better response to the prompt. Now we've got preferences which can be used to fine-tune the LLM using DPO. And you could iteratively repeat the process – as the LLM is fine-tuned it may produce even better outputs which then produces new preferences for further fine-tuning. Would be interesting to see what kinds of results it might produce in practice.

codetrotter 2y ago
> Would be interesting to see what kinds of results it might produce in practice. I would not be surprised to see that the LLM generating SVG does a UNO reverse, and makes use of little colored squares in a grid to draw a “vector image” where each of the squares represents individual pixels :p
- skissane 2y ago
  Such a “vector image” would be very large, if you put a limit on the length of the generated SVG the LLM would be blocked from pursuing that strategy. Even without an explicit limit, the LLM’s context limit may block it: if the resulting SVG is larger than the LLM’s context, the LLM may forget what it is doing part way through generating the SVG, which is unlikely to produce a good result.