3 ms·

The obvious use case, especially on HN, is frontend dev of any kind at all. The second most obvious one is OCR of paper documents.

by deaux 2mo ago

5kyn3t 2mo ago
Frontend Dev? I do not really understand. do you let the models analyze the webpages you are working on? or for testing?
- dandaka 2mo ago
  QA of course. You hook up your agent with CDP access to live product + let it screenshot and look into result. Also you could hook agent with CDP access to Figma to read/write, there a vision model is very useful as well.
- rpdillon 2mo ago
  Standard flow with a vision model in OMP is to write the front end code, fire up the server, fire up a headless browser and then take screenshots and examine and iterate. Works great. When I'm using DeepSeek V4 Flash, it always reminds me instead that I have to validate manually by loading up the page.
- deaux 2mo ago
  It closes the development loop. Without it a model can't check if the stuff it made actually visually renders like it's supposed to. It can only guess/assume.
- wongarsu 2mo ago
  LLMs are not great at aligning stuff on first try, they are however very good at taking screenshots and fixing their mistakes. Claude Design also does this all the time, as does regular Claude in the web UI if you tell it to make a powerpoint presentation I really missed this feature when I had DeepSeek code a small game for fun. When writing UI and rendering code it could execute the game and get screenshots back, but then had to rely on my feedback on what had gone wrong. Models with vision can do much better here, finding more issues on their own
dandaka 2mo ago
but for OCR there are much better suited models, I use mlx-community/PaddleOCR-VL-8bit
- deaux 2mo ago
  Sometimes you intentionally want to verbatim keep "mistakes", sometimes you don't and want them to be "fixed". OCR-only models tend to only do one of those two, in VLM cases often the latter. With multi-modal LLMs you can just tell them (adherence of course needing evals/differs per model).

The obvious use case, especially on HN, is frontend dev of any kind at all. The second most obvious one is OCR of paper documents.

by deaux 2mo ago

The obvious use case, especially on HN, is frontend dev of any kind at all. The second most obvious one is OCR of paper documents.

5kyn3t 2mo ago

Frontend Dev? I do not really understand. do you let the models analyze the webpages you are working on? or for testing?

dandaka 2mo ago
QA of course. You hook up your agent with CDP access to live product + let it screenshot and look into result. Also you could hook agent with CDP access to Figma to read/write, there a vision model is very useful as well.
rpdillon 2mo ago
Standard flow with a vision model in OMP is to write the front end code, fire up the server, fire up a headless browser and then take screenshots and examine and iterate. Works great. When I'm using DeepSeek V4 Flash, it always reminds me instead that I have to validate manually by loading up the page.
deaux 2mo ago
It closes the development loop. Without it a model can't check if the stuff it made actually visually renders like it's supposed to. It can only guess/assume.
wongarsu 2mo ago
LLMs are not great at aligning stuff on first try, they are however very good at taking screenshots and fixing their mistakes. Claude Design also does this all the time, as does regular Claude in the web UI if you tell it to make a powerpoint presentation I really missed this feature when I had DeepSeek code a small game for fun. When writing UI and rendering code it could execute the game and get screenshots back, but then had to rely on my feedback on what had gone wrong. Models with vision can do much better here, finding more issues on their own

dandaka 2mo ago

but for OCR there are much better suited models, I use mlx-community/PaddleOCR-VL-8bit

deaux 2mo ago
Sometimes you intentionally want to verbatim keep "mistakes", sometimes you don't and want them to be "fixed". OCR-only models tend to only do one of those two, in VLM cases often the latter. With multi-modal LLMs you can just tell them (adherence of course needing evals/differs per model).