5 ms·

Tangential question : did anyone ever use GPT4-V in production in visual tasks? It's never consistent enough for me to be useful

zacmps 2y ago
Nope, I tried it for graph and diagram understanding and it wasn't good enough. Planning to repeat the evaluation with 4o when I have time.
- SparkyMcUnicorn 2y ago
  I'm using 4o to convert visual diagrams into mermaid, and it's been almost perfectly accurate in my experience.
  - cuu508 2y ago
    This is the out of the box thinking I love about HN. What do you do with the mermaid?
    - SparkyMcUnicorn 2y ago
      The resulting mermaid is used for... more LLM processing. Converting to mermaid first is more cost-effective, consistent, and accurate for my purposes.
serjester 2y ago
Don’t use it for anything OCR related that needs perfect accuracy. Stuff where some errors are ok, we’ve had great success. Depending on your budget, you can also run it multiple times to catch errors.
- toomuchtodo 2y ago
  How does it compare to Tesseract? Edit: Thank you!
  - elanning 2y ago
    I’ve done a lot of OCR work and tesseract is nearly a decade out of date at this point. It is not a serious technology for anything requiring good accuracy or minor complexity. From what I’ve seen, GPT-4V completely smokes tesseract, but then again, most modern OCR systems do. If you want fast and pretty powerful OCR, check out paddle. If you want slower but higher accuracy, check out transformer based models such as TrOCR.
    - nh2 2y ago
      See this for a comparison of PaddleOCR, TrOCR, and various cloud ones (note: on documents of typed and handwritten text): https://news.ycombinator.com/item?id=32077375 https://news.ycombinator.com/item?id=32077375
      - authorfly 2y ago
        Caveat that being from 2022, the Tesseract version used was almost certainly v4 (if Linux), rather than v5 which is much better (and widely available on Windows in 2022, but not Linux yet). However Tesseract is quite behind still as you note, even with v5.
    - Zuiii 2y ago
      Tesseract's true value is being one apt-get command away (i.e. opensource). Does Debian host more modern OCR systems in their repos?
      - elanning 2y ago
        Big mistake on my part; I should clarify I fine-tuned both PaddleOCR and TrOCR on large amounts of data specific to my domain. I cannot speak on the best out of the box “ready to go” solutions (besides cloud ones, which were quite good with the right pre and post processing).
      - nunez 2y ago
        Tesseract the tool is one apt-get away but the trained models are not, and I've found that they are a starting point, not a final destination. You still have to do more training on top of them for anything that isn't black text on a crisp white background.
- nomel 2y ago
  > you can also run it multiple times to catch errors. Does this require a slight offset and/or rotation to the image, or just literal rerun, with seed seed/whatever giving a different result?
abrichr 2y ago
It's very reliable for GUI segment understanding; see e.g. https://github.com/OpenAdaptAI/OpenAdapt/pull/610 https://github.com/OpenAdaptAI/OpenAdapt/pull/610 (scroll down to `gpt-4-vision-preview`).
- amelius 2y ago
  Can it be used for automatic annotation? As in: you tell it that these and these parts should be masked such and such, and then it does that?
  - abrichr 2y ago
    We have not had success with that unfortunately.
    - amelius 2y ago
      Thank you, your comment will save me some trouble ;)
alexcnwy 2y ago
the coolest use case i've seen is this https://github.com/ddupont808/GPT-4V-Act https://github.com/ddupont808/GPT-4V-Act
- nucleative 2y ago
  I feel like this is the beginning of the end for all captchas
  - udev4096 2y ago
    Image based or any kind of visual captchas will never be extremely effective. I think we will see more of PoW captchas in the upcoming years (just like cloudflare's turnstile captcha)
    - fennecfoxy 2y ago
      I'm not suer about that, can't you give GPT4 a math problem in an image already and have it solve it correctly most of the time? And these haven't even been trained to defeat captchas/logic problem captchas yet, if it was fine tuned on the general pattern of them I imagine any form of captcha is bust.
    - mewpmewp2 2y ago
      Not sure about PoW. What about people with slower phones etc? I don't see how an older smartphone could meaningfully outcompute a spamming infra.
  - jorvi 2y ago
    Twitter / X has a very interesting captcha: you get to see 10 objects that have weird colors and are slightly deformed, and then you have to match them (1 at a time) with another row that has the same objects but seen from a different angle. Of course eventually this will be defeated too, but for now it seems to work pretty well.

zacmps 2y ago

Nope, I tried it for graph and diagram understanding and it wasn't good enough. Planning to repeat the evaluation with 4o when I have time.

SparkyMcUnicorn 2y ago
I'm using 4o to convert visual diagrams into mermaid, and it's been almost perfectly accurate in my experience.
- cuu508 2y ago
  This is the out of the box thinking I love about HN. What do you do with the mermaid?
  - SparkyMcUnicorn 2y ago
    The resulting mermaid is used for... more LLM processing. Converting to mermaid first is more cost-effective, consistent, and accurate for my purposes.

serjester 2y ago

Don’t use it for anything OCR related that needs perfect accuracy. Stuff where some errors are ok, we’ve had great success. Depending on your budget, you can also run it multiple times to catch errors.

toomuchtodo 2y ago
How does it compare to Tesseract? Edit: Thank you!
- elanning 2y ago
  I’ve done a lot of OCR work and tesseract is nearly a decade out of date at this point. It is not a serious technology for anything requiring good accuracy or minor complexity. From what I’ve seen, GPT-4V completely smokes tesseract, but then again, most modern OCR systems do. If you want fast and pretty powerful OCR, check out paddle. If you want slower but higher accuracy, check out transformer based models such as TrOCR.
  - nh2 2y ago
    See this for a comparison of PaddleOCR, TrOCR, and various cloud ones (note: on documents of typed and handwritten text): https://news.ycombinator.com/item?id=32077375 https://news.ycombinator.com/item?id=32077375
    - authorfly 2y ago
      Caveat that being from 2022, the Tesseract version used was almost certainly v4 (if Linux), rather than v5 which is much better (and widely available on Windows in 2022, but not Linux yet). However Tesseract is quite behind still as you note, even with v5.
  - Zuiii 2y ago
    Tesseract's true value is being one apt-get command away (i.e. opensource). Does Debian host more modern OCR systems in their repos?
    - elanning 2y ago
      Big mistake on my part; I should clarify I fine-tuned both PaddleOCR and TrOCR on large amounts of data specific to my domain. I cannot speak on the best out of the box “ready to go” solutions (besides cloud ones, which were quite good with the right pre and post processing).
    - nunez 2y ago
      Tesseract the tool is one apt-get away but the trained models are not, and I've found that they are a starting point, not a final destination. You still have to do more training on top of them for anything that isn't black text on a crisp white background.
nomel 2y ago
> you can also run it multiple times to catch errors. Does this require a slight offset and/or rotation to the image, or just literal rerun, with seed seed/whatever giving a different result?

abrichr 2y ago

It's very reliable for GUI segment understanding; see e.g. https://github.com/OpenAdaptAI/OpenAdapt/pull/610 https://github.com/OpenAdaptAI/OpenAdapt/pull/610 (scroll down to `gpt-4-vision-preview`).

amelius 2y ago
Can it be used for automatic annotation? As in: you tell it that these and these parts should be masked such and such, and then it does that?
- abrichr 2y ago
  We have not had success with that unfortunately.
  - amelius 2y ago
    Thank you, your comment will save me some trouble ;)

alexcnwy 2y ago

the coolest use case i've seen is this https://github.com/ddupont808/GPT-4V-Act https://github.com/ddupont808/GPT-4V-Act

nucleative 2y ago
I feel like this is the beginning of the end for all captchas
- udev4096 2y ago
  Image based or any kind of visual captchas will never be extremely effective. I think we will see more of PoW captchas in the upcoming years (just like cloudflare's turnstile captcha)
  - fennecfoxy 2y ago
    I'm not suer about that, can't you give GPT4 a math problem in an image already and have it solve it correctly most of the time? And these haven't even been trained to defeat captchas/logic problem captchas yet, if it was fine tuned on the general pattern of them I imagine any form of captcha is bust.
  - mewpmewp2 2y ago
    Not sure about PoW. What about people with slower phones etc? I don't see how an older smartphone could meaningfully outcompute a spamming infra.
- jorvi 2y ago
  Twitter / X has a very interesting captcha: you get to see 10 objects that have weird colors and are slightly deformed, and then you have to match them (1 at a time) with another row that has the same objects but seen from a different angle. Of course eventually this will be defeated too, but for now it seems to work pretty well.