5 ms·

Over Easter I asked GPT4 to count a mess of colored eggs on the floor that I was preparing for an egg hunt. They were mostly evenly separated and clearly visib

by ugh123 3y ago

Over Easter I asked GPT4 to count a mess of colored eggs on the floor that I was preparing for an egg hunt. They were mostly evenly separated and clearly visible (there were just 36). I gave it two tries to respond and it wasn't even close to the correct answer. Was it confused on colored eggs vs. "natural" eggs it might have been expecting? Should it have understood what I meant?

ravenstine 3y ago
I imagine it would be better at describing an image in a general sense, but probably isn't processing it in a sense where it would actually be counting individual features. I could be wrong about that, but it seems like a combination of traditional CV and an LLM might be what's needed for more precise feature identification.
- 0x008 3y ago
  The use case of “counting objects” is basically completely solved already by yolov8. There is no need to use an LLM for that.
  - ugh123 3y ago
    Seems like we're pretty far away from "AGI" or even anything resembling a legit "multi model"
    - ugh123 3y ago
      multi modal
  - philipwhiuk 3y ago
    I need an AI model to tell me which AI model/tool to use.
    - 0x008 3y ago
      Then you might want to look into query routing or function calling.
Sharlin 3y ago
I'm pretty sure LLM vision capabilities are right now limited to something similar to subitizing in humans at best, ie. being able to perceive the number of items less than ~7 without counting. Expecting it to be able to actually count objects is a bit too much.
fzzzy 3y ago
LLMs are very bad at counting.
tkgally 3y ago
I've been disappointed with both GPT-4's and Gemini 1.5's image recognition abilities in general, not just counting. When I have asked them to describe a photo containing multiple objects--a street scene, a room--they identify some of the objects correctly but invariably hallucinate others, naming things that are not present. Usually the hallucinations are of things that might appear in similar photos but definitely are not in the photo I gave them.

by ugh123 3y ago

ravenstine 3y ago

I imagine it would be better at describing an image in a general sense, but probably isn't processing it in a sense where it would actually be counting individual features. I could be wrong about that, but it seems like a combination of traditional CV and an LLM might be what's needed for more precise feature identification.

0x008 3y ago
The use case of “counting objects” is basically completely solved already by yolov8. There is no need to use an LLM for that.
- ugh123 3y ago
  Seems like we're pretty far away from "AGI" or even anything resembling a legit "multi model"
  - ugh123 3y ago
    multi modal
- philipwhiuk 3y ago
  I need an AI model to tell me which AI model/tool to use.
  - 0x008 3y ago
    Then you might want to look into query routing or function calling.

Sharlin 3y ago

I'm pretty sure LLM vision capabilities are right now limited to something similar to subitizing in humans at best, ie. being able to perceive the number of items less than ~7 without counting. Expecting it to be able to actually count objects is a bit too much.

fzzzy 3y ago

LLMs are very bad at counting.

tkgally 3y ago

I've been disappointed with both GPT-4's and Gemini 1.5's image recognition abilities in general, not just counting. When I have asked them to describe a photo containing multiple objects--a street scene, a room--they identify some of the objects correctly but invariably hallucinate others, naming things that are not present. Usually the hallucinations are of things that might appear in similar photos but definitely are not in the photo I gave them.