3 ms·

Is there read really a Python library called ImagePatch that can find any item in an image, and it works as well as in this video? Google didn’t find an obvious

by vosper 4y ago

Is there read really a Python library called ImagePatch that can find any item in an image, and it works as well as in this video? Google didn’t find an obvious match for “Python ImagePatch”

SCUSKU 4y ago
Looks like they haven't released their code yet, but my guess is that it's an in house wrapper around CLIP or something similar?
leobg 4y ago
There is a GitHub repo / Python lib called com2fun which exploits this. Allows you to get results from functions that you only pretend exist. (Am on mobile and can’t link to it right now.)
leobg 4y ago
I guess the idea is to trick the model into generating pseudo code. Which really doesn’t do much more than to act as a “scratchpad“ to focus the attention of the model to reason through the problem. Besides, the Codex models are free right now. So… one more reason to rephrase questions as coding questions ;-)
- vosper 4y ago
  Oh, so maybe I misunderstood what I was seeing. It wrote pseudo-code that makes sense conceptually, not code that I can paste in Jupyter and run (given the right imports)? That sure wasn't obvious from the video.
  - famouswaffles 4y ago
    It's not actully pseudocode. If you read the paper, these are functions/libraries they introduce that haven't been published to github yet.
- famouswaffles 4y ago
  It's not actully pseudocode. If you read the paper, these are functions/libraries they introduce that haven't been published to github yet.
make3 4y ago
it's just a separate vision model. you just have to use a state of the art instance segmentation model, the task shown are really not that hard. it's not "just a library"
- vosper 4y ago
  So the code that was written by the AI in the video doesn't actually work as written?
  - famouswaffles 4y ago
    It does. If you red the paper, these are functions/libraries they introduce that haven't been published to github yet.
VectorLock 4y ago
Almost as interesting as the GPT part of it.
drothlis 4y ago
According to the ViperGPT paper their "ImagePatch.find()" uses GLIP. According to the GLIP paper,† accuracy on a test-set not seen during training is around 60% so... neat demos but whether it'll be reliable enough depends on your application. † https://arxiv.org/abs/2206.05836 https://arxiv.org/abs/2206.05836

by vosper 4y ago

Is there read really a Python library called ImagePatch that can find any item in an image, and it works as well as in this video? Google didn’t find an obvious match for “Python ImagePatch”

SCUSKU 4y ago

Looks like they haven't released their code yet, but my guess is that it's an in house wrapper around CLIP or something similar?

leobg 4y ago

There is a GitHub repo / Python lib called com2fun which exploits this. Allows you to get results from functions that you only pretend exist. (Am on mobile and can’t link to it right now.)

leobg 4y ago

I guess the idea is to trick the model into generating pseudo code. Which really doesn’t do much more than to act as a “scratchpad“ to focus the attention of the model to reason through the problem. Besides, the Codex models are free right now. So… one more reason to rephrase questions as coding questions ;-)

vosper 4y ago
Oh, so maybe I misunderstood what I was seeing. It wrote pseudo-code that makes sense conceptually, not code that I can paste in Jupyter and run (given the right imports)? That sure wasn't obvious from the video.
- famouswaffles 4y ago
  It's not actully pseudocode. If you read the paper, these are functions/libraries they introduce that haven't been published to github yet.
famouswaffles 4y ago
It's not actully pseudocode. If you read the paper, these are functions/libraries they introduce that haven't been published to github yet.

make3 4y ago

it's just a separate vision model. you just have to use a state of the art instance segmentation model, the task shown are really not that hard. it's not "just a library"

vosper 4y ago
So the code that was written by the AI in the video doesn't actually work as written?
- famouswaffles 4y ago
  It does. If you red the paper, these are functions/libraries they introduce that haven't been published to github yet.

VectorLock 4y ago

Almost as interesting as the GPT part of it.

drothlis 4y ago

According to the ViperGPT paper their "ImagePatch.find()" uses GLIP. According to the GLIP paper,† accuracy on a test-set not seen during training is around 60% so... neat demos but whether it'll be reliable enough depends on your application. † https://arxiv.org/abs/2206.05836 https://arxiv.org/abs/2206.05836