3 ms·
Would it make sense to have the user click and then use that point as a SAM prompt? It might let you find a book even if the initial SAM query doesn't find it.
by Imnimo 3y ago
Would it make sense to have the user click and then use that point as a SAM prompt? It might let you find a book even if the initial SAM query doesn't find it.
- zerojames 3y agoPost author here. I like this idea. I plan to explore it and make a more generic solution. I'd love to have a point-and-click interface for annotating scenes. For example, I'd like to be able to click on pieces of coffee equipment in a photo of my coffee setup so I can add sticky note annotations when you hover over each item. For the bookshelves idea specifically, I would love to have a correction system in place. The problem isn't so much SAM as it is Grounding DINO, the model I'm using for object identification. I then pass each identified region to SAM and map the segmentation mask to the box. Grounding DINO detects a lot of book spines, but often misses 1-2. I am planning to try out YOLO-World (https://github.com/AILab-CVC/YOLO-World https://github.com/AILab-CVC/YOLO-World), which, in my limited testing, performs better for this task.