4 ms·
It's been done. For example it is possible to take an image through a CNN, generate box proposals and make a list of embeddings extracted from the boxes. That w
by visarga 6y ago
It's been done. For example it is possible to take an image through a CNN, generate box proposals and make a list of embeddings extracted from the boxes. That would be a bag of visual tokens. Then continue the sequence with text tokens and train it to solve a reasoning task.
Another example: Unicoder-VL https://arxiv.org/pdf/1908.06066.pdf https://arxiv.org/pdf/1908.06066.pdf