5 ms·

Visual question answering using CNN+RNN

108 points by abhshkdz 11y ago

KnightHawk3 11y ago
Is there a catch to the effectiveness of this? I haven't seen it before and it seems pretty magical.
- abhshkdz 11y ago
  Although there is no catch, it's far from perfect and hardly magical. Its accuracy goes up to ~55% on the VQA (http://visualqa.org/ http://visualqa.org/) dataset (which is short of state-of-the-art by ~7%).
- aidenn0 11y ago
  Did you look at the examples? Many of them are wrong, and others are guessable just from the question (e.g. "What shape is the plate?" I would say "round" without a picture)
n0us 11y ago
Amazing. I only had a chance to read the README.md but my question is this. What happens if you ask it questions that it could not possibly answer, as in if it were given a picture of the man playing tennis and you asked it what the score was? Is it capable of discerning between questions that cannot be answered (given a particular input) and those that can?
- abhshkdz 11y ago
  Priors from the language play a much bigger role in the answers that are predicted than the image itself. So for example, if you ask 'What color is ...?', irrespective of the image, it is more likely to spit out colors as the answer. The answers are usually well-aligned with the question that is being asked. 'Yes/no' for binary questions, 'red/blue/etc' for 'What color...', 'tennis/baseball/etc' for 'What sport...' and so on.
mrdrozdov 11y ago
How do you measure accuracy? Is this a new baseline?
- abhshkdz 11y ago
  Accuracy is measured as min((number of humans that provided that answer)/3, 1) i.e. 100% accurate if at least 3 humans provided that exact answer, as outlined here: http://visualqa.org/evaluation.html http://visualqa.org/evaluation.html. No, this model is from the NIPS15 paper by Ren et al (http://arxiv.org/abs/1505.02074 http://arxiv.org/abs/1505.02074).
arocks 11y ago
I have seen sites using captchas which ask such visual questions thinking that only a human can answer them. This project really makes me doubt the effectiveness of such techniques.
- abhshkdz 11y ago
  As it stands currently, it's quite far off from cracking captchas. :-)

108 points by abhshkdz 11y ago

KnightHawk3 11y ago

Is there a catch to the effectiveness of this? I haven't seen it before and it seems pretty magical.

abhshkdz 11y ago
Although there is no catch, it's far from perfect and hardly magical. Its accuracy goes up to ~55% on the VQA (http://visualqa.org/ http://visualqa.org/) dataset (which is short of state-of-the-art by ~7%).
aidenn0 11y ago
Did you look at the examples? Many of them are wrong, and others are guessable just from the question (e.g. "What shape is the plate?" I would say "round" without a picture)

n0us 11y ago

Amazing. I only had a chance to read the README.md but my question is this. What happens if you ask it questions that it could not possibly answer, as in if it were given a picture of the man playing tennis and you asked it what the score was? Is it capable of discerning between questions that cannot be answered (given a particular input) and those that can?

abhshkdz 11y ago
Priors from the language play a much bigger role in the answers that are predicted than the image itself. So for example, if you ask 'What color is ...?', irrespective of the image, it is more likely to spit out colors as the answer. The answers are usually well-aligned with the question that is being asked. 'Yes/no' for binary questions, 'red/blue/etc' for 'What color...', 'tennis/baseball/etc' for 'What sport...' and so on.

mrdrozdov 11y ago

How do you measure accuracy? Is this a new baseline?

abhshkdz 11y ago
Accuracy is measured as min((number of humans that provided that answer)/3, 1) i.e. 100% accurate if at least 3 humans provided that exact answer, as outlined here: http://visualqa.org/evaluation.html http://visualqa.org/evaluation.html. No, this model is from the NIPS15 paper by Ren et al (http://arxiv.org/abs/1505.02074 http://arxiv.org/abs/1505.02074).

arocks 11y ago

I have seen sites using captchas which ask such visual questions thinking that only a human can answer them. This project really makes me doubt the effectiveness of such techniques.

abhshkdz 11y ago
As it stands currently, it's quite far off from cracking captchas. :-)