5 ms·

Surely that's not true until someone achieves human-level performance with their entry.

abrichr 11y ago
Human level performance was surpassed earlier this year: http://arxiv.org/abs/1502.01852 http://arxiv.org/abs/1502.01852
- natch 11y ago
  I won't be surprised when human level performance is surpassed, but isn't that paper based on work that has the advantage of knowing the full test dataset in advance? You can say "yes but they only trained on the training data" but that doesn't rule out tweaking across several experiments and measuring them against the known test data, then cherry picking the experiment that was best overfit to that data, right? I'm not saying there are shenanigans here, just wondering how you know there are not.
  - argonaut 11y ago
    The test labels are not known to them. The test labels are held by a different team that tells them their test error rate. Furthermore, multiple, separate teams have surpassed human performance. FWIW, "human performance" is not as meaningful as most people make it out to be, because this is such a narrow task.
    - tim333 11y ago
      The paper mentions it's unfair to humans to have to tell say a coucal from an indigo bunting (both blue coloured birds). I've never heard of either so I guess I would have got that wrong but it's not the fault of my visual system.
      - p1esk 11y ago
        But you only have to guess right once in 5 tries!
    - natch 11y ago
      I replied to abrichr's link. Did you look at it at all? They used a dataset from 2012. Are you saying that the test labels from a 2012 dataset have been kept secret until now? Really?
      - argonaut 11y ago
        I am familiar with the ImageNet competition. Let me know if you find the ImageNet test labels. (Hint: they are not released, since they are re-used for competitions).
        natch 11y ago
        Interesting, did not know that. Do you think that proves this team was not able to get and exploit some inside knowledge? Judging from recent competitions, ethics does not seem to be a strong point in some cultures.
  - reacweb 11y ago
    Did they train also the human on the training data ?

Human level performance was surpassed earlier this year: http://arxiv.org/abs/1502.01852 http://arxiv.org/abs/1502.01852

natch 11y ago
I won't be surprised when human level performance is surpassed, but isn't that paper based on work that has the advantage of knowing the full test dataset in advance? You can say "yes but they only trained on the training data" but that doesn't rule out tweaking across several experiments and measuring them against the known test data, then cherry picking the experiment that was best overfit to that data, right? I'm not saying there are shenanigans here, just wondering how you know there are not.
- argonaut 11y ago
  The test labels are not known to them. The test labels are held by a different team that tells them their test error rate. Furthermore, multiple, separate teams have surpassed human performance. FWIW, "human performance" is not as meaningful as most people make it out to be, because this is such a narrow task.
  - tim333 11y ago
    The paper mentions it's unfair to humans to have to tell say a coucal from an indigo bunting (both blue coloured birds). I've never heard of either so I guess I would have got that wrong but it's not the fault of my visual system.
    - p1esk 11y ago
      But you only have to guess right once in 5 tries!
  - natch 11y ago
    I replied to abrichr's link. Did you look at it at all? They used a dataset from 2012. Are you saying that the test labels from a 2012 dataset have been kept secret until now? Really?
    - argonaut 11y ago
      I am familiar with the ImageNet competition. Let me know if you find the ImageNet test labels. (Hint: they are not released, since they are re-used for competitions).
      - natch 11y ago
        Interesting, did not know that. Do you think that proves this team was not able to get and exploit some inside knowledge? Judging from recent competitions, ethics does not seem to be a strong point in some cultures.
- reacweb 11y ago
  Did they train also the human on the training data ?