3 ms·

It's also stunning how poorly it performs in a real world (i.e., non-human-prefiltered photo) scenario. Like autonomous cars or security cameras or robotics.

by pakl 9y ago

jacquesm 9y ago
But it's stunningly better than it was 3 years ago. It's like going from crawling to walking. That takes a long time. But going from walking to running is much easier. We're not yet at a 'general vision is a solved problem' level and this may take a long time still - if it is ever solved. But many problems can be reduced to the point that they become tractable even absent 'general vision'. Affordable LIDAR is a huge step, so is radar. Those two reduce the problem of vision with two crappy stereoscopic cameras set 5" apart to something much closer to the domain. As far as I can see the race is on.
- Retric 9y ago
  We are currently overfitting the public datasets. The only valid test is a new never been used image set.
  - sgt101 9y ago
    try it for yourself - my team did, initially I sent the guy doing the tests away for screwing up the sums because I didn't believe it could be as good as he said. I was wrong - it was. This was a relatively early CNN as well. AGNN and RNN are revolutions in themselves and everyone is now scrambling to understand and harness them. Our GPU kit has got 6 times better in 2 years. Exciting times.
    - Retric 9y ago
      Hand someone ideally a 3-5 year old an average digital camera and have them walk around taking random pictures where half of them are a blurry mess and the results are much worse. Or drive around with a camera on you bumper etc. Public Datasets have style of photo that represent it's own bias.
      - jacquesm 9y ago
        > Hand someone ideally a 3-5 year old an average digital camera and have them walk around taking random pictures where half of them are a blurry mess and the results are much worse. Garbage in, garbage out. When was it any different? The question is: can a human do much better on that garbage? > Public Datasets have style of photo that represent it's own bias. That I readily agree with.
        Retric 9y ago
        The problem is self driving cars need to work with Garbage when their sensor is covered by slush from a puddle, scratches on the glass etc etc. The problem is we are not training them on Garbage which is it's own problem. Scoring is also critical, you want to gracefully degrade classification systems not getting the correct dog breed is insignificant vs calling a dog a sofa. Further, people base real world classification on stereoscopic video footage. Training robots to classify photos is a handicap for building robots to operate in the real world.
      - sgt101 9y ago
        Our data set was from surveys collected for human use using a variety of digital cameras - not DSLRs though. The surveyors did take specific images, blurry images would have meant not getting paid so none of those. We used transfer learning from the imagenet network vs a few hundred of the images well labelled and we trained it on a 6 gpu machine (titan x's) This is no place to claim results or do the detailed write up so all I would say is that do try for yourself if you haven't already, I was shocked by the quality of the classifier. Maybe you will be too! I started from the dataset bias position myself - billions of family snaps and selfies can't provide the reference for arbitrary images of stuff from arbitrary angles. My reading of the results that we got is that the claims of lower level features extracted by the cnn are correct, and that a network trained on massive public data can be the basis for specialised tools trained on more constrained proprietary datasets. Your milage may vary though...
      - sgt101 9y ago
        Our data set was from surveys collected for human use using a variety of digital cameras - not DSLRs though. The surveyors did take specific images, blurry images would have meant not getting paid so none of those. We used transfer learning from the imagenet network vs a few hundred of the images well labelled and we trained it on a 6 gpu machine (titan x's) This is no place to claim results or do the detailed write up so all I would say is that do try for yourself if you haven't already, I was shocked by the quality of the classifier. Maybe you will be too! I started from the dataset bias position myself - billions of family snaps and selfies can't provide the reference for arbitrary images of stuff from arbitrary angles. My reading of the results that we got is that the claims of lower level features extracted by the cnn are correct, and that a network trained on massive public data can be the basis for specialised tools trained on more constrained proprietary datasets. Your milage may vary though...
- ianamartin 9y ago
  "Better than three years ago" is a different measure of success than "better than humans" Different tasks have different error tolerances. When an MI tool can get within the acceptable margin of error for certain tasks, then there is a possibility for the MI to take jobs. As long as accuracy is critical to a classifier, in many cases it's better to pay for human eyeballs to look at the criteria. We're talking about a huge spectrum of applications here. Most of us probably don't care that much about the difference between human and MI error rates about classifying images used in broad-scale advertising on the internet. We do (or perhaps should) care very deeply about human vs. MI error rate in terms of granting or not-granting things like parole.

It's also stunning how poorly it performs in a real world (i.e., non-human-prefiltered photo) scenario. Like autonomous cars or security cameras or robotics.

by pakl 9y ago

It's also stunning how poorly it performs in a real world (i.e., non-human-prefiltered photo) scenario. Like autonomous cars or security cameras or robotics.

jacquesm 9y ago

But it's stunningly better than it was 3 years ago. It's like going from crawling to walking. That takes a long time. But going from walking to running is much easier. We're not yet at a 'general vision is a solved problem' level and this may take a long time still - if it is ever solved. But many problems can be reduced to the point that they become tractable even absent 'general vision'. Affordable LIDAR is a huge step, so is radar. Those two reduce the problem of vision with two crappy stereoscopic cameras set 5" apart to something much closer to the domain. As far as I can see the race is on.

Retric 9y ago
We are currently overfitting the public datasets. The only valid test is a new never been used image set.
- sgt101 9y ago
  try it for yourself - my team did, initially I sent the guy doing the tests away for screwing up the sums because I didn't believe it could be as good as he said. I was wrong - it was. This was a relatively early CNN as well. AGNN and RNN are revolutions in themselves and everyone is now scrambling to understand and harness them. Our GPU kit has got 6 times better in 2 years. Exciting times.
  - Retric 9y ago
    Hand someone ideally a 3-5 year old an average digital camera and have them walk around taking random pictures where half of them are a blurry mess and the results are much worse. Or drive around with a camera on you bumper etc. Public Datasets have style of photo that represent it's own bias.
    - jacquesm 9y ago
      > Hand someone ideally a 3-5 year old an average digital camera and have them walk around taking random pictures where half of them are a blurry mess and the results are much worse. Garbage in, garbage out. When was it any different? The question is: can a human do much better on that garbage? > Public Datasets have style of photo that represent it's own bias. That I readily agree with.
      - Retric 9y ago
        The problem is self driving cars need to work with Garbage when their sensor is covered by slush from a puddle, scratches on the glass etc etc. The problem is we are not training them on Garbage which is it's own problem. Scoring is also critical, you want to gracefully degrade classification systems not getting the correct dog breed is insignificant vs calling a dog a sofa. Further, people base real world classification on stereoscopic video footage. Training robots to classify photos is a handicap for building robots to operate in the real world.
    - sgt101 9y ago
      Our data set was from surveys collected for human use using a variety of digital cameras - not DSLRs though. The surveyors did take specific images, blurry images would have meant not getting paid so none of those. We used transfer learning from the imagenet network vs a few hundred of the images well labelled and we trained it on a 6 gpu machine (titan x's) This is no place to claim results or do the detailed write up so all I would say is that do try for yourself if you haven't already, I was shocked by the quality of the classifier. Maybe you will be too! I started from the dataset bias position myself - billions of family snaps and selfies can't provide the reference for arbitrary images of stuff from arbitrary angles. My reading of the results that we got is that the claims of lower level features extracted by the cnn are correct, and that a network trained on massive public data can be the basis for specialised tools trained on more constrained proprietary datasets. Your milage may vary though...
    - sgt101 9y ago
      Our data set was from surveys collected for human use using a variety of digital cameras - not DSLRs though. The surveyors did take specific images, blurry images would have meant not getting paid so none of those. We used transfer learning from the imagenet network vs a few hundred of the images well labelled and we trained it on a 6 gpu machine (titan x's) This is no place to claim results or do the detailed write up so all I would say is that do try for yourself if you haven't already, I was shocked by the quality of the classifier. Maybe you will be too! I started from the dataset bias position myself - billions of family snaps and selfies can't provide the reference for arbitrary images of stuff from arbitrary angles. My reading of the results that we got is that the claims of lower level features extracted by the cnn are correct, and that a network trained on massive public data can be the basis for specialised tools trained on more constrained proprietary datasets. Your milage may vary though...
ianamartin 9y ago
"Better than three years ago" is a different measure of success than "better than humans" Different tasks have different error tolerances. When an MI tool can get within the acceptable margin of error for certain tasks, then there is a possibility for the MI to take jobs. As long as accuracy is critical to a classifier, in many cases it's better to pay for human eyeballs to look at the criteria. We're talking about a huge spectrum of applications here. Most of us probably don't care that much about the difference between human and MI error rates about classifying images used in broad-scale advertising on the internet. We do (or perhaps should) care very deeply about human vs. MI error rate in terms of granting or not-granting things like parole.