5 ms·

> I tried, but in the end a straight up train-correct-retrain loop took care of all the edge cases much quicker and much more reliable than any feature engineer

> I tried, but in the end a straight up train-correct-retrain loop took care of all the edge cases much quicker and much more reliable than any feature engineering and database correlation that I tried I'd love to hear more about what you tried specifically. I'm considering doing this myself, and I was thinking of building a very large labeled dataset of 3d rendered images using the LDraw parts library and training on that. I could include hundreds of images per part by using different viewing angles, zoom levels, focus, etc in the rendering process. Did you try anything like that?

jacquesm 9y ago
That's fairly pointless for me. There are some intricacies in the optics that would need to be modeled as well as the specifics of the camera and the light path, and then you'd still have all the weird gravity related trickery: parts that end up on top of each other, parts that can only be found in one or two ways on the belt and so on. After endless messing around I finally bit the bullet and trained a neural net, from 0 to 100 in a few weeks and it is rapidly getting more usable now. The feature detection code may get a second life though: as a meta-data vector to be embedded in to the net. But only if it is really necessary. I'm quite curious though if you can get your method to work, especially for the parts that are very rare and rare colors.
- jncraton 9y ago
  That's very helpful. Thanks! I was assuming that at minimum I'd need to do a lot of filtering in order to get the camera images and renders into a state where they are similar enough to work for training. Any chance that you'll be releasing source code for this project and/or your labeled dataset?
  - jacquesm 9y ago
    > Any chance that you'll be releasing source code for this project and/or your labeled dataset? Yes, but not yet. It needs to get a lot better before I'm going to stamp my name on it as a release. Right now it is rather embarrassing from a code quality point of view, it has been ripped apart and put together several times now and every time it gets a lot better but we're not there yet.
    - sorenjan 9y ago
      Sounds like a great learning opportunity for beginners and people who haven't even started with machine learning yet, like me. I'm looking forward to it. If you're so inclined, a new blog post focusing on the machine learning process would be much appreciated, it's always interesting to read about the process of how somebody solved a real problem and implemented the solution. Just so I understand the process correctly, did you manually sort some pieces to get a labeled training set, feed those through the machine, train the NN with that, and then manually correct the errors when sorting unknown pieces, added all those pictures to the same training set and then finally run the full training again? How many labeled images do you need to start getting acceptable performance? Are you training the NN continuously with every new image, or from scratch with an increasing data set? Do you think a stereo camera would improve the classification in a meaningful way, or maybe a second camera from a different angle?
      - jacquesm 9y ago
        > did you manually sort some pieces to get a labeled training set, feed those through the machine, train the NN with that, and then manually correct the errors when sorting unknown pieces, added all those pictures to the same training set and then finally run the full training again? Yes, but that cycle repeats every day. So the training never really stops, it just runs at night and the machine runs during the day. Today it sorted close to 10K parts and those images will now be added to the training set and then I'll start the training overnight so tomorrow morning my error rate should be much better than it was today and so on. > How many labeled images do you need to start getting acceptable performance? Good question! Answer: I don't really know but judging by how fast the error rate is improving between 100 and 200 per 'class' so that will be 200K images or so when it is one with the 1000 most commonly found parts. > Are you training the NN continuously with every new image, or from scratch with an increasing data set? From scratch with every expanded set. I suspect that's the better way but I have no proof. My intution is that it is hard to make a neural net learn something entirely new that it has not seen before and every day totally new stuff gets added. So I re-train all the way from noise. > Do you think a stereo camera would improve the classification in a meaningful way, or maybe a second camera from a different angle? You're getting close to the secret sauce :)

jacquesm 9y ago

That's fairly pointless for me. There are some intricacies in the optics that would need to be modeled as well as the specifics of the camera and the light path, and then you'd still have all the weird gravity related trickery: parts that end up on top of each other, parts that can only be found in one or two ways on the belt and so on. After endless messing around I finally bit the bullet and trained a neural net, from 0 to 100 in a few weeks and it is rapidly getting more usable now. The feature detection code may get a second life though: as a meta-data vector to be embedded in to the net. But only if it is really necessary. I'm quite curious though if you can get your method to work, especially for the parts that are very rare and rare colors.

jncraton 9y ago
That's very helpful. Thanks! I was assuming that at minimum I'd need to do a lot of filtering in order to get the camera images and renders into a state where they are similar enough to work for training. Any chance that you'll be releasing source code for this project and/or your labeled dataset?
- jacquesm 9y ago
  > Any chance that you'll be releasing source code for this project and/or your labeled dataset? Yes, but not yet. It needs to get a lot better before I'm going to stamp my name on it as a release. Right now it is rather embarrassing from a code quality point of view, it has been ripped apart and put together several times now and every time it gets a lot better but we're not there yet.
  - sorenjan 9y ago
    Sounds like a great learning opportunity for beginners and people who haven't even started with machine learning yet, like me. I'm looking forward to it. If you're so inclined, a new blog post focusing on the machine learning process would be much appreciated, it's always interesting to read about the process of how somebody solved a real problem and implemented the solution. Just so I understand the process correctly, did you manually sort some pieces to get a labeled training set, feed those through the machine, train the NN with that, and then manually correct the errors when sorting unknown pieces, added all those pictures to the same training set and then finally run the full training again? How many labeled images do you need to start getting acceptable performance? Are you training the NN continuously with every new image, or from scratch with an increasing data set? Do you think a stereo camera would improve the classification in a meaningful way, or maybe a second camera from a different angle?
    - jacquesm 9y ago
      > did you manually sort some pieces to get a labeled training set, feed those through the machine, train the NN with that, and then manually correct the errors when sorting unknown pieces, added all those pictures to the same training set and then finally run the full training again? Yes, but that cycle repeats every day. So the training never really stops, it just runs at night and the machine runs during the day. Today it sorted close to 10K parts and those images will now be added to the training set and then I'll start the training overnight so tomorrow morning my error rate should be much better than it was today and so on. > How many labeled images do you need to start getting acceptable performance? Good question! Answer: I don't really know but judging by how fast the error rate is improving between 100 and 200 per 'class' so that will be 200K images or so when it is one with the 1000 most commonly found parts. > Are you training the NN continuously with every new image, or from scratch with an increasing data set? From scratch with every expanded set. I suspect that's the better way but I have no proof. My intution is that it is hard to make a neural net learn something entirely new that it has not seen before and every day totally new stuff gets added. So I re-train all the way from noise. > Do you think a stereo camera would improve the classification in a meaningful way, or maybe a second camera from a different angle? You're getting close to the secret sauce :)