4 ms·

Incredible Machine! Question: Were you able to utilise any data about Lego parts from Lego's own catalogues (current and historal) or technical specifications?

Incredible Machine! Question: Were you able to utilise any data about Lego parts from Lego's own catalogues (current and historal) or technical specifications? It sounds like you trained the classifier manually. I imagine if you want to sort into sets you need to know what makes up a particular set.....does Lego provide an API or anything regarding parts/sets? Further to that, if you have pricing data on sets you have a nice little optimisation problem - given my metric ton of parts, what are the most valuable complete sets I can make?

jacquesm 9y ago
> Were you able to utilise any data about Lego parts from Lego's own catalogues (current and historal) or technical specifications? I tried, but in the end a straight up train-correct-retrain loop took care of all the edge cases much quicker and much more reliable than any feature engineering and database correlation that I tried before. This is roughly the fourth incarnation of the software and by far the most clean and effective. HN pointed me in the direction of Keras a few weeks ago, that coupled with Jeremy Howard's course gave me the keys to finally crack the software in a decisive way. > It sounds like you trained the classifier manually. Only the first batch, after that it was mostly corrections. What it does is while it classifies one batch it saves a log which gives me more data to feed the classifier with for the next training session. There are so few errors now that I can add another 4K images to the training set in half an hour or so. > Further to that, if you have pricing data on sets you have a nice little optimisation problem - given my metric ton of parts, what are the most valuable complete sets I can make? I'm on that one :) And a few others that are not so obvious. There is a lot to know about lego. Far more than you'd think at first glance.
- ma2rten 9y ago
  You could also use mechanical turk to label your dataset.
  - jacquesm 9y ago
    Can't use MT from nl...
    - ma2rten 9y ago
      Oh, I forgot about that. Is there no European alternative?
      - jacquesm 9y ago
        Not that I know of. But in the end I found a much better and faster solution so in a way that constraint only pushed creativity.
- jncraton 9y ago
  > I tried, but in the end a straight up train-correct-retrain loop took care of all the edge cases much quicker and much more reliable than any feature engineering and database correlation that I tried I'd love to hear more about what you tried specifically. I'm considering doing this myself, and I was thinking of building a very large labeled dataset of 3d rendered images using the LDraw parts library and training on that. I could include hundreds of images per part by using different viewing angles, zoom levels, focus, etc in the rendering process. Did you try anything like that?
  - jacquesm 9y ago
    That's fairly pointless for me. There are some intricacies in the optics that would need to be modeled as well as the specifics of the camera and the light path, and then you'd still have all the weird gravity related trickery: parts that end up on top of each other, parts that can only be found in one or two ways on the belt and so on. After endless messing around I finally bit the bullet and trained a neural net, from 0 to 100 in a few weeks and it is rapidly getting more usable now. The feature detection code may get a second life though: as a meta-data vector to be embedded in to the net. But only if it is really necessary. I'm quite curious though if you can get your method to work, especially for the parts that are very rare and rare colors.
    - jncraton 9y ago
      That's very helpful. Thanks! I was assuming that at minimum I'd need to do a lot of filtering in order to get the camera images and renders into a state where they are similar enough to work for training. Any chance that you'll be releasing source code for this project and/or your labeled dataset?
      - jacquesm 9y ago
        > Any chance that you'll be releasing source code for this project and/or your labeled dataset? Yes, but not yet. It needs to get a lot better before I'm going to stamp my name on it as a release. Right now it is rather embarrassing from a code quality point of view, it has been ripped apart and put together several times now and every time it gets a lot better but we're not there yet.
- microcolonel 9y ago
  One thing that I think could work is generating imagery from the catalogue for parts you've not seen before. Initialize a random starting position, and use bullet to have them fall into a realistic position on a plane. then raytrace from the top. This way you could also cross-reference with part seller sites to see the going rate for a part and determine whether it's worth your time to separate it manually. Have a bin for rare parts worth separating by hand.
  - jacquesm 9y ago
    That's clever!
- make3 9y ago
  did you first download and use the pretrained net of an different classification task and use that as a default ? also, did you use batch normalization? also, did you try ResNets? you probably don't care at this point, but all of this would __massively__ decrease training time
  - jacquesm 9y ago
    > did you first download and use the pretrained net of an different classification task and use that as a default ? Yes, but that did not give me accuracy enough, so now I train from scratch. I had hoped to save having to train the conv layers. > also, did you use batch normalization? Yes. > also, did you try ResNets? No. > you probably don't care at this point, but all of this would __massively__ decrease training time Oh, I care all right :) I'm re-training the net every evening (it's running right now) after adding another batch of training images.

jacquesm 9y ago

> Were you able to utilise any data about Lego parts from Lego's own catalogues (current and historal) or technical specifications? I tried, but in the end a straight up train-correct-retrain loop took care of all the edge cases much quicker and much more reliable than any feature engineering and database correlation that I tried before. This is roughly the fourth incarnation of the software and by far the most clean and effective. HN pointed me in the direction of Keras a few weeks ago, that coupled with Jeremy Howard's course gave me the keys to finally crack the software in a decisive way. > It sounds like you trained the classifier manually. Only the first batch, after that it was mostly corrections. What it does is while it classifies one batch it saves a log which gives me more data to feed the classifier with for the next training session. There are so few errors now that I can add another 4K images to the training set in half an hour or so. > Further to that, if you have pricing data on sets you have a nice little optimisation problem - given my metric ton of parts, what are the most valuable complete sets I can make? I'm on that one :) And a few others that are not so obvious. There is a lot to know about lego. Far more than you'd think at first glance.

ma2rten 9y ago
You could also use mechanical turk to label your dataset.
- jacquesm 9y ago
  Can't use MT from nl...
  - ma2rten 9y ago
    Oh, I forgot about that. Is there no European alternative?
    - jacquesm 9y ago
      Not that I know of. But in the end I found a much better and faster solution so in a way that constraint only pushed creativity.
jncraton 9y ago
> I tried, but in the end a straight up train-correct-retrain loop took care of all the edge cases much quicker and much more reliable than any feature engineering and database correlation that I tried I'd love to hear more about what you tried specifically. I'm considering doing this myself, and I was thinking of building a very large labeled dataset of 3d rendered images using the LDraw parts library and training on that. I could include hundreds of images per part by using different viewing angles, zoom levels, focus, etc in the rendering process. Did you try anything like that?
- jacquesm 9y ago
  That's fairly pointless for me. There are some intricacies in the optics that would need to be modeled as well as the specifics of the camera and the light path, and then you'd still have all the weird gravity related trickery: parts that end up on top of each other, parts that can only be found in one or two ways on the belt and so on. After endless messing around I finally bit the bullet and trained a neural net, from 0 to 100 in a few weeks and it is rapidly getting more usable now. The feature detection code may get a second life though: as a meta-data vector to be embedded in to the net. But only if it is really necessary. I'm quite curious though if you can get your method to work, especially for the parts that are very rare and rare colors.
  - jncraton 9y ago
    That's very helpful. Thanks! I was assuming that at minimum I'd need to do a lot of filtering in order to get the camera images and renders into a state where they are similar enough to work for training. Any chance that you'll be releasing source code for this project and/or your labeled dataset?
    - jacquesm 9y ago
      > Any chance that you'll be releasing source code for this project and/or your labeled dataset? Yes, but not yet. It needs to get a lot better before I'm going to stamp my name on it as a release. Right now it is rather embarrassing from a code quality point of view, it has been ripped apart and put together several times now and every time it gets a lot better but we're not there yet.
microcolonel 9y ago
One thing that I think could work is generating imagery from the catalogue for parts you've not seen before. Initialize a random starting position, and use bullet to have them fall into a realistic position on a plane. then raytrace from the top. This way you could also cross-reference with part seller sites to see the going rate for a part and determine whether it's worth your time to separate it manually. Have a bin for rare parts worth separating by hand.
- jacquesm 9y ago
  That's clever!
make3 9y ago
did you first download and use the pretrained net of an different classification task and use that as a default ? also, did you use batch normalization? also, did you try ResNets? you probably don't care at this point, but all of this would __massively__ decrease training time
- jacquesm 9y ago
  > did you first download and use the pretrained net of an different classification task and use that as a default ? Yes, but that did not give me accuracy enough, so now I train from scratch. I had hoped to save having to train the conv layers. > also, did you use batch normalization? Yes. > also, did you try ResNets? No. > you probably don't care at this point, but all of this would __massively__ decrease training time Oh, I care all right :) I'm re-training the net every evening (it's running right now) after adding another batch of training images.