5 ms·

I would imagine that this is a hobby project and you're losing cash on it. But what would be the parameters of a profitable business? At what level of scale (

by jawns 10y ago

I would imagine that this is a hobby project and you're losing cash on it. But what would be the parameters of a profitable business? At what level of scale (if any) would it have to operate? And is there a lot of competition in this space?

jacquesm 10y ago
Well, yes. It is definitely not going to make money if I count my time. But I learned an awful lot about machine learning and the present state of computer vision, far more than if I had tried to do that on something abstract. And when I look at the Lego bought-and-sold it seems to work out ok. As for competition, yes, plenty, but all manual. Scaling up now that I have the software working is definitely an option but I have a good set of very well paying customers and not much can compete with that.
- dxbydt 10y ago
  Frankly, the "present state of cv" as you call it, is far far ahead of the 5% classification errors you are running into on a standardized textbook problem such as this - the Legos are all blocky discrete plastic with super-saturated colors and very distinct features. You can easily drop the error by taking multiple pictures from different angles, using ImageDataGenerator during training ( since you are using Keras), especially with horiz and vert flips, and the usual set of tuning tricks - going deeper, dropout, l2 etc. No, the real takeaway is that you have built something very impressive - a tangible physical classifier that uses vgg to get the error down to 5% with very little effort. Going from there to 0.5% would be quite easy if you post the training set so a bunch of people can try different things, see what works best for this situation. Last week I trained a car to drive around a circular track with just a stupid mnist classifier, only 9 classes, 10 lines of Keras! State of cv is very far ahead these days.
  - jacquesm 10y ago
    That 5% is based on a relatively small set of images that is expanding rapidly (4K or so per run), so I expect that error to get lower quickly. Also, these images are stitched and so are far from perfect, the colors are all over the map due to discoloration and so on. Then there are certain nasty aliasing effects where a part will look exactly like another part from the angles the camera can look at the part. This is fixable but will require (yet) another redesign of the optics. Dropout is at .5 both for conv and connected layers. All those things you mention have already been done and yet the error is as high as it is. But this does not surprise me at all, the difference is that the pre-trained models actually do very poorly on this dataset and so I'm having to train them up from scratch. That's getting there though, another week or two and I should be in the tens of thousands of images to train with and then life will get a lot easier. Data augmentation works well and has been enabled right from the get-go. I'll definitely do another write-up in a while about the software end of things, once it crystalizes to something a bit more stable, I'm still hacking on it daily. I will post the training set at some point, first it needs to become large enough. But with the machine helping now that is getting there, rapidly.
    - sorenjan 10y ago
      > Also, these images are stitched and so are far from perfect Maybe you could use a push broom scanner? I don't know if there's any hardware available for consumers, otherwise that seems ideal for a conveyor belt application.
      - jacquesm 10y ago
        This is a very rough equivalent of that. The area outside the stitched part is used to keep the belt synchronized if there are not parts in view, hence that wavy line on the belt in the video.
    - grogenaut 10y ago
      thoughts on tumbling/turning the parts on the belt once identified for more pictures? It wouldn't add anything to the sort speed but it'd get you way more pictures. Thinking of that machine that tumbled dice automatically. Does the addition/removal of the belt in the background come into the recognition? Otherwise in air shots from a tumble tower or belt with trip on it would get you a lot more images.
      - jacquesm 10y ago
        The belt movement is continuous, parts that do roll around on the belt are coming out crazily warped because in each image the belt has moved 3 to 5 mm and the part has moved the same but probably in a different direction than the one the belt is moving in. Those images are useless. This scheme would work if you could always get the parts out of a single frame on the camera but they're much too long for that (24 studs long is the maximum).
        grogenaut 10y ago
        What cameras are you using? I couldn't see it very well in the writeup. Also thanks very much for this post it's very cool. You should cross post to hackaday blog.
- pogo 10y ago
  We have a local Lego re-seller (https://bricksandminifigs.com/ https://bricksandminifigs.com/) which I visit occasionally. One time I spoke to the owner about the business economics. They try to buy complete kits. It is not economical to re-create kits from parts if you sort manually. When they buy unsorted Lego collections from collectors/enthusiasts, they do a limited manual sort to pull out the most valuable pieces and then they store the leftovers. When I spoke to her, I believe she said she had 2 storage units filled with Lego that aren't economical to sort further. Run-of-the-mill bricks and pieces have no value other than scrap. The value is primarily in mini-figs and rare specialty pieces like clear acrylic canopies. Obviously, kits are valuable, but only if you don't have to create them by manually sorting. Also, used kits are much more valuable if they are sold already assembled as proof that all of the pieces are included. This assembly labor must be factored into the business model. Buying lots off of Ebay, it seems like you risk buying presorted Lego which have already been picked over for the most valuable pieces.

by jawns 10y ago

jacquesm 10y ago

Well, yes. It is definitely not going to make money if I count my time. But I learned an awful lot about machine learning and the present state of computer vision, far more than if I had tried to do that on something abstract. And when I look at the Lego bought-and-sold it seems to work out ok. As for competition, yes, plenty, but all manual. Scaling up now that I have the software working is definitely an option but I have a good set of very well paying customers and not much can compete with that.

dxbydt 10y ago
Frankly, the "present state of cv" as you call it, is far far ahead of the 5% classification errors you are running into on a standardized textbook problem such as this - the Legos are all blocky discrete plastic with super-saturated colors and very distinct features. You can easily drop the error by taking multiple pictures from different angles, using ImageDataGenerator during training ( since you are using Keras), especially with horiz and vert flips, and the usual set of tuning tricks - going deeper, dropout, l2 etc. No, the real takeaway is that you have built something very impressive - a tangible physical classifier that uses vgg to get the error down to 5% with very little effort. Going from there to 0.5% would be quite easy if you post the training set so a bunch of people can try different things, see what works best for this situation. Last week I trained a car to drive around a circular track with just a stupid mnist classifier, only 9 classes, 10 lines of Keras! State of cv is very far ahead these days.
- jacquesm 10y ago
  That 5% is based on a relatively small set of images that is expanding rapidly (4K or so per run), so I expect that error to get lower quickly. Also, these images are stitched and so are far from perfect, the colors are all over the map due to discoloration and so on. Then there are certain nasty aliasing effects where a part will look exactly like another part from the angles the camera can look at the part. This is fixable but will require (yet) another redesign of the optics. Dropout is at .5 both for conv and connected layers. All those things you mention have already been done and yet the error is as high as it is. But this does not surprise me at all, the difference is that the pre-trained models actually do very poorly on this dataset and so I'm having to train them up from scratch. That's getting there though, another week or two and I should be in the tens of thousands of images to train with and then life will get a lot easier. Data augmentation works well and has been enabled right from the get-go. I'll definitely do another write-up in a while about the software end of things, once it crystalizes to something a bit more stable, I'm still hacking on it daily. I will post the training set at some point, first it needs to become large enough. But with the machine helping now that is getting there, rapidly.
  - sorenjan 10y ago
    > Also, these images are stitched and so are far from perfect Maybe you could use a push broom scanner? I don't know if there's any hardware available for consumers, otherwise that seems ideal for a conveyor belt application.
    - jacquesm 10y ago
      This is a very rough equivalent of that. The area outside the stitched part is used to keep the belt synchronized if there are not parts in view, hence that wavy line on the belt in the video.
  - grogenaut 10y ago
    thoughts on tumbling/turning the parts on the belt once identified for more pictures? It wouldn't add anything to the sort speed but it'd get you way more pictures. Thinking of that machine that tumbled dice automatically. Does the addition/removal of the belt in the background come into the recognition? Otherwise in air shots from a tumble tower or belt with trip on it would get you a lot more images.
    - jacquesm 10y ago
      The belt movement is continuous, parts that do roll around on the belt are coming out crazily warped because in each image the belt has moved 3 to 5 mm and the part has moved the same but probably in a different direction than the one the belt is moving in. Those images are useless. This scheme would work if you could always get the parts out of a single frame on the camera but they're much too long for that (24 studs long is the maximum).
      - grogenaut 10y ago
        What cameras are you using? I couldn't see it very well in the writeup. Also thanks very much for this post it's very cool. You should cross post to hackaday blog.
pogo 10y ago
We have a local Lego re-seller (https://bricksandminifigs.com/ https://bricksandminifigs.com/) which I visit occasionally. One time I spoke to the owner about the business economics. They try to buy complete kits. It is not economical to re-create kits from parts if you sort manually. When they buy unsorted Lego collections from collectors/enthusiasts, they do a limited manual sort to pull out the most valuable pieces and then they store the leftovers. When I spoke to her, I believe she said she had 2 storage units filled with Lego that aren't economical to sort further. Run-of-the-mill bricks and pieces have no value other than scrap. The value is primarily in mini-figs and rare specialty pieces like clear acrylic canopies. Obviously, kits are valuable, but only if you don't have to create them by manually sorting. Also, used kits are much more valuable if they are sold already assembled as proof that all of the pieces are included. This assembly labor must be factored into the business model. Buying lots off of Ebay, it seems like you risk buying presorted Lego which have already been picked over for the most valuable pieces.