17 ms·

Sorting 2 Tons of Lego, Many Questions, Results

322 points by darwhy 9y ago

nappy-doo 9y ago
I've thought about this machine a number of times since the previous post. In that post, there was a movie showing the machine's operation, but it was operating at a slow speed, and he's mentioned multiple times that there's been speed improvements. Does anyone know if there's a realtime movie anywhere?
- jacquesm 9y ago
  The speed of the machine is mostly a function of the mechanical feed speed of the parts. This is the hopper/belt mechanism and it is by far the slowest part of the machine. Once the parts are on the recognition belt things more very fast, a single part can be scanned and recognized in about 30 milliseconds, so roughly 30 parts / second. Making the machine much faster then means making that feed mechanism faster than it is today (roughly 1 part per second tops) and that's the challenge I'm looking at right now. Ideas more than welcome! Even bad ones, you never know which bad idea with a twist can become a good idea or even a very good idea.
  - maxerickson 9y ago
    Perhaps the dumbest idea would be to build a second feeder. Would double the speed and sounds like the rest of it is fast enough that you wouldn't even need to worry about parts falling on top of each other.
    - jacquesm 9y ago
      That's not a dumb idea, however: space is an issue, the rest is fast enough but even so parts do tend to fall in clumps and if two such clumps were to fall at the same time it could lead to a jam which requires manual intervention. Ideally parts would land on the belt in a very regular rhythm, without any kind of irregularity whether the parts are 8 mm round 1x1 plates or 16 long 1x1 bricks. You need about 3 cm of space on either side to make sure the right part gets blown off the belt, that's the most important thing that will eventually limit the speed. Oh, another factor limiting the speed is the maximum acceleration over the stretch towards the camera, otherwise the parts are still moving on the belt when they reach the camera and then you get bad images.
  - gboudrias 9y ago
    Not to state the obvious, but if the "hopper" is slow, why not just have multiple of them? Are they expensive? Take up too much space? I assume you've thought of this but I'm still curious. Really cool stuff by the way!
    - jacquesm 9y ago
      Space wise this is hard, also parts tend to fall in clusters so the second belt now has to be much faster than the first one to make sure there is enough separation. I do have a second belt that I have not used yet so if there is no better idea then that is what I'll end up doing.
      - gboudrias 9y ago
        So just to make sure I got this right, you need a way to get a single piece of any size out of a container full of random pieces, and onto the first moving belt, at a rate higher than 1 piece per second? I'm not familiar with the project but this seems like an interesting challenge.
        jacquesm 9y ago
        Yes, that's roughly the gist of it. This is a hard problem because Lego comes in many shapes and sizes, a pre-sieving step limits the pieces that hit the belt at 42 mm largest cross section. So anything below that (but possibly much longer than that, all the way up to 2x1x16) can be found in the hopper. There are some really good suggestions in this thread, the one that stands out for me most is the brush edged screw feeder, I think that one has most properties just right to work.
deleted 9y ago
[deleted]
filleokus 9y ago
Site down for me: https://webcache.googleusercontent.com/search?q=cache:h4AHuPZqxuoJ:https://jacquesmattheij.com/sorting-lego-many-questions-and-this-is-what-the-result-looks-like+&cd=1&hl=en&ct=clnk&gl=se&client=safari https://webcache.googleusercontent.com/search?q=cache:h4AHuP...
- jacquesm 9y ago
  Thanks for the heads up. Apparently serving up large numbers of images through ssl kills the server :( Working on it. I've disabled some of the images further down in the article to make it load again, so at least it should work. Ok, should be much better now. Ironically, your helpful cache link made things worse because it kept people clicking on a version of the page with huge images, I totally forgot to resize them. Now I've copied the small images over the larger ones so that this link works too and that appears to have solved the issue.
BillFranklin 9y ago
Jacques, John Tooker might be able to help you with the sorting issue: http://lego.jtooker.com/sorting/ http://lego.jtooker.com/sorting/.
- jacquesm 9y ago
  Thank you! That's a great hint.
zitterbewegung 9y ago
I wonder if Jaques tried to do dropout to increase the efficiency? Also if he made the trained neural net available you could made a lego identifier app.
- jacquesm 9y ago
  I've been thinking about making such an app. There are some problems in using the net as it is to power an app: the machine has a uniform background as well as multiple images of the same piece. I don't think the net as it is right now would easily work to recognize images taken with a smartphone. But if I had access to a large number of such smartphone taken images I do think that I could use the method I have right now to help label an initial training set from which a much better network could be derived.
  - zitterbewegung 9y ago
    Maybe search Flickr/google images / imgur with the lego tag ? Or ask the lego subreddit?
    - jacquesm 9y ago
      Good ideas, maybe just make the app, explain that it will work lousy in the beginning but will get better over time?
  - Jtsummers 9y ago
    Could you artificially vary the background of your current training set to create a larger and more varied set of backgrounds to train against initially. That is, if it's a uniform color or pattern just treat it like a green screen. Then use real photos with different background and lighting conditions to refine it later?
    - jacquesm 9y ago
      Yes, this is possible, already I know with fairly large accuracy where the part is in the image and changing or texturing the background is possible. But the optical setup is quite complex and hard to replicate with a smartphone, that would need some work.
mabbo 9y ago
With the machine working so well, you could run each sorted batch through it again, and sort into very specific pieces/colours, if you wanted. You just need to re-train on those classes instead- but I bet the front-end of the tensorflow network would already be trained well enough that re-training for new classes would be very fast.
- roel_v 9y ago
  IIRC the machine sorts like that already, but you can't have a bin for every separate type - you'd need a warehouse of boxes, and the boxes would have to be of several sizes. So I think the pictures just illustrate how he's chosen to cluster for now, as an example to people who try to decide how they'd actually want their purchase to be grouped.
  - jacquesm 9y ago
    Correct. A typical bin contains many different part ids but it would be trivial to make other divisions.
    - mabbo 9y ago
      Neat! So each item is actually identified right down to the size, colour, type, etc, but then literally bucketed into a group of similar items?
      - jacquesm 9y ago
        Yes, because otherwise I'd have to make very many passes through the machine to sort a particular batch.
        mabbo 9y ago
        I was thinking that you had trained the identifier just on the bucketized classes instead- fewer classes comprising all the similar parts. Maybe higher identification rate that way, or maybe not. Identifying right down to the specific colour and part means a heck of a lot of classes to train though. This kind of stuff fascinates me. I've worked on software for package sorting machines in Amazon warehouses. Very similar idea to this: identify, remove from conveyor at right place/time. Only the machines are millions of dollars, run at very high speeds, and use barcode scanning for identification.
andresgottlieb 9y ago
You should use https://www.cloudflare.com/ https://www.cloudflare.com/. It's free and very easy to implement.
- sillysaurus3 9y ago
  CloudFlare is getting rather large. I'd rather use a competitor. Does one exist? EDIT: Some discussion on alternatives: https://news.ycombinator.com/item?id=11913330 https://news.ycombinator.com/item?id=11913330
  - sargun 9y ago
    Fastly. Google sites.
    - tyingq 9y ago
      Fastly does have a good reputation, but they are currently experiencing a global outage. https://news.ycombinator.com/item?id=14654231 https://news.ycombinator.com/item?id=14654231
  - opendomain 9y ago
    Why exactly do you want to go to a competitor? CloudFlare is fantastic and they have a lot of Free services
    - amiga-workbench 9y ago
      No matter how fantastic and benevolent a company is at the moment, they always need competition to keep them in check and on their toes.
      - sillysaurus3 9y ago
        Pretty much. I also dislike how they handled the CloudBleed fiasco and how the CEO tried to offload some of the blame onto Google (incorrectly). https://news.ycombinator.com/item?id=13721644 https://news.ycombinator.com/item?id=13721644
throwanem 9y ago
I actually just gave away all my Lego to a colleague whose three sons are much better able to appreciate it than I am, and (because!) it's been quite a few years since I actually built anything. But, that said, were I looking to buy parts, I'd hesitate to do so in bulk by these categories, unless the price were very good indeed - between the inability to match a parts list to a purchase order, and the extensive manual labor required to sort parts sold by mass at a useful degree of granularity, they'd need to be very cheap in order to make the purchase worth my while. (On the other hand, most of my builds back in the day were spacecraft of various fictional types, and I heavily favored complexity, which means a large number of small parts, designing in wiring channels for LED lighting, and the like. I'm not sure how representative my comments here are of any other style.)
jkingsbery 9y ago
My five year daughter old mostly just wants to build castles, so if there was a less-expensive way to get large number of 2-peg width bricks (especially in pink and purple, which she reminds me daily are her favorite colors), I could see that being useful.
- mtu9001 9y ago
  You can order them. https://shop.lego.com/en-US/Pick-a-Brick https://shop.lego.com/en-US/Pick-a-Brick Order all of them.
  - throwanem 9y ago
    Bricklink has a much wider selection and better prices. https://www.bricklink.com/catalogTree.asp?itemType=P https://www.bricklink.com/catalogTree.asp?itemType=P
    - pavel_lishin 9y ago
      It seems wild to me that there's someone with a single black 2x4 brick in their inventory: https://store.bricklink.com/Teo.92&itemID=119350903#/shop?o={"invID":"119350903"} https://store.bricklink.com/Teo.92&itemID=119350903#/shop?o=...
      - throwanem 9y ago
        Testing, maybe? I might do likewise to get a sense of the UI, markup, sales flow, et cetera, before sinking a bunch of time listing everything.
        pavel_lishin 9y ago
        Maybe, but they've got 213 other items for sale.
        throwanem 9y ago
tinco 9y ago
Not a fanatic builder at all, been almost two decades since I built something with Lego, but really surprised to see that the categories are not color separated. If you're building any esthetic Lego project would have color restrictions. Also I imagine some colors are worth more than others, just because they're used in more projects. Also, color separated Lego bins just look pretty.
- PeterisP 9y ago
  If you have a bin of different Lego pieces of the same color, you have to carefully look for the piece you want. If you have a bin of the same pieces in various color, you just pick the color you want.
  - jacquesm 9y ago
    I suspect this is our evolutionary trained neural net looking for fruits in vegetation that helps us here.
    - CydeWeys 9y ago
      It makes me wonder if there are species (that have full color vision) that are better at the opposite. Is shape discrimination simply harder than color discrimination, or is it related to our evolutionary history as you posit?
      - jannes 9y ago
        I would say color discrimination is easier because it's basically 1-dimensional. In the best case you could do it with a resolution of a single photoreceptor. If you account for ambient lighting, shadows, etc you still just have a 2-dimensional task. Shape discrimination is a 3-dimensional task. In the worst case you need to change your point-of-view or the orientation of the object.
- unityByFreedom 9y ago
  I think this is more proof of concept. You could probably easily add that sort criteria yourself.
pveierland 9y ago
One idea for selling the parts would be to keep a precise inventory of everything entered into the system. A precise inventory + a database of the parts needed for any LEGO set would allow customers to order an arbitrary set for a nice price. A total system which would take random parts as input, and output packaged complete LEGO sets would be neat.
- jacquesm 9y ago
  That would require a robotic warehouse as well to pick the orders in a profitable way.
  - unityByFreedom 9y ago
    How many distinct Lego parts are there? Assuming they're all sorted into separate bins, picking them doesn't seem too hard. I guess sorting + putting them in the right bins would require something more complex. Surely there's a way.
    - jacquesm 9y ago
      About 40,000 shapes, about 100 different colors. And picking them is stupendously hard due to the large variety in shapes and sizes.
      - unityByFreedom 9y ago
        Ah I see. Looks like that ideal setup is a ways off. It would be really cool to see it happen some day!
  - mcorrand 9y ago
    Possibly not, right? If you don't sort but instead use the machine to do the inventory of your bins, then you could have your software tell you which bins you are going to need to fulfill your orders for the day. The big "if" in that scenario is that your machine is not categorizing anymore but identifying. I don't have the experience to assess how much harder it is. Then you'd go and pick the bins, run them through, and the machine assembles the sets from those bins. That's similar to the way Amazon does it. Now they have the shelves on robotic trolleys that bring them directly to the packers, but that's just a required efficiency at their scale. I guess the problem with this scheme is that you move the problem from classifying to identifying... twice. So the precision requirement goes up. I don't know how big your dataset would need to be to require minimum human intervention.
Marazan 9y ago
I suspect Mobile Frame Zero ( http://mobileframezero.com/mfz/ http://mobileframezero.com/mfz/ ) players would be delighted to get a source of the more esoteric bricks that go into their builds
- throwanem 9y ago
  Oh, for - of all the things to discover the month after I give away my entire collection...
cfeduke 9y ago
A few years ago a friend and I started work on a similar machine for sorting Magic: the Gathering cards. We weren't using any machine learning (this was beyond our capabilities) but our design was similar, except the bins were on a moving platform which slowed down as the weight of the cards in bins increased - due to the cheap parts/weakness of the motors we were using. Unfortunately it occupied an entire room in my house and was never completed and ultimately dismantled.
- setq 9y ago
  That's really interesting actually. In the distant past I spent more time sorting the cards than playing the game so there is probably a good trade off on automating it. I went and bought a duel deck the other day to play with the wife so this is going to become another issue in the future again.
- jacquesm 9y ago
  Do you have any pictures?
  - cfeduke 9y ago
    No; I had the bins and the conveyer/webcam/structure in my garage for years until I finally tossed non-Lego Mindstorms parts.
- masklinn 9y ago
  Fun, what were your sorting criterion? Set alone or more?
  - cfeduke 9y ago
    We were trying to it to sort by set at first - when ingesting a large collection of cards (my wife owned a brick and mortar card shop) they were often unsorted. Then with multiple passes we wanted to pull rarity, color and even alphabetize uncommons and rares [way, way too many passes]. The real goal was to be able to process a booster box quickly, so when a new set was released 40 cases in singles could be available very quickly - never even came close to that goal. It turns out that on release night customers are always willing to sort a booster box for you for a pack of cards...
unityByFreedom 9y ago
Pretty cool to keep seeing follow-ups on this. Jacques, any plans to publish your models or image dataset somewhere?
- jacquesm 9y ago
  Eventually yes, but there is a long way to go until it is complete. The software will be published as well. A university has asked for copies of the images that I have right now to make a project out of it.
  - unityByFreedom 9y ago
    That's cool. Good luck, hope to see part 4!
wand3r 9y ago
I have a suggestion but am not a Lego guy. Maybe you can do a few things now that the sorter works, potentially using it to automate or minimize order fulfillment. The idea is that if you can get a dataset of Lego packs / models you can use your inventory to make them. You can offer sales of the kits based on dynamic info. You can also provide inventory dynamically to users to purchase parts or upload a build manifest which will be filled. I think the 2 huge assets you have are sorting for processing AND fulfillment, and real-time inventory levels of verified authentic parts. Also, I would make some sort of contest, auction fun game where a person can win the whole pile of discarded stuff. Idk what's in there but I bet someone would want it.
mherrmann 9y ago
You should extend it so it can assemble the pieces ;)
- jacquesm 9y ago
  Hehe, ok, how about you do that bit and I'll stick to the sorting. The complexities of assembly are orders of magnitude larger for the general case than sorting. One guy made a little Lego car factory: https://www.youtube.com/watch?v=4Bp0GJSqfso https://www.youtube.com/watch?v=4Bp0GJSqfso
tzs 9y ago
> The next morning I woke up to a rather large number of emails congratulating me on having won almost every bid (lesson 1: if you win almost all bids you are bidding too high) That reminds me of the economist George Stigler's observation: "If you never miss a plane, you're spending too much time at the airport".
- Analemma_ 9y ago
  That's a strange observation. I would think an economist of all people would know that the negative utility of missing a plane is huge compared to that of spending a little unnecessary time at the airport. I'm rarely doing anything productive with the time prior to the airport arrival on "travel days" anyway.
  - Frenchgeek 9y ago
    Depend how often you travel by plane: all that little lost time do add up.
    - pavel_lishin 9y ago
      Do they add up to more than the time you spend waiting for the next flight, which may take a full 24 hours? I guess that's a rhetorical question, because the answer is the same to what you wrote; nvm.
    - beeforpork 9y ago
      The concept of 'lost time' should be abandoned, so our lives become less hectic again.
  - fegu 9y ago
    If you have lounge access, then the extra time at the airport is just as productive as at the office. No lost time, and plenty of peace of mind.
  - d--b 9y ago
    It's an analogy for understanding the idea of fair value. If you want to spend a "fair" amount of time at the airport, it means that you need to balance the utility of missing one plane against the utility of always spend a lot of time at the airport in order to never miss a plane. If you take 1000 planes in your life, and spend an extra 1 hour per plane in order to be sure that you will be on time, that's 1000 hours in your life. If, instead, you cut down that hour to 30 minutes, and that makes you miss 3 planes in your life, you'd still have "earned" 500 hours, while losing 3 planes. It's a better trade off overall. But yeah, just an analogy, I bet the guy never missed a plane...
jhassell 9y ago
On commercial pecan farms, the nuts need graded based on color, size, and variety. Currently, pecans are tumbled in rotating chutes composed of grates of increasing spacing, so that pecans are sorted based on size. Color and variety are not filtered, ultimately resulting in a lower sales price for the farmer. Your device could take this type of sorting to the next level.
- djb_hackernews 9y ago
  Optical sorting is pretty common in agriculture. Sounds like these guys do what you are proposing for pecans among their competitors: http://www.buhlergroup.com/global/en/process-technologies/optical-sorting/nut-sorting/pecan-sorting.htm#.WVPSgn7ytMM http://www.buhlergroup.com/global/en/process-technologies/op...
  - jhassell 9y ago
    Optical sorting in Ag with devices less than $1,000 ... not common.
    - jacquesm 9y ago
      That's a good point, industrial machinery is very expensive, smart industrial machinery is extremely expensive.
- lostlogin 9y ago
  That's a little grim. It's the method used for sorting rocks too, so that your gravel doesn't have bits that are too big.
ChuckMcM 9y ago
At some point you can completely reverse entropy and re-create the original sets :-) But in terms of just selling off excess Lego looking at the mix of the builder sets would let you figure out a sort of SKU you could put together and sell on Ebay or else where as a 'builder pack 1', 'builder pack 2' etc.
- jacquesm 9y ago
  > At some point you can completely reverse entropy and re-create the original sets :-) Already did that for a batch, about 150 partial sets as a result. Those would then need to be completed and checked by hand, I don't think that can be done profitably, but maybe there is some way. > But in terms of just selling off excess Lego looking at the mix of the builder sets would let you figure out a sort of SKU you could put together and sell on Ebay or else where as a 'builder pack 1', 'builder pack 2' etc. Yes, and right now I'm looking for what the contents of those packs should be.
  - pavel_lishin 9y ago
    > Already did that for a batch, about 150 partial sets as a result. Those would then need to be completed and checked by hand, I don't think that can be done profitably, but maybe there is some way. There's likely someone who's willing to pay X% of the set's price for a bucket that has a 95% chance of having all of the pieces, and a 100% chance of having too many pieces, including some that don't belong in the set, since you'd want to err on the side of caution. But even that might not be profitable.
    - vidarh 9y ago
      I think the problem with this would be to find sufficient sources of random bricks where the good stuff has not already been removed. E.g. minifigs can get high enough prices in some instances that the total value of a set can be higher if you sell the minifigs and higher value bricks separately, because some people want to buy just the minifig, or need some piece that only appear in a few sets. This would be my main issue with buying Lego to resell - I'd want to be very sure I'm not buying from other resellers where there'd be a risk it'd been stripped of the more valuable pieces.
JackC 9y ago
This series is the best thing ever. An idea: I have a three-month-old daughter who I'll definitely be buying Lego for when she gets old enough. What I'll be looking for is whatever's the least valuable in your collection -- big bins of cheap mixed stuff to see what she invents. Maybe that would be a cool side output for your project? Cheap chaff packs for kids of hackers ...
- jacquesm 9y ago
  Oh that's a great idea. Thank you! I have garbage bags full of 2x4, 2x3 and 2x2 bricks in common colors that are worth relatively little. I'll ask my panel of experts (5 and 7 years old) to see what they would do with such a treasure trove.
  - utexaspunk 9y ago
    I second that- when I was a kid I had tons of common bricks (1- and 2-by-N's) and regularly built decent-sized houses with full-height walls and multiple rooms and such. My 5 year-old daughter recently graduated from Duplo to Lego and I don't see a good way of building up a decent foundation of just "normal" bricks from which she can build lots of things. I definitely feel like there's a lack of sets with good quantities of them. Even the large "Classic" sets are pretty light on them compared to other pieces, and what they do have is spread out over like 15 colors, so you can forget about trying to make anything match. I've looked on eBay, but most of what I see in bulk there are lots of just one size or one color bricks, or just pounds of completely random pieces. I'd love it if I could find something like an assortment of 1,000 mixed 1-by-1/2/3/4/6/8 bricks in 4-5 colors, or the same with 2-by-N bricks.
tuna-piano 9y ago
I'm begging you, please post a full-speed video that shows a seemingly random assortment of pieces sorted into boxes with identical color/size/shape pieces. I think it would be insanely satisfying for many people to watch. I would literally grab popcorn and watch it for hours (and I think many others would too). Given your current setup, it might be easiest to just take one bin of identical pieces and then sort it by color into various bins. I'm salivating over what this would look like. If you posted that video, I bet it would rack up millions of YouTube views.
- jacquesm 9y ago
  Ok, will do but first I have to speed up that first feeder stage because I really don't want to stand there for an hour to drop pieces on the recognition belt. Any good ideas for that first stage are very welcome. I've looked at all kinds of mechanism (belts, pickers, vibrating drums) and none so far have the required speed, regularity and are quiet enough to be used in a home.
  - eigenvalue 9y ago
    What about a big subwoofer for a car, and you can try driving it with random sound waves until you find one that works. Or go through a set of tones that can hit the resonant frequencies of various pieces?
    - jacquesm 9y ago
      That's a very nice idea, I will try that. Thank you!
    - lostlogin 9y ago
      Noise is mentioned as a problem. That said, hearing it sing to its lego would be interesting.
  - trose 9y ago
    what about a screw feed?
munificent 9y ago
If, like me, you find photography of piles of sorted LEGO pieces hypnotically soothing, you will probably like this album I put together: http://imgur.com/a/OJaq6 http://imgur.com/a/OJaq6 A while back, I sorted my childhood LEGO pile before sending it to my brother. 73 pounds of LEGO sorted by hand. I'm sure it says things about how strangely wired my brain is, but this was an eminently enjoyable endeavor.
- jacquesm 9y ago
  That's very impressive to do by hand, 10 Kg or so back into sets takes a week by hand, how much time did this take you?
  - thefalcon 9y ago
    The album says it was about a week of work. It really is a very impressive album, too!
    - jacquesm 9y ago
      A week is insane speed for that quantity. That's day-and-night work.
      - munificent 9y ago
        Not that bad. Usually a couple of hours in the evening after the kids went to bed for a couple of weeks.
        jacquesm 9y ago
        Did you use any sieves or other tricks to speed it up?
        munificent 9y ago
        Nope. Spread it all out on a bedsheet and worked my way through the pile. I did (roughly) one category of piece at a time, starting with the bigger pieces.
sanj 9y ago
I would pay good money to send you a big box of Lego from mixed sets and have it come back sorted into one bag per set. You'd probably need the set numbers so that it'd be easier to choose which set to associate a part with. I'm more than happy to be a beta customer!
- jacquesm 9y ago
  With international shipping there and back I don't know if you'd still like that. But I'm game :) SaaS: Sorting as a Service.
  - NicoJuicy 9y ago
    I can generate a cheaper shipping ticket for PostNL for you. Mail in profile
    - jacquesm 9y ago
      Mail sent.
  - sanj 9y ago
    I make enough trips across the Atlantic to make it work, at least for me. To scale... I've always wanted to rent a cargo ship!
    - jacquesm 9y ago
      Well, feel free to drop by with a suitcase of Lego and I'll be happy to run the machine for you.
tuna-piano 9y ago
Has Lego approached you at all? If they don't have similar in-house technology, I could see this adding value to them.
- jacquesm 9y ago
  No, but I don't think I'm qualified to polish the shoes of the maintenance technicians that designed Lego's internal logistics, let alone measure myself with their engineers. There are some videos floating about of what Lego's manufacturing lines look like, very impressive stuff.
marze 9y ago
Why not go big? Write up a business plan that involves buying 100 tons of lego, making a bigger machine with 200 bins off the belt, or whatever, and making a new brickowl / bricklink site of your own. I'm sure you could find investors, you've got great publicity already. And you could propose it as a general sorting AI (or web sale of small item) firm in the long term, with the lego being your initial application. You'd need to solve the inverse problem of how to efficiently go from bin to packing container, which doesn't seem too hard. Rotating carousels of bins with little robots or something. Set the item on a tray, and if isn't the right item, puff it back into the bin.
- jacquesm 9y ago
  If it won't work small it won't work big. First learn to walk. If I can overcome all the challenges and there is a sufficiently large market for the end product and when all the costs (energy, for instance) have been accounted for there is still a business case left then this might be worth it. But there is a ton of work to be done before you could begin to answer those questions. For now it is a fun project and I'm trying hard to keep it that way. I've been approached by some of the larger bulk resellers that would love to get rid of the humans they employ, none of them made a good enough case why I should partner with them or even seemed to be nice people. The general nature of neural nets for sorting and classification I think is well recognized in industry and there are many companies that are capitalizing on this (and have been for some time). I highly doubt I'm doing anything original in that sense.
  - marze 9y ago
    If someone wants to give you 100k euros and develop some interesting technology, it seems like it would make the project more fun. What does anyone else think? Should Jacques open it up for investment?
    - jacquesm 9y ago
      I doubt anybody would give 100k euros. What is more likely to happen is that they would want equity in some company, a seat at the non existent board table, some wild eyed exit plan 3 years down the line and if they don't get their 20% ROI within the first year they'll be staging a minority shareholder revolt and possibly a lawsuit due to mis-management. For 100K I will gladly sidestep all those problems and self-finance.
pulse7 9y ago
You can really earn money by modifying this machine to sort various nuts and fruits by quality in the food industry...
arnsholt 9y ago
> Judging by the increase in accuracy from 16K images to 60K images there is something of a diminishing rate of return here. This is a well-known thing in machine learning. As far as I know all ML algorithms have this property, where the learning curve is basically logarithmic and each additive increase in performance requires multiplicatively more training material.
codewithcheese 9y ago
I think the sorted photos are great art. Maybe more valuable than the contents ;)
- jacquesm 9y ago
  Consider them public domain, if you want high res versions to print then just drop the .small. from the image filename and you should be good to go.
tzs 9y ago
Are there CAD models available for the various Lego components? I'm wondering if, given such models, you could use rendering software to generate labeled training images. That could allow you to get a large training set with many images for each kind of component in many different orientations, even for components you do not yet have. If the CAD models are not available, how much work is it to make a CAD model given an instance of a particular component? I'd expect that for some components, such as those NxM plates, you'd only need to build one model by hand in CAD software, and then could algorithmically generate the models for other sizes.
- ant6n 9y ago
  I don't know CAD models, but it would probably make sense to scrape the lego website where you can select all their bricks. Each named brick has 8 images (various orientations): https://shop.lego.com/Pick-a-Brick https://shop.lego.com/Pick-a-Brick I find it hard to believe that there are only 1421 bricks, including color combinations. Example: "Apple with leaf", Color Family: Dark Green, Exact Color: Bright green, Category: Foodstuff Element ID: 4107050, Design ID: 33051, orientation 8: https://sh-s7-live-s.legocdn.com/is/image/LEGOPCS/4107050_s8 https://sh-s7-live-s.legocdn.com/is/image/LEGOPCS/4107050_s8 ...This also makes me wonder whether it would make sense to set up multiple cameras, to get multiple orientations for every block. Use the image that has the lowest classification error, but use all orientations to train the network.
  - jacquesm 9y ago
    The Lego pick-a-brick store is very very limited. The images are not very useful to me because the images the machine takes are very different from those. But maybe it would somehow contribute. I'd be very ware of using other people's copyrighted content though. I do use multiple orientations.
    - ant6n 9y ago
      Point taken re copyright. Re similarity, it may be possible to write some sort of imagemagic filter that makes the bricks appear more realistic somehow. (But yeah, this would probably only be useful as a starting point for training the very rare bricks)
justinpombrio 9y ago
> Overall the machine works well but I’m still not happy with the hopper and the first stage belt. It is running much slower than the speed at which the parts recognizer works (30 parts / second) and I really would like to come up with something better. It sounds like your existing hopper is reliable but slow? If that's the case, could you just make two (perhaps somewhat smaller) hoppers and merge their outputs?
- jacquesm 9y ago
  It's not even all that reliable, plenty of times it will vomit 5 to 10 parts in one go because they got clumped together somehow and when one falls the rest will too. I've tried to improve on this by using a slide towards the second belt and this works sometimes but not always. Still plenty of 'multipart' images.
marze 9y ago
On speeding up your feed, how about adding another belt prior to the main belt, with a variable speed? You drop parts onto the variable speed belt from enough sources to get a good feed rate, and constantly vary the speed of that belt with video feedback to drop them onto the main belt with reasonably even spacing.
jokoon 9y ago
Impressive, but I'm still wondering what 2 tons of lego look like.
- jacquesm 9y ago
  I really should make a picture of the storage, it's essentially a garage full of Lego. The garage is nearby, not at the same location where the machine is (that's at my house), I don't have room enough here to store that much stuff.
stuaxo 9y ago
Now you have a bunch of pieces each with different classifications. Possibly you could feed in parts lists from a huge amount of existing sets - have some sort of (handwavey learning thing or even markov chain), and generate mixes of pieces based on data sets from spaceships / cars / etc.
Poiesis 9y ago
Technics and larger wheels are useful for Lego robotics (FLL) teams everywhere; certainly useful for our team. Edit: this is in response to your asking for feedback about categories/demand.
- jacquesm 9y ago
  Larger wheels is a good category to have. Anything larger than 42 mm diameter gets filtered out very early on so that's a trivial class to add. Thank you!
Gravityloss 9y ago
In part 1 you mention that under vibration the bricks self assemble into bridges that obstruct the conveyor belt, and you want to minimize that. How about maximizing it instead? Try to build bridges as long as possible? Or some other fitness function? In theory you could control the vibration sequence and the part feeding sequence (once you have sorted all the parts :) ) and use an evolutionary algorithm...
- jacquesm 9y ago
  That's a hilarious idea. I'm somewhat tempted to try this. I've managed to bridge 35 cm 'by accident', I'm pretty sure you could go substantially larger than that with relatively little work.
punnerud 9y ago
Why not sort Lego the same way we sort plasic in Norway? Watch this video. Drawing of the technique at 0:35 and shown from 0:52 https://youtu.be/Mi0FHNLkim0 https://youtu.be/Mi0FHNLkim0
phaedrus 9y ago
Something I've wanted to try for years is what about a Lego refraction distillation tower? Similar to how oil refineries use a tower to sort hydrocarbon compounds by length based on their volatility. Except in this case instead of using heat we would use air blowing upward in a long tube to make the pieces "volatile". Picture a 6" diameter clear tube stretching floor to ceiling with a powerful blower at the base. Or a smaller version could be prototyped using an air popcorn popper with the heating element disabled. Essentially it would sort pieces according to their free-fall terminal velocity. You could exploit the Venturi effect to make the bottom of the tube have a higher velocity than the top, or simply add bleed-air holes at progressive levels. There could be take-off doors at different levels of the tube. A nice property is that if the ingress and egress mechanisms are designed elegantly, it could run continuously. (Although, that last point is one reason I've never built it: it seems like there would be a fine line between "air powered sorter" and "air powered rock tumbler" and I worry it could wear the corners of all my Lego bricks if it goes wrong.) Another potential issue is that there might not be much logical relationship between the pieces which share similar terminal velocities, and different pieces may have different or metastable terminal velocities depending on initial orientation. Obviously this wouldn't be a perfect sorter, but it could be a good pre- or post- sorter for your visual system. And for my purposes it might work well since instead of adapting the system to suit my sorting preferences I could simply train my sorting expectations based on what I learn the machine tends to sort together. Hmmm now that I have an industrial dust collector installed in my workshop, I actually already have the bulkiest / most expensive part of this invention. I just need a tall acrylic tube and I could try this!
- schoen 9y ago
  How do asymmetry and angular momentum affect this? Or how do they affect terminal velocity in general? (For example, what can happen to a Lego plate which has a very different terminal velocity in one orientation than in another?)

322 points by darwhy 9y ago

I've thought about this machine a number of times since the previous post. In that post, there was a movie showing the machine's operation, but it was operating at a slow speed, and he's mentioned multiple times that there's been speed improvements. Does anyone know if there's a realtime movie anywhere?

jacquesm 9y ago
The speed of the machine is mostly a function of the mechanical feed speed of the parts. This is the hopper/belt mechanism and it is by far the slowest part of the machine. Once the parts are on the recognition belt things more very fast, a single part can be scanned and recognized in about 30 milliseconds, so roughly 30 parts / second. Making the machine much faster then means making that feed mechanism faster than it is today (roughly 1 part per second tops) and that's the challenge I'm looking at right now. Ideas more than welcome! Even bad ones, you never know which bad idea with a twist can become a good idea or even a very good idea.
- maxerickson 9y ago
  Perhaps the dumbest idea would be to build a second feeder. Would double the speed and sounds like the rest of it is fast enough that you wouldn't even need to worry about parts falling on top of each other.
  - jacquesm 9y ago
    That's not a dumb idea, however: space is an issue, the rest is fast enough but even so parts do tend to fall in clumps and if two such clumps were to fall at the same time it could lead to a jam which requires manual intervention. Ideally parts would land on the belt in a very regular rhythm, without any kind of irregularity whether the parts are 8 mm round 1x1 plates or 16 long 1x1 bricks. You need about 3 cm of space on either side to make sure the right part gets blown off the belt, that's the most important thing that will eventually limit the speed. Oh, another factor limiting the speed is the maximum acceleration over the stretch towards the camera, otherwise the parts are still moving on the belt when they reach the camera and then you get bad images.
- gboudrias 9y ago
  Not to state the obvious, but if the "hopper" is slow, why not just have multiple of them? Are they expensive? Take up too much space? I assume you've thought of this but I'm still curious. Really cool stuff by the way!
  - jacquesm 9y ago
    Space wise this is hard, also parts tend to fall in clusters so the second belt now has to be much faster than the first one to make sure there is enough separation. I do have a second belt that I have not used yet so if there is no better idea then that is what I'll end up doing.
    - gboudrias 9y ago
      So just to make sure I got this right, you need a way to get a single piece of any size out of a container full of random pieces, and onto the first moving belt, at a rate higher than 1 piece per second? I'm not familiar with the project but this seems like an interesting challenge.
      - jacquesm 9y ago
        Yes, that's roughly the gist of it. This is a hard problem because Lego comes in many shapes and sizes, a pre-sieving step limits the pieces that hit the belt at 42 mm largest cross section. So anything below that (but possibly much longer than that, all the way up to 2x1x16) can be found in the hopper. There are some really good suggestions in this thread, the one that stands out for me most is the brush edged screw feeder, I think that one has most properties just right to work.

deleted 9y ago

[deleted]

filleokus 9y ago

Site down for me: https://webcache.googleusercontent.com/search?q=cache:h4AHuPZqxuoJ:https://jacquesmattheij.com/sorting-lego-many-questions-and-this-is-what-the-result-looks-like+&cd=1&hl=en&ct=clnk&gl=se&client=safari https://webcache.googleusercontent.com/search?q=cache:h4AHuP...

jacquesm 9y ago
Thanks for the heads up. Apparently serving up large numbers of images through ssl kills the server :( Working on it. I've disabled some of the images further down in the article to make it load again, so at least it should work. Ok, should be much better now. Ironically, your helpful cache link made things worse because it kept people clicking on a version of the page with huge images, I totally forgot to resize them. Now I've copied the small images over the larger ones so that this link works too and that appears to have solved the issue.

BillFranklin 9y ago

Jacques, John Tooker might be able to help you with the sorting issue: http://lego.jtooker.com/sorting/ http://lego.jtooker.com/sorting/.

jacquesm 9y ago
Thank you! That's a great hint.

zitterbewegung 9y ago

I wonder if Jaques tried to do dropout to increase the efficiency? Also if he made the trained neural net available you could made a lego identifier app.

jacquesm 9y ago
I've been thinking about making such an app. There are some problems in using the net as it is to power an app: the machine has a uniform background as well as multiple images of the same piece. I don't think the net as it is right now would easily work to recognize images taken with a smartphone. But if I had access to a large number of such smartphone taken images I do think that I could use the method I have right now to help label an initial training set from which a much better network could be derived.
- zitterbewegung 9y ago
  Maybe search Flickr/google images / imgur with the lego tag ? Or ask the lego subreddit?
  - jacquesm 9y ago
    Good ideas, maybe just make the app, explain that it will work lousy in the beginning but will get better over time?
- Jtsummers 9y ago
  Could you artificially vary the background of your current training set to create a larger and more varied set of backgrounds to train against initially. That is, if it's a uniform color or pattern just treat it like a green screen. Then use real photos with different background and lighting conditions to refine it later?
  - jacquesm 9y ago
    Yes, this is possible, already I know with fairly large accuracy where the part is in the image and changing or texturing the background is possible. But the optical setup is quite complex and hard to replicate with a smartphone, that would need some work.

mabbo 9y ago

With the machine working so well, you could run each sorted batch through it again, and sort into very specific pieces/colours, if you wanted. You just need to re-train on those classes instead- but I bet the front-end of the tensorflow network would already be trained well enough that re-training for new classes would be very fast.

roel_v 9y ago
IIRC the machine sorts like that already, but you can't have a bin for every separate type - you'd need a warehouse of boxes, and the boxes would have to be of several sizes. So I think the pictures just illustrate how he's chosen to cluster for now, as an example to people who try to decide how they'd actually want their purchase to be grouped.
- jacquesm 9y ago
  Correct. A typical bin contains many different part ids but it would be trivial to make other divisions.
  - mabbo 9y ago
    Neat! So each item is actually identified right down to the size, colour, type, etc, but then literally bucketed into a group of similar items?
    - jacquesm 9y ago
      Yes, because otherwise I'd have to make very many passes through the machine to sort a particular batch.
      - mabbo 9y ago
        I was thinking that you had trained the identifier just on the bucketized classes instead- fewer classes comprising all the similar parts. Maybe higher identification rate that way, or maybe not. Identifying right down to the specific colour and part means a heck of a lot of classes to train though. This kind of stuff fascinates me. I've worked on software for package sorting machines in Amazon warehouses. Very similar idea to this: identify, remove from conveyor at right place/time. Only the machines are millions of dollars, run at very high speeds, and use barcode scanning for identification.

andresgottlieb 9y ago

You should use https://www.cloudflare.com/ https://www.cloudflare.com/. It's free and very easy to implement.

sillysaurus3 9y ago
CloudFlare is getting rather large. I'd rather use a competitor. Does one exist? EDIT: Some discussion on alternatives: https://news.ycombinator.com/item?id=11913330 https://news.ycombinator.com/item?id=11913330
- sargun 9y ago
  Fastly. Google sites.
  - tyingq 9y ago
    Fastly does have a good reputation, but they are currently experiencing a global outage. https://news.ycombinator.com/item?id=14654231 https://news.ycombinator.com/item?id=14654231
- opendomain 9y ago
  Why exactly do you want to go to a competitor? CloudFlare is fantastic and they have a lot of Free services
  - amiga-workbench 9y ago
    No matter how fantastic and benevolent a company is at the moment, they always need competition to keep them in check and on their toes.
    - sillysaurus3 9y ago
      Pretty much. I also dislike how they handled the CloudBleed fiasco and how the CEO tried to offload some of the blame onto Google (incorrectly). https://news.ycombinator.com/item?id=13721644 https://news.ycombinator.com/item?id=13721644

throwanem 9y ago

I actually just gave away all my Lego to a colleague whose three sons are much better able to appreciate it than I am, and (because!) it's been quite a few years since I actually built anything. But, that said, were I looking to buy parts, I'd hesitate to do so in bulk by these categories, unless the price were very good indeed - between the inability to match a parts list to a purchase order, and the extensive manual labor required to sort parts sold by mass at a useful degree of granularity, they'd need to be very cheap in order to make the purchase worth my while. (On the other hand, most of my builds back in the day were spacecraft of various fictional types, and I heavily favored complexity, which means a large number of small parts, designing in wiring channels for LED lighting, and the like. I'm not sure how representative my comments here are of any other style.)

jkingsbery 9y ago

My five year daughter old mostly just wants to build castles, so if there was a less-expensive way to get large number of 2-peg width bricks (especially in pink and purple, which she reminds me daily are her favorite colors), I could see that being useful.

mtu9001 9y ago
You can order them. https://shop.lego.com/en-US/Pick-a-Brick https://shop.lego.com/en-US/Pick-a-Brick Order all of them.
- throwanem 9y ago
  Bricklink has a much wider selection and better prices. https://www.bricklink.com/catalogTree.asp?itemType=P https://www.bricklink.com/catalogTree.asp?itemType=P
  - pavel_lishin 9y ago
    It seems wild to me that there's someone with a single black 2x4 brick in their inventory: https://store.bricklink.com/Teo.92&itemID=119350903#/shop?o={"invID":"119350903"} https://store.bricklink.com/Teo.92&itemID=119350903#/shop?o=...
    - throwanem 9y ago
      Testing, maybe? I might do likewise to get a sense of the UI, markup, sales flow, et cetera, before sinking a bunch of time listing everything.
      - pavel_lishin 9y ago
        Maybe, but they've got 213 other items for sale.
        throwanem 9y ago

tinco 9y ago

Not a fanatic builder at all, been almost two decades since I built something with Lego, but really surprised to see that the categories are not color separated. If you're building any esthetic Lego project would have color restrictions. Also I imagine some colors are worth more than others, just because they're used in more projects. Also, color separated Lego bins just look pretty.

PeterisP 9y ago
If you have a bin of different Lego pieces of the same color, you have to carefully look for the piece you want. If you have a bin of the same pieces in various color, you just pick the color you want.
- jacquesm 9y ago
  I suspect this is our evolutionary trained neural net looking for fruits in vegetation that helps us here.
  - CydeWeys 9y ago
    It makes me wonder if there are species (that have full color vision) that are better at the opposite. Is shape discrimination simply harder than color discrimination, or is it related to our evolutionary history as you posit?
    - jannes 9y ago
      I would say color discrimination is easier because it's basically 1-dimensional. In the best case you could do it with a resolution of a single photoreceptor. If you account for ambient lighting, shadows, etc you still just have a 2-dimensional task. Shape discrimination is a 3-dimensional task. In the worst case you need to change your point-of-view or the orientation of the object.
unityByFreedom 9y ago
I think this is more proof of concept. You could probably easily add that sort criteria yourself.

pveierland 9y ago

One idea for selling the parts would be to keep a precise inventory of everything entered into the system. A precise inventory + a database of the parts needed for any LEGO set would allow customers to order an arbitrary set for a nice price. A total system which would take random parts as input, and output packaged complete LEGO sets would be neat.

jacquesm 9y ago
That would require a robotic warehouse as well to pick the orders in a profitable way.
- unityByFreedom 9y ago
  How many distinct Lego parts are there? Assuming they're all sorted into separate bins, picking them doesn't seem too hard. I guess sorting + putting them in the right bins would require something more complex. Surely there's a way.
  - jacquesm 9y ago
    About 40,000 shapes, about 100 different colors. And picking them is stupendously hard due to the large variety in shapes and sizes.
    - unityByFreedom 9y ago
      Ah I see. Looks like that ideal setup is a ways off. It would be really cool to see it happen some day!
- mcorrand 9y ago
  Possibly not, right? If you don't sort but instead use the machine to do the inventory of your bins, then you could have your software tell you which bins you are going to need to fulfill your orders for the day. The big "if" in that scenario is that your machine is not categorizing anymore but identifying. I don't have the experience to assess how much harder it is. Then you'd go and pick the bins, run them through, and the machine assembles the sets from those bins. That's similar to the way Amazon does it. Now they have the shelves on robotic trolleys that bring them directly to the packers, but that's just a required efficiency at their scale. I guess the problem with this scheme is that you move the problem from classifying to identifying... twice. So the precision requirement goes up. I don't know how big your dataset would need to be to require minimum human intervention.

Marazan 9y ago

I suspect Mobile Frame Zero ( http://mobileframezero.com/mfz/ http://mobileframezero.com/mfz/ ) players would be delighted to get a source of the more esoteric bricks that go into their builds

throwanem 9y ago
Oh, for - of all the things to discover the month after I give away my entire collection...

cfeduke 9y ago

A few years ago a friend and I started work on a similar machine for sorting Magic: the Gathering cards. We weren't using any machine learning (this was beyond our capabilities) but our design was similar, except the bins were on a moving platform which slowed down as the weight of the cards in bins increased - due to the cheap parts/weakness of the motors we were using. Unfortunately it occupied an entire room in my house and was never completed and ultimately dismantled.

setq 9y ago
That's really interesting actually. In the distant past I spent more time sorting the cards than playing the game so there is probably a good trade off on automating it. I went and bought a duel deck the other day to play with the wife so this is going to become another issue in the future again.
jacquesm 9y ago
Do you have any pictures?
- cfeduke 9y ago
  No; I had the bins and the conveyer/webcam/structure in my garage for years until I finally tossed non-Lego Mindstorms parts.
masklinn 9y ago
Fun, what were your sorting criterion? Set alone or more?
- cfeduke 9y ago
  We were trying to it to sort by set at first - when ingesting a large collection of cards (my wife owned a brick and mortar card shop) they were often unsorted. Then with multiple passes we wanted to pull rarity, color and even alphabetize uncommons and rares [way, way too many passes]. The real goal was to be able to process a booster box quickly, so when a new set was released 40 cases in singles could be available very quickly - never even came close to that goal. It turns out that on release night customers are always willing to sort a booster box for you for a pack of cards...

unityByFreedom 9y ago

Pretty cool to keep seeing follow-ups on this. Jacques, any plans to publish your models or image dataset somewhere?

jacquesm 9y ago
Eventually yes, but there is a long way to go until it is complete. The software will be published as well. A university has asked for copies of the images that I have right now to make a project out of it.
- unityByFreedom 9y ago
  That's cool. Good luck, hope to see part 4!

wand3r 9y ago

I have a suggestion but am not a Lego guy. Maybe you can do a few things now that the sorter works, potentially using it to automate or minimize order fulfillment. The idea is that if you can get a dataset of Lego packs / models you can use your inventory to make them. You can offer sales of the kits based on dynamic info. You can also provide inventory dynamically to users to purchase parts or upload a build manifest which will be filled. I think the 2 huge assets you have are sorting for processing AND fulfillment, and real-time inventory levels of verified authentic parts. Also, I would make some sort of contest, auction fun game where a person can win the whole pile of discarded stuff. Idk what's in there but I bet someone would want it.

mherrmann 9y ago

You should extend it so it can assemble the pieces ;)

jacquesm 9y ago
Hehe, ok, how about you do that bit and I'll stick to the sorting. The complexities of assembly are orders of magnitude larger for the general case than sorting. One guy made a little Lego car factory: https://www.youtube.com/watch?v=4Bp0GJSqfso https://www.youtube.com/watch?v=4Bp0GJSqfso

tzs 9y ago

> The next morning I woke up to a rather large number of emails congratulating me on having won almost every bid (lesson 1: if you win almost all bids you are bidding too high) That reminds me of the economist George Stigler's observation: "If you never miss a plane, you're spending too much time at the airport".

Analemma_ 9y ago
That's a strange observation. I would think an economist of all people would know that the negative utility of missing a plane is huge compared to that of spending a little unnecessary time at the airport. I'm rarely doing anything productive with the time prior to the airport arrival on "travel days" anyway.
- Frenchgeek 9y ago
  Depend how often you travel by plane: all that little lost time do add up.
  - pavel_lishin 9y ago
    Do they add up to more than the time you spend waiting for the next flight, which may take a full 24 hours? I guess that's a rhetorical question, because the answer is the same to what you wrote; nvm.
  - beeforpork 9y ago
    The concept of 'lost time' should be abandoned, so our lives become less hectic again.
- fegu 9y ago
  If you have lounge access, then the extra time at the airport is just as productive as at the office. No lost time, and plenty of peace of mind.
- d--b 9y ago
  It's an analogy for understanding the idea of fair value. If you want to spend a "fair" amount of time at the airport, it means that you need to balance the utility of missing one plane against the utility of always spend a lot of time at the airport in order to never miss a plane. If you take 1000 planes in your life, and spend an extra 1 hour per plane in order to be sure that you will be on time, that's 1000 hours in your life. If, instead, you cut down that hour to 30 minutes, and that makes you miss 3 planes in your life, you'd still have "earned" 500 hours, while losing 3 planes. It's a better trade off overall. But yeah, just an analogy, I bet the guy never missed a plane...

jhassell 9y ago

On commercial pecan farms, the nuts need graded based on color, size, and variety. Currently, pecans are tumbled in rotating chutes composed of grates of increasing spacing, so that pecans are sorted based on size. Color and variety are not filtered, ultimately resulting in a lower sales price for the farmer. Your device could take this type of sorting to the next level.

djb_hackernews 9y ago
Optical sorting is pretty common in agriculture. Sounds like these guys do what you are proposing for pecans among their competitors: http://www.buhlergroup.com/global/en/process-technologies/optical-sorting/nut-sorting/pecan-sorting.htm#.WVPSgn7ytMM http://www.buhlergroup.com/global/en/process-technologies/op...
- jhassell 9y ago
  Optical sorting in Ag with devices less than $1,000 ... not common.
  - jacquesm 9y ago
    That's a good point, industrial machinery is very expensive, smart industrial machinery is extremely expensive.
lostlogin 9y ago
That's a little grim. It's the method used for sorting rocks too, so that your gravel doesn't have bits that are too big.

ChuckMcM 9y ago

At some point you can completely reverse entropy and re-create the original sets :-) But in terms of just selling off excess Lego looking at the mix of the builder sets would let you figure out a sort of SKU you could put together and sell on Ebay or else where as a 'builder pack 1', 'builder pack 2' etc.

jacquesm 9y ago
> At some point you can completely reverse entropy and re-create the original sets :-) Already did that for a batch, about 150 partial sets as a result. Those would then need to be completed and checked by hand, I don't think that can be done profitably, but maybe there is some way. > But in terms of just selling off excess Lego looking at the mix of the builder sets would let you figure out a sort of SKU you could put together and sell on Ebay or else where as a 'builder pack 1', 'builder pack 2' etc. Yes, and right now I'm looking for what the contents of those packs should be.
- pavel_lishin 9y ago
  > Already did that for a batch, about 150 partial sets as a result. Those would then need to be completed and checked by hand, I don't think that can be done profitably, but maybe there is some way. There's likely someone who's willing to pay X% of the set's price for a bucket that has a 95% chance of having all of the pieces, and a 100% chance of having too many pieces, including some that don't belong in the set, since you'd want to err on the side of caution. But even that might not be profitable.
  - vidarh 9y ago
    I think the problem with this would be to find sufficient sources of random bricks where the good stuff has not already been removed. E.g. minifigs can get high enough prices in some instances that the total value of a set can be higher if you sell the minifigs and higher value bricks separately, because some people want to buy just the minifig, or need some piece that only appear in a few sets. This would be my main issue with buying Lego to resell - I'd want to be very sure I'm not buying from other resellers where there'd be a risk it'd been stripped of the more valuable pieces.

JackC 9y ago

This series is the best thing ever. An idea: I have a three-month-old daughter who I'll definitely be buying Lego for when she gets old enough. What I'll be looking for is whatever's the least valuable in your collection -- big bins of cheap mixed stuff to see what she invents. Maybe that would be a cool side output for your project? Cheap chaff packs for kids of hackers ...

jacquesm 9y ago
Oh that's a great idea. Thank you! I have garbage bags full of 2x4, 2x3 and 2x2 bricks in common colors that are worth relatively little. I'll ask my panel of experts (5 and 7 years old) to see what they would do with such a treasure trove.
- utexaspunk 9y ago
  I second that- when I was a kid I had tons of common bricks (1- and 2-by-N's) and regularly built decent-sized houses with full-height walls and multiple rooms and such. My 5 year-old daughter recently graduated from Duplo to Lego and I don't see a good way of building up a decent foundation of just "normal" bricks from which she can build lots of things. I definitely feel like there's a lack of sets with good quantities of them. Even the large "Classic" sets are pretty light on them compared to other pieces, and what they do have is spread out over like 15 colors, so you can forget about trying to make anything match. I've looked on eBay, but most of what I see in bulk there are lots of just one size or one color bricks, or just pounds of completely random pieces. I'd love it if I could find something like an assortment of 1,000 mixed 1-by-1/2/3/4/6/8 bricks in 4-5 colors, or the same with 2-by-N bricks.

tuna-piano 9y ago

I'm begging you, please post a full-speed video that shows a seemingly random assortment of pieces sorted into boxes with identical color/size/shape pieces. I think it would be insanely satisfying for many people to watch. I would literally grab popcorn and watch it for hours (and I think many others would too). Given your current setup, it might be easiest to just take one bin of identical pieces and then sort it by color into various bins. I'm salivating over what this would look like. If you posted that video, I bet it would rack up millions of YouTube views.

jacquesm 9y ago
Ok, will do but first I have to speed up that first feeder stage because I really don't want to stand there for an hour to drop pieces on the recognition belt. Any good ideas for that first stage are very welcome. I've looked at all kinds of mechanism (belts, pickers, vibrating drums) and none so far have the required speed, regularity and are quiet enough to be used in a home.
- eigenvalue 9y ago
  What about a big subwoofer for a car, and you can try driving it with random sound waves until you find one that works. Or go through a set of tones that can hit the resonant frequencies of various pieces?
  - jacquesm 9y ago
    That's a very nice idea, I will try that. Thank you!
  - lostlogin 9y ago
    Noise is mentioned as a problem. That said, hearing it sing to its lego would be interesting.
- trose 9y ago
  what about a screw feed?

munificent 9y ago

If, like me, you find photography of piles of sorted LEGO pieces hypnotically soothing, you will probably like this album I put together: http://imgur.com/a/OJaq6 http://imgur.com/a/OJaq6 A while back, I sorted my childhood LEGO pile before sending it to my brother. 73 pounds of LEGO sorted by hand. I'm sure it says things about how strangely wired my brain is, but this was an eminently enjoyable endeavor.

jacquesm 9y ago
That's very impressive to do by hand, 10 Kg or so back into sets takes a week by hand, how much time did this take you?
- thefalcon 9y ago
  The album says it was about a week of work. It really is a very impressive album, too!
  - jacquesm 9y ago
    A week is insane speed for that quantity. That's day-and-night work.
    - munificent 9y ago
      Not that bad. Usually a couple of hours in the evening after the kids went to bed for a couple of weeks.
      - jacquesm 9y ago
        Did you use any sieves or other tricks to speed it up?
        munificent 9y ago
        Nope. Spread it all out on a bedsheet and worked my way through the pile. I did (roughly) one category of piece at a time, starting with the bigger pieces.

sanj 9y ago

I would pay good money to send you a big box of Lego from mixed sets and have it come back sorted into one bag per set. You'd probably need the set numbers so that it'd be easier to choose which set to associate a part with. I'm more than happy to be a beta customer!

jacquesm 9y ago
With international shipping there and back I don't know if you'd still like that. But I'm game :) SaaS: Sorting as a Service.
- NicoJuicy 9y ago
  I can generate a cheaper shipping ticket for PostNL for you. Mail in profile
  - jacquesm 9y ago
    Mail sent.
- sanj 9y ago
  I make enough trips across the Atlantic to make it work, at least for me. To scale... I've always wanted to rent a cargo ship!
  - jacquesm 9y ago
    Well, feel free to drop by with a suitcase of Lego and I'll be happy to run the machine for you.

tuna-piano 9y ago

Has Lego approached you at all? If they don't have similar in-house technology, I could see this adding value to them.

jacquesm 9y ago
No, but I don't think I'm qualified to polish the shoes of the maintenance technicians that designed Lego's internal logistics, let alone measure myself with their engineers. There are some videos floating about of what Lego's manufacturing lines look like, very impressive stuff.

marze 9y ago

Why not go big? Write up a business plan that involves buying 100 tons of lego, making a bigger machine with 200 bins off the belt, or whatever, and making a new brickowl / bricklink site of your own. I'm sure you could find investors, you've got great publicity already. And you could propose it as a general sorting AI (or web sale of small item) firm in the long term, with the lego being your initial application. You'd need to solve the inverse problem of how to efficiently go from bin to packing container, which doesn't seem too hard. Rotating carousels of bins with little robots or something. Set the item on a tray, and if isn't the right item, puff it back into the bin.

jacquesm 9y ago
If it won't work small it won't work big. First learn to walk. If I can overcome all the challenges and there is a sufficiently large market for the end product and when all the costs (energy, for instance) have been accounted for there is still a business case left then this might be worth it. But there is a ton of work to be done before you could begin to answer those questions. For now it is a fun project and I'm trying hard to keep it that way. I've been approached by some of the larger bulk resellers that would love to get rid of the humans they employ, none of them made a good enough case why I should partner with them or even seemed to be nice people. The general nature of neural nets for sorting and classification I think is well recognized in industry and there are many companies that are capitalizing on this (and have been for some time). I highly doubt I'm doing anything original in that sense.
- marze 9y ago
  If someone wants to give you 100k euros and develop some interesting technology, it seems like it would make the project more fun. What does anyone else think? Should Jacques open it up for investment?
  - jacquesm 9y ago
    I doubt anybody would give 100k euros. What is more likely to happen is that they would want equity in some company, a seat at the non existent board table, some wild eyed exit plan 3 years down the line and if they don't get their 20% ROI within the first year they'll be staging a minority shareholder revolt and possibly a lawsuit due to mis-management. For 100K I will gladly sidestep all those problems and self-finance.

pulse7 9y ago

You can really earn money by modifying this machine to sort various nuts and fruits by quality in the food industry...

arnsholt 9y ago

> Judging by the increase in accuracy from 16K images to 60K images there is something of a diminishing rate of return here. This is a well-known thing in machine learning. As far as I know all ML algorithms have this property, where the learning curve is basically logarithmic and each additive increase in performance requires multiplicatively more training material.

codewithcheese 9y ago

I think the sorted photos are great art. Maybe more valuable than the contents ;)

jacquesm 9y ago
Consider them public domain, if you want high res versions to print then just drop the .small. from the image filename and you should be good to go.

tzs 9y ago

Are there CAD models available for the various Lego components? I'm wondering if, given such models, you could use rendering software to generate labeled training images. That could allow you to get a large training set with many images for each kind of component in many different orientations, even for components you do not yet have. If the CAD models are not available, how much work is it to make a CAD model given an instance of a particular component? I'd expect that for some components, such as those NxM plates, you'd only need to build one model by hand in CAD software, and then could algorithmically generate the models for other sizes.

ant6n 9y ago
I don't know CAD models, but it would probably make sense to scrape the lego website where you can select all their bricks. Each named brick has 8 images (various orientations): https://shop.lego.com/Pick-a-Brick https://shop.lego.com/Pick-a-Brick I find it hard to believe that there are only 1421 bricks, including color combinations. Example: "Apple with leaf", Color Family: Dark Green, Exact Color: Bright green, Category: Foodstuff Element ID: 4107050, Design ID: 33051, orientation 8: https://sh-s7-live-s.legocdn.com/is/image/LEGOPCS/4107050_s8 https://sh-s7-live-s.legocdn.com/is/image/LEGOPCS/4107050_s8 ...This also makes me wonder whether it would make sense to set up multiple cameras, to get multiple orientations for every block. Use the image that has the lowest classification error, but use all orientations to train the network.
- jacquesm 9y ago
  The Lego pick-a-brick store is very very limited. The images are not very useful to me because the images the machine takes are very different from those. But maybe it would somehow contribute. I'd be very ware of using other people's copyrighted content though. I do use multiple orientations.
  - ant6n 9y ago
    Point taken re copyright. Re similarity, it may be possible to write some sort of imagemagic filter that makes the bricks appear more realistic somehow. (But yeah, this would probably only be useful as a starting point for training the very rare bricks)

justinpombrio 9y ago

> Overall the machine works well but I’m still not happy with the hopper and the first stage belt. It is running much slower than the speed at which the parts recognizer works (30 parts / second) and I really would like to come up with something better. It sounds like your existing hopper is reliable but slow? If that's the case, could you just make two (perhaps somewhat smaller) hoppers and merge their outputs?

jacquesm 9y ago
It's not even all that reliable, plenty of times it will vomit 5 to 10 parts in one go because they got clumped together somehow and when one falls the rest will too. I've tried to improve on this by using a slide towards the second belt and this works sometimes but not always. Still plenty of 'multipart' images.

marze 9y ago

On speeding up your feed, how about adding another belt prior to the main belt, with a variable speed? You drop parts onto the variable speed belt from enough sources to get a good feed rate, and constantly vary the speed of that belt with video feedback to drop them onto the main belt with reasonably even spacing.

jokoon 9y ago

Impressive, but I'm still wondering what 2 tons of lego look like.

jacquesm 9y ago
I really should make a picture of the storage, it's essentially a garage full of Lego. The garage is nearby, not at the same location where the machine is (that's at my house), I don't have room enough here to store that much stuff.

stuaxo 9y ago

Now you have a bunch of pieces each with different classifications. Possibly you could feed in parts lists from a huge amount of existing sets - have some sort of (handwavey learning thing or even markov chain), and generate mixes of pieces based on data sets from spaceships / cars / etc.

Poiesis 9y ago

Technics and larger wheels are useful for Lego robotics (FLL) teams everywhere; certainly useful for our team. Edit: this is in response to your asking for feedback about categories/demand.

jacquesm 9y ago
Larger wheels is a good category to have. Anything larger than 42 mm diameter gets filtered out very early on so that's a trivial class to add. Thank you!

Gravityloss 9y ago

In part 1 you mention that under vibration the bricks self assemble into bridges that obstruct the conveyor belt, and you want to minimize that. How about maximizing it instead? Try to build bridges as long as possible? Or some other fitness function? In theory you could control the vibration sequence and the part feeding sequence (once you have sorted all the parts :) ) and use an evolutionary algorithm...

jacquesm 9y ago
That's a hilarious idea. I'm somewhat tempted to try this. I've managed to bridge 35 cm 'by accident', I'm pretty sure you could go substantially larger than that with relatively little work.

punnerud 9y ago

Why not sort Lego the same way we sort plasic in Norway? Watch this video. Drawing of the technique at 0:35 and shown from 0:52 https://youtu.be/Mi0FHNLkim0 https://youtu.be/Mi0FHNLkim0

phaedrus 9y ago

Something I've wanted to try for years is what about a Lego refraction distillation tower? Similar to how oil refineries use a tower to sort hydrocarbon compounds by length based on their volatility. Except in this case instead of using heat we would use air blowing upward in a long tube to make the pieces "volatile". Picture a 6" diameter clear tube stretching floor to ceiling with a powerful blower at the base. Or a smaller version could be prototyped using an air popcorn popper with the heating element disabled. Essentially it would sort pieces according to their free-fall terminal velocity. You could exploit the Venturi effect to make the bottom of the tube have a higher velocity than the top, or simply add bleed-air holes at progressive levels. There could be take-off doors at different levels of the tube. A nice property is that if the ingress and egress mechanisms are designed elegantly, it could run continuously. (Although, that last point is one reason I've never built it: it seems like there would be a fine line between "air powered sorter" and "air powered rock tumbler" and I worry it could wear the corners of all my Lego bricks if it goes wrong.) Another potential issue is that there might not be much logical relationship between the pieces which share similar terminal velocities, and different pieces may have different or metastable terminal velocities depending on initial orientation. Obviously this wouldn't be a perfect sorter, but it could be a good pre- or post- sorter for your visual system. And for my purposes it might work well since instead of adapting the system to suit my sorting preferences I could simply train my sorting expectations based on what I learn the machine tends to sort together. Hmmm now that I have an industrial dust collector installed in my workshop, I actually already have the bulkiest / most expensive part of this invention. I just need a tall acrylic tube and I could try this!

schoen 9y ago
How do asymmetry and angular momentum affect this? Or how do they affect terminal velocity in general? (For example, what can happen to a Lego plate which has a very different terminal velocity in one orientation than in another?)