4 ms·

RebrickNet – Lego Part Detector

107 points by wonger_ 2y ago

bigiain 2y ago
Nice. Reminds me of this: https://jacquesmattheij.com/sorting-two-metric-tons-of-lego/ https://jacquesmattheij.com/sorting-two-metric-tons-of-lego/
denysvitali 2y ago
(I'm not a LEGO expert.) It looks like LEGO provides 3D models for their components (e.g: via Bricklink [1]). Wouldn't it be easier to generate the training data with a rendering pipeline of some sort that randomizes the brick position in a 3D scene with different lighting? Of course you can still collect the user submissions for the test set [1]: https://www.bricklink.com/v3/studio/download.page https://www.bricklink.com/v3/studio/download.page
- prybak 2y ago
  From personal experience, it doesn't work that well. The available 3D models [1] are not detailed enough and the effort to make realistic and diverse renders is huge. It will kinda work but it won't generalize well into real images. [1] https://www.ldraw.org/ https://www.ldraw.org/
  - tirant 2y ago
    That is something I don't really understand exactly why. 3D Models provide in one dataset the whole geometry information of each part, making it easier to learn and recognize that part in the real world, wouldn't it? Is it because these ML algos lack a way to internally interpret 3D models for learning? On top of that you reduce effort in data labeling, as each model would come with the relevant part ID, shape and color. Of course, finetuning can be done afterwards with real world photos to increase robustness.
    - prybak 2y ago
      [dead]
    - loa_in_ 2y ago
      I think it's because we classify single images which lose 3d information. We expect them to work with single image but not even humans do that (without stereoscopic vision) if we can help it.
      - voiper1 2y ago
        Right, I would assume we'd render multiple 2D images from different angles, and train on that. But as I'm writing this, why can't we do ML directly with the 3D models...
        amelius 2y ago
        Yes, I think it should be possible without any technological hurdles. It's just some work to set it up. (DL models are trained with one type of camera and used with another type of camera all the time, and that's sort of similar; plus renderers as used in blender3d are pretty good and they should work well with LEGO bricks which are relatively simple objects; if despite all this it doesn't work you could degrade both the real and the generated images e.g. by quantization, with the aim of bringing them closer together.)
        rlnvlc 2y ago
        Because the input is a 2D picture not a 3D scene with all the coordinates and geometry information.
        lostdog 2y ago
        It's really hard to do in practice. Yes, you can train an ML model on rendered data, but the model tends to fixate on rendering artifacts, and performance doesn't transfer to real world images. Plus, it's very difficult to make generated scenes with the variety and complexity of the real world. Your trained model will fail to generalize to all the distractions in natural scenes. Yes, there are techniques for all these problems, but none of them are good, reliable, or easy to get right.
- juliangamble 2y ago
  There is some code to generate Lego images from Lego 3d files here and use it to train models: https://github.com/jtheiner/LegoBrickClassification https://github.com/jtheiner/LegoBrickClassification It is based on this post (which others have mentioned) https://jacquesmattheij.com/sorting-lego-the-software-side/ https://jacquesmattheij.com/sorting-lego-the-software-side/
- michaelt 2y ago
  > Wouldn't it be easier to generate the training data with a rendering pipeline of some sort that randomizes the brick position in a 3D scene with different lighting? You might enjoy the 2023 paper "Brickognize: Applying Photo-Realistic Image Synthesis for Lego Bricks Recognition with Limited Data" https://www.mdpi.com/1424-8220/23/4/1898 https://www.mdpi.com/1424-8220/23/4/1898
  - denysvitali 2y ago
    Wow. Exactly my idea. Thanks for linking it, I'm glad it's a successful approach!
prybak 2y ago
RebrickNet is an abandoned project that supports only 300 parts. I recommend Brickognize [1], which recognizes all 85k Lego parts, minifigures, and sets. Disclosure: I'm the creator. [1] https://brickognize.com/ https://brickognize.com/
- cgeier 2y ago
  Which works really well! Thanks for a great project! Is there any more information available on how it works?
- pragmatick 2y ago
  That's a weirdly minimalistic site. I mean it's nice that the front page is exactly what you want to see but I wish the About page would show some more information about the what the site can do, some examples, technology bits etc.
  - rkagerer 2y ago
    Sometimes a hammer can just be a hammer. I wish the whole web was so clean.
- majke 2y ago
  Piotr has an amazing video on the inner workings of his model https://www.youtube.com/watch?v=bzyG4Wf1Nkc https://www.youtube.com/watch?v=bzyG4Wf1Nkc
- fxtentacle 2y ago
  Are you willing to share some info on architecture and data set?
byrolew 2y ago
Does it use brickognize underneath?
shagrath 2y ago
Neat, I happpened to rediscover and sort my childhood lego collection two weeks ago, and had good success for the "no so common parts" with the brickowl camera search (I don't know what kind of engine is behind)
primitivesuave 2y ago
I built a rudimentary brick sorting machine with the Lego color sensor many years ago, and I've seen some awesome brick sorting machines pop up over the years [1]. I started planning a sorting machine with a larger array of buckets in CAD but never got around to building it, hopefully the intrepid Lego fans over time will keep advancing the state of the art. 1. https://www.youtube.com/watch?v=04JkdHEX3Yk https://www.youtube.com/watch?v=04JkdHEX3Yk
Aissen 2y ago
No mention of license on contributed data, or the model.
juliangamble 2y ago
I’ve tried different part detectors, and nothing has been as good as the Google Image search (download the Google App if you’re on an iPhone.) I’ve found it surprisingly good at identifying LEGO parts.
mrbigbob 2y ago
Another neat app if you have an iphone is Brickit which scans a large pile of your lego pieces and gives you build ideas. https://brickit.app/ https://brickit.app/
- elephantum 2y ago
  This one actually works :) Source: I'm responsible for ML development at Brickit
  - htrp 2y ago
    would love to hear more on the architecture choices you made. do your models run on device? what's the general CV backbone?
    - elephantum 2y ago
      Everything runs on device in tflite and it gives us some headaches, especially Android ecosystem. We do not use anything fancy, detector is ssd-like, and classifier is either resnet or efficientnet depending on your device capability
  - julianpye 2y ago
    One question about Brickit - the main usecase I and many many dads of Lego kids have is that our kids want to reassemble sets and we spend ages searching for these pieces. Yet Brickit works by identifying and recommending its own mini-set lists. Is this a usecase that is not in scope (because of business model) or is technically difficult or unsatisfactory in execution (colors, accuracy)?
    - elephantum 2y ago
      The use case "let me reassemble all the parts that I have" is out of scope of Brickit app. It works explicitly with the contents of a single scan, that's why you see small ideas to build: it's just what fits into the parts you scanned. But! We recognize the intent, and we have something in the works which will be released very soon, stay tuned!
vincvinc 2y ago
related: see HN user jacquesm's work on a LEGO sorting machine https://jacquesmattheij.com/sorting-lego-many-questions-and-this-is-what-the-result-looks-like/ https://jacquesmattheij.com/sorting-lego-many-questions-and-...
amelius 2y ago
I'm curious how far away we are from a robot that can pick up LEGO pieces and stack them together according to the assembly-booklet. And what kind of DL models would it use?
- fragmede 2y ago
  We're a ways away from a hand and arm that dexterous, though there's been promising developments given all the money going into AI lately. We're also a ways away from having a machine "read" a PDF of LEGO instructions and then modeling it as a digital twin. It's being worked on though, so maybe in our lifetimes. That's an AI I'd rather not see though - that's the fun part OF LEGOs. If you really want a model completed, pay a local teenage (or laid off software dev) to do it.
  - amelius 2y ago
    > We're a ways away from a hand and arm that dexterous So you are saying it is not possible to stack LEGO pieces by a remotely controlled arm (with a human operator)? > That's an AI I'd rather not see though - that's the fun part OF LEGOs AI is already claiming all the fun tasks like writing poems and drawing kittens ;)
    - Freak_NL 2y ago
      Have you ever assembled a Lego set rated 8+ or older? It's not about stacking bricks, it's about applying just enough pressure at exactly the right angle, whilst supporting the assembly in the right places. (Something humans can do remarkably well.) This is something you can probably program an expensive assembly robot to do if it has to repeat that same procedure thousands of times, but doing it for a whole set human operated? That only makes sense if you haven't got a set of nimble appendices and opposable thumbs, and you have the time to spare and inclination to train with the remotely controlled arm. Having an LLM do it seems fruitless at this point — and why bother?
      - amelius 2y ago

107 points by wonger_ 2y ago

Nice. Reminds me of this: https://jacquesmattheij.com/sorting-two-metric-tons-of-lego/ https://jacquesmattheij.com/sorting-two-metric-tons-of-lego/

denysvitali 2y ago

(I'm not a LEGO expert.) It looks like LEGO provides 3D models for their components (e.g: via Bricklink [1]). Wouldn't it be easier to generate the training data with a rendering pipeline of some sort that randomizes the brick position in a 3D scene with different lighting? Of course you can still collect the user submissions for the test set [1]: https://www.bricklink.com/v3/studio/download.page https://www.bricklink.com/v3/studio/download.page

prybak 2y ago
From personal experience, it doesn't work that well. The available 3D models [1] are not detailed enough and the effort to make realistic and diverse renders is huge. It will kinda work but it won't generalize well into real images. [1] https://www.ldraw.org/ https://www.ldraw.org/
- tirant 2y ago
  That is something I don't really understand exactly why. 3D Models provide in one dataset the whole geometry information of each part, making it easier to learn and recognize that part in the real world, wouldn't it? Is it because these ML algos lack a way to internally interpret 3D models for learning? On top of that you reduce effort in data labeling, as each model would come with the relevant part ID, shape and color. Of course, finetuning can be done afterwards with real world photos to increase robustness.
  - prybak 2y ago
    [dead]
  - loa_in_ 2y ago
    I think it's because we classify single images which lose 3d information. We expect them to work with single image but not even humans do that (without stereoscopic vision) if we can help it.
    - voiper1 2y ago
      Right, I would assume we'd render multiple 2D images from different angles, and train on that. But as I'm writing this, why can't we do ML directly with the 3D models...
      - amelius 2y ago
        Yes, I think it should be possible without any technological hurdles. It's just some work to set it up. (DL models are trained with one type of camera and used with another type of camera all the time, and that's sort of similar; plus renderers as used in blender3d are pretty good and they should work well with LEGO bricks which are relatively simple objects; if despite all this it doesn't work you could degrade both the real and the generated images e.g. by quantization, with the aim of bringing them closer together.)
      - rlnvlc 2y ago
        Because the input is a 2D picture not a 3D scene with all the coordinates and geometry information.
      - lostdog 2y ago
        It's really hard to do in practice. Yes, you can train an ML model on rendered data, but the model tends to fixate on rendering artifacts, and performance doesn't transfer to real world images. Plus, it's very difficult to make generated scenes with the variety and complexity of the real world. Your trained model will fail to generalize to all the distractions in natural scenes. Yes, there are techniques for all these problems, but none of them are good, reliable, or easy to get right.
juliangamble 2y ago
There is some code to generate Lego images from Lego 3d files here and use it to train models: https://github.com/jtheiner/LegoBrickClassification https://github.com/jtheiner/LegoBrickClassification It is based on this post (which others have mentioned) https://jacquesmattheij.com/sorting-lego-the-software-side/ https://jacquesmattheij.com/sorting-lego-the-software-side/
michaelt 2y ago
> Wouldn't it be easier to generate the training data with a rendering pipeline of some sort that randomizes the brick position in a 3D scene with different lighting? You might enjoy the 2023 paper "Brickognize: Applying Photo-Realistic Image Synthesis for Lego Bricks Recognition with Limited Data" https://www.mdpi.com/1424-8220/23/4/1898 https://www.mdpi.com/1424-8220/23/4/1898
- denysvitali 2y ago
  Wow. Exactly my idea. Thanks for linking it, I'm glad it's a successful approach!

prybak 2y ago

RebrickNet is an abandoned project that supports only 300 parts. I recommend Brickognize [1], which recognizes all 85k Lego parts, minifigures, and sets. Disclosure: I'm the creator. [1] https://brickognize.com/ https://brickognize.com/

cgeier 2y ago
Which works really well! Thanks for a great project! Is there any more information available on how it works?
pragmatick 2y ago
That's a weirdly minimalistic site. I mean it's nice that the front page is exactly what you want to see but I wish the About page would show some more information about the what the site can do, some examples, technology bits etc.
- rkagerer 2y ago
  Sometimes a hammer can just be a hammer. I wish the whole web was so clean.
majke 2y ago
Piotr has an amazing video on the inner workings of his model https://www.youtube.com/watch?v=bzyG4Wf1Nkc https://www.youtube.com/watch?v=bzyG4Wf1Nkc
fxtentacle 2y ago
Are you willing to share some info on architecture and data set?

byrolew 2y ago

Does it use brickognize underneath?

shagrath 2y ago

Neat, I happpened to rediscover and sort my childhood lego collection two weeks ago, and had good success for the "no so common parts" with the brickowl camera search (I don't know what kind of engine is behind)

primitivesuave 2y ago

I built a rudimentary brick sorting machine with the Lego color sensor many years ago, and I've seen some awesome brick sorting machines pop up over the years [1]. I started planning a sorting machine with a larger array of buckets in CAD but never got around to building it, hopefully the intrepid Lego fans over time will keep advancing the state of the art. 1. https://www.youtube.com/watch?v=04JkdHEX3Yk https://www.youtube.com/watch?v=04JkdHEX3Yk

Aissen 2y ago

No mention of license on contributed data, or the model.

juliangamble 2y ago

I’ve tried different part detectors, and nothing has been as good as the Google Image search (download the Google App if you’re on an iPhone.) I’ve found it surprisingly good at identifying LEGO parts.

mrbigbob 2y ago

Another neat app if you have an iphone is Brickit which scans a large pile of your lego pieces and gives you build ideas. https://brickit.app/ https://brickit.app/

elephantum 2y ago
This one actually works :) Source: I'm responsible for ML development at Brickit
- htrp 2y ago
  would love to hear more on the architecture choices you made. do your models run on device? what's the general CV backbone?
  - elephantum 2y ago
    Everything runs on device in tflite and it gives us some headaches, especially Android ecosystem. We do not use anything fancy, detector is ssd-like, and classifier is either resnet or efficientnet depending on your device capability
- julianpye 2y ago
  One question about Brickit - the main usecase I and many many dads of Lego kids have is that our kids want to reassemble sets and we spend ages searching for these pieces. Yet Brickit works by identifying and recommending its own mini-set lists. Is this a usecase that is not in scope (because of business model) or is technically difficult or unsatisfactory in execution (colors, accuracy)?
  - elephantum 2y ago
    The use case "let me reassemble all the parts that I have" is out of scope of Brickit app. It works explicitly with the contents of a single scan, that's why you see small ideas to build: it's just what fits into the parts you scanned. But! We recognize the intent, and we have something in the works which will be released very soon, stay tuned!

vincvinc 2y ago

related: see HN user jacquesm's work on a LEGO sorting machine https://jacquesmattheij.com/sorting-lego-many-questions-and-this-is-what-the-result-looks-like/ https://jacquesmattheij.com/sorting-lego-many-questions-and-...

amelius 2y ago

I'm curious how far away we are from a robot that can pick up LEGO pieces and stack them together according to the assembly-booklet. And what kind of DL models would it use?

fragmede 2y ago
We're a ways away from a hand and arm that dexterous, though there's been promising developments given all the money going into AI lately. We're also a ways away from having a machine "read" a PDF of LEGO instructions and then modeling it as a digital twin. It's being worked on though, so maybe in our lifetimes. That's an AI I'd rather not see though - that's the fun part OF LEGOs. If you really want a model completed, pay a local teenage (or laid off software dev) to do it.
- amelius 2y ago
  > We're a ways away from a hand and arm that dexterous So you are saying it is not possible to stack LEGO pieces by a remotely controlled arm (with a human operator)? > That's an AI I'd rather not see though - that's the fun part OF LEGOs AI is already claiming all the fun tasks like writing poems and drawing kittens ;)
  - Freak_NL 2y ago
    Have you ever assembled a Lego set rated 8+ or older? It's not about stacking bricks, it's about applying just enough pressure at exactly the right angle, whilst supporting the assembly in the right places. (Something humans can do remarkably well.) This is something you can probably program an expensive assembly robot to do if it has to repeat that same procedure thousands of times, but doing it for a whole set human operated? That only makes sense if you haven't got a set of nimble appendices and opposable thumbs, and you have the time to spare and inclination to train with the remotely controlled arm. Having an LLM do it seems fruitless at this point — and why bother?
    - amelius 2y ago