6 ms·

Image2Lego: Customized Lego Set Generation from Images

98 points by hubraumhugo 5y ago

fxtentacle 5y ago
"Finally, we test these automatically generated LEGO sets by constructing physical builds using real LEGO bricks." That reads to me like: "I'm not playing honey, this is work!" I sure hope the team doing this had a lot of fun :) The algorithm itself is OK but doesn't appear all that novel to me. It looks like they're using ShapeNet (or similar dataset) to train a 3D voxel autoencoder which can predict geometry from an image. And then they convert those voxels to LEGO instructions. For the first part, I've seen equally good papers before, for example by the ShapeNet team. For the second part, there's working explicit algorithms. So their main contribution appears to be to combine two working things into a (somewhat) useful whole.
- coldacid 5y ago
  I think a big improvement here would be supporting all the specialty pieces rather than the generic LEGO blocks and panels. Would certainly make for more interesting builds!
thih9 5y ago
Wouldn’t it be simpler to focus on the “3d model” -> “lego model” part of the conversion? What are the benefits of generating a lego set from an image, as opposed to generating a lego set from a 3d model (and doing the “image to 3d model” conversion with a different tool)?
- stavros 5y ago
  Because then it's more steps and more things to go wrong, I'd assume.
  - deleted 5y ago
    [deleted]
- cruano 5y ago
  > doing the “image to 3d model” conversion with a different tool Or using your phone's LIDAR sensors
mobilemidget 5y ago
Should work fine for minecraft creations too then I guess :)
tda 5y ago
TLDR: Image2voxel algorithm with a trivially simple voxels2lego bit tagged on for fun and/or clickbait
homarp 5y ago
see also https://krlennon.github.io/image2lego/ https://krlennon.github.io/image2lego/
- city41 5y ago
  I made a tool that converts pixel art to Lego mosaics that people in this thread might find interesting: https://city41.github.io/pxtobrx/ https://city41.github.io/pxtobrx/
vletal 5y ago
Could be seen as "The image to 3d models produce mostly unusable garbage. Looks a bit like Lego though..."
tomjuggler 5y ago
Translation: we took the fun part out if building Lego and gave it to a machine... Seriously, though, I have a real-world problem which needs a solution: given the roughly 3000 loose bricks in my son's Lego collection, tell me which ones go with which sets? Thank you - we can take it from here.
- Cthulhu_ 5y ago
  It probably can be done, if someone hasn't already done it; put the bricks in a bin that pulls it through a conveyor belt, which uses image recognition to determine color + shape, matching that up with known lego pieces. And the list of parts for every set is stored online somewhere as well. The difficulty is that some bricks will look like other bricks, close enough that the image recognition might be accurate. But, some margin for error is fine.
  - tomjuggler 5y ago
    I did see a sorting project somewhere, with image recognition. Apparently there is money to be made buying old Lego collections and selling sorted. Just the part where it puts the sets together was missing. If I had the time..
  - klausjensen 5y ago
    This is a video about such a project - of course done in LEGO! https://www.youtube.com/watch?v=04JkdHEX3Yk https://www.youtube.com/watch?v=04JkdHEX3Yk
  - mbreese 5y ago
    Here are a few... (and yes, they were on HN at the time) 2017 - Sorting 2 Metric Tons of Lego https://jacquesmattheij.com/sorting-two-metric-tons-of-lego/ https://jacquesmattheij.com/sorting-two-metric-tons-of-lego/ https://news.ycombinator.com/item?id=14226889 https://news.ycombinator.com/item?id=14226889 2019 - AI-powered Lego sorting machine built with Lego bricks https://www.brothers-brick.com/2019/12/06/this-fan-created-the-ultimate-lego-brick-sorting-machine-powered-by-ai-exclusive-feature/ https://www.brothers-brick.com/2019/12/06/this-fan-created-t... https://news.ycombinator.com/item?id=21741834 https://news.ycombinator.com/item?id=21741834
pavlovskyi 5y ago
Large amount of work, it gives me a nice insight about Image2Voxel, which I never test out before. What will be worth to add is an real case scenario or some actual benefit that it'll produce in your opinion.
- sumtechguy 5y ago
  Ages ago I built a 'lego' system. Basically I figured out you could build any lego shape from 5 basic voxel types, cube, bevel, cone segment, sphere segment, cylinder. Think I was calling them lego atoms, as you could scale the resolution depending on the amount of detail you needed. Not sure if that is true anymore with the more interesting shapes they have introduced. Not sure only 4 would cut it anymore. I only took it to the paper/whiteboard design stage. The problem I had was voxels became wildly more memory consuming when you introduce more than a binary there not there type. I was looking at 10-15 megabytes in memory just to describe a simple 2x4 brick with the min resolution and it still worked as a lego. At the time I was lucky I had 96MB... The exe was pretty small (maybe 1-2MB total). The data on the other hand I could not find a nice way to compress it enough to work without a few dozen bricks filling my memory space. Mostly I got bored with the project and moved on. The 'voxel' style has one very nice quality about it over the ldraw cad system. Unintentional gaps are nearly impossible to have and overlap is dead easy to find. Correct overlap lets you do interesting things in that you can intersect things correctly and it will just work on the snap grid. The ldraw system has to jump thru a few hoops to make it look like it is working. The effect is the same but more of a pain making sure your floating point is correct. ldraw has a nice quality in that rotating an object in freespace is 'cheap' and you only have to rotate a few points. Whereas a voxel system like I designed you in effect have to rotate everything. At this time the ldraw system is the choice to use if you are doing this though. When I built my system there were maybe 200 different bricks total. Now there are thousands.
  - pavlovskyi 5y ago
    Love it! I started to working at my side project on lego classification and detection algorithm to identify what set could you build from bricks scattered on the floor. What change is that I am not that aware of memory consumption as in your project in the past. All best!
nikkinana 5y ago
Where's the code? Anybody can write up an idea, that's the easy part.
dane-pgp 5y ago
> In order to demonstrate the broad applicability of our system, we generate step-by-step building instructions and animations for LEGO models Then perhaps the AI experts are on track for a successful prediction: > The team also predicts that by the year 2023, a machine will be able to “physically assemble any Lego set given the pieces and instructions, using non-specialized robotics hardware.” https://www.inverse.com/article/44660-lego-building-ai-robots https://www.inverse.com/article/44660-lego-building-ai-robot...

98 points by hubraumhugo 5y ago

"Finally, we test these automatically generated LEGO sets by constructing physical builds using real LEGO bricks." That reads to me like: "I'm not playing honey, this is work!" I sure hope the team doing this had a lot of fun :) The algorithm itself is OK but doesn't appear all that novel to me. It looks like they're using ShapeNet (or similar dataset) to train a 3D voxel autoencoder which can predict geometry from an image. And then they convert those voxels to LEGO instructions. For the first part, I've seen equally good papers before, for example by the ShapeNet team. For the second part, there's working explicit algorithms. So their main contribution appears to be to combine two working things into a (somewhat) useful whole.

coldacid 5y ago
I think a big improvement here would be supporting all the specialty pieces rather than the generic LEGO blocks and panels. Would certainly make for more interesting builds!

thih9 5y ago

Wouldn’t it be simpler to focus on the “3d model” -> “lego model” part of the conversion? What are the benefits of generating a lego set from an image, as opposed to generating a lego set from a 3d model (and doing the “image to 3d model” conversion with a different tool)?

stavros 5y ago
Because then it's more steps and more things to go wrong, I'd assume.
- deleted 5y ago
  [deleted]
cruano 5y ago
> doing the “image to 3d model” conversion with a different tool Or using your phone's LIDAR sensors

mobilemidget 5y ago

Should work fine for minecraft creations too then I guess :)

tda 5y ago

TLDR: Image2voxel algorithm with a trivially simple voxels2lego bit tagged on for fun and/or clickbait

homarp 5y ago

see also https://krlennon.github.io/image2lego/ https://krlennon.github.io/image2lego/

city41 5y ago
I made a tool that converts pixel art to Lego mosaics that people in this thread might find interesting: https://city41.github.io/pxtobrx/ https://city41.github.io/pxtobrx/

vletal 5y ago

Could be seen as "The image to 3d models produce mostly unusable garbage. Looks a bit like Lego though..."

tomjuggler 5y ago

Translation: we took the fun part out if building Lego and gave it to a machine... Seriously, though, I have a real-world problem which needs a solution: given the roughly 3000 loose bricks in my son's Lego collection, tell me which ones go with which sets? Thank you - we can take it from here.

Cthulhu_ 5y ago
It probably can be done, if someone hasn't already done it; put the bricks in a bin that pulls it through a conveyor belt, which uses image recognition to determine color + shape, matching that up with known lego pieces. And the list of parts for every set is stored online somewhere as well. The difficulty is that some bricks will look like other bricks, close enough that the image recognition might be accurate. But, some margin for error is fine.
- tomjuggler 5y ago
  I did see a sorting project somewhere, with image recognition. Apparently there is money to be made buying old Lego collections and selling sorted. Just the part where it puts the sets together was missing. If I had the time..
- klausjensen 5y ago
  This is a video about such a project - of course done in LEGO! https://www.youtube.com/watch?v=04JkdHEX3Yk https://www.youtube.com/watch?v=04JkdHEX3Yk
- mbreese 5y ago
  Here are a few... (and yes, they were on HN at the time) 2017 - Sorting 2 Metric Tons of Lego https://jacquesmattheij.com/sorting-two-metric-tons-of-lego/ https://jacquesmattheij.com/sorting-two-metric-tons-of-lego/ https://news.ycombinator.com/item?id=14226889 https://news.ycombinator.com/item?id=14226889 2019 - AI-powered Lego sorting machine built with Lego bricks https://www.brothers-brick.com/2019/12/06/this-fan-created-the-ultimate-lego-brick-sorting-machine-powered-by-ai-exclusive-feature/ https://www.brothers-brick.com/2019/12/06/this-fan-created-t... https://news.ycombinator.com/item?id=21741834 https://news.ycombinator.com/item?id=21741834

pavlovskyi 5y ago

Large amount of work, it gives me a nice insight about Image2Voxel, which I never test out before. What will be worth to add is an real case scenario or some actual benefit that it'll produce in your opinion.

sumtechguy 5y ago
Ages ago I built a 'lego' system. Basically I figured out you could build any lego shape from 5 basic voxel types, cube, bevel, cone segment, sphere segment, cylinder. Think I was calling them lego atoms, as you could scale the resolution depending on the amount of detail you needed. Not sure if that is true anymore with the more interesting shapes they have introduced. Not sure only 4 would cut it anymore. I only took it to the paper/whiteboard design stage. The problem I had was voxels became wildly more memory consuming when you introduce more than a binary there not there type. I was looking at 10-15 megabytes in memory just to describe a simple 2x4 brick with the min resolution and it still worked as a lego. At the time I was lucky I had 96MB... The exe was pretty small (maybe 1-2MB total). The data on the other hand I could not find a nice way to compress it enough to work without a few dozen bricks filling my memory space. Mostly I got bored with the project and moved on. The 'voxel' style has one very nice quality about it over the ldraw cad system. Unintentional gaps are nearly impossible to have and overlap is dead easy to find. Correct overlap lets you do interesting things in that you can intersect things correctly and it will just work on the snap grid. The ldraw system has to jump thru a few hoops to make it look like it is working. The effect is the same but more of a pain making sure your floating point is correct. ldraw has a nice quality in that rotating an object in freespace is 'cheap' and you only have to rotate a few points. Whereas a voxel system like I designed you in effect have to rotate everything. At this time the ldraw system is the choice to use if you are doing this though. When I built my system there were maybe 200 different bricks total. Now there are thousands.
- pavlovskyi 5y ago
  Love it! I started to working at my side project on lego classification and detection algorithm to identify what set could you build from bricks scattered on the floor. What change is that I am not that aware of memory consumption as in your project in the past. All best!

nikkinana 5y ago

Where's the code? Anybody can write up an idea, that's the easy part.

dane-pgp 5y ago

> In order to demonstrate the broad applicability of our system, we generate step-by-step building instructions and animations for LEGO models Then perhaps the AI experts are on track for a successful prediction: > The team also predicts that by the year 2023, a machine will be able to “physically assemble any Lego set given the pieces and instructions, using non-specialized robotics hardware.” https://www.inverse.com/article/44660-lego-building-ai-robots https://www.inverse.com/article/44660-lego-building-ai-robot...