7 ms·

MIT, Autodesk develop AI that can figure out Lego instructions

138 points by samizdis 4y ago

stinkytaco 4y ago
This is entirely beside the point, but I think Lego instructions were my first real foray into technical documentation. It's surprising how well they worked. Their ability to boil down a complicated process into something a 10 year old could follow was impressive. As I got older I sort of dismissed them as antithetical to creativity, but after doing Lego robotics with some kids I began to realize the value I got from them.
- rkangel 4y ago
  An enormous amount of effort has gone into making Lego instructions as clear as they are, regardless of what age you are and what language you speak. They really show what is possible.
  - corrral 4y ago
    The new ones are much clearer than they used to be in the 80s and 90s. I'm convinced the old-style ones were the reason I buried the needle the one time I was given a cognitive test that specifically tested spatial reasoning. The old sets' instructions were basically 3d spot-the-difference puzzles, in which orientation might even change between states, so they were pretty much training for that kind of test. The new ones have a little of that, but a lot more arrows pointing things out, little insets highlighting important parts, et c. Plus they put the pieces in numbered bags so you don't have them all out at once, for medium-sized or larger sets! Much, much easier than they used to be. Flip side is the old ones used larger pieces, on average. The modern sets are all fiddly, tiny bricks, which does make them a little harder to assemble (and much harder—nearly impossible—to repair from memory if they're damaged during play)
  - Swizec 4y ago
    Now if they could print them in higher quality so I’m not constantky guessing if that’s a black piece or dark blue … Maybe I’m older and my color vision isn’t as good as it used to be, but damn it’s frustrating.
    - TylerE 4y ago
      I don’t think it’s the printing. The downloadable pdfs have the same issues
  - deleted 4y ago
    [deleted]
- mlyle 4y ago
  > As I got older I sort of dismissed them as antithetical to creativity Yah, there's a lot of criticism of the "directions". My kids build big elaborate modular buildings from the directions... and then leave them as-is... And then try to build equivalently complicated things from loose bricks (sometimes copying a little bit of a mechanism or a trick they've seen elsewhere). I think the instructions are great. They just shouldn't be the be-all, end-all.
  - disconcision 4y ago
    yup. the blocks themselves give you the first-order building components, and then the instructions give you an increasingly library of higher-order building components to work with... like reading code
- samstave 4y ago
  AS AN ASIDE; When I was a young'n I had a lot of legos - but my granfather was a nuclear engineer with GE... (helped build Hanford) There was a technical design/drafting book in his book case I picked up where I was around ~6 or so... I drew every single diagram/ortho/iso whatever in that book. Multiple times. When I got into middle school, I was opted to elect a highschool class, and I chose technical drafting... When I went to class, the textbook was the same one from my grandfathers bookcase - I sailed through that, and then... This was what got me into design and architecture and CAD... I then setup the first cad network in my highschool, while I was still in 7th grade. I was in VOCAD by 9th, and came in second in the CAD Olympics in 1993 (I was the fastest ever to finish the test in 30 minutes, on a 3 hour deadline.) THE ENTIRE POINT BEING ; Get kids in front of technical design/docs/whatever is complex as EARLY as possible. There is no loss in teaching a toddler calculus. They may wind up being a flamboyant broadway star, but at least they can maths.
  - amelius 4y ago
    > They may wind up being a flamboyant broadway star, but at least they can maths. What fun is it to be a broadway star when you know maths?
    - asadlionpk 4y ago
      You can calculate probability of success as a broadway star.
    - beambot 4y ago
      Understanding that the world and your interests may not be one dimensional... ;)
    - samstave 4y ago
      AAAAAND TWO AND THREE AND FOUR AND FIVE AND SIX AND SASHE AND TURN THAT NOMIAL INTO A PRIME AND FIVE SIX SEVEN EIGHT! HANDS LADIES!!! LET ME SEE YOUR JAZZ HANDS TIMES THE SQUARE ROOT OF SEVEN DIVIDED BY THE OUTPUT OF THE GLITTER FACTORY!!! Sounds like it could be fun.
    - mrguyorama 4y ago
- paulryanrogers 4y ago
  More so that they do it with few words and mostly images and icons.
intrasight 4y ago
Brings back an old memory. I took CMU's AI class in 1986. Most CMU classes were really two classes - one about the lectures and one about learning to hack. I did my project for this class in Lisp and wrote an engine to solve Lego assembly problems. While testing, I gave it a problem to solve, and it came up with a better solution than mine. Was my first AI eureka moment.
- LionTamer 4y ago
  That sounds really interesting. It was awhile ago so you may not remember, but how did you “operationalize” the problem? I wonder how that kind of problem / task is structured.
  - intrasight 4y ago
    The problem was structured with two inputs. 1. the available Lego pieces. 2. the structure to be built. At least that's what I remember.
upupandup 4y ago
a bit off topic, but is there like a more affordable lego like alternative? it really disgusts me how expensive lego and litigous they are. so many Americans live in poverty who would love to get their kids to play with lego but can't because its ridiculously expensive.
- SequoiaHope 4y ago
  I’m not sure how they compare in price but I really enjoyed erector set and k’nex when I was a kid 30 years ago. And now I’m a robotics engineer! Also, an Ender 3 3D printer provides a lot of similar construction skills and learning (with the printed objects) for the price of one high end Lego set.
  - londons_explore 4y ago
    +1 to buying a child a 3d printer. There are okayish ones for about $140 now. About age 6-8 seems like a good time. Set them up with an easy to use bit of cad software and show them how to 3d print a cube... Then let them 3d print whatever they want to make. 3d printers have hot parts - let them burn their fingers once or twice and they'll soon learn which bits not to touch. After that, a 3d printer is pretty safe. Test it first, but generally the motors aren't strong enough to crush a finger placed in the wrong spot. The only downside is many 3d prints take 1 hour + to print, which is pretty frustrating for a child. You need another activity to entertain them during printing breaks.
    - TaylorAlexander 4y ago
      I have no kids but I definitely found that waiting for the print was tough! Often what I do now is I get all my design work done at night. have fun getting design features sorted, or searching online libraries for printable parts, then get everything configured and start the print before bed. Watch the first layer or two to make sure it's gonna work, and then go to sleep while its printing. I wake up and the print is done! idk if this would work for kids but it works for me! it also means that prints which are 8-10 hours don't feel long, because I start it an hour before bed time, then as I am waking up and doing my morning routine the print is on its last few bits!
rkagerer 4y ago
This is begging for someone to make a robot using MEPNet that can assemble any kit for which you show it the instructions. Bonus points if it's made out of Lego and can build itself.
mech422 4y ago
I'd be more impressed if they can figure out Ikea directions :-P
- pchristensen 4y ago
  Try assembling furniture from another maker (e.g. anything from Target, Home Depot, etc) and you'll be thirsting for the clarity, completeness, and the large, legible print of Ikea instructions.
  - jiggunjer 4y ago
    Step 1: build half of the bed. Step 2: build the other half.
- sschueller 4y ago
  I actually find ikea instructions quite clear and reasonable. I don't know why so many people have a hard time with it. Maybe spacial awareness is not something you can learn easily.
  - stevenjgarner 4y ago
    I get a real B&W experience with Ikea - it would be interesting to have a technical writing expert compare the Ikea instructions and the Lego instructions. Lego seems to generally be accepted as more universally simple, and yet Ikea products frequently have fewer parts and steps.
    - breuleux 4y ago
      I find both quite clear. I think the main difference is that it's a lot more frustrating if you mess up a step in Ikea instructions than in a Lego build. So if you're bad at spatial reasoning or following instructions, you can still have fun with Lego, you'll just need to redo some steps sometimes and you might not even remember it, but the same kind of mistake with Ikea will throw you in a fit of rage.
dekhn 4y ago
I developed an AI to do this- well, I guess it wasn't artificial. I had a kid, and after showing him a few times (and correcting some mistakes due to the instructions sometimes being... as sparse as technically possible....) he had no problem. Now the issue is getting rid of all the 5-year-old Hogwarts Castles, etc...
- javajosh 4y ago
  Feel free to break them down and ship them to me, or some other parent of a young kid. Those sets are expensive!
O__________O 4y ago
Official Research Page: https://cs.stanford.edu/~rcwang/projects/lego_manual/ https://cs.stanford.edu/~rcwang/projects/lego_manual/ - which includes the paper, code and video.
eatsyourtacos 4y ago
I wonder what they taught the AI first to make enough money to be able to afford legos?
russellbeattie 4y ago
There is an app that's been linked on HN a few times called Brickit. It analyzes a picture you took of a pile of Legos, identifies pieces with object recognition, then uses AR to highlight exactly which pieces to grab to build a model. If there's such thing as a beautiful app, this is it. https://brickit.app/ https://brickit.app/
- jiggunjer 4y ago
  Don't people get their kids Lego to get them away from phones?
dane-pgp 4y ago
Interestingly, people were predicting[0] in 2018 that the robotic equivalent of this task would be solved by 2023: > The team also predicts that by the year 2023, a machine will be able to “physically assemble any Lego set given the pieces and instructions, using non-specialized robotics hardware.” The paper cites University of California Berkeley research that showed a machine joining two Lego pieces together in 2016 through reinforcement learning. [0] https://www.inverse.com/article/44660-lego-building-ai-robots https://www.inverse.com/article/44660-lego-building-ai-robot...
peterlk 4y ago
Side note about MITs marketing... The original paper [0] was published by 5 authors. 3 from Stanford, 1 from MIT, and 1 from Autodesk. I find MITs aggressive marketing pretty off-putting. Any discovery related to MIT _always_ has MIT in the headline, even if their involvement was secondary. It's annoying and makes me feel like MIT cares more about optics than good research. If I'm not alone in this belief, MIT will erode their brand by doing so. They certainly have for me. [0] https://cs.stanford.edu/~rcwang/projects/lego_manual/ https://cs.stanford.edu/~rcwang/projects/lego_manual/
- RcouF1uZ4gsC 4y ago
  > If I'm not alone in this belief, MIT will erode their brand by doing so. They certainly have for me. They already eroded their brand when they provided cover for Jeffrey Epstein.

138 points by samizdis 4y ago

This is entirely beside the point, but I think Lego instructions were my first real foray into technical documentation. It's surprising how well they worked. Their ability to boil down a complicated process into something a 10 year old could follow was impressive. As I got older I sort of dismissed them as antithetical to creativity, but after doing Lego robotics with some kids I began to realize the value I got from them.

rkangel 4y ago
An enormous amount of effort has gone into making Lego instructions as clear as they are, regardless of what age you are and what language you speak. They really show what is possible.
- corrral 4y ago
  The new ones are much clearer than they used to be in the 80s and 90s. I'm convinced the old-style ones were the reason I buried the needle the one time I was given a cognitive test that specifically tested spatial reasoning. The old sets' instructions were basically 3d spot-the-difference puzzles, in which orientation might even change between states, so they were pretty much training for that kind of test. The new ones have a little of that, but a lot more arrows pointing things out, little insets highlighting important parts, et c. Plus they put the pieces in numbered bags so you don't have them all out at once, for medium-sized or larger sets! Much, much easier than they used to be. Flip side is the old ones used larger pieces, on average. The modern sets are all fiddly, tiny bricks, which does make them a little harder to assemble (and much harder—nearly impossible—to repair from memory if they're damaged during play)
- Swizec 4y ago
  Now if they could print them in higher quality so I’m not constantky guessing if that’s a black piece or dark blue … Maybe I’m older and my color vision isn’t as good as it used to be, but damn it’s frustrating.
  - TylerE 4y ago
    I don’t think it’s the printing. The downloadable pdfs have the same issues
- deleted 4y ago
  [deleted]
mlyle 4y ago
> As I got older I sort of dismissed them as antithetical to creativity Yah, there's a lot of criticism of the "directions". My kids build big elaborate modular buildings from the directions... and then leave them as-is... And then try to build equivalently complicated things from loose bricks (sometimes copying a little bit of a mechanism or a trick they've seen elsewhere). I think the instructions are great. They just shouldn't be the be-all, end-all.
- disconcision 4y ago
  yup. the blocks themselves give you the first-order building components, and then the instructions give you an increasingly library of higher-order building components to work with... like reading code
samstave 4y ago
AS AN ASIDE; When I was a young'n I had a lot of legos - but my granfather was a nuclear engineer with GE... (helped build Hanford) There was a technical design/drafting book in his book case I picked up where I was around ~6 or so... I drew every single diagram/ortho/iso whatever in that book. Multiple times. When I got into middle school, I was opted to elect a highschool class, and I chose technical drafting... When I went to class, the textbook was the same one from my grandfathers bookcase - I sailed through that, and then... This was what got me into design and architecture and CAD... I then setup the first cad network in my highschool, while I was still in 7th grade. I was in VOCAD by 9th, and came in second in the CAD Olympics in 1993 (I was the fastest ever to finish the test in 30 minutes, on a 3 hour deadline.) THE ENTIRE POINT BEING ; Get kids in front of technical design/docs/whatever is complex as EARLY as possible. There is no loss in teaching a toddler calculus. They may wind up being a flamboyant broadway star, but at least they can maths.
- amelius 4y ago
  > They may wind up being a flamboyant broadway star, but at least they can maths. What fun is it to be a broadway star when you know maths?
  - asadlionpk 4y ago
    You can calculate probability of success as a broadway star.
  - beambot 4y ago
    Understanding that the world and your interests may not be one dimensional... ;)
  - samstave 4y ago
    AAAAAND TWO AND THREE AND FOUR AND FIVE AND SIX AND SASHE AND TURN THAT NOMIAL INTO A PRIME AND FIVE SIX SEVEN EIGHT! HANDS LADIES!!! LET ME SEE YOUR JAZZ HANDS TIMES THE SQUARE ROOT OF SEVEN DIVIDED BY THE OUTPUT OF THE GLITTER FACTORY!!! Sounds like it could be fun.
  - mrguyorama 4y ago
paulryanrogers 4y ago
More so that they do it with few words and mostly images and icons.

intrasight 4y ago

Brings back an old memory. I took CMU's AI class in 1986. Most CMU classes were really two classes - one about the lectures and one about learning to hack. I did my project for this class in Lisp and wrote an engine to solve Lego assembly problems. While testing, I gave it a problem to solve, and it came up with a better solution than mine. Was my first AI eureka moment.

LionTamer 4y ago
That sounds really interesting. It was awhile ago so you may not remember, but how did you “operationalize” the problem? I wonder how that kind of problem / task is structured.
- intrasight 4y ago
  The problem was structured with two inputs. 1. the available Lego pieces. 2. the structure to be built. At least that's what I remember.

upupandup 4y ago

a bit off topic, but is there like a more affordable lego like alternative? it really disgusts me how expensive lego and litigous they are. so many Americans live in poverty who would love to get their kids to play with lego but can't because its ridiculously expensive.

SequoiaHope 4y ago
I’m not sure how they compare in price but I really enjoyed erector set and k’nex when I was a kid 30 years ago. And now I’m a robotics engineer! Also, an Ender 3 3D printer provides a lot of similar construction skills and learning (with the printed objects) for the price of one high end Lego set.
- londons_explore 4y ago
  +1 to buying a child a 3d printer. There are okayish ones for about $140 now. About age 6-8 seems like a good time. Set them up with an easy to use bit of cad software and show them how to 3d print a cube... Then let them 3d print whatever they want to make. 3d printers have hot parts - let them burn their fingers once or twice and they'll soon learn which bits not to touch. After that, a 3d printer is pretty safe. Test it first, but generally the motors aren't strong enough to crush a finger placed in the wrong spot. The only downside is many 3d prints take 1 hour + to print, which is pretty frustrating for a child. You need another activity to entertain them during printing breaks.
  - TaylorAlexander 4y ago
    I have no kids but I definitely found that waiting for the print was tough! Often what I do now is I get all my design work done at night. have fun getting design features sorted, or searching online libraries for printable parts, then get everything configured and start the print before bed. Watch the first layer or two to make sure it's gonna work, and then go to sleep while its printing. I wake up and the print is done! idk if this would work for kids but it works for me! it also means that prints which are 8-10 hours don't feel long, because I start it an hour before bed time, then as I am waking up and doing my morning routine the print is on its last few bits!

rkagerer 4y ago

This is begging for someone to make a robot using MEPNet that can assemble any kit for which you show it the instructions. Bonus points if it's made out of Lego and can build itself.

mech422 4y ago

I'd be more impressed if they can figure out Ikea directions :-P

pchristensen 4y ago
Try assembling furniture from another maker (e.g. anything from Target, Home Depot, etc) and you'll be thirsting for the clarity, completeness, and the large, legible print of Ikea instructions.
- jiggunjer 4y ago
  Step 1: build half of the bed. Step 2: build the other half.
sschueller 4y ago
I actually find ikea instructions quite clear and reasonable. I don't know why so many people have a hard time with it. Maybe spacial awareness is not something you can learn easily.
- stevenjgarner 4y ago
  I get a real B&W experience with Ikea - it would be interesting to have a technical writing expert compare the Ikea instructions and the Lego instructions. Lego seems to generally be accepted as more universally simple, and yet Ikea products frequently have fewer parts and steps.
  - breuleux 4y ago
    I find both quite clear. I think the main difference is that it's a lot more frustrating if you mess up a step in Ikea instructions than in a Lego build. So if you're bad at spatial reasoning or following instructions, you can still have fun with Lego, you'll just need to redo some steps sometimes and you might not even remember it, but the same kind of mistake with Ikea will throw you in a fit of rage.

dekhn 4y ago

I developed an AI to do this- well, I guess it wasn't artificial. I had a kid, and after showing him a few times (and correcting some mistakes due to the instructions sometimes being... as sparse as technically possible....) he had no problem. Now the issue is getting rid of all the 5-year-old Hogwarts Castles, etc...

javajosh 4y ago
Feel free to break them down and ship them to me, or some other parent of a young kid. Those sets are expensive!

O__________O 4y ago

Official Research Page: https://cs.stanford.edu/~rcwang/projects/lego_manual/ https://cs.stanford.edu/~rcwang/projects/lego_manual/ - which includes the paper, code and video.

eatsyourtacos 4y ago

I wonder what they taught the AI first to make enough money to be able to afford legos?

russellbeattie 4y ago

There is an app that's been linked on HN a few times called Brickit. It analyzes a picture you took of a pile of Legos, identifies pieces with object recognition, then uses AR to highlight exactly which pieces to grab to build a model. If there's such thing as a beautiful app, this is it. https://brickit.app/ https://brickit.app/

jiggunjer 4y ago
Don't people get their kids Lego to get them away from phones?

dane-pgp 4y ago

Interestingly, people were predicting[0] in 2018 that the robotic equivalent of this task would be solved by 2023: > The team also predicts that by the year 2023, a machine will be able to “physically assemble any Lego set given the pieces and instructions, using non-specialized robotics hardware.” The paper cites University of California Berkeley research that showed a machine joining two Lego pieces together in 2016 through reinforcement learning. [0] https://www.inverse.com/article/44660-lego-building-ai-robots https://www.inverse.com/article/44660-lego-building-ai-robot...

peterlk 4y ago

Side note about MITs marketing... The original paper [0] was published by 5 authors. 3 from Stanford, 1 from MIT, and 1 from Autodesk. I find MITs aggressive marketing pretty off-putting. Any discovery related to MIT _always_ has MIT in the headline, even if their involvement was secondary. It's annoying and makes me feel like MIT cares more about optics than good research. If I'm not alone in this belief, MIT will erode their brand by doing so. They certainly have for me. [0] https://cs.stanford.edu/~rcwang/projects/lego_manual/ https://cs.stanford.edu/~rcwang/projects/lego_manual/

RcouF1uZ4gsC 4y ago
> If I'm not alone in this belief, MIT will erode their brand by doing so. They certainly have for me. They already eroded their brand when they provided cover for Jeffrey Epstein.