7 ms·
Translating a Visual Lego Manual to a Machine-Executable Plan
- triggercut 4y agoWould be interesting to see a solution of the reverse process i.e. given an image or definition of a final model, create a bill of materials and (within various bounds) output step-by-step instructions to create it.
- sporkl 4y agoI think LEGO used to have a publicly-available CAD which could give the building steps, given a model.
- pclmulqdq 4y agoLego did have this, and I think they still do, but the building instructions it created were fairly formulaic, layer-by-layer instructions. The manuals you get with a normal Lego set are much better.
- colordrops 4y agoConnect this to DALL-E, and instaneously create new Lego sets with text prompts.
- usrusr 4y agoSo that's how the world ends, a computer going full paperclip maximizer trying to build Lego sheep that are in some way "electric"
- laserbeam 4y agoThis is awesome and cool! It's hard to say how well it would translate to other kinds of manuals. I say this just knowing how exquisitely high quality the design is for lego manuals, and how much effort went into ensuring 3yr olds can accuracy understand them.
- nuccy 4y agoIt is interesting how it is done by Lego themselves. Given the quality and accuracy of the manuals I always assumed that they are machine generated.
- proto_lambda 4y agoI remember them offering a software (or maybe a webpage, though it would've likely been a Java of Flash applet at the time) that allowed users to create official-looking manuals for their own creations by interactively specifying the build steps.
- tencentshill 4y ago
- _trampeltier 4y agoLego manuals exist much longer than computer could do the task. I mean manuals from the 60s or 70s had to be like hand drawn but still look the same as today.
- nuccy 4y agoSure. I never had a chance to assemble Lego from those days, but from unboxing/assembling videos [1] it seems that the manuals were either hand drawn or photos, so definitely it was a manual work. 1. https://youtu.be/xuJ2OulhC1M?t=564 https://youtu.be/xuJ2OulhC1M?t=564
- amelius 4y agoThe article is not about generating manuals. It is about reading them.
- rob74 4y agoNow they just need a robot that can grab the required parts out of a box full of Legos and execute the plan. But I guess that's left for the next paper...
- wwilim 4y ago"Two more papers down the line..." - dr Karoly Zsolnai-Feher
- msl 4y agoJacques Mattheij has got the first part covered (kind of): https://jacquesmattheij.com/sorting-two-metric-tons-of-lego/ https://jacquesmattheij.com/sorting-two-metric-tons-of-lego/ https://jacquesmattheij.com/sorting-lego-many-questions-and-this-is-what-the-result-looks-like/ https://jacquesmattheij.com/sorting-lego-many-questions-and-... for a follow up and more links.
- seanmcdirmid 4y agoMore importantly, a robot that can pick out required parts from a bunch of sets mixed all together. I would so kill for something like that.
- euroderf 4y agoAnd then, progressing upstream, the next version will spring into a Lego store, grab the required box(es), and sprint back out again.
- YeGoblynQueenne 4y agoGiven that this approach needs a manual written by somebody who already understands how to assemble a lego kit, this is not practically useful. The person who wrote the manual can also specify a set of machine-readable instructions (for example, in a programming language). In short, this looks cool, but it's pointless.
- soraki_soladead 4y agoMany more people can put together a manual of step-by-step pictures than can program.
- YeGoblynQueenne 4y agoYes, but if the step-by-step procedure is known, it can be programmed manually and does not need to be machine-learned from examples. It doesn't have to be the same person who writes the manual and does the coding. An approach like the one in the article above would be useful if it could derive a procedure we don't know how to code manually.
- outsidetheparty 4y agoCouldn’t possibly disagree more; that’s akin to saying that feature identification is not useful, because if we really wanted to we could just deliver a list of identifiable features along with every photograph. The point of this, obviously, is not to make it easier for machines to construct LEGO sets. It’s just a really clever problem space to do research into improving machine vision and image interpretation: it’s complicated enough to run into meaningful challenges, but constrained enough to make progress achievable. (The “Minecraft” dataset mentioned sounds like a simpler baby-steps version of the same, since there’s a much more limited set of possible shapes and connections to work with.) Plus they get to be the fun lab with all the LEGOs in it
- YeGoblynQueenne 4y ago>> Couldn’t possibly disagree more; that’s akin to saying that feature identification is not useful, because if we really wanted to we could just deliver a list of identifiable features along with every photograph. These are not arbitrary photographs but images of legos. The features needed are few and finite and there's no need for machine vision to find them. This is pointless and has no real-world usefulness, other than to look like "the fun lab with all the legos", like you say.