8 ms·
LegoGPT: Generating Physically Stable and Buildable Lego
- W0lfEagle 1y agoGreat. Please do cabinets next. Constrain to some specified material such as 2.5m by 1.25m 18mm ply. Iterate designs by text and output the model, cutlist and assembly instructions. Simple right?
- 9dev 1y agoWhen I was a kid, I proudly exclaimed I wanted to become a professional lego builder. Not in my wildest dreams would I have assumed how close to that career path I could have come.
- jader201 1y agoThere’s a bug on the page (on iPhone, at least) once you scroll to the gifs that it starts to auto load them without doing anything, making it really hard to navigate anywhere at that point.
- Aeolun 1y agoWhen will people finally learn to never autoplay.
- vachina 1y agoAutoplay is fine, it’s Safari opting to autoplay in FULLSCREEN. Firefox et. al. play them in the respective video containers.
- deleted 1y ago[deleted]
- pragmatick 1y agoThe opposite for me on Firefox Desktop - I didn't realize they were gifs and wondered what the pictures were supposed to tell me.
- MangoTec 1y agothis should be fixable with `playsinline` on the video element: https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/video#playsinline https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/... annoying that this is the default behaviour on iOS though
- gilgoomesh 1y agoIt's hilarious watching $50,000 worth of robots take so long to assemble a couple dollars worth of Lego. It's like peering into the old folks home for robots.
- FirmwareBurner 1y agoThat should tell you why stuff is still hand assembled in Asia instead of by robots in the west.
- smikhanov 1y agoAlso why it’s OK to stop worrying about our future robotic (or AI) overlords.
- FirmwareBurner 1y agoThose are already an issue. AI is a bigger threat to cognitive tasks than to physical ones. Skynet isn't goanna attack you with Terminators wielding a "phased plasma rifle in the 40W range", but will be auto-rejecting your job application, your health insurance claims, your credit score and brain washing your relatives on social media.
- smikhanov 1y agoAbsolutely, that’s without any doubt. There’s a difference though. The “cool” Terminator Skynet pursues its own goals, and wasn’t programmed by humans to kill. The “boring” insurance-rejecting Skynet is explicitly programmed to reject insurance claims by other humans, unfortunately. So still, no need to worry about our AI overlords, worry about people running the AI systems.
- imtringued 1y ago> AI is a bigger threat to cognitive tasks than to physical ones. I don't see how you could possibly think this is true. Physical automation is easier to scale since you only need to solve a single problem instance and then just keep applying it on a bigger scale.
- benob 1y agoIt looks like it is an extension of 3d model generation techniques.
- yathaid 1y agoThis is super cool! The GIFs showing the object being built are just yummy; I have no other way to describe it. If anyone else was searching for the dataset, it is at https://huggingface.co/datasets/AvaLovelace/StableText2Lego https://huggingface.co/datasets/AvaLovelace/StableText2Lego It contains " contains 47,000+ different LEGO structures, covering 28,000+ unique 3D objects from 21 common object categories of the ShapeNetCore dataset". Local inference instructions are over at their github page - https://github.com/AvaLovelace1/LegoGPT/?tab=readme-ov-file https://github.com/AvaLovelace1/LegoGPT/?tab=readme-ov-file
- carstenhag 1y agoHave the authors never heard of Lego being one of the companies that are super strict about their trademark? They file takedown notices etc on every project they see. Even if the stone design has the little thingies on top/bottom...
- sschueller 1y agoThis is probably going to get a letter from LEGO's lawyers. If you want to be safe do not use the word LEGO. Use Bricks or in German "Klemmbausteine". Many people have had to deal with LEGO's lawyers and it ain't pretty.
- edoceo 1y agoWhy are they like Nintendo when they could be like Sega? Embrace your community where they are.
- raverbashing 1y ago(Not saying it's related) But, which one of those are still running?
- 71bw 1y agoBoth?
- vanderZwan 1y agoBoth are. Sega just lost a console war a few decades ago and decided not to pursue that any more.
- Cthulhu_ 1y agoSega is generating more than $1.5 billion a year, they're fine.
- Philpax 1y agoI'd go as far as to say that Sega's embracing of their fans is a big part of why they're still around: https://en.wikipedia.org/wiki/Sonic_Mania#Development https://en.wikipedia.org/wiki/Sonic_Mania#Development
- Freak_NL 1y agoTrademark law leaves no space for that. The Lego Group has to actively defend their trademark. That means a name like LegoGPT is really on the obvious end of 'don't do that'.
- nurettin 1y agoThis could prove useful for minecraft bot scene
- davidthewatson 1y agoIndeed. I thought blocks world stuff would be amazing for early childhood education. I'm guessing some labs are already there since minecraft supports user-programmable models for years though I dunno the details. I'd be happy to learn if anybody knows of their evolution since the rise of AI.
- kilimounjaro 1y agoDoesn’t seem to add much to just converting a 3d model into voxels and therefore bricks. Using bricks other than 2x2 and 2x4 blocks creatively to make interesting things is really important, i’m not sure what type if algorithm would best auto generate beautiful MOCs however? Was thinking of doing a $50000 kaggle comp for this, what do others think?
- yunusabd 1y agoCool project, but judging from the videos, it looks like some of them can't actually be built using those instructions. E.g. "A backless bench with armrest" would require some bricks to float in the air with no support while you're assembling the rest.
- biofox 1y agoThe design is sound, just not the order of assembly shown. For the bench, the lower bricks are suspended by the upper ones, so would need to be assembled separately before connecting to the legs.
- yunusabd 1y agoTrue, easy for a human, not so easy for a robot to go through those extra steps. I wonder if they made it work with the robots, because in the video they only show the robots building from the bottom up.
- Etherlord87 1y agoStill, I think you wouldn't find such a design in actual Lego set, because such floating bricks make the construction weak.
- haberman 1y ago> To improve the stability of the resulting designs, we employ an efficient validity check and physics-aware rollback during autoregressive inference, which prunes infeasible token predictions using physics laws and assembly constraints. I'm far from an AI expert, but I've long felt that this is one of the most interesting ways to use AI: to generate and optimize possibilities within a set of domain-specific constraints that are programmed manually. For example, imagine an AI that is designed to optimize traffic light patterns. You want a hard constraint that no intersection gives a combination of green lights that could cause collisions. But within that set of constraints, which you could manually specify, the AI could go wild trying whatever ideas it can come up with. At that point, the interesting work is deciding how to design the problem space and the set of constraints. In this case it's a set of lego bricks and how they can be built (and be stable).
- zelos 1y agoYou'd probably use some kind of MILP or CLP based model for that kind of thing, wouldn't you? The constraints define the search space and the solver algorithm then explores it.
- Narew 1y agoI haven't read how they apply the constraint. But there is similar stuff when you force llm to generate structured output like Json format. llama.cpp allow to match a custom grammar for example.
- benterix 1y ago> to generate and optimize possibilities within a set of domain-specific constraints Well, yes, we've been doing this for several decades, many people call it metaheuristics. There is a wide array of algorithms in there. An excellent and light intro can be found here: https://cs.gmu.edu/~sean/book/metaheuristics/ https://cs.gmu.edu/~sean/book/metaheuristics/
- eurekin 1y agoMetaheurestics? I always thought it's similar to "I don't know how many neurons to put in the hidden layer... and I also don't know how many hidden layers I need, so, let's make it a part of the optimisation problem to find out on it's own".
- Traubenfuchs 1y agoIs this aligned, safeguarded, censored? Can it produce an ample bossom made of lego? And indecent protrusion? Weapons?
- b3lvedere 1y agoWeapons has been kinda done earlier: https://nostarch.com/flego https://nostarch.com/flego https://nostarch.com/legoguns.htm https://nostarch.com/legoguns.htm
- foobahhhhh 1y ago[flagged]
- xeyownt 1y agoSo, besides training a LLM to generate build instructions for lego model, they have robots to assemble these models, and they applied 3D texture on 3D generated model (what for?). Sometimes the amount of money and energy that are spent in "recreation" projects just amazes me.
- thenaturalist 1y agoYou do realize that a system like LEGO is just an extremely efficient and cheap proof of concept with a proxy material (LEGO) for later real life applications of building X from standardized components Y right? This is interesting and seemingly quite applicable base research and we move forward by being curious.
- vladde 1y agoLooks amazing, with the arms building the piece! However, the model "A high-backed chair" has some floating pieces in the middle of the seat, that are fastened from above. Can these robots handle building these?
- psiops 1y agoI noticed that "a basic sofa" involves some placing some floating bricks if built in the order of the animation. It hints at the way this model generates the designs. The automated assembly of generated LEGO structures using robots would have serious trouble creating these designs I reckon.
- sdoering 1y agoI came here to say that. I immediately thought: Wow, this works in the assembled version, but not the way the assembly is being animated. You would need to first build the base sofa layer from two levels so that the upper layer keeps the lower layer bricks in place. Only afterwards could it be put onto the legs.
- paulluuk 1y agoIndeed, I would be very curious to see how their robots would actually build that sofa. Although the robots aren't really part of the model of course, they're just a little extra.
- josefx 1y agoDoes it have a sense of scale? Can it make a matching bookcase, table and chair or are they all going to end up with a random size?
- oaiey 1y agoReal challenge are not LEGO System pieces but LEGO Technic pieces where you do not have to build layers bottom-up but everything inside out
- londons_explore 1y ago> voxelizing it onto a grid and applying legolization I guess I learned a word today...
- stevage 1y agoThis does not seem like a very impressive result. It's using such a small set of bricks and the results don't really look much like the intended thing. It feels like a hand-crafted algorithm would get a much better result.
- otabdeveloper4 1y agoWhat we need is an AI where you feed it some photos of your pile of bricks and it generates you instructions based on the bricks you have. (Totally feasible with today's technology, but you'll need to train your own specialized models.)
- stevage 1y agohttps://au.lifehacker.com/toys-board-games/50948/news/scan-your-lego-with-this-app-to-figure-out-what-to-build https://au.lifehacker.com/toys-board-games/50948/news/scan-y...
- amelius 1y agoWhat I'd be interested in most is a robot that can assemble a model from a pile of bricks/parts.
- dspillett 1y agoThere already exists an app that will, from photos of your pile, pick out models you can make from a large library of existing models. Though IIRC that has been around long enough that it isn't quite using what people are currently calling AI (instead using older ML techniques for brick identification, and a basic DB search to pick out the valid plans for the resulting list of bricks).
- Spellman 1y agoAlready exists, put out by LEGO https://brickit.app/ https://brickit.app/
- Loughla 1y agoMy experience with that is that it gives you unbelievably simplistic builds, not the complicated things you might be imagining.
- RaSoJo 1y agoI don’t need automation to build LEGO sets — that’s the fun part, and I want to do it myself. What I need is automation after the build: to clean up, sort the bricks by color and shape, and store them properly. I just wish scientists would start by solving problems that actually exist in the real world. There’s real value — and real money — in that.
- KurSix 1y agoYou're totally right: sometimes the real innovation isn't in making the fun parts easier, it's in making the boring parts disappear
- lee-rhapsody 1y agoThe issue with solving real-world problems is that it distracts from publishing, which is all scientists are taught to care about.
- joeyparsons 1y agocame here to write this comment. clean up and sorting is the top problem in lego land.
- shagie 1y agoSorting 2 Metric Tons of Lego https://jacquesmattheij.com/sorting-two-metric-tons-of-lego/ https://jacquesmattheij.com/sorting-two-metric-tons-of-lego/ (2017) https://news.ycombinator.com/item?id=14226889 https://news.ycombinator.com/item?id=14226889 World’s first AI-powered LEGO sorting machine built with 10,000 LEGO bricks https://www.brothers-brick.com/2019/12/06/this-fan-created-the-ultimate-lego-brick-sorting-machine-powered-by-ai-exclusive-feature/ https://www.brothers-brick.com/2019/12/06/this-fan-created-t... (2019) https://news.ycombinator.com/item?id=21741834 https://news.ycombinator.com/item?id=21741834
- z3t4 1y agoThe results are a bit underwhelming, considering what I'm used to see in image generation, world generation in games, etc.
- dwighttk 1y agoQuit trying to read the article after the 15th video went to full screen and had to be dismissed hitting the tiny x in the upper left… 3 more interfered with me trying to go back to this page
- beklein 1y agoYou can also grab the paper as a PDF here: https://arxiv.org/pdf/2505.05469 https://arxiv.org/pdf/2505.05469. Keep in mind that these sites are run by AI researchers, not dedicated UX teams at major tech companies—so the interface can feel a bit rough around the edges. That said, your critique is still valid; it’s just fair to cut them a little slack given their priorities.
- deleted 1y ago[deleted]
- belter 1y agoCan I have IKEA-GPT ? This weekend if possible?
- davidthewatson 1y agoIndeed. I'm guessing legal is the only reason we don't have 3d-printed ikea. Raymond Loewy FTW. But then, we'd have garages full of bespoke n-of-1 junk instead of mass-made LLM (liminal-labor-made) junk.
- necovek 1y agoI love this: it'd be great if it also worked to give age-appropriate designs as well (eg. 5 yo has less patience and ability than an 8 yo).
- aayushmaan45 1y ago[flagged]
- aayushmaan45 1y ago[flagged]
- soared 1y agoThe high backed chair gif example is interesting - the way it’s animated it would completely fall apart and be unstable. But if you built it in reverse, it would work fine. But it also shows the weirdness of the solution - in places where larger bricks make sense, multiple smaller bricks are used instead. In a section where a 2x6 should be repeated, in on instance of the repetition it uses tow 1x6s. It’s weird. Cool idea.
- m3kw9 1y agoThese legos doesn't look very intricate even by 200 pc box standards.
- 6stringmerc 1y agoIs this use of the trademark approved by the LEGO owners by any chance? I skimmed it and there didn’t seem to be an indication this has been endorsed by the company. It kind of makes me suspicious of the integrity of Carnegie Mellon if they will allow trademark infringement of this type because, well, it does make me feel like I can shit in a bag and call it a Carnegie Mellon Socking Stuffer without consequence.
- curtisszmania 1y ago[dead]
- andyjohnson0 1y agoIts been a long day and and I can think of nothing to add other than that those blue robot arms are both calming and visually pleasant.
- serial_dev 1y agoIt would make sense for a car to have proper LEGO wheels.
- sargstuff 1y ago?? 'type'-less / block structured language/design ??