7 ms·

Text-to-CAD: Risks and Opportunities

66 points by danboarder 3y ago

syntaxing 3y ago
I don’t do mechanical design anymore but did prior for about a decade. A lot of the examples the author has is text to 3D model which is not the same as text to CAD. Think of AutoCAD (even though it’s 2D CAD, the example still stands). You would design a blueprint of your house with it but you won’t make an ice cream sundae model. You would use Maya for that.
- brudgers 3y ago
  AutoCad has been 3d for more than thirty years. R10 had simple surfaces. R11 had Advanced Modeling Extension for solid modeling. Rendering was added in r12.
cozzyd 3y ago
openscAId?
onthecanposting 3y ago
I'm not that optimistic about an algorithm producing useful results from a text-based description given the solution space is any possible combination of any number of shapes. The data structure for 3D objects also depends on your software. Where CAD matters professionally, there usually are strict requirements or business rules that would rule out an "average .OBJ files that had these tags" Now, I AM optimistic about algorithms solving for useful relationships between pre-defined objects, like routing conduit through a building without collisions or optimizing a lumber cut-list. Finch and Hypar are interesting small companies in this space.
- techdragon 3y ago
  Having an expert (or other more specialised software) optimise an otherwise satisfactory 3d model is going to be vastly more simple a process than what can be a painstaking process of getting the “original vision” for a shape into a digital form. I’m better with CAD style workflows than I am with Freeform modelling, but I’m also not an experienced enough user of any particular CAD program that I don’t spend ages frustratingly trying to understand where to go to edit the correct part of the chain of operations that built up a particular fillet, chamfer, or sweep, every program is a little different… just different enough to throw me off since I don’t use them often enough to really get over that learning threshold, for instance I’ve used Onshape one time in the last 12 months… spent an hour trying to work something out… and then 5 minutes actually making my edit… If I could have opened up the design in 2d blueprints form, and put in some kind of multi modal query to the effect of “these are my blueprints, can you convert them to (CAD program of choice), and while you are converting them, increase cross section CS1 vertically to from 8mm to 10mm keeping everything else fixed” … would save me a lot of time making adjustments to peoples 3d models for printing. It can be really annoying to fuck around as much as is sometimes necessary due to individual cad workflow preferences just to make conceptually simple edits like “this hanging pot, but stretched 50% taller, and with all the holes/cutouts kept the same size please” The ML models might not be able to do a perfect curve for a gorgeous arcing buttress or a complex bolt hole pattern for attaching multiple elements to a primary support structure… but it should be able to do a lot of the sorts of work that people use OpenSCAD for.
whelp_24 3y ago
The author really does not like image generators.
carterschonwald 3y ago
So I’ve been collaborating with some mechanical engineering friends over the past year, and one experiment we did was exploring ways to make cad better. One conclusion we came to is that large language models do not understand geometry, in a pretty fundamental way! They do understand pair wise relationships that are more topological in character. But that’s not quite the same thing. The blocker on the text to concept art to 3D model to cad route is the meshes and geometry you’ll generate won’t have the easy adjusting and parameterization manipulations you can take for granted in human authored cad! That’s also ignoring developing a robust geometry kernel that would live underneath all this!
acyou 3y ago
Tons of great text and 2D image training data. If the same were true for 3D, the problem would be "easy". Not anticipating any great source 3D model training data will be made available anytime soon.
digdugdirk 3y ago
The world doesn't need Text-to-CAD. The world needs a fully capable open source parametric 3D geometric CAD kernel. Solidworks, Creo, AutoCAD, Fusion, etc., can all take their bug ridden unoptimized single threaded rent-seeking monstrosities and stick em where the sun don't shine. Seriously - if anyone wants to create an absolutely world-changing piece of software, start working on a new CAD kernel that takes the last 50 years of computer science advances into account, because none of the entrenched industry standards have done so. Don't worry about having to provide customer service, because none of the entrenched industry standards worry about that either. And no - while openCascade and solvespace are impressive, they aren't fully capable, nor do they start from a modern foundation.
- dymk 3y ago
  https://www.fornjot.app/ https://www.fornjot.app/ is an attempt, and also good at showing what a large undertaking such a goal is
- auggierose 3y ago
  So what's so difficult about such a kernel? What does a modern foundation look like?
  - dymk 3y ago
    The math is quite difficult to do right, and there's a billion corner cases to make a kernel useful for real world designs. Take a fillet: It needs to handle inside corners, outside corners, compound angles coming in from arbitrary numbers of directions, it probably needs the ability to vary along its distance, create more geometry when adjacent faces don't leave enough room, etc etc. That's just the start of a single feature type. Now you need a bunch more feature types, and they all need to interact well with each other. The kernel also needs some way of solving the topological naming problem to be useful (FreeCAD might get a basic version of this after a decade(?) of work). It's probably tantamount to writing a modern-day browser in terms of complexity.
    - auggierose 3y ago
Renaud 3y ago
To me, the more interesting AI/CAD capabilities integration will be when specialist AIs can analyse models and understand real-world constraints. Imagine an AI helping an architect ensure theirs drawings are compliant with local code ordinances, and it can produce some of the paperwork itself. Imagine an AI that understands the manufacturing processes well enough to guide you or give useful advice about how to manufacture or modify your parts so they become easier and cheaper to build. Imagine and AI that can know more about an electronic or mechanical project and can get outputs from various simulation tools to advise you on your design compliance with regulation, or that would recognise weak design choices pulling from a knowledge base of part failure or other real-world constraints. It could propel computer-aided design in ways we can't imagine today, but this integration will probably be hard and not be just text-based.
- lionkor 3y ago
  I dont think you're looking for an AI, some of those are very much possible without AI (and then theyre even - gasp - testable!)
- Kosirich 3y ago
  Try a combination of taking screenshots and GPT4 image input. You would be surprised how good it is in analyzing part even without prompting. This is one of the things I'm investigating if it makes sense putting effort in. Just basic company process/rules following (for our own internal products). Anything else, we are talking about computational design + AI for interpretability. I haven't checked all the examples in the link posted, but I know AutodeskAI Lab has some seriously impressive papers out. (code too)
bredren 3y ago
I’ve been working on light weight text to cad as part of my participation in Paddle’s AI Launchpad accelerator. I am an amateur woodworker and wanted easier ways to quickly prototype ideas. The sweet spot for me is more accurate measurements and better drawings than my pen and paper but without the overhead of firing up Fusion 360 and trying to lay out the 2D then 3D process. Neither of the above is great for iteratively exploring designs either. My last project was a custom drill press workbench, and I did the 3D in sketchup and Fusion to get a feel for both tools popular with woodworker hobbyists. These types of designs are often sold for a few bucks with the project assembly videos posted on YouTube. I did my initial testing of this using iterative prompts to OpenAI models asking them to refine the design of an outdoor wooden bench with dimensions appropriate for a toddler. I had some live edge donor wood and wanted it to comply with the thickness of the materials as input. I was able to prove to myself it could be done with generated scripts for the blender-API. I set aim at single page that can record spoken audio, perform STT, process it into valid blender Python, export a .glb and display it on the same page. Making a great demo is a lot of integration work and a lot of LLM programming, pre and post processing, system context refinement etc. But it’s pretty awesome. In my experience generating for Fusion is dicier than blender, but I suspect with specialized model training and a bunch of dark art LLM incantations this could become a prosumer tool, and possibly speed along professional work as described in the blog. So far this is not stuff w complex mechanics or fancy hinges, so it might not meet the threshold of CAD for some. But there are a lot of folks who want “cad like” experiences without having to muck w the tools. I’d love feedback if anyone is interested in checking it out. Demo day is toward end of this month, I can be reached at the email in my HN profile.
- longtimelistnr 3y ago
  To your point about iterative design - something I learned in school for industrial design is 3D modeling will never be used for iterative design. The closest I’ve seen is VR modeling. But even then, the commitment is too much while instead I can sketch on a napkin instantly. Move to 3D only when I’m satisfied with what I want to construct.
deleted 3y ago
[deleted]
rlt 3y ago
Deepfakes, weapons, racist content, copyright infringement, and "digital colonialism"? The author is really grasping at straws for downsides of using generative AI, while downplaying the benefits. Sure it may help others produce lots of low quality content you don't enjoy, but it will also help you generate content you really enjoy. Especially things you can then manifest in the physical world. If anything I think it will help elevate more obscure styles, as it creates and effectively infinite number of artists that can produce such styles.
llogiq 3y ago
There is so much wrong with this article. Throw a little bit of ML pixie dust on everything for more hype? Check. Compare wildly different things as if they were the same? Double check. digdugdirk has the right idea, and AFAIR, there is some work on that front (https://www.fornjot.app/ https://www.fornjot.app/). Also the Fiat 500 goes 100km on about 4l of gas, while the Ford F150 uses 7l. No clue where the author gets the idea that the Fiat would get worse mileage, perhaps he's dividing by weight? The rest, I don't even.
HenryBemis 3y ago
Looking at the images of comparisons (i.e. pig wearing a backpack) I am dreaming of the moment I can 'talk' to an AI engine, and ask it "design XYZ with LEGO bricks, then print me the list so I can order them, and instructions to build it". (yeah I hate it that Star Trek universe is not signing a deal with Lego)(I would much rather have a Enterprise than any Star Wars item)(https://xkcd.com/1563/ https://xkcd.com/1563/)
gene-h 3y ago
I wouldn't be so worried about LLMs making guns. The SketchIT project found that it's very difficult to describe mechanical device to other humans using just text, images are needed too[0]. I'd also worry about the gun produced being structurally sound, how can one be so sure that the gun barrel hasn't been hallucinated to be too thin? Guns and other mechanical devices don't exist alone. A gun must interface with a bullet, a part of an aircraft must interface with other parts. So CAD AI must be able to understand the geometric context of the parts it is making. That being said, I think AI will soon be capable of making mechanical devices. There has been some improvement in physical reasoning benchmarks like PHYRE[1]. Understanding physical reasoning and how multiple objects move with respect to each other is important in the synthesis of new mechanical devices. SketchIt[0] demonstrated that by making reduced 2DoF description of how pairs of objects in a device may move with respect to each other, it's possible to synthesize a new device which performs the same function. Solving PHYRE problems requires reasoning with larger degrees of freedom. The first example on the homepage has something like 5 objects which each have 3 positional DoF (translation and rotation). Even reasoning with 3DoF is quite difficult for approaches like those used in SketchIT. Given that approaches like slotformer[2] already do somewhat well at solving these huge DoF problems, I don't think we're very far from AI being able to design complicated mechanical devices. [0]https://dspace.mit.edu/bitstream/handle/1721.1/6773/AITR-1573.pdf https://dspace.mit.edu/bitstream/handle/1721.1/6773/AITR-157... [1]https://phyre.ai https://phyre.ai [2]https://slotformer.github.io https://slotformer.github.io
kwhitefoot 3y ago
If they are lying, or merely ludicrously mistaken, about the Fiat 500 and the F-150 what credence can be placed in the rest?

66 points by danboarder 3y ago

I don’t do mechanical design anymore but did prior for about a decade. A lot of the examples the author has is text to 3D model which is not the same as text to CAD. Think of AutoCAD (even though it’s 2D CAD, the example still stands). You would design a blueprint of your house with it but you won’t make an ice cream sundae model. You would use Maya for that.

brudgers 3y ago
AutoCad has been 3d for more than thirty years. R10 had simple surfaces. R11 had Advanced Modeling Extension for solid modeling. Rendering was added in r12.

cozzyd 3y ago

openscAId?

onthecanposting 3y ago

I'm not that optimistic about an algorithm producing useful results from a text-based description given the solution space is any possible combination of any number of shapes. The data structure for 3D objects also depends on your software. Where CAD matters professionally, there usually are strict requirements or business rules that would rule out an "average .OBJ files that had these tags" Now, I AM optimistic about algorithms solving for useful relationships between pre-defined objects, like routing conduit through a building without collisions or optimizing a lumber cut-list. Finch and Hypar are interesting small companies in this space.

techdragon 3y ago
Having an expert (or other more specialised software) optimise an otherwise satisfactory 3d model is going to be vastly more simple a process than what can be a painstaking process of getting the “original vision” for a shape into a digital form. I’m better with CAD style workflows than I am with Freeform modelling, but I’m also not an experienced enough user of any particular CAD program that I don’t spend ages frustratingly trying to understand where to go to edit the correct part of the chain of operations that built up a particular fillet, chamfer, or sweep, every program is a little different… just different enough to throw me off since I don’t use them often enough to really get over that learning threshold, for instance I’ve used Onshape one time in the last 12 months… spent an hour trying to work something out… and then 5 minutes actually making my edit… If I could have opened up the design in 2d blueprints form, and put in some kind of multi modal query to the effect of “these are my blueprints, can you convert them to (CAD program of choice), and while you are converting them, increase cross section CS1 vertically to from 8mm to 10mm keeping everything else fixed” … would save me a lot of time making adjustments to peoples 3d models for printing. It can be really annoying to fuck around as much as is sometimes necessary due to individual cad workflow preferences just to make conceptually simple edits like “this hanging pot, but stretched 50% taller, and with all the holes/cutouts kept the same size please” The ML models might not be able to do a perfect curve for a gorgeous arcing buttress or a complex bolt hole pattern for attaching multiple elements to a primary support structure… but it should be able to do a lot of the sorts of work that people use OpenSCAD for.

whelp_24 3y ago

The author really does not like image generators.

carterschonwald 3y ago

So I’ve been collaborating with some mechanical engineering friends over the past year, and one experiment we did was exploring ways to make cad better. One conclusion we came to is that large language models do not understand geometry, in a pretty fundamental way! They do understand pair wise relationships that are more topological in character. But that’s not quite the same thing. The blocker on the text to concept art to 3D model to cad route is the meshes and geometry you’ll generate won’t have the easy adjusting and parameterization manipulations you can take for granted in human authored cad! That’s also ignoring developing a robust geometry kernel that would live underneath all this!

acyou 3y ago

Tons of great text and 2D image training data. If the same were true for 3D, the problem would be "easy". Not anticipating any great source 3D model training data will be made available anytime soon.

digdugdirk 3y ago

The world doesn't need Text-to-CAD. The world needs a fully capable open source parametric 3D geometric CAD kernel. Solidworks, Creo, AutoCAD, Fusion, etc., can all take their bug ridden unoptimized single threaded rent-seeking monstrosities and stick em where the sun don't shine. Seriously - if anyone wants to create an absolutely world-changing piece of software, start working on a new CAD kernel that takes the last 50 years of computer science advances into account, because none of the entrenched industry standards have done so. Don't worry about having to provide customer service, because none of the entrenched industry standards worry about that either. And no - while openCascade and solvespace are impressive, they aren't fully capable, nor do they start from a modern foundation.

dymk 3y ago
https://www.fornjot.app/ https://www.fornjot.app/ is an attempt, and also good at showing what a large undertaking such a goal is
auggierose 3y ago
So what's so difficult about such a kernel? What does a modern foundation look like?
- dymk 3y ago
  The math is quite difficult to do right, and there's a billion corner cases to make a kernel useful for real world designs. Take a fillet: It needs to handle inside corners, outside corners, compound angles coming in from arbitrary numbers of directions, it probably needs the ability to vary along its distance, create more geometry when adjacent faces don't leave enough room, etc etc. That's just the start of a single feature type. Now you need a bunch more feature types, and they all need to interact well with each other. The kernel also needs some way of solving the topological naming problem to be useful (FreeCAD might get a basic version of this after a decade(?) of work). It's probably tantamount to writing a modern-day browser in terms of complexity.
  - auggierose 3y ago

Renaud 3y ago

To me, the more interesting AI/CAD capabilities integration will be when specialist AIs can analyse models and understand real-world constraints. Imagine an AI helping an architect ensure theirs drawings are compliant with local code ordinances, and it can produce some of the paperwork itself. Imagine an AI that understands the manufacturing processes well enough to guide you or give useful advice about how to manufacture or modify your parts so they become easier and cheaper to build. Imagine and AI that can know more about an electronic or mechanical project and can get outputs from various simulation tools to advise you on your design compliance with regulation, or that would recognise weak design choices pulling from a knowledge base of part failure or other real-world constraints. It could propel computer-aided design in ways we can't imagine today, but this integration will probably be hard and not be just text-based.

lionkor 3y ago
I dont think you're looking for an AI, some of those are very much possible without AI (and then theyre even - gasp - testable!)
Kosirich 3y ago
Try a combination of taking screenshots and GPT4 image input. You would be surprised how good it is in analyzing part even without prompting. This is one of the things I'm investigating if it makes sense putting effort in. Just basic company process/rules following (for our own internal products). Anything else, we are talking about computational design + AI for interpretability. I haven't checked all the examples in the link posted, but I know AutodeskAI Lab has some seriously impressive papers out. (code too)

bredren 3y ago

I’ve been working on light weight text to cad as part of my participation in Paddle’s AI Launchpad accelerator. I am an amateur woodworker and wanted easier ways to quickly prototype ideas. The sweet spot for me is more accurate measurements and better drawings than my pen and paper but without the overhead of firing up Fusion 360 and trying to lay out the 2D then 3D process. Neither of the above is great for iteratively exploring designs either. My last project was a custom drill press workbench, and I did the 3D in sketchup and Fusion to get a feel for both tools popular with woodworker hobbyists. These types of designs are often sold for a few bucks with the project assembly videos posted on YouTube. I did my initial testing of this using iterative prompts to OpenAI models asking them to refine the design of an outdoor wooden bench with dimensions appropriate for a toddler. I had some live edge donor wood and wanted it to comply with the thickness of the materials as input. I was able to prove to myself it could be done with generated scripts for the blender-API. I set aim at single page that can record spoken audio, perform STT, process it into valid blender Python, export a .glb and display it on the same page. Making a great demo is a lot of integration work and a lot of LLM programming, pre and post processing, system context refinement etc. But it’s pretty awesome. In my experience generating for Fusion is dicier than blender, but I suspect with specialized model training and a bunch of dark art LLM incantations this could become a prosumer tool, and possibly speed along professional work as described in the blog. So far this is not stuff w complex mechanics or fancy hinges, so it might not meet the threshold of CAD for some. But there are a lot of folks who want “cad like” experiences without having to muck w the tools. I’d love feedback if anyone is interested in checking it out. Demo day is toward end of this month, I can be reached at the email in my HN profile.

longtimelistnr 3y ago
To your point about iterative design - something I learned in school for industrial design is 3D modeling will never be used for iterative design. The closest I’ve seen is VR modeling. But even then, the commitment is too much while instead I can sketch on a napkin instantly. Move to 3D only when I’m satisfied with what I want to construct.

deleted 3y ago

[deleted]

rlt 3y ago

Deepfakes, weapons, racist content, copyright infringement, and "digital colonialism"? The author is really grasping at straws for downsides of using generative AI, while downplaying the benefits. Sure it may help others produce lots of low quality content you don't enjoy, but it will also help you generate content you really enjoy. Especially things you can then manifest in the physical world. If anything I think it will help elevate more obscure styles, as it creates and effectively infinite number of artists that can produce such styles.

llogiq 3y ago

There is so much wrong with this article. Throw a little bit of ML pixie dust on everything for more hype? Check. Compare wildly different things as if they were the same? Double check. digdugdirk has the right idea, and AFAIR, there is some work on that front (https://www.fornjot.app/ https://www.fornjot.app/). Also the Fiat 500 goes 100km on about 4l of gas, while the Ford F150 uses 7l. No clue where the author gets the idea that the Fiat would get worse mileage, perhaps he's dividing by weight? The rest, I don't even.

HenryBemis 3y ago

Looking at the images of comparisons (i.e. pig wearing a backpack) I am dreaming of the moment I can 'talk' to an AI engine, and ask it "design XYZ with LEGO bricks, then print me the list so I can order them, and instructions to build it". (yeah I hate it that Star Trek universe is not signing a deal with Lego)(I would much rather have a Enterprise than any Star Wars item)(https://xkcd.com/1563/ https://xkcd.com/1563/)

gene-h 3y ago

I wouldn't be so worried about LLMs making guns. The SketchIT project found that it's very difficult to describe mechanical device to other humans using just text, images are needed too[0]. I'd also worry about the gun produced being structurally sound, how can one be so sure that the gun barrel hasn't been hallucinated to be too thin? Guns and other mechanical devices don't exist alone. A gun must interface with a bullet, a part of an aircraft must interface with other parts. So CAD AI must be able to understand the geometric context of the parts it is making. That being said, I think AI will soon be capable of making mechanical devices. There has been some improvement in physical reasoning benchmarks like PHYRE[1]. Understanding physical reasoning and how multiple objects move with respect to each other is important in the synthesis of new mechanical devices. SketchIt[0] demonstrated that by making reduced 2DoF description of how pairs of objects in a device may move with respect to each other, it's possible to synthesize a new device which performs the same function. Solving PHYRE problems requires reasoning with larger degrees of freedom. The first example on the homepage has something like 5 objects which each have 3 positional DoF (translation and rotation). Even reasoning with 3DoF is quite difficult for approaches like those used in SketchIT. Given that approaches like slotformer[2] already do somewhat well at solving these huge DoF problems, I don't think we're very far from AI being able to design complicated mechanical devices. [0]https://dspace.mit.edu/bitstream/handle/1721.1/6773/AITR-1573.pdf https://dspace.mit.edu/bitstream/handle/1721.1/6773/AITR-157... [1]https://phyre.ai https://phyre.ai [2]https://slotformer.github.io https://slotformer.github.io

kwhitefoot 3y ago

If they are lying, or merely ludicrously mistaken, about the Fiat 500 and the F-150 what credence can be placed in the rest?