4 ms·

Anyone who has ever done CAD knows that a picture is worth a thousand words. Describing a 3D object you want in words is much more effort to get right than draw

Anyone who has ever done CAD knows that a picture is worth a thousand words. Describing a 3D object you want in words is much more effort to get right than drawing a simple sketch. Wouldn’t it be better to have image input for an application like this?

jsheard 2y ago
Yeah but when all you have is an LLM hammer, everything looks like a text2text nail.
- CSMastermind 2y ago
  LLMs are fundamentally one dimensional which works fine when you're generating next tokens for text which because that's a 1D problem. I do wonder how much progress we could make on a problem like this with a 3D transformer architecture.
  - abotsis 2y ago
    I’m not sure I follow this. Isn’t an LLMs dimensionality measured by how many parameters the model supports? Ie 10s of billions in some cases? If I understand it correctly, then, the model is already evaluating things in lots of dimensions and reducing it down to 1, as you say in the case of text, 2 dimensions in image generation, 3 should be pretty straightforward.
    - btbuildem 2y ago
      I think they're referring to the dimensionality of the input / output space, not the intermediate internal representation.
  - btbuildem 2y ago
    The neat thing is, you can rasterize 1D space into 2D, 3D and so on. Trick as old as analog TV signal processing.
    - throwgfgfd25 2y ago
      If I am understanding you right... I don't think this gets you anywhere useful. Even if you could do what you're suggesting with an LLM (I have my doubts) this result would be a mesh or 3D pixel grid or something, yes? This is terrible for interoperability and it's the opposite of what mainstream CAD packages do.
DonnyV 2y ago
Yeah, a way to turn a flat sketch to a 3D model would be a better way to do this.
godelski 2y ago
That's exactly what I was thinking. I can't describe what I want because I don't know it yet. It comes to life as I design. I was an engineer in a former life but still do a fair amount of printing. But when I design parts there's not even ways I __could__ know what I want before hand. As I build I realize I made wrong assumptions, but also not enough, that there are better ways to do things, that I can solve other problems, that I didn't think how things would interact together, that I could modify things to be better for the manufacturing process (this is such a big on in 3d printing and so many online files get this VERY wrong. But it is a hard skill to learn), and so many other things. In part this is because I've had more time to think, but there's more to it, when you see the thing "coming alive". This is much the same way I code, though I guess that's not common. I'm not sure if they'll address these problems, but I think anyone working in this space should make sure they also spend a lot of time in CAD themselves. It isn't clear to me that any of the authors do (looking at their websites. The main author mentions interest in AI-CAD but only has work from this year. Get this man a 3D printer). It's quite possible that they do, but they look like they've been computer scientists their whole career and that is probably not enough to understand the the intricacies of the problems they're trying to solve. There's a classic problem in CS where people get that you can learn a lot of things quickly but it is missed that getting the nuance and mastery takes time, that you should talk to experts. The first part is useful because it gives you the language to talk to experts, not because it makes you a replacement for one.
- throwgfgfd25 2y ago
  > In part this is because I've had more time to think, but there's more to it, when you see the thing "coming alive". Yep. I have two printed prototypes of different approaches to a mechanism on my desk that only exist because of months of staring at CAD in the evenings, learning new things, doing research. They are not radical (they may be slightly novel in places; I have never seen 3D printed mechanisms like them). I don't know if I could describe them in words at all, but if I could, it would only be because I worked through them in CAD in the first place. For anything other than a trivial object I just can't see how you'd even come up with the words without having worked through the design -- what, on paper in 2D in pencil? After doing the maths? That's CAD in reverse.
trenchgun 2y ago
What you really want is draw a sketch + describe constraints in symbols

jsheard 2y ago

Yeah but when all you have is an LLM hammer, everything looks like a text2text nail.

CSMastermind 2y ago
LLMs are fundamentally one dimensional which works fine when you're generating next tokens for text which because that's a 1D problem. I do wonder how much progress we could make on a problem like this with a 3D transformer architecture.
- abotsis 2y ago
  I’m not sure I follow this. Isn’t an LLMs dimensionality measured by how many parameters the model supports? Ie 10s of billions in some cases? If I understand it correctly, then, the model is already evaluating things in lots of dimensions and reducing it down to 1, as you say in the case of text, 2 dimensions in image generation, 3 should be pretty straightforward.
  - btbuildem 2y ago
    I think they're referring to the dimensionality of the input / output space, not the intermediate internal representation.
- btbuildem 2y ago
  The neat thing is, you can rasterize 1D space into 2D, 3D and so on. Trick as old as analog TV signal processing.
  - throwgfgfd25 2y ago
    If I am understanding you right... I don't think this gets you anywhere useful. Even if you could do what you're suggesting with an LLM (I have my doubts) this result would be a mesh or 3D pixel grid or something, yes? This is terrible for interoperability and it's the opposite of what mainstream CAD packages do.

DonnyV 2y ago

Yeah, a way to turn a flat sketch to a 3D model would be a better way to do this.

godelski 2y ago

That's exactly what I was thinking. I can't describe what I want because I don't know it yet. It comes to life as I design. I was an engineer in a former life but still do a fair amount of printing. But when I design parts there's not even ways I __could__ know what I want before hand. As I build I realize I made wrong assumptions, but also not enough, that there are better ways to do things, that I can solve other problems, that I didn't think how things would interact together, that I could modify things to be better for the manufacturing process (this is such a big on in 3d printing and so many online files get this VERY wrong. But it is a hard skill to learn), and so many other things. In part this is because I've had more time to think, but there's more to it, when you see the thing "coming alive". This is much the same way I code, though I guess that's not common. I'm not sure if they'll address these problems, but I think anyone working in this space should make sure they also spend a lot of time in CAD themselves. It isn't clear to me that any of the authors do (looking at their websites. The main author mentions interest in AI-CAD but only has work from this year. Get this man a 3D printer). It's quite possible that they do, but they look like they've been computer scientists their whole career and that is probably not enough to understand the the intricacies of the problems they're trying to solve. There's a classic problem in CS where people get that you can learn a lot of things quickly but it is missed that getting the nuance and mastery takes time, that you should talk to experts. The first part is useful because it gives you the language to talk to experts, not because it makes you a replacement for one.

throwgfgfd25 2y ago
> In part this is because I've had more time to think, but there's more to it, when you see the thing "coming alive". Yep. I have two printed prototypes of different approaches to a mechanism on my desk that only exist because of months of staring at CAD in the evenings, learning new things, doing research. They are not radical (they may be slightly novel in places; I have never seen 3D printed mechanisms like them). I don't know if I could describe them in words at all, but if I could, it would only be because I worked through them in CAD in the first place. For anything other than a trivial object I just can't see how you'd even come up with the words without having worked through the design -- what, on paper in 2D in pencil? After doing the maths? That's CAD in reverse.

trenchgun 2y ago

What you really want is draw a sketch + describe constraints in symbols