6 ms·

Diagrams AI can, and cannot, generate

214 points by billyp-rva 2y ago

enoeht 2y ago
Have more success with asking for a detailed workflow print then a d2/mermaid output. No problems with creating a ASCI diagram either and using that for a manual d2 can be done fast enough.
vunderba 2y ago
Related - a nice time saver that I've been using since they added image recognition support to ChatGPT has been taking a quick snap of my crudely drawn hand sketched diagrams (on graph paper) with my phone and asking ChatGPT to convert them to mermaid UML syntax.
- seeingnature 2y ago
  Comments like these are why I come to hacker news! I'm working on a project right now where I've been learning mermaid, but have gotten to the point where it would be easier for me to draw it out and convert this way by a lot. I'll try this!
cadamsdotcom 2y ago
Thanks for writing this up. Some questions for the author: Interesting perspective but it’s a bit incomplete without a comparison of various models and how they perform. Kind of like Simon Willison’s now-famous “pelican on a bicycle” test, these diagrams might be done better by some models than others. Second, this presents a static picture of things, but AI moves really fast! It’d also be great to understand how this capability is improving over time.
- ndr_ 2y ago
  I talk about this, kind-of, in my article about process visualization (in German, available behind paywall and in print). It‘s not rigorous in the sense that I give points, but a picture emerges along the way. Based on the full set of practical examples there, I would recommend the „v1“ of Claude 3.5 Sonnet. GPT 4.5 also looks good, but I haven‘t run the full suite. https://www.heise.de/ratgeber/Prozessvisualisierung-mit-generativer-KI-im-Praxistest-10266093.html https://www.heise.de/ratgeber/Prozessvisualisierung-mit-gene...
submeta 2y ago
Try asking llm to generate plantuml markup (use case, statechart, etc) which has some other diagram types in addition to mermaid markup. Then paste it into the free plantuml renderer. Works pretty well. I also experimented with bpmn markup (xml). Realized there are already repos on GitHub creating bpmn diagrams from prompt. You can also ask llms to create svg.
- cptnmrvl 2y ago
  plantuml also supports AWS icons
- monkeydust 2y ago
  Plantuml works pretty well with openai models.
- antman 2y ago
  Bpmn diagrams from prompt? Did you try any that was good?
LASR 2y ago
We use mermaidjs as a supercharged version of chain-of-thought for generating some sophisticated decompositions of the intent. Then we injected the generated mermaid diagrams back into subsequent requests. Reasoning performance improves for a whole variety of applications.
- Garlef 2y ago
  Neat idea! Could you go into a bit more detail on how you encode the intent?
- BOOSTERHIDROGEN 2y ago
  Any simple examples?
mulmboy 2y ago
I have found LLMs to be very good at the kind of code -> diagram task presented here. Fire up superwhisper[1] and stream-of-consciousness away about why you want the diagram, which bits are important, who the audience is, and so on. Then iterate a few times. Works brilliantly for even very complex things, including 5000 line CDK files. It's disingenuous to conclude that AI is no good at diagramming after using an impotent prompt AND refusing to iterate with it. A human would do no better with the same instructions, LLMs aren't magic. This is the same as my previous comment https://news.ycombinator.com/item?id=42524125 https://news.ycombinator.com/item?id=42524125 [1] https://superwhisper.com/ https://superwhisper.com/
- andra_nl 2y ago
  Fair point. What the author does mention is that if you have to do a lot of work getting proper results out of AI (and potentially contend with hallucinations), you may as well do the actual work yourself and be more confident about the end result. That being said, I think part of the potential is repeatability. Once you've done the work of property prompting for the desired result, you can often save the adjusted prompts (or a variation of it) for later use, giving you a flying start on subsequent occasions.
peter_retief 2y ago
I ask AI to generate diagrams in LaTeX, works well for me.
notTooFarGone 2y ago
I used AI to generate some UML diagrams on a loosely coupled system - just fed it the actual classes where only names identify the actual links. It did quite a good job there. It was a well defined domain so I guess the training data argument doesn't fit for stuff that is within a "natural" domain like graphs. LLMs can infer the behavior based on naming quite well.
victorbjorklund 2y ago
I have had good success with D2 diagrams with Claude: https://victorbjorklund.com/build-diagrams-as-code-with-d2-d2lang https://victorbjorklund.com/build-diagrams-as-code-with-d2-d... They have icons for common things like cloud things.
- chillingeffect 2y ago
  Ive had similar results asking chatgpt to generate inout files for graphviz "dot". Pretty good. E.g. i asked it to summarize a complex article and draw the people named and their relationships. I also got it embellish the diagrams a little but, but it needed a lot of guidance to know what kind of nodes to add. But it was good at arranging the elements in timeline order for example.
stared 2y ago
GPT 4o is not particularly good at this kind of logic, at least compared to other current models. Trying something that is at least in the top 10 from this WebDev Areans leaderboard: https://web.lmarena.ai/leaderboard https://web.lmarena.ai/leaderboard would help. Make sure it is allowed to think before doing (not necessarily in a dedicated thinking mode, it can be a regular prompt to design a graph before implementing it; make sure to add in a prompt who the graph is for (e.g. "a clean graph, suitable for a blog post for technical audience").
james-bcn 2y ago
Why just stick to Mermaid? I expect that there is a lot more material with regards to SVG that large models have been trained on. And it's a fairly simple format. Asking it to create diagrams in SVG format gives it much more flexibility. Of course there may be a bit less consistency, but there are ways around that (e.g. giving an example/template to follow). Simon Willison has shown that current models aren't very good at creating an SVG of a pelican on a bicycle, but drawing a box diagram in SVG is a much simpler task.
- TeMPOraL 2y ago
  Because few ever write diagrams in SVG. SVG is an output format, not an input format, and asking for a diagram in SVG is asking for the model to translate whatever you're asking about (e.g. names of systems you expect the model to know, like AWS, or regular code that needs to be turned into a diagram) into some hidden diagram-ish form, and then generate an SVG out of it. Can't see it working without letting the model output an intermediary form in PlantUML or Mermaid or Dot - going straight to SVG is cramming too much work into too few tokens. For the same reason, textual diagram formats are better for iterative work. SVG is too open-ended, and carries little to no semantics. Diagramming language are all about semantics, have fewer degrees of freedom, and much less spurious token noise.
  - Terretta 2y ago
    > Because few ever write diagrams in SVG. SVG is an output format, not an input format... Aside from "try Inkscape", that sounds like a human problem not an LLM problem. LLMs output what they input, and if diagrams in blog articles or docs are SVG, they merrily input SVG, and associate it with the adjacencies. One might as well say MidJourney won't work because few ever make paintings using pixels. You're asking it to translate whatever you're asking about (e.g. scenes and names of painters you'd expect the model to know, like Escher or DaVinci), into some hidden imagined scene, render that as brush strokes of types of paint on textured media, and then generate a PNG out of it.
    - robjan 2y ago
mharig 2y ago
[dead]
trash_cat 2y ago
Sonnet 3.7 is perticularly good to generated xml diagrams that can be imported into draw.io. If you are using Cline, Windusurf or Cursor, you can ask it to create the xml file and immediately open it up. Combine it together with CONTEXT.md or ARCHITECTURE.md and you can get a very good overview of the codebase and have discussions around it.
graphviz 2y ago
Random thoughts: Sketching backed by automated cleanup can be good for entering small diagrams. There used to be an iOS app based on graphviz: http://instaviz.com http://instaviz.com Constraint-based interactive layout may be underinvested, as a consequence of too many disappointments and false starts in the 1980s. LLMs seem ill-suited to solving the optimization of combinatorial and geometric constraints and objectives required for good diagram layout. Overall, one has to admire the directness and simplicity of mermaid. Also, it would be great to someday see a practical tool with the quality and generality of the ultra-compact grid layout prototype from the Monash group, https://ialab.it.monash.edu/~dwyer/papers/gridlayout2015.pdf https://ialab.it.monash.edu/~dwyer/papers/gridlayout2015.pdf (2015!!)
- ttd 2y ago
  Oh wow, thank you for linking that paper. I've been working an interactive tool for a while and have been musing on new constraint and layout types to add. Anecdotally it seems a lot of mainstream graph layout algorithms work well for small to mediumish complexity inputs, but then quickly start generating visual spaghetti. So this looks incredibly apropos for me.
- teleforce 2y ago
  Thanks for link to the Monash's paper. >LLMs seem ill-suited to solving the optimization of combinatorial and geometric constraints and objectives required for good diagram layout. I think this is where LLM distance NLP cousin can be of help namely CUE since fundamentally it's based on feature structure from the deterministic approach of NLP unlike LLM that's stochastic NLP [1],[2],[3]. Based on the Monash's paper, Constraint Programming (CP) is one of the popular approaches that's being used for the automatic grid layout. Since CUE is a constraint configuration language belong to CP, and its NLP background should make it easier and seamless to integrate with LLM. If someone somehow can crack this then it will be a new generation LLM that can perform good and accurate diagramming via prompts and it will be a boon for the architect, designer and engineer. Talking about engineer, if this approach can also be used for IC layout design (analog and digital) not only for diagrams, it will easily disrupt the multi-billion dollars industry for the very expensive software for IC design and man powers. I hope I'm not getting ahead of myself, but ultimately this combo can probably solve the "holy grails" problem mentioned towards the end of the paper's conclusions regarding layout model that somehow incorporates routing in a way that is efficiently solvable to optimality. After all some people in computer science consider CP as "holy grails" of programming [4]. Please someone somehow make a start up, or any existing YC startup like JITX (Hi Patrick) can look into this potential fruitful endeavor of hybrid LLM combo for automated IC design [5]. Perhaps your random thoughts are not so random but deterministic non-random in nature, pardon the pun. [1] Cue – A language for defining, generating, and validating data: https://news.ycombinator.com/item?id=20847943 https://news.ycombinator.com/item?id=20847943 [2] Feature structure: https://en.m.wikipedia.org/wiki/Feature_structure https://en.m.wikipedia.org/wiki/Feature_structure [3] The Logic of CUE: https://cuelang.org/docs/concept/the-logic-of-cue/ https://cuelang.org/docs/concept/the-logic-of-cue/ [4] Solving Combinatorial Optimization Problems with Constraint Programming and OscaR [video]: https://m.youtube.com/watch?v=opXBR00z_QM https://m.youtube.com/watch?v=opXBR00z_QM [5] JITX: Automatic circuit board design: https://www.ycombinator.com/companies/jitx https://www.ycombinator.com/companies/jitx
WesleyLivesay 2y ago
Given the pace of development in this space, it is probably worth noting in the title that this is from November 2024 so the results might be a bit dated.
- billyp-rva 2y ago
  Author here. It's a fair point and it would be worth revisiting, if not now then within the year. That said, I wouldn't expect things to change too drastically. TFA goes into details, but in short LLMs are already quite good at whiteboarding (where you interactively describe the diagram you want). They're also really bad at generating a diagram from an existing system. In either case, small, incremental improvements won't really help; you'd need a large change to move the needle.
  - Terretta 2y ago
    See also: https://simonwillison.net/2024/Oct/25/pelicans-on-a-bicycle/ https://simonwillison.net/2024/Oct/25/pelicans-on-a-bicycle/
jbverschoor 2y ago
[yet]
sachinaag 2y ago
[dead]
larodi 2y ago
Claude does quite alright. Across one and a half year I did more than several dozens of Mermaid diagrams of all kinds, and only the most complex perhaps were out of reach. It also really depends on the printing.
- layer8 2y ago
  Printing?
  - larodi 2y ago
    hmmm let me remember what did I want to say. hmmm hmm hmmm. depends on the prompting I guess :D sorry
ndr_ 2y ago
I wrote about the same general topic (or more narrowly: process visualization) in German iX magazine, also available here: https://www.heise.de/ratgeber/Prozessvisualisierung-mit-generativer-KI-im-Praxistest-10266093.html https://www.heise.de/ratgeber/Prozessvisualisierung-mit-gene... (€) Rather than relying on end-user products like ChatGPT or Claude.ai, this article is based on the „pure“ model offerings via API and frontends that build on these. While the Ilograph blog ponders „AI’s ability to create generic diagrams“, I‘d conclude: do it, but avoid the „open“ models and low-cost offerings.
RKFADU_UOFCCLEL 2y ago
The "AI" we have now is just a tweening algorithm on a different medium. You won't be able to get it to do anything specific, except when that's a point between 2 existing works. As for this blog, it's nigh unreadable for those not following the current fad web frameworks. Who's to say the user doesn't have to log in to get to the gateway? Gateway can mean different things. Why can the user choose to upload images instead of logging in? What was the purpose of the log in?
diggan 2y ago
A mistake I see people repeating over and over, is never restarting their conversations with a edited initial message. Instead of doing what the author is doing here, and sending messages back and forward, leading to a longer and longer conversation, where each messages leads to worse and worse quality replies, until the LLM seems like a dumb rock, rewrite your initial message with everything that went wrong/was misunderstood, and aim to have whatever you want solved in the first message, and you'll get a lot higher quality answers. If the LLM misunderstood, don't reply "No, what I mean was..." but instead rewrite the first message so it's clearer. This is at least true for all ChatGPT, Claude and DeepSeek models, YMMV with other models.
- bpodgursky 2y ago
  100% To be honest, this would help a lot of person-implemented iteration too, if it was biologically feasible to erase a conversation from a brain.
  - dingnuts 2y ago
    alright, time for you to go watch Eternal Sunshine of the Spotless Mind so that you can disabuse yourself of that notion
- swatcoder 2y ago
  Yup. Inasmuch as these are collaborative document generators at their core, "minimally ambiguous prompt and conforming reply" is a strongly represented document structure and so we benefit by setting them up to complete one. Likewise, "tragi-comic dialog between increasingly frustrated instructor and bumbling pupil" is also a widely represented document structure that we benefit by trying to avoid. Chatbot training works to minimize the chance of an LLM engaging in the latter, because dialog is a intuitive interface that users enjoy, but we can avoid the problem more successfully by just providing a new and less ambiguous prompt in a new session, as you suggest.
  - dartos 2y ago
    > dialog is a intuitive interface that users enjoy Do people enjoy chat interfaces in their workflows? I always thought that cursor/copilot/copy.ai/v0.dev were so popular because they break away from the chat UI. Dialog is cool when exploring but, imo, really painful when trying to accomplish a task. LLMs are far too slow to make a real fluid conversation.
30minAdayHN 2y ago
I was thinking about the similar topic and started to wonder if I can generated a diagram of a large codebase. I thought that LLMs are great at compressing information and thought of putting it to good use by compressing a large codebase into a single diagram. Since entire codebase doesn't fit in the context window, I built a recursive LLM tool that calls itself. It takes two params: * current diagram state, * new files it needs to expand the diagram. The seed set would be an empty diagram and an entry point to source code. And I also extended it to complexity analysis. It worked magically well. Here are couple of diagrams it generated: * https://gist.github.com/priyankc/27eb786e50e41c32d332390a42e56cd1 https://gist.github.com/priyankc/27eb786e50e41c32d332390a42e... * https://gist.github.com/priyankc/0ca04f09a32f6d91c6b42bd8b180ae4b https://gist.github.com/priyankc/0ca04f09a32f6d91c6b42bd8b18... If you are interested in trying out, I've blogged here: https://updates.priyank.ch/projects/2025/03/12/complexity-analysis.html https://updates.priyank.ch/projects/2025/03/12/complexity-an...
McNutty 2y ago
You have got more patience than me. I have tried to use these tools to generate (basic) network diagrams and by the time I reached your third step I already knew that it was time to quit and draw it out myself. Diagrams need to be correct and accurate otherwise they're just art. I also need any amendments to be made to the same diagram, not to have it regenerated each time. I do like the idea of another commenter here who takes a photo of their whiteboard and instructs the AI tool to turn it into a structured diagram. That seems to be well within reach of these tools.
melagonster 2y ago
Although searching provides better results, it is certain that attempting to copy or directly use these images would infringe on someone's copyright. In a broader sense, using AI can also be related to copyright infringement; the court must first defeat the AI provider before it can reach the user.
giberson 2y ago
FWIW, I think this article could just as accurately be titled "Diagrams Developers can, and cannot, generate". I'm mainly speaking to the ability to read IaC code ([probably of any library but at LEAST in my case] cdk, pulumi, terraform, cloudformation, serverless) and be able to infer architectural flow from it. It's really not conducive to that use case. I could also, kidding/not kidding, be speaking to the range of abilities for "mid" and "senior" developers to know and convey such flows in diagrams. But really my point is this feels like more validation that AI doesn't provide increased ability, it provides existing (and demonstrated) ability faster with less formalized context. The "less formalized context" is what distinguishes it from programs/code.

214 points by billyp-rva 2y ago

Have more success with asking for a detailed workflow print then a d2/mermaid output. No problems with creating a ASCI diagram either and using that for a manual d2 can be done fast enough.

vunderba 2y ago

Related - a nice time saver that I've been using since they added image recognition support to ChatGPT has been taking a quick snap of my crudely drawn hand sketched diagrams (on graph paper) with my phone and asking ChatGPT to convert them to mermaid UML syntax.

seeingnature 2y ago
Comments like these are why I come to hacker news! I'm working on a project right now where I've been learning mermaid, but have gotten to the point where it would be easier for me to draw it out and convert this way by a lot. I'll try this!

cadamsdotcom 2y ago

Thanks for writing this up. Some questions for the author: Interesting perspective but it’s a bit incomplete without a comparison of various models and how they perform. Kind of like Simon Willison’s now-famous “pelican on a bicycle” test, these diagrams might be done better by some models than others. Second, this presents a static picture of things, but AI moves really fast! It’d also be great to understand how this capability is improving over time.

ndr_ 2y ago
I talk about this, kind-of, in my article about process visualization (in German, available behind paywall and in print). It‘s not rigorous in the sense that I give points, but a picture emerges along the way. Based on the full set of practical examples there, I would recommend the „v1“ of Claude 3.5 Sonnet. GPT 4.5 also looks good, but I haven‘t run the full suite. https://www.heise.de/ratgeber/Prozessvisualisierung-mit-generativer-KI-im-Praxistest-10266093.html https://www.heise.de/ratgeber/Prozessvisualisierung-mit-gene...

submeta 2y ago

Try asking llm to generate plantuml markup (use case, statechart, etc) which has some other diagram types in addition to mermaid markup. Then paste it into the free plantuml renderer. Works pretty well. I also experimented with bpmn markup (xml). Realized there are already repos on GitHub creating bpmn diagrams from prompt. You can also ask llms to create svg.

cptnmrvl 2y ago
plantuml also supports AWS icons
monkeydust 2y ago
Plantuml works pretty well with openai models.
antman 2y ago
Bpmn diagrams from prompt? Did you try any that was good?

LASR 2y ago

We use mermaidjs as a supercharged version of chain-of-thought for generating some sophisticated decompositions of the intent. Then we injected the generated mermaid diagrams back into subsequent requests. Reasoning performance improves for a whole variety of applications.

Garlef 2y ago
Neat idea! Could you go into a bit more detail on how you encode the intent?
BOOSTERHIDROGEN 2y ago
Any simple examples?

mulmboy 2y ago

I have found LLMs to be very good at the kind of code -> diagram task presented here. Fire up superwhisper[1] and stream-of-consciousness away about why you want the diagram, which bits are important, who the audience is, and so on. Then iterate a few times. Works brilliantly for even very complex things, including 5000 line CDK files. It's disingenuous to conclude that AI is no good at diagramming after using an impotent prompt AND refusing to iterate with it. A human would do no better with the same instructions, LLMs aren't magic. This is the same as my previous comment https://news.ycombinator.com/item?id=42524125 https://news.ycombinator.com/item?id=42524125 [1] https://superwhisper.com/ https://superwhisper.com/

andra_nl 2y ago
Fair point. What the author does mention is that if you have to do a lot of work getting proper results out of AI (and potentially contend with hallucinations), you may as well do the actual work yourself and be more confident about the end result. That being said, I think part of the potential is repeatability. Once you've done the work of property prompting for the desired result, you can often save the adjusted prompts (or a variation of it) for later use, giving you a flying start on subsequent occasions.

peter_retief 2y ago

I ask AI to generate diagrams in LaTeX, works well for me.

notTooFarGone 2y ago

I used AI to generate some UML diagrams on a loosely coupled system - just fed it the actual classes where only names identify the actual links. It did quite a good job there. It was a well defined domain so I guess the training data argument doesn't fit for stuff that is within a "natural" domain like graphs. LLMs can infer the behavior based on naming quite well.

victorbjorklund 2y ago

I have had good success with D2 diagrams with Claude: https://victorbjorklund.com/build-diagrams-as-code-with-d2-d2lang https://victorbjorklund.com/build-diagrams-as-code-with-d2-d... They have icons for common things like cloud things.

chillingeffect 2y ago
Ive had similar results asking chatgpt to generate inout files for graphviz "dot". Pretty good. E.g. i asked it to summarize a complex article and draw the people named and their relationships. I also got it embellish the diagrams a little but, but it needed a lot of guidance to know what kind of nodes to add. But it was good at arranging the elements in timeline order for example.

stared 2y ago

GPT 4o is not particularly good at this kind of logic, at least compared to other current models. Trying something that is at least in the top 10 from this WebDev Areans leaderboard: https://web.lmarena.ai/leaderboard https://web.lmarena.ai/leaderboard would help. Make sure it is allowed to think before doing (not necessarily in a dedicated thinking mode, it can be a regular prompt to design a graph before implementing it; make sure to add in a prompt who the graph is for (e.g. "a clean graph, suitable for a blog post for technical audience").

james-bcn 2y ago

Why just stick to Mermaid? I expect that there is a lot more material with regards to SVG that large models have been trained on. And it's a fairly simple format. Asking it to create diagrams in SVG format gives it much more flexibility. Of course there may be a bit less consistency, but there are ways around that (e.g. giving an example/template to follow). Simon Willison has shown that current models aren't very good at creating an SVG of a pelican on a bicycle, but drawing a box diagram in SVG is a much simpler task.

TeMPOraL 2y ago
Because few ever write diagrams in SVG. SVG is an output format, not an input format, and asking for a diagram in SVG is asking for the model to translate whatever you're asking about (e.g. names of systems you expect the model to know, like AWS, or regular code that needs to be turned into a diagram) into some hidden diagram-ish form, and then generate an SVG out of it. Can't see it working without letting the model output an intermediary form in PlantUML or Mermaid or Dot - going straight to SVG is cramming too much work into too few tokens. For the same reason, textual diagram formats are better for iterative work. SVG is too open-ended, and carries little to no semantics. Diagramming language are all about semantics, have fewer degrees of freedom, and much less spurious token noise.
- Terretta 2y ago
  > Because few ever write diagrams in SVG. SVG is an output format, not an input format... Aside from "try Inkscape", that sounds like a human problem not an LLM problem. LLMs output what they input, and if diagrams in blog articles or docs are SVG, they merrily input SVG, and associate it with the adjacencies. One might as well say MidJourney won't work because few ever make paintings using pixels. You're asking it to translate whatever you're asking about (e.g. scenes and names of painters you'd expect the model to know, like Escher or DaVinci), into some hidden imagined scene, render that as brush strokes of types of paint on textured media, and then generate a PNG out of it.
  - robjan 2y ago

mharig 2y ago

[dead]

trash_cat 2y ago

Sonnet 3.7 is perticularly good to generated xml diagrams that can be imported into draw.io. If you are using Cline, Windusurf or Cursor, you can ask it to create the xml file and immediately open it up. Combine it together with CONTEXT.md or ARCHITECTURE.md and you can get a very good overview of the codebase and have discussions around it.

graphviz 2y ago

Random thoughts: Sketching backed by automated cleanup can be good for entering small diagrams. There used to be an iOS app based on graphviz: http://instaviz.com http://instaviz.com Constraint-based interactive layout may be underinvested, as a consequence of too many disappointments and false starts in the 1980s. LLMs seem ill-suited to solving the optimization of combinatorial and geometric constraints and objectives required for good diagram layout. Overall, one has to admire the directness and simplicity of mermaid. Also, it would be great to someday see a practical tool with the quality and generality of the ultra-compact grid layout prototype from the Monash group, https://ialab.it.monash.edu/~dwyer/papers/gridlayout2015.pdf https://ialab.it.monash.edu/~dwyer/papers/gridlayout2015.pdf (2015!!)

ttd 2y ago
Oh wow, thank you for linking that paper. I've been working an interactive tool for a while and have been musing on new constraint and layout types to add. Anecdotally it seems a lot of mainstream graph layout algorithms work well for small to mediumish complexity inputs, but then quickly start generating visual spaghetti. So this looks incredibly apropos for me.
teleforce 2y ago
Thanks for link to the Monash's paper. >LLMs seem ill-suited to solving the optimization of combinatorial and geometric constraints and objectives required for good diagram layout. I think this is where LLM distance NLP cousin can be of help namely CUE since fundamentally it's based on feature structure from the deterministic approach of NLP unlike LLM that's stochastic NLP [1],[2],[3]. Based on the Monash's paper, Constraint Programming (CP) is one of the popular approaches that's being used for the automatic grid layout. Since CUE is a constraint configuration language belong to CP, and its NLP background should make it easier and seamless to integrate with LLM. If someone somehow can crack this then it will be a new generation LLM that can perform good and accurate diagramming via prompts and it will be a boon for the architect, designer and engineer. Talking about engineer, if this approach can also be used for IC layout design (analog and digital) not only for diagrams, it will easily disrupt the multi-billion dollars industry for the very expensive software for IC design and man powers. I hope I'm not getting ahead of myself, but ultimately this combo can probably solve the "holy grails" problem mentioned towards the end of the paper's conclusions regarding layout model that somehow incorporates routing in a way that is efficiently solvable to optimality. After all some people in computer science consider CP as "holy grails" of programming [4]. Please someone somehow make a start up, or any existing YC startup like JITX (Hi Patrick) can look into this potential fruitful endeavor of hybrid LLM combo for automated IC design [5]. Perhaps your random thoughts are not so random but deterministic non-random in nature, pardon the pun. [1] Cue – A language for defining, generating, and validating data: https://news.ycombinator.com/item?id=20847943 https://news.ycombinator.com/item?id=20847943 [2] Feature structure: https://en.m.wikipedia.org/wiki/Feature_structure https://en.m.wikipedia.org/wiki/Feature_structure [3] The Logic of CUE: https://cuelang.org/docs/concept/the-logic-of-cue/ https://cuelang.org/docs/concept/the-logic-of-cue/ [4] Solving Combinatorial Optimization Problems with Constraint Programming and OscaR [video]: https://m.youtube.com/watch?v=opXBR00z_QM https://m.youtube.com/watch?v=opXBR00z_QM [5] JITX: Automatic circuit board design: https://www.ycombinator.com/companies/jitx https://www.ycombinator.com/companies/jitx

WesleyLivesay 2y ago

Given the pace of development in this space, it is probably worth noting in the title that this is from November 2024 so the results might be a bit dated.

billyp-rva 2y ago
Author here. It's a fair point and it would be worth revisiting, if not now then within the year. That said, I wouldn't expect things to change too drastically. TFA goes into details, but in short LLMs are already quite good at whiteboarding (where you interactively describe the diagram you want). They're also really bad at generating a diagram from an existing system. In either case, small, incremental improvements won't really help; you'd need a large change to move the needle.
- Terretta 2y ago
  See also: https://simonwillison.net/2024/Oct/25/pelicans-on-a-bicycle/ https://simonwillison.net/2024/Oct/25/pelicans-on-a-bicycle/

jbverschoor 2y ago

[yet]

sachinaag 2y ago

[dead]

larodi 2y ago

Claude does quite alright. Across one and a half year I did more than several dozens of Mermaid diagrams of all kinds, and only the most complex perhaps were out of reach. It also really depends on the printing.

layer8 2y ago
Printing?
- larodi 2y ago
  hmmm let me remember what did I want to say. hmmm hmm hmmm. depends on the prompting I guess :D sorry

ndr_ 2y ago

I wrote about the same general topic (or more narrowly: process visualization) in German iX magazine, also available here: https://www.heise.de/ratgeber/Prozessvisualisierung-mit-generativer-KI-im-Praxistest-10266093.html https://www.heise.de/ratgeber/Prozessvisualisierung-mit-gene... (€) Rather than relying on end-user products like ChatGPT or Claude.ai, this article is based on the „pure“ model offerings via API and frontends that build on these. While the Ilograph blog ponders „AI’s ability to create generic diagrams“, I‘d conclude: do it, but avoid the „open“ models and low-cost offerings.

RKFADU_UOFCCLEL 2y ago

The "AI" we have now is just a tweening algorithm on a different medium. You won't be able to get it to do anything specific, except when that's a point between 2 existing works. As for this blog, it's nigh unreadable for those not following the current fad web frameworks. Who's to say the user doesn't have to log in to get to the gateway? Gateway can mean different things. Why can the user choose to upload images instead of logging in? What was the purpose of the log in?

diggan 2y ago

A mistake I see people repeating over and over, is never restarting their conversations with a edited initial message. Instead of doing what the author is doing here, and sending messages back and forward, leading to a longer and longer conversation, where each messages leads to worse and worse quality replies, until the LLM seems like a dumb rock, rewrite your initial message with everything that went wrong/was misunderstood, and aim to have whatever you want solved in the first message, and you'll get a lot higher quality answers. If the LLM misunderstood, don't reply "No, what I mean was..." but instead rewrite the first message so it's clearer. This is at least true for all ChatGPT, Claude and DeepSeek models, YMMV with other models.

bpodgursky 2y ago
100% To be honest, this would help a lot of person-implemented iteration too, if it was biologically feasible to erase a conversation from a brain.
- dingnuts 2y ago
  alright, time for you to go watch Eternal Sunshine of the Spotless Mind so that you can disabuse yourself of that notion
swatcoder 2y ago
Yup. Inasmuch as these are collaborative document generators at their core, "minimally ambiguous prompt and conforming reply" is a strongly represented document structure and so we benefit by setting them up to complete one. Likewise, "tragi-comic dialog between increasingly frustrated instructor and bumbling pupil" is also a widely represented document structure that we benefit by trying to avoid. Chatbot training works to minimize the chance of an LLM engaging in the latter, because dialog is a intuitive interface that users enjoy, but we can avoid the problem more successfully by just providing a new and less ambiguous prompt in a new session, as you suggest.
- dartos 2y ago
  > dialog is a intuitive interface that users enjoy Do people enjoy chat interfaces in their workflows? I always thought that cursor/copilot/copy.ai/v0.dev were so popular because they break away from the chat UI. Dialog is cool when exploring but, imo, really painful when trying to accomplish a task. LLMs are far too slow to make a real fluid conversation.

30minAdayHN 2y ago

I was thinking about the similar topic and started to wonder if I can generated a diagram of a large codebase. I thought that LLMs are great at compressing information and thought of putting it to good use by compressing a large codebase into a single diagram. Since entire codebase doesn't fit in the context window, I built a recursive LLM tool that calls itself. It takes two params: * current diagram state, * new files it needs to expand the diagram. The seed set would be an empty diagram and an entry point to source code. And I also extended it to complexity analysis. It worked magically well. Here are couple of diagrams it generated: * https://gist.github.com/priyankc/27eb786e50e41c32d332390a42e56cd1 https://gist.github.com/priyankc/27eb786e50e41c32d332390a42e... * https://gist.github.com/priyankc/0ca04f09a32f6d91c6b42bd8b180ae4b https://gist.github.com/priyankc/0ca04f09a32f6d91c6b42bd8b18... If you are interested in trying out, I've blogged here: https://updates.priyank.ch/projects/2025/03/12/complexity-analysis.html https://updates.priyank.ch/projects/2025/03/12/complexity-an...

McNutty 2y ago

You have got more patience than me. I have tried to use these tools to generate (basic) network diagrams and by the time I reached your third step I already knew that it was time to quit and draw it out myself. Diagrams need to be correct and accurate otherwise they're just art. I also need any amendments to be made to the same diagram, not to have it regenerated each time. I do like the idea of another commenter here who takes a photo of their whiteboard and instructs the AI tool to turn it into a structured diagram. That seems to be well within reach of these tools.

melagonster 2y ago

Although searching provides better results, it is certain that attempting to copy or directly use these images would infringe on someone's copyright. In a broader sense, using AI can also be related to copyright infringement; the court must first defeat the AI provider before it can reach the user.

giberson 2y ago

FWIW, I think this article could just as accurately be titled "Diagrams Developers can, and cannot, generate". I'm mainly speaking to the ability to read IaC code ([probably of any library but at LEAST in my case] cdk, pulumi, terraform, cloudformation, serverless) and be able to infer architectural flow from it. It's really not conducive to that use case. I could also, kidding/not kidding, be speaking to the range of abilities for "mid" and "senior" developers to know and convey such flows in diagrams. But really my point is this feels like more validation that AI doesn't provide increased ability, it provides existing (and demonstrated) ability faster with less formalized context. The "less formalized context" is what distinguishes it from programs/code.