4 ms·
DSLs like Semgrep are one of my top use-cases for LLMs generally. It used to be that tools like Semgrep and jq and Tree Sitter and zsh all required you to lear
by simonw 2y ago
DSLs like Semgrep are one of my top use-cases for LLMs generally.
It used to be that tools like Semgrep and jq and Tree Sitter and zsh all required you to learn quite a bit of syntax before you could start using them productively.
Thanks to LLMs you can focus on learning what they can do for you without also having to learn the fiddly syntax.
- Helmut10001 2y agoYes, the same applies to many niche syntaxes: Influx' flux language (I was able to Design my dream Grafana dashboards now!) or Auto Hot Key (AHK) for Windows automation are only two examples.
- phito 2y agoI love using LLMs to generate plantUML diagrams
- pletnes 2y agoWhat other tool do you use? I guess you want an svg or png output at some point?
- mdaniel 2y agoIf you mean "I have a .puml and need .svg or .png", the tool does that natively, as do most online versions (this one is their example https://www.plantuml.com/plantuml/uml/SyfFKj2rKt3CoKnELR1Io4ZDoSa70000 https://www.plantuml.com/plantuml/uml/SyfFKj2rKt3CoKnELR1Io4... and there are PNG and SVG links under the textarea). It also works offline https://plantuml.com/command-line#:~:text=Types%20of%20Output%20File https://plantuml.com/command-line#:~:text=Types%20of%20Outpu...
- pletnes 2y agoI didn’t know plantuml.com existed. Thanks!
- binary132 2y agoI had an absolutely terrible time today trying to get chatgpt to write me a very simple awk oneliner. It “thought” I was specifying a much more complicated requirement than I actually was.
- coder543 2y ago"ChatGPT" can mean many things... If you meant the free 4o-mini model, then yes, this outcome is not surprising, since it is bad at basically anything related to code. If you meant the more powerful o1 or o3-mini models (great naming, openai...), then that outcome would be surprising.
- s12mi 2y ago[flagged]
- rtpg 2y agoOver and over again I have witnessed people just drive themselves in circles downstream of a refusal to just step out of the “make the LLM fix the issue for me” loop. At one point the syntax and the specifics matter! * and + have different meanings in the regex. Overly-specified LLM output is worth trimming down to what your problem actually needed. I appreciate LLM output able to draft out sketches of results (and yes, a lot of the time, getting exactly the right result). And it’s great as a learning tool (especially if you’re diligent in the trust + verify department). But I worry that people are not taking opportunities to sit down and actually use the output as the sketch, and to insert the sort of precision that comes from the “infinite context” of the human working on the problem. Devs can’t just decide to opt out of getting into the details IMO
- scottlamb 2y ago> Over and over again I have witnessed people just drive themselves in circles downstream of a refusal to just step out of the “make the LLM fix the issue for me” loop. You're reminding me of a 1997 talk by an American Airlines trainer about automation dependency and when dropping down a level of automation reduces workload: <https://www.youtube.com/watch?v=WITLR_qSPXk https://www.youtube.com/watch?v=WITLR_qSPXk>
- rtpg 2y agocool talk, respect him making the point in 20 minutes. There's a lot of levels of applying this when programming, not just with LLM usage.
- PoignardAzur 2y ago> Over and over again I have witnessed people just drive themselves in circles downstream of a refusal to just step out of the “make the LLM fix the issue for me” loop. I think that's an amazing encapsulation of my experience with LLMs. They can be great tools, but you need to extremely willing to throw away their output and start over instead of trying to incrementally fix what they give you.
- gorgoiler 2y agoExactly the same for me: the major breakout success was having ChatGPT teach me how to use pandas. It really shone as an interactive manual with worked examples. Taking the training wheels off isn’t something I’ve really nailed though: for example, I keep coming back with the same questions about how to melt and pivot. I can self diagnose as this showing I didn’t really spend enough time understanding the answers the first time around.
- FridgeSeal 2y agoTo be fair, that also could be because the pandas API is…not great. When I used to write lots of python and pandas I basically always had the pandas docs open intuitive and memorable it is not.
- jjice 2y agoI kind of agree. I’ve had very mixed experiences with LLMs and DSLs. I was writing an NRQL query (New Relic’s log query language) and wanted to essentially do a GROUP BY date_trunc. It kept giving me options that I was eager for, and then the functions it gave me just didn’t exist. After like four back and forths of me telling it that the functions it was giving me didn’t exist - it worked. Then I needed it to split on the second forward slash of a string and just give me the first piece. It gave me the foundation to fill in the gaps of the function, but the LLM never got it. In that case, I assume it’s a lack of training data since NRQL is pretty niche. I catch myself swinging from “holy shit this is impressive” to “wow this sucks” and back regularly for code.
- darkwater 2y agoThis is similar to my experience with LLMs and DSLs. They tend to hallucinate functions that will magically work in the situation you are describing. My pet theory here is that they are fooled by many forum posts/issues "why doesn't a function exist in this DSLs called ABC that does this?"
- mycall 2y agoDid you try asking the LLM to writ the missing functions? It might work in some cases.
- obiefernandez 2y agoIn that case putting NRQL reference into context would work wonders
- jjice 2y agoOh goodness I’m a fool to not have tried that. I’ll absolutely give that a go next time!
- michaelcampbell 2y agoNRQL's abandonment of GROUP BY for FACET always throws me.