6 ms·

Did Semgrep Just Get a Lot More Interesting?

190 points by ghuntley 2y ago

deleted 2y ago
[deleted]
fizx 2y ago
I'd put a $1k long bet that a 3B param model, cleverly orchestrated, will achieve AGI* in the next ten years. These are the sorts of ideas that would help get us there. Any takers? *AGI defined as smarter than a FAANG staff engineer on similar tasks.
- chubot 2y ago
  I liked the original article - since I've looked at semgrep, and I'm also hoping "closing the loop" can fix some of the downsides of LLMs I'm also willing to bet money, and I'd even thought of a challenge for 10x or 20x that amount But if you want to bet, then you have to have something well-defined and interesting to bet on: - leave out the term "AGI" - this only confuses things, because everyone has a different definition of it. Just say what the problem is, precisely - leave out "FAANG staff engineer". Because computers are already better than staff engineers on dozens and dozens of tasks, like adding two 32 bit numbers, or compiling C++ code, or running Python code. Not to mention certain things involving LLMs. i.e. it's extremely obvious that LLMs are better at engineers at certain things -- the ones they choose to use LLMs for. That doesn't mean LLMs can replace them, which is often what people mean by "AGI".
sho_hn 2y ago
> But I’m burying the lead. It's "lede". There's a few other typos too. I'm not sure I like the "This one trick they don't want you to know about!" writing style of these (e.g. the Cursor/malpractice hot take, that sort of thing).
- tptacek 2y ago
  It's "lede" if you're showing off or writing for a publication, where they deliberately search for "lede" and "hed" and "tk", which exist specifically to stick out from ordinary text.
  - dqv 2y ago
    I love how fickle HN is. This is the etymological equivalent of saying "it's expresso, not espresso" (espresso -> expresso, lead -> lede both happened in the 50s) and yet claiming that would be highly controversial.
- mulmboy 2y ago
  I like it! It's provocative and opinionated, just what I want out of a blog. Blogs are casual, personal, subjective. I read a lot of blogs like this and appreciate being able to take in many broad opinions, many of which yeah I think are dumb, but I like to have them all bounce around my head. Gives me a window into how people are thinking across the community
  - lucianbr 2y ago
    Oh yeah, the provocative opinion "If you're not using AI you're missing out. It is the future!" Never heard that one before.
neom 2y ago
Love the illustrator. And love linking out and supporting her.
- tptacek 2y ago
  That's Annie Ruygt. She predates me at Fly.io.
  - neom 2y ago
    Yeah I just spent some time going through the blog and looking at them all, she's really great. Need art.fly.io plz, would love a spot to look at them all together. :)
- esafak 2y ago
  In the age of generative AI, handmade art stands out all the more.
ignoramous 2y ago
r2c / semgrep has truly come a long way since its incubation at Facebook: https://github.com/facebookarchive/pfff https://github.com/facebookarchive/pfff Remember using soot, kythe.io, & pfff to find the exact CTS (compatibility test suite) tests to run given code diff between two AOSP builds.
mmsc 2y ago
I've been trying to do something similar to create CodeQL queries recently, and found that chatgpt is completely unable to create even simple queries. I assume it's because training is based on old query language or just completely missing, but being able to feed the rules and the errors which they produce when run has been a complete failure for me.
zamalek 2y ago
So the idea is that LLM1 looks at the output of LLM0 and builds a new set of constraints, and then LLM0 has to try again, rinse and repeat? (LLM0 could be the same as LLM1, and I think it is in the article?)
simonw 2y ago
DSLs like Semgrep are one of my top use-cases for LLMs generally. It used to be that tools like Semgrep and jq and Tree Sitter and zsh all required you to learn quite a bit of syntax before you could start using them productively. Thanks to LLMs you can focus on learning what they can do for you without also having to learn the fiddly syntax.
- Helmut10001 2y ago
  Yes, the same applies to many niche syntaxes: Influx' flux language (I was able to Design my dream Grafana dashboards now!) or Auto Hot Key (AHK) for Windows automation are only two examples.
  - phito 2y ago
    I love using LLMs to generate plantUML diagrams
    - pletnes 2y ago
      What other tool do you use? I guess you want an svg or png output at some point?
      - mdaniel 2y ago
        If you mean "I have a .puml and need .svg or .png", the tool does that natively, as do most online versions (this one is their example https://www.plantuml.com/plantuml/uml/SyfFKj2rKt3CoKnELR1Io4ZDoSa70000 https://www.plantuml.com/plantuml/uml/SyfFKj2rKt3CoKnELR1Io4... and there are PNG and SVG links under the textarea). It also works offline https://plantuml.com/command-line#:~:text=Types%20of%20Output%20File https://plantuml.com/command-line#:~:text=Types%20of%20Outpu...
        pletnes 2y ago
        I didn’t know plantuml.com existed. Thanks!
waynenilsen 2y ago
That's Devin / replit agent Not there yet but it is inevitable
kubb 2y ago
OK, hear me out. The future isn’t o4 or whatever. The future is when everyone, every language, every tool, every single library and codebase can train their own custom model tailored to their needs and acting as a smart documentation which you can tell what you want to do and it will tell you how to do it. People have been trying with fine tuning, RAG, using the context window. That’s not enough. The model needs to be trained on countless examples of question-answer for this particular area of knowledge starting from a base model aware of comp sci concepts and language (just English is fine). This implies that such examples have to be created by humans - each such community will need its own „Stack Overflow”. Smaller, specialized models are the future of productivity. But of course that can’t be monetized, right? Well, the technology just needs to get cheaper so that people can just afford to train such models themselves. That’s the next major breakthrough. Could be anyway.
skirge 2y ago
"Generate patterns for language X and framework Y which can lead to vulnerability V, generate Semgrep/Joern rule for it" longest chats with ChatGPT.
ksec 2y ago
LOL but i cant help but think about the comment from tptacek [1]. >"We wrote all sorts of stuff this week and this is what gets to the front page. :P" And how they write content specifically for HN [2]. [1] https://news.ycombinator.com/item?id=43053985 https://news.ycombinator.com/item?id=43053985 [2] https://fly.io/blog/a-blog-if-kept/ https://fly.io/blog/a-blog-if-kept/
eitland 2y ago
I am reminded if this IMO timeless classic: https://news.ycombinator.com/item?id=5397797 https://news.ycombinator.com/item?id=5397797 A short snippet (the whole thing is very funny and interestingly written in 2013 long before the modern ai craze): By now I had started moving on to doing my own consulting work, but I never disabled the hill-climbing algorithm. I'd closed and forgotten about the Amazon account, had no idea what the password to the free vps was anymore, and simply appreciated the free money. But there was a time bomb. That hill climbing algorithm would fudge variables left and right. To avoid local maxima, it would sometimes try something very different. One day it decided to stop paying me. Its reviews did not suffer. It's balance increased. So it said, great change, let's keep it. It now has over $28,000 of my money, is not answering my mail, and we have been locked in an equity battle over the past 18 months. The worst part is that I still have to clean up all its answers to protect our reputation. Who's running who anyway?
gorgoiler 2y ago
How are people collaborating on code when using AI tools to generate patches? We hold code review dear as a tool to make sure more than one set of eyeballs has been over a change before it goes into production, and more than one person has the context behind the code to be able to fix it in future. As model generated code becomes the norm I’m seeing code from junior engineers that they haven’t read and possible nor do they understand. For example, one Python script calling another using exec instead of importing it as a module, or writing code that is already available as a very common part of the standard library. In such cases, are we asking people to mark their code as auto generated? Should we review their prompts instead of the code? Should we require the prompt to code step be deterministic? Should we see their entire prompt context and not just the prompt they used to build the finished patch? I feel like a lot of the value of code review is to bring junior engineers up to higher levels. To that extent each review feels like an end of week school test, and I’m getting handed plagiarised AI slop to mark instead of something that maps properly to what the student does or does not know. Pair programming is another great teaching tool. Soon, it might be the only one left.
- nicoburns 2y ago
  That level of submission sounds like a "reject on sight, go back and do it again without Ai, if you need help then ask" to me.
  - silvestrov 2y ago
    Then you get pushback from management: we don't have time for the perfect solution, if it works, it's good.
    - scottlamb 2y ago
      Pathologically short-term thinking from management is a problem that far predates AI (computers, even), and the solution hasn't changed: go somewhere better and watch the old place burn.
      - 2y ago
miki123211 2y ago
I think an even more interesting use case for semgrep, and also LSP or something like LSP, is querying for exactly what an AI needs to know to fix a specific problem. Unlike humans, LLMs have no memory, so they can't just learn where things are in the code by remembering the work they did in the past. In a way, they need to re-learn the relevant parts of your codebase from scratch on every change, always keeping context window limitations in mind. Humans learn by scrolling and clicking around and remembering what's important to them; LLMs can't do that. We try to give them autogenerated codebase maps and tools that can inject specific files into the context window, but that doesn't seem to be nearly enough. Semantic queries look like a much better idea. I thought you couldn't really teach an LLM how to use something like that effectively, as that's not how humans work and there's no data to train on, but the recent breakthroughs with RL made me change my mind.
- mycall 2y ago
  > LLMs have no memory, so they can't just learn where things are in the code by remembering the work they did in the past. Isn't this what Google Titan is trying to achieve?
technion 2y ago
I have to ask if this semgreo rule for relock bugs is public, because the first google hit for me is this blog.
xg15 2y ago
> What interests me is this: it seems obvious that we’re going to do more and more “closed-loop” LLM agent code generation stuff. By “closed loop”, I mean that the thingy that generates code is going to get to run the code and watch what happens when it’s interacted with. Well, at least we have a credible pathway into the Terminator or Matrix universes now...
antirez 2y ago
Take a large context frontier model. Upload 200k tokens of code for each query. Ask about what code pattern you want it to highlight for you. Works better than any other system, but costs token on API services.
0x696C6961 2y ago
I wrote a tool for rewriting semgrep matches using an LLM https://github.com/icholy/semgrepx https://github.com/icholy/semgrepx
scottlamb 2y ago
> But I just checked and, unsurprisingly, 4o seems to do reasonably well at generating Semgrep rules? Like: I have no idea if this rule is actually any good. But it looks like a Semgrep rule? I don't know about semgrep syntax, but the chat it generated is bad in at least a couple other ways. E.g. their "how to fix" instruction is wrong: if let Some(Load::Local(load)) = self.load.read().get(...) { // do a bunch of stuff with `load` } else { drop(self.load.read()); // Explicitly drop before taking write lock let mut w = self.load.write(); self.init_for(&w); } That actually acquires and then drops a second read lock. It doesn't solve the problem that the first read lock is still active and thus the write lock will deadlock. Speaking of which, acquiring two read locks from the same thread can also deadlock, as shown in the "Potential deadlock example" at <https://doc.rust-lang.org/std/sync/struct.RwLock.html https://doc.rust-lang.org/std/sync/struct.RwLock.html>. It can happen in the code above (one line before the other deadlock). It can also slip through their rule because they're incorrectly looking for just a write lock in the else block. I've been playing with AI code generation tools like everyone else, and they are okay as autocomplete, but I don't see them as trustworthy. For a while I thought I just wasn't prompting well enough, but when other people show me their AI output, I can see it's wrong, so maybe I'm just looking more closely?
mcqueenjordan 2y ago
> But I just checked and, unsurprisingly, 4o seems to do reasonably well at generating Semgrep rules? Like: I have no idea if this rule is actually any good. But it looks like a Semgrep rule? This is the thing with LLMs. When you’re not an expert, the output always looks incredible. It’s similar to the fluency paradox — if you’re not native in a language, anyone you hear speak it at a higher level than yourself appears to be fluent to you. Even if for example they’re actually just a beginner. The problem with LLMs is that they’re very good at appearing to speak “a language” at a higher level than you, even if they totally aren’t.
- tptacek 2y ago
  Hold on, hold on. You're missing a step here. I agree completely that an LLM's first attempt to write a Semgrep rule is likely as not to be horseshit. That's true of everything an LLM generates. But I'm talking about closed-loop LLM code generation. Unlike legal arguments and medical diagnoses, you can hook an LLM up to an execution environment and let it see what happens when the code it generates runs. It then iterates, until it has something that works. Which, when you think about it, is how a lot of human-generated code gets written too. So my thesis here does not depend on LLMs getting things right the first time, or without assistance.
  - danielbln 2y ago
    That's also the problem with these conversations. Some people evaluate zero-shot promoted code oozing out of gpt-3.5, others plug Sonnet into an IDE with access to terminal, LSP, diagnostics etc. crunching through a problem in an agentic self improvement loop. Those two approaches will generate very different quality levels of code.
  - bambax 2y ago
    The problem is what one means by "works". Is it just that it runs without triggering exceptions here and there? One has to know, and understand, what the code is supposed to be doing, to evaluate it. Or use tests. But LLMs love to lie so they can't be trusted to write the tests, or even to report how the code they wrote passed the tests. In my experience the way to use LLMs for coding is exactly the opposite: the user should already have very good knowledge of the problem domain as well as the language used, and just needs to have a conversation with someone on how to approach a specific implementation detail (or help with an obscure syntax quirk). Then LLMs can be very useful. But having them directly output code for things one doesn't know, in a language one doesn't know either, hoping they will magically solve the problem by iterating in "closed loops", will result in chaos.
spamfilter247 2y ago
I’ve built something for a solution that takes you most of the way there, using Semgrep’s SARIF output and prompted LLMs to help prioritize triage. We’ve used this for the past year at Microsoft to help prioritize the “most likely interesting” 5% of a large set of results for human triage. It works quite well… https://github.com/247arjun/ai-secure-code-review https://github.com/247arjun/ai-secure-code-review
bhouston 2y ago
I have a closed loop coding agent working here, you can try it out: https://mycoder.ai https://mycoder.ai
awinter-py 2y ago
'closed loop' concept in here is important the point that a unit of code is a thing that is maintained, rather than a thing that is generated once, is where codegen has always lost me (both AI codegen and ruby-on-rails boilerplate generators) iterative improvement, including factoring useful things out to standard libraries, is where it's at
timewizard 2y ago
> makes me think that more of the future of our field belongs to people who figure out how to use this weird bags of model weights than any of us are comfortable with. Until you find a way to improve self guided training, no, this will never happen. New things get invented and need to be implemented before your "bag of weights" has any idea how to approach it, which is, of course, by simply stealing something that already existed. People who think this way blow my mind. Is it that you don't actually like your day job and dream about having a machine do it for you while, somehow, still earning the salary you currently command? Laughable.
hamilyon2 2y ago
I just tried my latest task with it and o1 readily hallucinated non-existent semgrep functions.
j45 2y ago
I think the author is missing one part about cursor, aider, etc. Out of the box it is decent. Watching only the basic optimizations on YouTube developers are doing proper to starting a project puts the experience and consistency to a far higher level Maybe this casual surface testing if I’m not Mia reading is why so many tech people are missing what tools like cursor, aider, etc are doing.
jasonjmcghee 2y ago
I'm quite surprised "autofix" functionality wasn't mentioned. https://semgrep.dev/docs/writing-rules/autofix https://semgrep.dev/docs/writing-rules/autofix Seems like the natural thing to do for cases that support it.

190 points by ghuntley 2y ago

deleted 2y ago

[deleted]

fizx 2y ago

I'd put a $1k long bet that a 3B param model, cleverly orchestrated, will achieve AGI* in the next ten years. These are the sorts of ideas that would help get us there. Any takers? *AGI defined as smarter than a FAANG staff engineer on similar tasks.

chubot 2y ago
I liked the original article - since I've looked at semgrep, and I'm also hoping "closing the loop" can fix some of the downsides of LLMs I'm also willing to bet money, and I'd even thought of a challenge for 10x or 20x that amount But if you want to bet, then you have to have something well-defined and interesting to bet on: - leave out the term "AGI" - this only confuses things, because everyone has a different definition of it. Just say what the problem is, precisely - leave out "FAANG staff engineer". Because computers are already better than staff engineers on dozens and dozens of tasks, like adding two 32 bit numbers, or compiling C++ code, or running Python code. Not to mention certain things involving LLMs. i.e. it's extremely obvious that LLMs are better at engineers at certain things -- the ones they choose to use LLMs for. That doesn't mean LLMs can replace them, which is often what people mean by "AGI".

sho_hn 2y ago

> But I’m burying the lead. It's "lede". There's a few other typos too. I'm not sure I like the "This one trick they don't want you to know about!" writing style of these (e.g. the Cursor/malpractice hot take, that sort of thing).

tptacek 2y ago
It's "lede" if you're showing off or writing for a publication, where they deliberately search for "lede" and "hed" and "tk", which exist specifically to stick out from ordinary text.
- dqv 2y ago
  I love how fickle HN is. This is the etymological equivalent of saying "it's expresso, not espresso" (espresso -> expresso, lead -> lede both happened in the 50s) and yet claiming that would be highly controversial.
mulmboy 2y ago
I like it! It's provocative and opinionated, just what I want out of a blog. Blogs are casual, personal, subjective. I read a lot of blogs like this and appreciate being able to take in many broad opinions, many of which yeah I think are dumb, but I like to have them all bounce around my head. Gives me a window into how people are thinking across the community
- lucianbr 2y ago
  Oh yeah, the provocative opinion "If you're not using AI you're missing out. It is the future!" Never heard that one before.

neom 2y ago

Love the illustrator. And love linking out and supporting her.

tptacek 2y ago
That's Annie Ruygt. She predates me at Fly.io.
- neom 2y ago
  Yeah I just spent some time going through the blog and looking at them all, she's really great. Need art.fly.io plz, would love a spot to look at them all together. :)
esafak 2y ago
In the age of generative AI, handmade art stands out all the more.

ignoramous 2y ago

r2c / semgrep has truly come a long way since its incubation at Facebook: https://github.com/facebookarchive/pfff https://github.com/facebookarchive/pfff Remember using soot, kythe.io, & pfff to find the exact CTS (compatibility test suite) tests to run given code diff between two AOSP builds.

mmsc 2y ago

I've been trying to do something similar to create CodeQL queries recently, and found that chatgpt is completely unable to create even simple queries. I assume it's because training is based on old query language or just completely missing, but being able to feed the rules and the errors which they produce when run has been a complete failure for me.

zamalek 2y ago

So the idea is that LLM1 looks at the output of LLM0 and builds a new set of constraints, and then LLM0 has to try again, rinse and repeat? (LLM0 could be the same as LLM1, and I think it is in the article?)

simonw 2y ago

DSLs like Semgrep are one of my top use-cases for LLMs generally. It used to be that tools like Semgrep and jq and Tree Sitter and zsh all required you to learn quite a bit of syntax before you could start using them productively. Thanks to LLMs you can focus on learning what they can do for you without also having to learn the fiddly syntax.

Helmut10001 2y ago
Yes, the same applies to many niche syntaxes: Influx' flux language (I was able to Design my dream Grafana dashboards now!) or Auto Hot Key (AHK) for Windows automation are only two examples.
- phito 2y ago
  I love using LLMs to generate plantUML diagrams
  - pletnes 2y ago
    What other tool do you use? I guess you want an svg or png output at some point?
    - mdaniel 2y ago
      If you mean "I have a .puml and need .svg or .png", the tool does that natively, as do most online versions (this one is their example https://www.plantuml.com/plantuml/uml/SyfFKj2rKt3CoKnELR1Io4ZDoSa70000 https://www.plantuml.com/plantuml/uml/SyfFKj2rKt3CoKnELR1Io4... and there are PNG and SVG links under the textarea). It also works offline https://plantuml.com/command-line#:~:text=Types%20of%20Output%20File https://plantuml.com/command-line#:~:text=Types%20of%20Outpu...
      - pletnes 2y ago
        I didn’t know plantuml.com existed. Thanks!

waynenilsen 2y ago

That's Devin / replit agent Not there yet but it is inevitable

kubb 2y ago

OK, hear me out. The future isn’t o4 or whatever. The future is when everyone, every language, every tool, every single library and codebase can train their own custom model tailored to their needs and acting as a smart documentation which you can tell what you want to do and it will tell you how to do it. People have been trying with fine tuning, RAG, using the context window. That’s not enough. The model needs to be trained on countless examples of question-answer for this particular area of knowledge starting from a base model aware of comp sci concepts and language (just English is fine). This implies that such examples have to be created by humans - each such community will need its own „Stack Overflow”. Smaller, specialized models are the future of productivity. But of course that can’t be monetized, right? Well, the technology just needs to get cheaper so that people can just afford to train such models themselves. That’s the next major breakthrough. Could be anyway.

skirge 2y ago

"Generate patterns for language X and framework Y which can lead to vulnerability V, generate Semgrep/Joern rule for it" longest chats with ChatGPT.

ksec 2y ago

LOL but i cant help but think about the comment from tptacek [1]. >"We wrote all sorts of stuff this week and this is what gets to the front page. :P" And how they write content specifically for HN [2]. [1] https://news.ycombinator.com/item?id=43053985 https://news.ycombinator.com/item?id=43053985 [2] https://fly.io/blog/a-blog-if-kept/ https://fly.io/blog/a-blog-if-kept/

eitland 2y ago

I am reminded if this IMO timeless classic: https://news.ycombinator.com/item?id=5397797 https://news.ycombinator.com/item?id=5397797 A short snippet (the whole thing is very funny and interestingly written in 2013 long before the modern ai craze): By now I had started moving on to doing my own consulting work, but I never disabled the hill-climbing algorithm. I'd closed and forgotten about the Amazon account, had no idea what the password to the free vps was anymore, and simply appreciated the free money. But there was a time bomb. That hill climbing algorithm would fudge variables left and right. To avoid local maxima, it would sometimes try something very different. One day it decided to stop paying me. Its reviews did not suffer. It's balance increased. So it said, great change, let's keep it. It now has over $28,000 of my money, is not answering my mail, and we have been locked in an equity battle over the past 18 months. The worst part is that I still have to clean up all its answers to protect our reputation. Who's running who anyway?

gorgoiler 2y ago

How are people collaborating on code when using AI tools to generate patches? We hold code review dear as a tool to make sure more than one set of eyeballs has been over a change before it goes into production, and more than one person has the context behind the code to be able to fix it in future. As model generated code becomes the norm I’m seeing code from junior engineers that they haven’t read and possible nor do they understand. For example, one Python script calling another using exec instead of importing it as a module, or writing code that is already available as a very common part of the standard library. In such cases, are we asking people to mark their code as auto generated? Should we review their prompts instead of the code? Should we require the prompt to code step be deterministic? Should we see their entire prompt context and not just the prompt they used to build the finished patch? I feel like a lot of the value of code review is to bring junior engineers up to higher levels. To that extent each review feels like an end of week school test, and I’m getting handed plagiarised AI slop to mark instead of something that maps properly to what the student does or does not know. Pair programming is another great teaching tool. Soon, it might be the only one left.

nicoburns 2y ago
That level of submission sounds like a "reject on sight, go back and do it again without Ai, if you need help then ask" to me.
- silvestrov 2y ago
  Then you get pushback from management: we don't have time for the perfect solution, if it works, it's good.
  - scottlamb 2y ago
    Pathologically short-term thinking from management is a problem that far predates AI (computers, even), and the solution hasn't changed: go somewhere better and watch the old place burn.
    - 2y ago

miki123211 2y ago

I think an even more interesting use case for semgrep, and also LSP or something like LSP, is querying for exactly what an AI needs to know to fix a specific problem. Unlike humans, LLMs have no memory, so they can't just learn where things are in the code by remembering the work they did in the past. In a way, they need to re-learn the relevant parts of your codebase from scratch on every change, always keeping context window limitations in mind. Humans learn by scrolling and clicking around and remembering what's important to them; LLMs can't do that. We try to give them autogenerated codebase maps and tools that can inject specific files into the context window, but that doesn't seem to be nearly enough. Semantic queries look like a much better idea. I thought you couldn't really teach an LLM how to use something like that effectively, as that's not how humans work and there's no data to train on, but the recent breakthroughs with RL made me change my mind.

mycall 2y ago
> LLMs have no memory, so they can't just learn where things are in the code by remembering the work they did in the past. Isn't this what Google Titan is trying to achieve?

technion 2y ago

I have to ask if this semgreo rule for relock bugs is public, because the first google hit for me is this blog.

xg15 2y ago

> What interests me is this: it seems obvious that we’re going to do more and more “closed-loop” LLM agent code generation stuff. By “closed loop”, I mean that the thingy that generates code is going to get to run the code and watch what happens when it’s interacted with. Well, at least we have a credible pathway into the Terminator or Matrix universes now...

antirez 2y ago

Take a large context frontier model. Upload 200k tokens of code for each query. Ask about what code pattern you want it to highlight for you. Works better than any other system, but costs token on API services.

0x696C6961 2y ago

I wrote a tool for rewriting semgrep matches using an LLM https://github.com/icholy/semgrepx https://github.com/icholy/semgrepx

scottlamb 2y ago

> But I just checked and, unsurprisingly, 4o seems to do reasonably well at generating Semgrep rules? Like: I have no idea if this rule is actually any good. But it looks like a Semgrep rule? I don't know about semgrep syntax, but the chat it generated is bad in at least a couple other ways. E.g. their "how to fix" instruction is wrong: if let Some(Load::Local(load)) = self.load.read().get(...) { // do a bunch of stuff with `load` } else { drop(self.load.read()); // Explicitly drop before taking write lock let mut w = self.load.write(); self.init_for(&w); } That actually acquires and then drops a second read lock. It doesn't solve the problem that the first read lock is still active and thus the write lock will deadlock. Speaking of which, acquiring two read locks from the same thread can also deadlock, as shown in the "Potential deadlock example" at <https://doc.rust-lang.org/std/sync/struct.RwLock.html https://doc.rust-lang.org/std/sync/struct.RwLock.html>. It can happen in the code above (one line before the other deadlock). It can also slip through their rule because they're incorrectly looking for just a write lock in the else block. I've been playing with AI code generation tools like everyone else, and they are okay as autocomplete, but I don't see them as trustworthy. For a while I thought I just wasn't prompting well enough, but when other people show me their AI output, I can see it's wrong, so maybe I'm just looking more closely?

mcqueenjordan 2y ago

> But I just checked and, unsurprisingly, 4o seems to do reasonably well at generating Semgrep rules? Like: I have no idea if this rule is actually any good. But it looks like a Semgrep rule? This is the thing with LLMs. When you’re not an expert, the output always looks incredible. It’s similar to the fluency paradox — if you’re not native in a language, anyone you hear speak it at a higher level than yourself appears to be fluent to you. Even if for example they’re actually just a beginner. The problem with LLMs is that they’re very good at appearing to speak “a language” at a higher level than you, even if they totally aren’t.

tptacek 2y ago
Hold on, hold on. You're missing a step here. I agree completely that an LLM's first attempt to write a Semgrep rule is likely as not to be horseshit. That's true of everything an LLM generates. But I'm talking about closed-loop LLM code generation. Unlike legal arguments and medical diagnoses, you can hook an LLM up to an execution environment and let it see what happens when the code it generates runs. It then iterates, until it has something that works. Which, when you think about it, is how a lot of human-generated code gets written too. So my thesis here does not depend on LLMs getting things right the first time, or without assistance.
- danielbln 2y ago
  That's also the problem with these conversations. Some people evaluate zero-shot promoted code oozing out of gpt-3.5, others plug Sonnet into an IDE with access to terminal, LSP, diagnostics etc. crunching through a problem in an agentic self improvement loop. Those two approaches will generate very different quality levels of code.
- bambax 2y ago
  The problem is what one means by "works". Is it just that it runs without triggering exceptions here and there? One has to know, and understand, what the code is supposed to be doing, to evaluate it. Or use tests. But LLMs love to lie so they can't be trusted to write the tests, or even to report how the code they wrote passed the tests. In my experience the way to use LLMs for coding is exactly the opposite: the user should already have very good knowledge of the problem domain as well as the language used, and just needs to have a conversation with someone on how to approach a specific implementation detail (or help with an obscure syntax quirk). Then LLMs can be very useful. But having them directly output code for things one doesn't know, in a language one doesn't know either, hoping they will magically solve the problem by iterating in "closed loops", will result in chaos.

spamfilter247 2y ago

I’ve built something for a solution that takes you most of the way there, using Semgrep’s SARIF output and prompted LLMs to help prioritize triage. We’ve used this for the past year at Microsoft to help prioritize the “most likely interesting” 5% of a large set of results for human triage. It works quite well… https://github.com/247arjun/ai-secure-code-review https://github.com/247arjun/ai-secure-code-review

bhouston 2y ago

I have a closed loop coding agent working here, you can try it out: https://mycoder.ai https://mycoder.ai

awinter-py 2y ago

'closed loop' concept in here is important the point that a unit of code is a thing that is maintained, rather than a thing that is generated once, is where codegen has always lost me (both AI codegen and ruby-on-rails boilerplate generators) iterative improvement, including factoring useful things out to standard libraries, is where it's at

timewizard 2y ago

> makes me think that more of the future of our field belongs to people who figure out how to use this weird bags of model weights than any of us are comfortable with. Until you find a way to improve self guided training, no, this will never happen. New things get invented and need to be implemented before your "bag of weights" has any idea how to approach it, which is, of course, by simply stealing something that already existed. People who think this way blow my mind. Is it that you don't actually like your day job and dream about having a machine do it for you while, somehow, still earning the salary you currently command? Laughable.

hamilyon2 2y ago

I just tried my latest task with it and o1 readily hallucinated non-existent semgrep functions.

j45 2y ago

I think the author is missing one part about cursor, aider, etc. Out of the box it is decent. Watching only the basic optimizations on YouTube developers are doing proper to starting a project puts the experience and consistency to a far higher level Maybe this casual surface testing if I’m not Mia reading is why so many tech people are missing what tools like cursor, aider, etc are doing.

jasonjmcghee 2y ago

I'm quite surprised "autofix" functionality wasn't mentioned. https://semgrep.dev/docs/writing-rules/autofix https://semgrep.dev/docs/writing-rules/autofix Seems like the natural thing to do for cases that support it.