3 ms·

I'd agree, I've been building a personal assistant (https://github.com/skorokithakis/stavrobot https://github.com/skorokithakis/stavrobot) and I'm amazed that,

by stavros 8mo ago

I'd agree, I've been building a personal assistant (https://github.com/skorokithakis/stavrobot https://github.com/skorokithakis/stavrobot) and I'm amazed that, for the first time ever, LLMs manage to build reliably, with much fewer bugs than I'd expect from a human, and without the repo devolving to unmaintainability after a few cycles. It's really amazing, we've crossed a threshold, and I don't know what that means for our jobs.

Grimblewald 8mo ago
No bugs means nothing if bugs get hidden and llms are great at hiding bugs and will absolutely fail to find some fairly critical ones. Your own repo, which is slop at best, fails to meet its core premise > Another AI agent. This one is awesome, though, and very secure. it isn't secure. It took me less than three minutes to find a vulnerability. Start engaging with your own code, it isn't as good as you think it is. edit: i had kimi "red team" it out of curiosity, it found the main critical vulnerability i did and several others Severity - Count - Categories Critical - 2 - SQL Injection, Path Traversal High - 4 - SSRF, Auth Bypass, Privilege Escalation, Secret Exposure Medium - 3 - DoS, Information Disclosure, Injection You need to sit down and really think about what people who do know what they're doing are saying. You're going to get yourself into deep trouble with this. I'm not a security specialist, i take a recreational interest in security, and llm's are by no means expert. A human with skill and intent would, i would gamble, be able fuck your shit up in a major way.
- reedf1 8mo ago
  Build a redteam into your feedback mechanism. Seriously. You've identified the problem and even solved it. Now automate it.
  - Grimblewald 8mo ago
    it sure can help, but it shouldn't be considered solved. Realistically you should treat it as vulnerable until it's received some attention from natural human folks who do know what they're doing. LLM's are great but they're not experts in any field as of yet.
- stavros 8mo ago
  Right yeah, thanks for the constructive comment. Mind filing those vulnerabilities, or are you just making a point? How do you know these are actual vulnerabilities? You just ran an LLM and it told you something and you came back to dunk on me, with zero context on the project. Maybe you need to sit down and really think that you have no idea who you're talking to or what the project does. Next time you make a "omg this code is so shit" comment, include something more than "well my LLM says your LLM is bad" so we can have a discussion with facts rather than LLM-aided trashtalk. EDIT: Out of curiosity, I've ran Kimi K2.5 on the codebase, and all the things it found are invalid, or explicit design decisions. So, next time you decide to tell someone their project "is slop" by running an LLM and relaying its verdict, consider a) the irony of what you're doing, and b) that the other person might know more than you about their own project that you spent "three minutes" running an LLM on.
  - Grimblewald 8mo ago
    The three minutes was my own time. There are thousands of projects like yours, why would i sink serious time into yours over another? I found one myself in three minutes, then ran kimi to see if it finds more. Someone being able to hijack your agent to run arbitrary code on your device seems like an odd design decision. The constructive feedbakc here is to take it seriously and engage with your codebase. Give a man a fish /Teach a man fish, you know?
    - stavros 8mo ago
      Yeah except you didn't actually find a vulnerability, you just searched for a pattern, found it, and thought "vulnerability" because you aren't a security researcher and don't realize that context is important. You should educate yourself more before you go around slandering people.
      - Grimblewald 8mo ago
        Attacker payload -> Message to bot -> LLM processes -> arbitrary execute_sql triggered This can be done via a signal/telegram message, via compromised plug-ins, via file uploads etc. At no stage does there appear to be, as far as I can tell, any attempt to mitigate or limit attack surface. That's just one of the problems. If that's by design, my bad, you do you king.

by stavros 8mo ago

Grimblewald 8mo ago

No bugs means nothing if bugs get hidden and llms are great at hiding bugs and will absolutely fail to find some fairly critical ones. Your own repo, which is slop at best, fails to meet its core premise > Another AI agent. This one is awesome, though, and very secure. it isn't secure. It took me less than three minutes to find a vulnerability. Start engaging with your own code, it isn't as good as you think it is. edit: i had kimi "red team" it out of curiosity, it found the main critical vulnerability i did and several others Severity - Count - Categories Critical - 2 - SQL Injection, Path Traversal High - 4 - SSRF, Auth Bypass, Privilege Escalation, Secret Exposure Medium - 3 - DoS, Information Disclosure, Injection You need to sit down and really think about what people who do know what they're doing are saying. You're going to get yourself into deep trouble with this. I'm not a security specialist, i take a recreational interest in security, and llm's are by no means expert. A human with skill and intent would, i would gamble, be able fuck your shit up in a major way.

reedf1 8mo ago
Build a redteam into your feedback mechanism. Seriously. You've identified the problem and even solved it. Now automate it.
- Grimblewald 8mo ago
  it sure can help, but it shouldn't be considered solved. Realistically you should treat it as vulnerable until it's received some attention from natural human folks who do know what they're doing. LLM's are great but they're not experts in any field as of yet.
stavros 8mo ago
Right yeah, thanks for the constructive comment. Mind filing those vulnerabilities, or are you just making a point? How do you know these are actual vulnerabilities? You just ran an LLM and it told you something and you came back to dunk on me, with zero context on the project. Maybe you need to sit down and really think that you have no idea who you're talking to or what the project does. Next time you make a "omg this code is so shit" comment, include something more than "well my LLM says your LLM is bad" so we can have a discussion with facts rather than LLM-aided trashtalk. EDIT: Out of curiosity, I've ran Kimi K2.5 on the codebase, and all the things it found are invalid, or explicit design decisions. So, next time you decide to tell someone their project "is slop" by running an LLM and relaying its verdict, consider a) the irony of what you're doing, and b) that the other person might know more than you about their own project that you spent "three minutes" running an LLM on.
- Grimblewald 8mo ago
  The three minutes was my own time. There are thousands of projects like yours, why would i sink serious time into yours over another? I found one myself in three minutes, then ran kimi to see if it finds more. Someone being able to hijack your agent to run arbitrary code on your device seems like an odd design decision. The constructive feedbakc here is to take it seriously and engage with your codebase. Give a man a fish /Teach a man fish, you know?
  - stavros 8mo ago
    Yeah except you didn't actually find a vulnerability, you just searched for a pattern, found it, and thought "vulnerability" because you aren't a security researcher and don't realize that context is important. You should educate yourself more before you go around slandering people.
    - Grimblewald 8mo ago
      Attacker payload -> Message to bot -> LLM processes -> arbitrary execute_sql triggered This can be done via a signal/telegram message, via compromised plug-ins, via file uploads etc. At no stage does there appear to be, as far as I can tell, any attempt to mitigate or limit attack surface. That's just one of the problems. If that's by design, my bad, you do you king.