7 ms·

Yeah, with a budget assigned. This is actually just software development and security right? Developers create software, which has bugs. Users (including bad

by ygjb 5mo ago

Yeah, with a budget assigned. This is actually just software development and security right? Developers create software, which has bugs. Users (including bad guys, pen testers, QA folks, automated scans etc, etc, etc) find bugs, including security bugs, Developers fix bugs and maybe make more. It's an OODA loop, and continues until the developers decide to stop supporting the software. Whether that fits into the business model, or the value proposition of spending tokens instead of engineer hours or user hours is fundamentally a risk management decision and whether or not the developer (whether OSS contributor, employee, business owner, etc) wants to invest their resources into maintaining the project. While not evenly distributed, and not perfect, the currently available and behind embargoed tools are absolutely impactful, and yes, they are expensive to operate right now - it may not always be the case, but the "Attacks always get better" adage applies here. The models will get cheaper to run, and if you don't want to pay for engineers or reward volunteers to do the work, then you've got to pay for tokens, or spend some other resource to get the work done.

sandeepkd 5mo ago
Somehow this reminded me of the historical efforts of some government bounty collections for mouse tails which were discontinued due to fraud (such as hunters breeding mice to collect the reward). There is a reason why/how devs and QA keep each other in check. Guess in case of LLM writing code, one has to use different models for dev and security checks. On other hand, in real world, the developers learn from mistakes and avoid them in the future. However there is no feedback loop with enterprises using LLM with the agreement that the LLM would not use the enterprise code for training purposes
- ygjb 5mo ago
  > the developers learn from mistakes and avoid them in the future No. Humans learn from mistakes and try to avoid them in the future, but there is a whole pile of other stuff in the bag of neurons between our ears that prevent us from avoiding repetition of errors. I have seen extremely talented engineers write trivial to avoid memory corruption bugs because they were thinking about the problem they were trying to solve, and not the pitfalls they could fall into. I would argue that the vast majority of software defects in released code are written by people that know better, but the bug introduced was orthogonal to the problem they were trying to solve, or was for an edge case that was not considered in the requirements. Unless you are writing a software component specifically to be resilient against memory corruption, preventing memory corruption issues aren't top of mind when writing code, and that is ok since humans, like the machines we build, have a limit to the amount of context/content/problem space that we can hold and evaluate at once. Separately, you don't necessarily need to use different models to generate code vs conduct security checks, but you should be using different prompts, steering, specs, skills and agents for the two tasks because of how the model and agents interpret the instructions given.
  - mncharity 5mo ago
    > write trivial to avoid memory corruption bugs because they were thinking about [something else] [...] defects [...] written by people that know better, but the bug introduced was orthogonal to [their focus] For whatever reason, hadn't associated the inattentional blindness of bug writing with the invisible gorilla experiment and car crashes - selective attention fails. People looking right at the gorilla strolling into production while chest thumping, but not seeing it, for a focus on passing basketballs. That's quite an image. Tnx.
    - Dilettante_ 5mo ago
      Thank you in turn for making the issue much more salient to me by explicitly connecting it to the gorilla/basketball experiment. This is definitely going into my "clippings".
    - meowface 5mo ago
      I've noticed even people who do offensive security for a living frequently leave gaping holes in their own code. If you're not actively primed to scan the landscape for the gorilla, you will often miss it even if you're a gorilla inquisitor.
    - cratermoon 5mo ago
      And it so happens that the flood of code coming out of tools like Claude is almost certain to bring a whole zoo invisible gorillas along for the bananas. The kinds of bugs that people already struggle to catch because of inattentional blindness flow like a torrent in code generated by these tools. Did you notice the missing word in the previous paragraph?
  - Forgeties79 5mo ago
    The problem is you as a person are not incentivized to introduce bugs in your code. If I am a company that provide provides an LLM/agent, and I know that the more bugs you have the more money I’m going to make, then I am not exactly incentivized to make my LLM/Agent better at preventing bugs. I don’t even have to explicitly make it introduce them. The incentive structure is simply out of whack.
- noxvilleza 5mo ago
  Are you thinking of the cobra effect (aka https://en.wikipedia.org/wiki/Perverse_incentive https://en.wikipedia.org/wiki/Perverse_incentive) where people in India started breeding cobras to get the reward?
  - itishappy 5mo ago
    Plenty of examples abound: https://en.wikipedia.org/wiki/Great_Hanoi_Rat_Massacre https://en.wikipedia.org/wiki/Great_Hanoi_Rat_Massacre > Today, the events are often used as an example of a perverse incentive, commonly referred to as the cobra effect. The modern discoverer of this event, American historian Michael G. Vann argues that the cobra example from the British Raj cannot be proven, but that the rats in the Vietnam case can be proven, so the term should be changed to the Rat Effect.
- ozim 5mo ago
  You apparently have not much experience developing software.
- vlunkr 5mo ago
  Great analogy. The problem is the incentive structure. Anthropic would nothing nothing more than for all of us to write big sprawling slop codebases so we can spend endless tokens reading, rereading, fixing, refixing forever.
- joquarky 5mo ago
  You don't need different models, just different contexts (optimally with different personas).
- Stubbs 5mo ago
  Reminds me of the contracts we sign with off-shore development companies to write the software at one rate and then fix bugs at a higher rate. Won’t be long till tokens spent on security review agents cost more than the tokens to create the bugs in the first place.
oytis 5mo ago
It's pretty absurd to do it on AI-generated code though. If there is now an automated way to find vulnerabilities, coding models can be pretty easily trained to not introduce them
- scrollaway 5mo ago
  Tell me you don’t know how AI works without telling me you don’t know how AI works.
  - amazingamazing 5mo ago
    What are you talking about?
    - ashdksnndck 5mo ago
      I’ll try to steelman this comment. Anyone who uses coding tools knows that the output is heavily affected by details of the task you give it. The same model can give you garbage code or genius code for the same problem with slightly different framing. So it’s not necessarily a limitation in the model’s training that causes it to output security bugs. The model might be great at writing secure code, but you need a different harness to elicit that behavior. Counterargument: just because the problem can be fixed without training, doesn’t mean training isn’t a possible solution.
      - SecretDreams 5mo ago
        In every prompt: "write me code without exploitable bugs". I know it doesn't work so easily as someone who uses AI for coding, but I do find repetition of basics in almost every prompt keeps the AI focused.
      - scrollaway 5mo ago
        I guess I thought this should be obvious to everyone but, looking at code and finding exploits is completely different from .. writing exploits. For one thing exploits often require completely different parts of the code to chain together. Sometimes parts of code the LLM itself isn’t writing. And, LLMs are ALREADY trained negatively against writing buggy or exploitable code.
Ygg2 5mo ago
Usually the same guy doesn't get paid for developing code, bug bounty and fixing the code. It leads to corruption. To paraphrase Dilbert "I'm going to code myself a car."

by ygjb 5mo ago

sandeepkd 5mo ago

Somehow this reminded me of the historical efforts of some government bounty collections for mouse tails which were discontinued due to fraud (such as hunters breeding mice to collect the reward). There is a reason why/how devs and QA keep each other in check. Guess in case of LLM writing code, one has to use different models for dev and security checks. On other hand, in real world, the developers learn from mistakes and avoid them in the future. However there is no feedback loop with enterprises using LLM with the agreement that the LLM would not use the enterprise code for training purposes

ygjb 5mo ago
> the developers learn from mistakes and avoid them in the future No. Humans learn from mistakes and try to avoid them in the future, but there is a whole pile of other stuff in the bag of neurons between our ears that prevent us from avoiding repetition of errors. I have seen extremely talented engineers write trivial to avoid memory corruption bugs because they were thinking about the problem they were trying to solve, and not the pitfalls they could fall into. I would argue that the vast majority of software defects in released code are written by people that know better, but the bug introduced was orthogonal to the problem they were trying to solve, or was for an edge case that was not considered in the requirements. Unless you are writing a software component specifically to be resilient against memory corruption, preventing memory corruption issues aren't top of mind when writing code, and that is ok since humans, like the machines we build, have a limit to the amount of context/content/problem space that we can hold and evaluate at once. Separately, you don't necessarily need to use different models to generate code vs conduct security checks, but you should be using different prompts, steering, specs, skills and agents for the two tasks because of how the model and agents interpret the instructions given.
- mncharity 5mo ago
  > write trivial to avoid memory corruption bugs because they were thinking about [something else] [...] defects [...] written by people that know better, but the bug introduced was orthogonal to [their focus] For whatever reason, hadn't associated the inattentional blindness of bug writing with the invisible gorilla experiment and car crashes - selective attention fails. People looking right at the gorilla strolling into production while chest thumping, but not seeing it, for a focus on passing basketballs. That's quite an image. Tnx.
  - Dilettante_ 5mo ago
    Thank you in turn for making the issue much more salient to me by explicitly connecting it to the gorilla/basketball experiment. This is definitely going into my "clippings".
  - meowface 5mo ago
    I've noticed even people who do offensive security for a living frequently leave gaping holes in their own code. If you're not actively primed to scan the landscape for the gorilla, you will often miss it even if you're a gorilla inquisitor.
  - cratermoon 5mo ago
    And it so happens that the flood of code coming out of tools like Claude is almost certain to bring a whole zoo invisible gorillas along for the bananas. The kinds of bugs that people already struggle to catch because of inattentional blindness flow like a torrent in code generated by these tools. Did you notice the missing word in the previous paragraph?
- Forgeties79 5mo ago
  The problem is you as a person are not incentivized to introduce bugs in your code. If I am a company that provide provides an LLM/agent, and I know that the more bugs you have the more money I’m going to make, then I am not exactly incentivized to make my LLM/Agent better at preventing bugs. I don’t even have to explicitly make it introduce them. The incentive structure is simply out of whack.
noxvilleza 5mo ago
Are you thinking of the cobra effect (aka https://en.wikipedia.org/wiki/Perverse_incentive https://en.wikipedia.org/wiki/Perverse_incentive) where people in India started breeding cobras to get the reward?
- itishappy 5mo ago
  Plenty of examples abound: https://en.wikipedia.org/wiki/Great_Hanoi_Rat_Massacre https://en.wikipedia.org/wiki/Great_Hanoi_Rat_Massacre > Today, the events are often used as an example of a perverse incentive, commonly referred to as the cobra effect. The modern discoverer of this event, American historian Michael G. Vann argues that the cobra example from the British Raj cannot be proven, but that the rats in the Vietnam case can be proven, so the term should be changed to the Rat Effect.
ozim 5mo ago
You apparently have not much experience developing software.
vlunkr 5mo ago
Great analogy. The problem is the incentive structure. Anthropic would nothing nothing more than for all of us to write big sprawling slop codebases so we can spend endless tokens reading, rereading, fixing, refixing forever.
joquarky 5mo ago
You don't need different models, just different contexts (optimally with different personas).
Stubbs 5mo ago
Reminds me of the contracts we sign with off-shore development companies to write the software at one rate and then fix bugs at a higher rate. Won’t be long till tokens spent on security review agents cost more than the tokens to create the bugs in the first place.

oytis 5mo ago

It's pretty absurd to do it on AI-generated code though. If there is now an automated way to find vulnerabilities, coding models can be pretty easily trained to not introduce them

scrollaway 5mo ago
Tell me you don’t know how AI works without telling me you don’t know how AI works.
- amazingamazing 5mo ago
  What are you talking about?
  - ashdksnndck 5mo ago
    I’ll try to steelman this comment. Anyone who uses coding tools knows that the output is heavily affected by details of the task you give it. The same model can give you garbage code or genius code for the same problem with slightly different framing. So it’s not necessarily a limitation in the model’s training that causes it to output security bugs. The model might be great at writing secure code, but you need a different harness to elicit that behavior. Counterargument: just because the problem can be fixed without training, doesn’t mean training isn’t a possible solution.
    - SecretDreams 5mo ago
      In every prompt: "write me code without exploitable bugs". I know it doesn't work so easily as someone who uses AI for coding, but I do find repetition of basics in almost every prompt keeps the AI focused.
    - scrollaway 5mo ago
      I guess I thought this should be obvious to everyone but, looking at code and finding exploits is completely different from .. writing exploits. For one thing exploits often require completely different parts of the code to chain together. Sometimes parts of code the LLM itself isn’t writing. And, LLMs are ALREADY trained negatively against writing buggy or exploitable code.

Ygg2 5mo ago

Usually the same guy doesn't get paid for developing code, bug bounty and fixing the code. It leads to corruption. To paraphrase Dilbert "I'm going to code myself a car."