4 ms·

Script kiddies will have a field day with this. Conversely, it might start making companies take security more seriously.

nostrademons 3y ago
I've heard the trend is going the opposite direction - companies are using LLMs to write code, and the resulting code has terrible security properties because the LLM knows nothing about tracing every possible code path and a lot about what code other developers would write to solve a problem. One of my teammates introduced a NPE into prod the other day through LLM-suggested code. The LLM suggested a construct that is safe to use everywhere other than initialization, but did so in a method called during initialization, when one of its dependent variables would be null. Was syntactically valid and looked correct, so both the author and reviewer figured it was fine, but crashed real devices. The fact that a "corp-blessed" LLM suggested it also gave a false sense of security, while really the LLM's level of actual understanding is worse than a college student's.
- mlinhares 3y ago
  Isn't that missing a linter? Pretty sure any null-safety linter in Java could pick this up.
  - vitiral 3y ago
    The idea that we just need more piles of tools to fix the problems caused by our shiny new pile of tools is, I think, very demonstrative of how it's all going to come crashing down.
    - web3-is-a-scam 3y ago
      Linters/static analysis should be in “the pile” regardless if you’re using LLM-generated code or not.
  - nostrademons 3y ago
    No, the NPE was implicit in the Android frameworks, not in user code. Basically it called a method that can be called only after onCreate() completes, from a method that was itself called from onCreate().
    - web3-is-a-scam 3y ago
      This seems like something that static analysis that is android aware should be able to find?
      - nostrademons 3y ago
        One of the notes in the postmortem was that if it were Kotlin, it would've been caught by the nullability checker. It wasn't, though, and much of the core Android code won't compile with nullability checks because it was written before @Nullable/@NotNull annotations were a thing. Pre-2010 Java code basically has to assume everything is nullable. My point is that LLM-generated code often doesn't, because it's trained on StackOverflow code where the author either doesn't care or had implicit knowledge about which variables could be null and which couldn't. Hence it generates code that is valid in most situations but can lead to a crash when used in situations where its data dependencies may be null or uninitialized. Exactly the stuff of security nightmares.
        mlinhares 3y ago
        Or, put in another way, a job guarantee program LOL.
ianhawes 3y ago
Ahem we call them threat actors now.
- starttoaster 3y ago
  Script kiddy was adopted to make them sound more like what they are: people that have no clue what they're doing but mostly just exploit things that smarter people found out they could exploit. The "smarter person" being a naive language model in this case. The idea being that being called something "cool" like "hacker" or "threat actor" actually incentivizes script kiddies to put down their Xbox controllers and do bad things.
krainboltgreene 3y ago
I seriously doubt this will change anything for malicious actors. The bottle neck was never "coming up with the attack".

nostrademons 3y ago

I've heard the trend is going the opposite direction - companies are using LLMs to write code, and the resulting code has terrible security properties because the LLM knows nothing about tracing every possible code path and a lot about what code other developers would write to solve a problem. One of my teammates introduced a NPE into prod the other day through LLM-suggested code. The LLM suggested a construct that is safe to use everywhere other than initialization, but did so in a method called during initialization, when one of its dependent variables would be null. Was syntactically valid and looked correct, so both the author and reviewer figured it was fine, but crashed real devices. The fact that a "corp-blessed" LLM suggested it also gave a false sense of security, while really the LLM's level of actual understanding is worse than a college student's.

mlinhares 3y ago
Isn't that missing a linter? Pretty sure any null-safety linter in Java could pick this up.
- vitiral 3y ago
  The idea that we just need more piles of tools to fix the problems caused by our shiny new pile of tools is, I think, very demonstrative of how it's all going to come crashing down.
  - web3-is-a-scam 3y ago
    Linters/static analysis should be in “the pile” regardless if you’re using LLM-generated code or not.
- nostrademons 3y ago
  No, the NPE was implicit in the Android frameworks, not in user code. Basically it called a method that can be called only after onCreate() completes, from a method that was itself called from onCreate().
  - web3-is-a-scam 3y ago
    This seems like something that static analysis that is android aware should be able to find?
    - nostrademons 3y ago
      One of the notes in the postmortem was that if it were Kotlin, it would've been caught by the nullability checker. It wasn't, though, and much of the core Android code won't compile with nullability checks because it was written before @Nullable/@NotNull annotations were a thing. Pre-2010 Java code basically has to assume everything is nullable. My point is that LLM-generated code often doesn't, because it's trained on StackOverflow code where the author either doesn't care or had implicit knowledge about which variables could be null and which couldn't. Hence it generates code that is valid in most situations but can lead to a crash when used in situations where its data dependencies may be null or uninitialized. Exactly the stuff of security nightmares.
      - mlinhares 3y ago
        Or, put in another way, a job guarantee program LOL.

ianhawes 3y ago

Ahem we call them threat actors now.

starttoaster 3y ago
Script kiddy was adopted to make them sound more like what they are: people that have no clue what they're doing but mostly just exploit things that smarter people found out they could exploit. The "smarter person" being a naive language model in this case. The idea being that being called something "cool" like "hacker" or "threat actor" actually incentivizes script kiddies to put down their Xbox controllers and do bad things.

krainboltgreene 3y ago

I seriously doubt this will change anything for malicious actors. The bottle neck was never "coming up with the attack".