5 ms·

The Rise of 'Vibe Hacking' Is the Next AI Nightmare

33 points by muzz 1y ago

vouaobrasil 1y ago
> victory will belong to the savvy blackhat hacker who uses AI to generate code at scale This is just it: AI, while providing some efficiency gains for the average user, will become simply too powerful. Imagine a superpower that allows you to move objects with your mind. That could be a very nice thing to have for many to have, because you could probably help people with it. That's the attitude many hacker-types take. The problem is, it allows people to also kill instantly, which means that telekinesis would just be too powerful to juxtapose against our animal instincts. AI is just too powerful – and if more people took a serious stand against it, it might actually be shut down.
- rco8786 1y ago
  Is it even possible to shut it down?
  - vouaobrasil 1y ago
    Of course it is. If enough people were truly enraged by it, if some leader were to rile up the mob enough, it could be shut down. Revolts have occurred in other parts of the world, and things are getting sufficiently bad that a sizable enough revolt could shut AI down. All we need is a sufficient number of people who are angry enough at AI.
    - sgjohnson 1y ago
      Good luck shutting down the LLM running on my MacBook. The Pandora’s Box is open. It’s over.
      - blibble 1y ago
        a software update could easily cripple its ability to run on your local machine unless you plan to never update again
        diggan 1y ago
        > a software update could easily cripple its ability to run on your local machine A software update collaborated on by Microsoft, Apple + countless of volunteer groups managing various other distributions? The cat really is out of the bag. You could probably make it a death penalty in the whole world and some people would still use it secretly. Once things like this run on consumer hardware, I think it's already too late to pull it down fully. You could regulate it though and probably have a better chance of limiting the damages, not sure an outright ban could even have the effect you want with a ban.
        blibble 1y ago
        the nvidia/AMD/apple chips all require proprietary firmware blobs, it can be enforced in there yes you won't get people that won't ever update, but you'll get the overwhelming majority and the hardware the never-updaters use will eventually fail and won't be able to be replaced also: ban the release of new "open" models, they will slowly become out of date and useless combine these, and the problem will solve itself over time
        diggan 1y ago
        > they will slowly become out of date and useless Models released today are already useful for a bunch of stuff, maybe over the course of 100 year they could be considered "out of date", but they don't exactly bitrot by themselves because they sit on a disk, not sure why'd they suddenly "expire" or whatever you try to hint at. And even over the course of 100 year, people will continue the machine learning science, regardless if it's legal or not, the potential benefits (for a select few) seems to be too good for people to ignore, which is why the current bubble is happening in the first place.
        1y ago
      - vouaobrasil 1y ago
        Plenty of past civilizations have thought they were invulnerable. In fact, most entities with power think that they can never be taken down. But countless empires in the past have fallen, and countless powerful people have lost their wealth and power overnight.
        svachalek 1y ago
        There's a big difference between a civilization being taken down, and civilization being taken down.
    - rco8786 1y ago
      But it's just like, math. The math is out there now. You can't shut down math.
  - rglover 1y ago
    It's just software running on a server...this isn't a Johnny Depp movie [1]. Just flip the power switch on the racks. [1] https://www.youtube.com/watch?v=0jg3mSf561w https://www.youtube.com/watch?v=0jg3mSf561w
    - hluska 1y ago
      Rather, it’s many different types of software running on many different systems around the world, each funded by a different party with its own motives. This is no movie…
      - vouaobrasil 1y ago
        True, but the system only exists because it is currently economically viable. A mass taboo against AI would change that. And many people outside of tech already dislike AI a lot, so it's not inconceivable that this dislike could be fuelled into a worldwide taboo.
        diggan 1y ago
        > True, but the system only exists because it is currently economically viable. The "system" isn't a thing, but more like running apps, some run on servers, other consumer hardware. And the parts that run on consumer hardware will be around even if 99% of the current hyped up ecosystem dies overnight, people won't suddenly stop trying to run these things locally.
      - rglover 1y ago
        And every single one has a power switch. I get the general "too many variables" argument, but the idea that humans have no means of stopping any of these apps/systems/algorithms/etc if they get "out of control" (a farce in itself as it's a chat bot) is ridiculous. It's very interesting to see how badly people want to be living in and being an active participant in a sci-fi flick. I think that's far more concerning than the AI itself.
        vouaobrasil 1y ago
tptacek 1y ago
Software security has been running off large-scale automation for over a decade. LLMs might or might not be a step change in that automation (I'm optimistic but uncertain), but, unlike in conventional software development, the standard arguments about craft and thoughtfulness aren't operative here. If there was an argument to be had, it would have been had around the time Google stood up the mega fuzzing farms. A fun thing to keep in mind about software security is that it's premised on the existence of a determined and amoral adversary. In the long-long ago, Richard Stallman's host at MIT had no password; anybody could log into it. It was a statement about the ethics of locking down computing resources. That's approximately the position any security practitioner would be taking if they attempted to moralize against LLM-assisted offensive computing.
- bradyriddle 1y ago
  Say more about these mega fuzzing farms. I haven't heard anything about this.
  - navanchauhan 1y ago
    Something like https://security.googleblog.com/2011/08/fuzzing-at-scale.html https://security.googleblog.com/2011/08/fuzzing-at-scale.htm... ?
  - tptacek 1y ago
    There are fields, endless fields, where kernel zero days are no longer born. They are grown.
  - cibyr 1y ago
    ClusterFuzz: https://google.github.io/clusterfuzz/ https://google.github.io/clusterfuzz/
- AStonesThrow 1y ago
  rms@gnu.ai.mit.edu It was actually the A.I. Lab at M.I.T. and they already had their own dedicated subdomain for it. This had to have been around 1990-91. And IIRC, the actual admins made a valiant effort to keep all the shell users away from "root" privileges, so it wasn't a total dumpster fire and the system stayed alive, mostly https://en.wikipedia.org/wiki/MIT_Computer_Science_and_Artificial_Intelligence_Laboratory#AI_Lab_and_LCS https://en.wikipedia.org/wiki/MIT_Computer_Science_and_Artif...
unstablediffusi 1y ago
[flagged]
deleted 1y ago
[deleted]
Animats 1y ago
The AI nightmare after that is "vibe capitalism". You put in a pitch deck and an operating business comes out. Somebody should pitch that to YC.
- blibble 1y ago
  this already exists, it's called a venture capital fund
byt3bl33d3r 1y ago
I've written tailored offensive security tools and malware for Red Teams for around a decade and now work in the AI space. The argument that LLMs will enable "super powered" malware and that existing security solutions won't be able to keep up, is completely overblown. I see 0 evidence of this being possible with the current incarnation of "AI" or LLMs. "Vide coded" malware will be easier to detect if the people creating it don't understand what the code is actually doing and will result in incredible amount of OpSec fails when the malware actually hits the target systems. I do agree that "vide coding" will accelerate malware development and generally increase the amount of attacks to orgs. However if you're already applying bog-standard security practices like defense in depth, you shouldn't be concerned about this. If anything, you might want to start thinking about SOC automations in order to reduce alert fatigue. Stay far away from anyone trying to sell you products to defend against "AI enabled malware". As of right now it's 100% snake oil. Also, this is probably one of the cringiest articles on the subject I've ever read and is only meant to spread FUD. I do find the banner video extremely entertaining however.
- DanMcInerney 1y ago
  I too write automated offensive tooling. We actually wrote a project, vulnhuntr, that found the first autonomously-discovered 0day using AI. Feed it a GitHub repo and it tracks down user input from source to sink and analyzes for web-based vulnerabilities. Agree this article is incredibly cringy and standard best practices in network and development security will use the same AI efficiency gains to keep up (more or less). What bothers me the most about this article is that the tools that attackers use to do stuff like find 0days in code are the same tools that defenders can use to find the 0day first and fix it. It's not like offensive tooling is being developed in a vacuum and the world is ending as "armies of script kiddies" will suddenly drain every bank account in the world. Automated defense and code analysis is improving at a similar rate as automated offense. In this awful article's defense though, I would argue that red team will always have an advantage over blue team because blue team is by definition reactionary. So as tech continues it's exponential advancements, the advantage gap for the top 1% red teamers is likely to scale accordingly.
  - 1y ago
jdefr89 1y ago
Alright folks.. To qualify myself. I am a vulnerability Researcher @ MIT. My day to day research concerns embedded hardware/software security. Some of my current/past endeavors involve AI/ML integration and understanding just how useful it actually is for finding/exploiting vulnerabilities. Just last week my lab hosted a conference that included MIT folks and the outsiders we invite. One talk was on the current state of AI/LLM. To keep things short, this article is sensationalized and overstates the utility of AI/ML on finding actual novel vulnerabilities. As it currently stands, LLMs cannot even reliably find bugs other less sophisticated tools could have found in much less time. Binary Exploitation is a great place for illustrating the wall you’ll hit using LLMs hoping for a 0day. While LLMs can help with things like setting up fuzzers or maybe giving you a place to start manual analysis, their utility kind of stops there. They cannot reliably catch memory corruption bugs that a basic fuzzers or sanitizers could have found within seconds. This makes sense for that class of bugs. LLMs are fuzzy logic and these issues aren’t reliably found with that paradigm. That’s the whole reason we have fuzzers; they find subtle bugs worth triaging. You’ve seen how well LLMs count, it’s no surprise they might miss many of the same things a humans would but fuzzers wouldn’t (think UaF, OOB, etc). All the other tools you see written for script kiddies yield the same amount of false positives they could have gotten with other tools that already exist.. I can go on and on but I am on shuttle, typing on small phone. TLDR: Article is trying to say LLMs are super hackers already and that’s simply false. They definitely have allure for script kiddies. In the future this might change. LLMs time saving aspects are definitely worth checking out for static binary analysis. Binary Ninja with Sidekick saves a lot of time! But again. You still need to double check important things!
- tptacek 1y ago
  I'm a vuln researcher too, and we just had an article here about another vuln researcher using o3 to find a zero-day remote Linux kernel vulnerability. And not in an especially human-directed way: they literally set up 100 runs of o3, using the 'simonw `llm` tool, and sifted through the results. I'm having trouble reconciling what you wrote here with that result. Also with my own experiences, not necessarily of finding kernel vulnerabilities (I haven't had any need to do that for the last couple years), but of rapidly comprehending and analyzing kernel code (which I do need to do), and realizing how potent that ability would have been on projects 10 years ago. I think you're wrong about this.
parliament32 1y ago
We are no closer to this than "vibe-businessing", where you vibe your way into a profit-generating business powered by nothing than AI agents.
- tptacek 1y ago
  (Looks around) You know, there are some pretty crazy run rates out there.

33 points by muzz 1y ago

> victory will belong to the savvy blackhat hacker who uses AI to generate code at scale This is just it: AI, while providing some efficiency gains for the average user, will become simply too powerful. Imagine a superpower that allows you to move objects with your mind. That could be a very nice thing to have for many to have, because you could probably help people with it. That's the attitude many hacker-types take. The problem is, it allows people to also kill instantly, which means that telekinesis would just be too powerful to juxtapose against our animal instincts. AI is just too powerful – and if more people took a serious stand against it, it might actually be shut down.

rco8786 1y ago
Is it even possible to shut it down?
- vouaobrasil 1y ago
  Of course it is. If enough people were truly enraged by it, if some leader were to rile up the mob enough, it could be shut down. Revolts have occurred in other parts of the world, and things are getting sufficiently bad that a sizable enough revolt could shut AI down. All we need is a sufficient number of people who are angry enough at AI.
  - sgjohnson 1y ago
    Good luck shutting down the LLM running on my MacBook. The Pandora’s Box is open. It’s over.
    - blibble 1y ago
      a software update could easily cripple its ability to run on your local machine unless you plan to never update again
      - diggan 1y ago
        > a software update could easily cripple its ability to run on your local machine A software update collaborated on by Microsoft, Apple + countless of volunteer groups managing various other distributions? The cat really is out of the bag. You could probably make it a death penalty in the whole world and some people would still use it secretly. Once things like this run on consumer hardware, I think it's already too late to pull it down fully. You could regulate it though and probably have a better chance of limiting the damages, not sure an outright ban could even have the effect you want with a ban.
        blibble 1y ago
        the nvidia/AMD/apple chips all require proprietary firmware blobs, it can be enforced in there yes you won't get people that won't ever update, but you'll get the overwhelming majority and the hardware the never-updaters use will eventually fail and won't be able to be replaced also: ban the release of new "open" models, they will slowly become out of date and useless combine these, and the problem will solve itself over time
        diggan 1y ago
        > they will slowly become out of date and useless Models released today are already useful for a bunch of stuff, maybe over the course of 100 year they could be considered "out of date", but they don't exactly bitrot by themselves because they sit on a disk, not sure why'd they suddenly "expire" or whatever you try to hint at. And even over the course of 100 year, people will continue the machine learning science, regardless if it's legal or not, the potential benefits (for a select few) seems to be too good for people to ignore, which is why the current bubble is happening in the first place.
        1y ago
    - vouaobrasil 1y ago
      Plenty of past civilizations have thought they were invulnerable. In fact, most entities with power think that they can never be taken down. But countless empires in the past have fallen, and countless powerful people have lost their wealth and power overnight.
      - svachalek 1y ago
        There's a big difference between a civilization being taken down, and civilization being taken down.
  - rco8786 1y ago
    But it's just like, math. The math is out there now. You can't shut down math.
- rglover 1y ago
  It's just software running on a server...this isn't a Johnny Depp movie [1]. Just flip the power switch on the racks. [1] https://www.youtube.com/watch?v=0jg3mSf561w https://www.youtube.com/watch?v=0jg3mSf561w
  - hluska 1y ago
    Rather, it’s many different types of software running on many different systems around the world, each funded by a different party with its own motives. This is no movie…
    - vouaobrasil 1y ago
      True, but the system only exists because it is currently economically viable. A mass taboo against AI would change that. And many people outside of tech already dislike AI a lot, so it's not inconceivable that this dislike could be fuelled into a worldwide taboo.
      - diggan 1y ago
        > True, but the system only exists because it is currently economically viable. The "system" isn't a thing, but more like running apps, some run on servers, other consumer hardware. And the parts that run on consumer hardware will be around even if 99% of the current hyped up ecosystem dies overnight, people won't suddenly stop trying to run these things locally.
    - rglover 1y ago
      And every single one has a power switch. I get the general "too many variables" argument, but the idea that humans have no means of stopping any of these apps/systems/algorithms/etc if they get "out of control" (a farce in itself as it's a chat bot) is ridiculous. It's very interesting to see how badly people want to be living in and being an active participant in a sci-fi flick. I think that's far more concerning than the AI itself.
      - vouaobrasil 1y ago

tptacek 1y ago

Software security has been running off large-scale automation for over a decade. LLMs might or might not be a step change in that automation (I'm optimistic but uncertain), but, unlike in conventional software development, the standard arguments about craft and thoughtfulness aren't operative here. If there was an argument to be had, it would have been had around the time Google stood up the mega fuzzing farms. A fun thing to keep in mind about software security is that it's premised on the existence of a determined and amoral adversary. In the long-long ago, Richard Stallman's host at MIT had no password; anybody could log into it. It was a statement about the ethics of locking down computing resources. That's approximately the position any security practitioner would be taking if they attempted to moralize against LLM-assisted offensive computing.

bradyriddle 1y ago
Say more about these mega fuzzing farms. I haven't heard anything about this.
- navanchauhan 1y ago
  Something like https://security.googleblog.com/2011/08/fuzzing-at-scale.html https://security.googleblog.com/2011/08/fuzzing-at-scale.htm... ?
- tptacek 1y ago
  There are fields, endless fields, where kernel zero days are no longer born. They are grown.
- cibyr 1y ago
  ClusterFuzz: https://google.github.io/clusterfuzz/ https://google.github.io/clusterfuzz/
AStonesThrow 1y ago
rms@gnu.ai.mit.edu It was actually the A.I. Lab at M.I.T. and they already had their own dedicated subdomain for it. This had to have been around 1990-91. And IIRC, the actual admins made a valiant effort to keep all the shell users away from "root" privileges, so it wasn't a total dumpster fire and the system stayed alive, mostly https://en.wikipedia.org/wiki/MIT_Computer_Science_and_Artificial_Intelligence_Laboratory#AI_Lab_and_LCS https://en.wikipedia.org/wiki/MIT_Computer_Science_and_Artif...

unstablediffusi 1y ago

[flagged]

deleted 1y ago

[deleted]

Animats 1y ago

The AI nightmare after that is "vibe capitalism". You put in a pitch deck and an operating business comes out. Somebody should pitch that to YC.

blibble 1y ago
this already exists, it's called a venture capital fund

byt3bl33d3r 1y ago

I've written tailored offensive security tools and malware for Red Teams for around a decade and now work in the AI space. The argument that LLMs will enable "super powered" malware and that existing security solutions won't be able to keep up, is completely overblown. I see 0 evidence of this being possible with the current incarnation of "AI" or LLMs. "Vide coded" malware will be easier to detect if the people creating it don't understand what the code is actually doing and will result in incredible amount of OpSec fails when the malware actually hits the target systems. I do agree that "vide coding" will accelerate malware development and generally increase the amount of attacks to orgs. However if you're already applying bog-standard security practices like defense in depth, you shouldn't be concerned about this. If anything, you might want to start thinking about SOC automations in order to reduce alert fatigue. Stay far away from anyone trying to sell you products to defend against "AI enabled malware". As of right now it's 100% snake oil. Also, this is probably one of the cringiest articles on the subject I've ever read and is only meant to spread FUD. I do find the banner video extremely entertaining however.

DanMcInerney 1y ago
I too write automated offensive tooling. We actually wrote a project, vulnhuntr, that found the first autonomously-discovered 0day using AI. Feed it a GitHub repo and it tracks down user input from source to sink and analyzes for web-based vulnerabilities. Agree this article is incredibly cringy and standard best practices in network and development security will use the same AI efficiency gains to keep up (more or less). What bothers me the most about this article is that the tools that attackers use to do stuff like find 0days in code are the same tools that defenders can use to find the 0day first and fix it. It's not like offensive tooling is being developed in a vacuum and the world is ending as "armies of script kiddies" will suddenly drain every bank account in the world. Automated defense and code analysis is improving at a similar rate as automated offense. In this awful article's defense though, I would argue that red team will always have an advantage over blue team because blue team is by definition reactionary. So as tech continues it's exponential advancements, the advantage gap for the top 1% red teamers is likely to scale accordingly.
- 1y ago

jdefr89 1y ago

Alright folks.. To qualify myself. I am a vulnerability Researcher @ MIT. My day to day research concerns embedded hardware/software security. Some of my current/past endeavors involve AI/ML integration and understanding just how useful it actually is for finding/exploiting vulnerabilities. Just last week my lab hosted a conference that included MIT folks and the outsiders we invite. One talk was on the current state of AI/LLM. To keep things short, this article is sensationalized and overstates the utility of AI/ML on finding actual novel vulnerabilities. As it currently stands, LLMs cannot even reliably find bugs other less sophisticated tools could have found in much less time. Binary Exploitation is a great place for illustrating the wall you’ll hit using LLMs hoping for a 0day. While LLMs can help with things like setting up fuzzers or maybe giving you a place to start manual analysis, their utility kind of stops there. They cannot reliably catch memory corruption bugs that a basic fuzzers or sanitizers could have found within seconds. This makes sense for that class of bugs. LLMs are fuzzy logic and these issues aren’t reliably found with that paradigm. That’s the whole reason we have fuzzers; they find subtle bugs worth triaging. You’ve seen how well LLMs count, it’s no surprise they might miss many of the same things a humans would but fuzzers wouldn’t (think UaF, OOB, etc). All the other tools you see written for script kiddies yield the same amount of false positives they could have gotten with other tools that already exist.. I can go on and on but I am on shuttle, typing on small phone. TLDR: Article is trying to say LLMs are super hackers already and that’s simply false. They definitely have allure for script kiddies. In the future this might change. LLMs time saving aspects are definitely worth checking out for static binary analysis. Binary Ninja with Sidekick saves a lot of time! But again. You still need to double check important things!

tptacek 1y ago
I'm a vuln researcher too, and we just had an article here about another vuln researcher using o3 to find a zero-day remote Linux kernel vulnerability. And not in an especially human-directed way: they literally set up 100 runs of o3, using the 'simonw `llm` tool, and sifted through the results. I'm having trouble reconciling what you wrote here with that result. Also with my own experiences, not necessarily of finding kernel vulnerabilities (I haven't had any need to do that for the last couple years), but of rapidly comprehending and analyzing kernel code (which I do need to do), and realizing how potent that ability would have been on projects 10 years ago. I think you're wrong about this.

parliament32 1y ago

We are no closer to this than "vibe-businessing", where you vibe your way into a profit-generating business powered by nothing than AI agents.

tptacek 1y ago
(Looks around) You know, there are some pretty crazy run rates out there.