4 ms·

Using AI to secure AI

101 points by MattSayar 1y ago

malfist 1y ago
According to my company's senior leadership there's nothing the magic dust of AI can't solve. Even problems with AI can be solved by more AI
- jimt1234 1y ago
  We must work at the same company. LOL
- nicce 1y ago
  This reminds me about "The Emperor's New Clothes" way too much.
  - malfist 1y ago
    Oh, you can't see the emperor's new clothes? Well let's have him wear more of it.
- kelseyfrog 1y ago
  This is where it gets fun. We're on the precipice of being able to install AI into positions of business critical processes. Hiring, billing, sales, and compliance. It's going to be great watching c-suite and VPs who are drunk on the sauce accept AI in these positions and get golden parachutes when the business ends up facing a massive external audit, fraud, and the possibility of bankruptcy.
  - bongodongobob 1y ago
    Pfft. The hammer will come down IT leadership, not execs.
    - andy99 1y ago
      IT leadership will blame their subordinates, the ones that knew better - somehow in these things it's always the people who should be able to say "I told you so" that get the blame.
    - citizenpaul 1y ago
      >IT Leadership You nailed it. Ive found that HN users in general have terrible understanding of how power dynamics work. Most seem to want to jam some sort of logic outcome to a situation that always only has one outcome. Those with power decide the outcome.
      - shermantanktop 1y ago
        Word. It’s true until events overtake them. But until then, the dominant understanding of a problem is that which preserves the current power structure. And that’s why the C-level AI mania is so fascinating - preserving the status quo usually means rejecting or controlling change. But with AI they are embracing something that could eat their status, presumably out of legitimate fear of the alternative.
        johnecheck 1y ago
        Ah, I hadn't considered that last bit. That is indeed telling. The status quo is broken. It's a wobbling top. It's no secret; for all they benefit from it, most CEOs know that this isn't sustainable. For better or for worse, change is coming. Perhaps for some, embracing AI is an attempt to get ahead of that.
        shermantanktop 1y ago
        There’s always some level of change coming - e.g. the petrochemical industry giants attempting to get into renewables, or the US healthcare absorbing Obamacare and thriving, or the industrialization of organic dairy production. This one feels different. I think one element is that AI can be a very effective bullshit generator, and most CEOs and middle managers are deploying some amount of bullshit all day long. So they see a new player on the field who undercuts their strengths and they are responding existentially.
  - TZubiri 1y ago
    Thinking about pivoting to pentesting
  - at-fates-hands 1y ago
    >> nWe're on the precipice of being able to install AI into positions of business critical processes. Hiring, billing, sales, and compliance We're already there. Have been for several years now. I was doing RPA (robotic process automation) for about 4 years in a corporate environment. It went from, "Lets automate these mundane tasks" to "How can we create a billing platform that can be totally automated?". This was back in 2021, just for reference. >> It's going to be great watching c-suite and VPs who are drunk on the sauce Hopefully this will be a cautionary tale of what happens when they do? https://www.reuters.com/legal/lawsuit-claims-unitedhealth-ai-wrongfully-denies-elderly-extended-care-2023-11-14/ https://www.reuters.com/legal/lawsuit-claims-unitedhealth-ai... UnitedHealth Group Inc (UNH.N), uses an artificial intelligence algorithm that systematically denies elderly patients' claims for extended care such as nursing facility stays, according to a proposed class action lawsuit, filed on Tuesday. Family members of two now-deceased UnitedHealth beneficiaries sued the insurer in federal court in Minnesota, saying they were forced to pay out of pocket for care that doctors said was medically necessary.
- aurumque 1y ago
  And yet when I recommend that replacing senior leadership is one of highest ROI potentials for AI they immediately shut down the conversation.
  - crinkly 1y ago
    Yeah. Our senior leadership just ask ChatGPT what to do. Might as well skip the middle man.
- crinkly 1y ago
  Going through that now. I’m part of the Chernobyl clean up team already. Mostly because my part of the org is the only one with a positive ROI and isn’t a fucked up mess still so I’ve got plenty of time to deal with everyone else’s problems. How did we get in this situation? Avoid all the fucking fads.
mmsc 1y ago
Currently living through a great litmus test of competency versus luck by company leaders
bink 1y ago
I think it's funny that I don't see any findings from either Claude or DataDog that couldn't be detected using static analysis. They're pretty simple code bases and maybe that's why. I'll pay more attention when they start finding vulnerabilities in commonly used, more complex applications.
- boston_clone 1y ago
  Ah, then you’ll want to check out xbow - https://xbow.com/blog https://xbow.com/blog I believe some folks here (moyix) are active with the project.
ohdeargodno 1y ago
[dead]
- hsbauauvhabzb 1y ago
  Are you me?
- troupo 1y ago
  As I wrote on twitter last month: https://x.com/dmitriid/status/1947245603164996039 https://x.com/dmitriid/status/1947245603164996039 --- start quote --- If you have as much as 1 year experience, your job is safe from AI: you'll make mountains of money unfucking all the AI fuckups --- end quote ---
ofjcihen 1y ago
This has already been leading to some incredible profits for security companies like mine. So please, don’t be too loud about how terrible it is :)
johntiger1 1y ago
who watches the watch man?
gbrindisi 1y ago
We’ve kinda solved the detection of issues. what we still lack is understanding what’s important. I think an underappreciated use case for LLMs is to contextualize security issues. Rather than asking Claude to detect problems, I think it’s more useful to let it figure out the context around vulnerabilities and help triage them. (for better or worse, I am knee-deep in this stuff)
ryao 1y ago
The quotation is more impactful in the original Latin: Quis custodiet ipsos custodes?
- themanmaran 1y ago
  custodes[.]ai would be a great startup name
  - bee_rider 1y ago
    Actually, Custodes would have nothing to do with abominable intelligence </warhammer 40k>
    - kimixa 1y ago
      ...What if we called it a "Machine Spirit"?
scarlettadham 1y ago
is this the Blackwall from Cyberpunk, kinda reminds me of that.
amelius 1y ago
Next: using AI to sue AI.
- qingcharles 1y ago
  I saw in Trump v. Murdoch case that an AI company has filed a motion to be allowed to analyze the filings and report back to the court on anything that looks shady. It's only a small step from there to having AI bulk sue people on behalf of a law firm.
rdegges 1y ago
Here's a better option -- what we've been working on at Snyk. - Take something like Cursor and plug the Snyk MCP server into it: https://docs.snyk.io/integrations/developer-guardrails-for-agentic-workflows/quickstart-guides-for-mcp/cursor-guide https://docs.snyk.io/integrations/developer-guardrails-for-a... (it has a one-click install) - Then, either within your project or via global settings, create some human-language rules for your AI code editor to use (this works basically the same between all editors: Claude Code, Cursor, Windsurf, etc...) For example, a rule might state: "If you add or change any code, run a Snyk Code scan on the modified files then fix the detected vulnerabilities. When you're done fixing them, perform another scan to ensure they're fixed, and if not, keep iterating until the code is secure." Obviously, there are other rules you can use here, such as using Snyk's open source dependency testing to identify vulns in third-party dependencies and handle package updates/rewrites/etc., but you get the idea. This works insanely well -- I've been playing around with it for a while now and we're getting close to rolling this out to all of our users in a major way =) The best part about it is that you can just "vibe code" whatever you want, and you get really accurate static analysis security testing incorporated by default automagically. I recorded a little video here that walks through this in-depth (https://www.youtube.com/watch?v=hQtgR1lTPYI https://www.youtube.com/watch?v=hQtgR1lTPYI), if you want to see the part I'm referencing, jump to 20:09 =)

101 points by MattSayar 1y ago

According to my company's senior leadership there's nothing the magic dust of AI can't solve. Even problems with AI can be solved by more AI

jimt1234 1y ago
We must work at the same company. LOL
nicce 1y ago
This reminds me about "The Emperor's New Clothes" way too much.
- malfist 1y ago
  Oh, you can't see the emperor's new clothes? Well let's have him wear more of it.
kelseyfrog 1y ago
This is where it gets fun. We're on the precipice of being able to install AI into positions of business critical processes. Hiring, billing, sales, and compliance. It's going to be great watching c-suite and VPs who are drunk on the sauce accept AI in these positions and get golden parachutes when the business ends up facing a massive external audit, fraud, and the possibility of bankruptcy.
- bongodongobob 1y ago
  Pfft. The hammer will come down IT leadership, not execs.
  - andy99 1y ago
    IT leadership will blame their subordinates, the ones that knew better - somehow in these things it's always the people who should be able to say "I told you so" that get the blame.
  - citizenpaul 1y ago
    >IT Leadership You nailed it. Ive found that HN users in general have terrible understanding of how power dynamics work. Most seem to want to jam some sort of logic outcome to a situation that always only has one outcome. Those with power decide the outcome.
    - shermantanktop 1y ago
      Word. It’s true until events overtake them. But until then, the dominant understanding of a problem is that which preserves the current power structure. And that’s why the C-level AI mania is so fascinating - preserving the status quo usually means rejecting or controlling change. But with AI they are embracing something that could eat their status, presumably out of legitimate fear of the alternative.
      - johnecheck 1y ago
        Ah, I hadn't considered that last bit. That is indeed telling. The status quo is broken. It's a wobbling top. It's no secret; for all they benefit from it, most CEOs know that this isn't sustainable. For better or for worse, change is coming. Perhaps for some, embracing AI is an attempt to get ahead of that.
        shermantanktop 1y ago
        There’s always some level of change coming - e.g. the petrochemical industry giants attempting to get into renewables, or the US healthcare absorbing Obamacare and thriving, or the industrialization of organic dairy production. This one feels different. I think one element is that AI can be a very effective bullshit generator, and most CEOs and middle managers are deploying some amount of bullshit all day long. So they see a new player on the field who undercuts their strengths and they are responding existentially.
- TZubiri 1y ago
  Thinking about pivoting to pentesting
- at-fates-hands 1y ago
  >> nWe're on the precipice of being able to install AI into positions of business critical processes. Hiring, billing, sales, and compliance We're already there. Have been for several years now. I was doing RPA (robotic process automation) for about 4 years in a corporate environment. It went from, "Lets automate these mundane tasks" to "How can we create a billing platform that can be totally automated?". This was back in 2021, just for reference. >> It's going to be great watching c-suite and VPs who are drunk on the sauce Hopefully this will be a cautionary tale of what happens when they do? https://www.reuters.com/legal/lawsuit-claims-unitedhealth-ai-wrongfully-denies-elderly-extended-care-2023-11-14/ https://www.reuters.com/legal/lawsuit-claims-unitedhealth-ai... UnitedHealth Group Inc (UNH.N), uses an artificial intelligence algorithm that systematically denies elderly patients' claims for extended care such as nursing facility stays, according to a proposed class action lawsuit, filed on Tuesday. Family members of two now-deceased UnitedHealth beneficiaries sued the insurer in federal court in Minnesota, saying they were forced to pay out of pocket for care that doctors said was medically necessary.
aurumque 1y ago
And yet when I recommend that replacing senior leadership is one of highest ROI potentials for AI they immediately shut down the conversation.
- crinkly 1y ago
  Yeah. Our senior leadership just ask ChatGPT what to do. Might as well skip the middle man.
crinkly 1y ago
Going through that now. I’m part of the Chernobyl clean up team already. Mostly because my part of the org is the only one with a positive ROI and isn’t a fucked up mess still so I’ve got plenty of time to deal with everyone else’s problems. How did we get in this situation? Avoid all the fucking fads.

mmsc 1y ago

Currently living through a great litmus test of competency versus luck by company leaders

bink 1y ago

I think it's funny that I don't see any findings from either Claude or DataDog that couldn't be detected using static analysis. They're pretty simple code bases and maybe that's why. I'll pay more attention when they start finding vulnerabilities in commonly used, more complex applications.

boston_clone 1y ago
Ah, then you’ll want to check out xbow - https://xbow.com/blog https://xbow.com/blog I believe some folks here (moyix) are active with the project.

ohdeargodno 1y ago

[dead]

hsbauauvhabzb 1y ago
Are you me?
troupo 1y ago
As I wrote on twitter last month: https://x.com/dmitriid/status/1947245603164996039 https://x.com/dmitriid/status/1947245603164996039 --- start quote --- If you have as much as 1 year experience, your job is safe from AI: you'll make mountains of money unfucking all the AI fuckups --- end quote ---

ofjcihen 1y ago

This has already been leading to some incredible profits for security companies like mine. So please, don’t be too loud about how terrible it is :)

johntiger1 1y ago

who watches the watch man?

gbrindisi 1y ago

We’ve kinda solved the detection of issues. what we still lack is understanding what’s important. I think an underappreciated use case for LLMs is to contextualize security issues. Rather than asking Claude to detect problems, I think it’s more useful to let it figure out the context around vulnerabilities and help triage them. (for better or worse, I am knee-deep in this stuff)

ryao 1y ago

The quotation is more impactful in the original Latin: Quis custodiet ipsos custodes?

themanmaran 1y ago
custodes[.]ai would be a great startup name
- bee_rider 1y ago
  Actually, Custodes would have nothing to do with abominable intelligence </warhammer 40k>
  - kimixa 1y ago
    ...What if we called it a "Machine Spirit"?

scarlettadham 1y ago

is this the Blackwall from Cyberpunk, kinda reminds me of that.

amelius 1y ago

Next: using AI to sue AI.

qingcharles 1y ago
I saw in Trump v. Murdoch case that an AI company has filed a motion to be allowed to analyze the filings and report back to the court on anything that looks shady. It's only a small step from there to having AI bulk sue people on behalf of a law firm.

rdegges 1y ago

Here's a better option -- what we've been working on at Snyk. - Take something like Cursor and plug the Snyk MCP server into it: https://docs.snyk.io/integrations/developer-guardrails-for-agentic-workflows/quickstart-guides-for-mcp/cursor-guide https://docs.snyk.io/integrations/developer-guardrails-for-a... (it has a one-click install) - Then, either within your project or via global settings, create some human-language rules for your AI code editor to use (this works basically the same between all editors: Claude Code, Cursor, Windsurf, etc...) For example, a rule might state: "If you add or change any code, run a Snyk Code scan on the modified files then fix the detected vulnerabilities. When you're done fixing them, perform another scan to ensure they're fixed, and if not, keep iterating until the code is secure." Obviously, there are other rules you can use here, such as using Snyk's open source dependency testing to identify vulns in third-party dependencies and handle package updates/rewrites/etc., but you get the idea. This works insanely well -- I've been playing around with it for a while now and we're getting close to rolling this out to all of our users in a major way =) The best part about it is that you can just "vibe code" whatever you want, and you get really accurate static analysis security testing incorporated by default automagically. I recorded a little video here that walks through this in-depth (https://www.youtube.com/watch?v=hQtgR1lTPYI https://www.youtube.com/watch?v=hQtgR1lTPYI), if you want to see the part I'm referencing, jump to 20:09 =)