3 ms·
For areas that require a great deal of confidence in conclusions, like forensics and incident triage, I can only see generative AI making matters worse. There’s
by _23sd 3y ago
For areas that require a great deal of confidence in conclusions, like forensics and incident triage, I can only see generative AI making matters worse. There’s already a ton of over-automation and humans making mistakes because they trust their tools too much. Giving that industry access to more tools that confidently give wrong answers almost seems irresponsible.
I am excited for AI-based threat intelligence products, though.
- notif1 3y agoI could see the exact opposite happening. Forensics and IR often require a ton of complex information parsing that an LLM could accomplish very quickly and then point an analyst in the right direction to verify. It's not meant to be a panacea just an accelerant. Threat intel has purported to use AI for ages and I remain unconvinced of its effectiveness. Program analysis to support TI could be exciting however.
- toomuchtodo 3y agoAgree, but having the LLM explain how it arrived at the output will be crucial for provenance and anti-hallucination purposes. Judges don’t care for fabricated evidence or citations.
- chaxor 3y agoThis language bothers me deeply. So many people ask the system itself to describe what it is doing, for which it is absolutely incapable of doing. The system has no access to its internals. Even academics are asking the LLM itself to describe what it's doing like idiots. I know that academia's intellect is slipping due to allowing too many people in, but this type of stupidity is just mind boggling.
- dinvlad 3y agoI like to preserve skepticism here as well, though probably blaming them won’t help the issue. But absolutely agreed, just listened to Gary Marcus’s podcast episode on how LLMs are already capable of making dad jokes, but they can’t do neither “deep” humor generation nor “deep” humor explanation very well, in part because they lack all of the cultural and experiential (“touchy-feely”) context of the humans, and an even bigger part since they are optimized for generation, not explanation, while humans are almost the opposite.
- toomuchtodo 3y agoI think you misunderstood my comment or I didn’t articulate the idea clearly. I’m simply arguing for a verbose or debug mode that can be toggled on to show how output was derived, not expecting an LLM to have awareness or understanding of itself or it’s internal workings.
- chaxor 3y agoThat's fair, an encoder system for which the full attention values and weights available is a useful tool. IMO encoder models are far more powerful and useful for devs than any of the decoder models.
- ignoramous 3y ago> I am excited for AI-based threat intelligence products, though. Have examples of existing product or project in this space that are interesting to you?
- dinvlad 3y agoThere’s traceable.ai, for one (not affiliated, but testing it). I’d say none of traditional ML products are very good, at all (at least among the ones available publicly). Most of them are just snake oil. But I’d err on the side of plausibility that LLMs might finally add fuel to that, at least on the pentesting side (might, not yet will). Additionally, even OpenAI’s own advertisement “paper” acknowledges they couldn’t make it work for pentesting well (hence probably why they created this bounty program, among other reasons).
- dinvlad 3y agoFWIW there’s this paper that just came out, and promises a low hallucination factor (not quite for pentesting, but this could be a useful starting point): “Gorilla: Large Language Model Connected with Massive APIs” https://arxiv.org/abs/2305.15334 https://arxiv.org/abs/2305.15334