6 ms·
> This is the fundamental problem: AI can generate the form of security research without the substance. I think this is the fundamental problem of LLMs in gene
by goalieca 1y ago
> This is the fundamental problem: AI can generate the form of security research without the substance.
I think this is the fundamental problem of LLMs in general. Some of the time looks just enough right to seem legitimate. Luckily the rest of the time it doesn’t.
- jsheard 1y agoThe other fundamental problem is that to a grifter, it's not a fundamental problem for the output to be plausible but often wrong. Plausible is all they need.
- gdulli 1y agoThat's an important one. Another fundamental problem with plausible output is that it makes a manager, or a junior, or some other unsophisticated end user think the technology is almost there, and a reliably correct version is just around the corner.
- jacquesm 1y agoIndeed. The elderly in my family are seeing a substantial uptick of AI generated stuff that looks extremely plausible. Fortunately they're old but not stupid, so far nobody has fallen for any of these but I have to admit: they look good enough to pass a first casual inspection.
- Sharlin 1y agoUnfortunately, to a majority of the population approximately 100% of LLM output seems entirely legitimate.
- dafelst 1y agoI agree wholeheartedly, and this is the core problem - many of the people evangelizing LLMs for a particular task (especially investors and AI gold rush "entrepreneurs") do not have enough expertise in that particular field to effectively evaluate the quality of the output. It sure looks the part though, and for those with a shallow understanding, it is often enough.
- captainkrtek 1y agoThat, combined with the confidence any of its output is communicated back to the user.
- jimbokun 1y agoExcept for things they happen to know something about.
- gdulli 1y agoUnfortunately, too few people are making the obvious leap from "LLMs aren't great for topics I have expertise in" to "maybe that means LLMs aren't actually great for the other topics either."
- deleted 1y ago[deleted]
- dtech 1y agoWe as humans aren't good at it. Before AI it was already coined as the "Gell-Mann Amnesia" effect
- deleted 1y ago[deleted]
- Rebuff5007 1y agoAnd a sizable portion of the population believe vaccines don't work and/or have 5G! I feel like I'm watching a tsunami about to hit while literally already drowning from a different tsunami.
- acchow 1y agoI’ve been trying ChatGPT for transit directions on Shanghai’s metro and it has been absolutely terrible. Hallucinating connections and routes. But all of it’s responses definitely seem convincing (as it has been trained to do)
- everybodyknows 1y agoA parallel to AI-slop has existed for generations now out here in meatspace: Administrative/legal people on the periphery of a technical field (though possibly alas, at the top of the org's command chain) who do not at all understand what technical terms signify, but having seen hundreds of sentences produced by real experts, become able to themselves string together plausible-looking assertions.
- breakpointalpha 1y agoWe call these people "compliance flies". Any large enough organization gathers them en mass to cloud real development work with "compliance."
- fisf 1y agoIt's also a fundamental problem of security research. Lot's of irrelevant, highly contextual "vulnerabilities", submitted to farm internet points (driven by a broken cve system). AI only amplifies this.
- lukev 1y ago"Form without substance" is quite possibly the most accurate way to describe LLMs in general. Extremely powerful (and extremely useful) pattern finders and replicators... and little else.
- 8note 1y agoits certianly why they pair well with tool calls - substance without form
- spwa4 1y agoBut the problem is fundamentally slop, and grifters. It is possible to generate valid security bug reports with an AI agent, and there certainly is value in that. I'd even say a lot of value. But of course producing fake ones is far easier and cheaper.
- seanmcdirmid 1y agoNo, it is the problem of any ceremonial barrier in existence. If substance wasn’t required in the first place, people were faking it already.
- godelski 1y agoThere's another term for this that I think should catch on: Cargo Culting Everything looks right but misses the underlying details that actually matter. There is a larger problem that I think we like to pretend that everything is so simple you don't need expertise. This is especially bad in our CS communities where there's a tendency of thinking intelligence in one domain cleanly transfers to others. In this respect I generally advise people not to first ask LLMs what they don't know but what they are experts in. That way they can properly evaluate their responses. Least we all fall for Murry Gelmann amnesia lol https://en.wikipedia.org/wiki/Cargo_cult https://en.wikipedia.org/wiki/Cargo_cult