4 ms·
Wow, this announcement is good content marketing. Don't get me wrong, it's interesting. But there is no technical discussion as to how they did it. It's simply
by melvinroest 1mo ago
Wow, this announcement is good content marketing.
Don't get me wrong, it's interesting. But there is no technical discussion as to how they did it. It's simply: we did it and Mythos and Codex didn't.
It's good to know that it's possible, but I'd have already expected it. Put a base model versus a base model + harness + whatever else, and yea, if you do it right then you have a better system to find vulnerabilities.
> We then ran AISLE's autonomous AI system against curl.
They don't even mention what models the use under the hood. It wouldn't surprise me if they are from Anthropic and OpenAI.
- drdrd 1mo ago> what models the use under the hood Presumably their own, wouldn’t they?
- melvinroest 1mo agoYou mean their own trained models, or do you think it's an open source model that they fine-tuned? If they use their own, I'd guess it's the latter.
- catlifeonmars 1mo agoMaybe the model doesn’t matter, maybe you just need something minimally intelligent to seed the fuzzer, generate a test case, and rinse and repeat when the fuzzer gets stuck.
- vorticalbox 1mo agoDefault to gpt 5.4 nano https://github.com/weareaisle/nano-analyzer/blob/main/scan.py#L43 https://github.com/weareaisle/nano-analyzer/blob/main/scan.p...
- tux3 1mo agoThe homepage says something about AI guided fuzzing based on libfuzzer or AFL. Looks like they have the LLMs identify a bunch of interesting functions to test, generate some test harnesses, and then sort through the fuzzer findings at a high level, which sounds like a pretty good idea.
- melvinroest 1mo agoThanks for figuring that out. Sort of sounds like AI programming programs to find vulnerabilities, of which fuzzing is one of the proven techniques to do it.
- bch 1mo agoAlso sounds incredibly compute intensive.
- matherial 1mo agoSetting a swarm of agents loose for hours to look for software vulnerabilities is far more compute-expensive than fuzzing. The industry has never thrown this kind of compute resources at pure fuzzing, in part because you can't get much VC money for that.
- lukeschlather 1mo agoThis sounds like a swarm of agents with particular prompting that happens to guide the LLMs toward doing a lot of fuzzing, so it's not either/or; you're getting all the compute requirements of both.
- rcxdude 1mo agoYou also quickly get into diminishing returns with fuzzing. Generally a bug is either going to be found relatively quickly with a given fuzzing approach or it's going to be nearly impossible to find. You're usually better coming up with more intelligent fuzzing approaches than you are just dumping compute into it.
- zamadatix 1mo agoFuzzing or having the LLM sort through where might be most useful to fuzz & sorting the results? Neither seem particularly compute intensive to me, fuzzing is a pretty standard step and having the LLM read through to find the most interesting areas to fuzz sounds a lot more efficient than leaving the whole task to the LLM.
- 1mo ago
- whizzter 1mo agoTheir system can run with various models, they go into more details in this article. https://aisle.com/blog/system-over-model-zero-day-discovery-at-the-jagged-frontier https://aisle.com/blog/system-over-model-zero-day-discovery-...
- wky 1mo agoIt wouldn’t surprise me if AISLE uses many different providers’ models, and what’s holding back OpenAI and Anthropic is only using first-party models. Just because OpenAI and Anthropic have arguably the strongest models overall doesn’t mean their models are the strongest at finding any given class of vulnerability or lead to follow.
- vorticalbox 1mo agoIt defaults to gpt5.4 nano https://github.com/weareaisle/nano-analyzer/blob/main/scan.py#L43 https://github.com/weareaisle/nano-analyzer/blob/main/scan.p...
- bryanlarsen 1mo agoA repo named "nano-analyzer" unsurprisingly uses gpt5.4 nano. I doubt their "pay them money" version uses nano.
- grumpy-swe-9000 1mo agoI am pretty sure that the nano-analyzer is just a limited open source demo of their "System over Model" thesis from https://aisle.com/blog/system-over-model-zero-day-discovery-at-the-jagged-frontier https://aisle.com/blog/system-over-model-zero-day-discovery-..., not the main product.
- 1970-01-01 1mo ago>All six are rated Low severity This says it all. Nothing important was missing. This is marketing hype.
- alephnerd 29d ago> Wow, this announcement is good content marketing. Why do you think companies hire PMMs?