3 ms·

The best case scenario for AI companies is, people receive those bug reports, look at the model that produced it and not even look at the details, just apply th

The best case scenario for AI companies is, people receive those bug reports, look at the model that produced it and not even look at the details, just apply the fix mindlessly This gives Anthropic a staggering amount of power. Oh it came from Mythos? We will just lose time trying to analyze it, better apply the fix ASAP

stingraycharles 4mo ago
> The best case scenario for AI companies is, people receive those bug reports, look at the model that produced it and not even look at the details, just apply the fix mindlessly Do people maintaining serious software do this, though?
- pixl97 4mo ago
  Define 'serious'. There is a lot of software in serious places written by very unserious people.
- nextaccountic 4mo ago
  The problem is that serious software is drowning in AI vulnerability reports. There is not enough manpower to analyze them properly. And if you ignore the reports (like curl is doing in their 1-month vacation), malicious actors will just exploit them. At some point it's inevitable to just rubber stamp whatever is coming from AI. The actual, underlying problem is that software is buggy and current programming languages aren't fit for writing reliable software. There's a wide gap between the state of art in formal verification, and what is actually practiced in the industry. It's because of this general unreliability that AI has a large supply of vulnerabilities to find. The situation will only get better if software becomes reliable and written in solid foundations. My guess is that AI will be even more useful to verify software (something like, write Lean or Coq proofs that the software is not vulnerable, things like that), rather than finding vulnerabilities piecemeal but still letting software be written in unsuitable languages, with no formal verification to prevent bugs from sneaking through.
  - cbzbc 4mo ago
    That gap explains much of the spike. Companies who never used any scanning tools on much of their codebase are suddenly having that gap closed.
  - re-thc 4mo ago
    > At some point it's inevitable to just rubber stamp whatever is coming from AI. To make it worse? AI and even Fable can make things +50% and then -50% in different places. You can trade 1 bug for another. So just "rubber stamp" doesn't make it better.
  - fweimer 4mo ago
    We have plenty of functions that convert one byte array to another byte array. Both arrays have specified bounds. The functions are total (an error return indicates if the input or output arrays are incomplete). Most of them do not even have state that is preserved between calls. Complete source code is available in the same build for all the functions they call. In theory, this should be very straightforward to prove correct with many of the current tools. In practice, no one has shown us how to do it. We could even rewrite the code from a macro/#include maze to proper function calls if that's a prerequisite for analysis. At this point, I would even take a one-off analysis.
realusername 4mo ago
Currently it looks like the opposite is happening haha, "oh it came from AI, let's discard it ASAP" is the trend in open source
- cmxch 4mo ago
  And if it’s Mythos or Claude derived (a strong hint for Mythos), discard it faster.
cmxch 4mo ago
TBH, I’d reject Mythos or similar reports and require full reproduction on a publicly available model before considering them valid.

stingraycharles 4mo ago

> The best case scenario for AI companies is, people receive those bug reports, look at the model that produced it and not even look at the details, just apply the fix mindlessly Do people maintaining serious software do this, though?

pixl97 4mo ago
Define 'serious'. There is a lot of software in serious places written by very unserious people.
nextaccountic 4mo ago
The problem is that serious software is drowning in AI vulnerability reports. There is not enough manpower to analyze them properly. And if you ignore the reports (like curl is doing in their 1-month vacation), malicious actors will just exploit them. At some point it's inevitable to just rubber stamp whatever is coming from AI. The actual, underlying problem is that software is buggy and current programming languages aren't fit for writing reliable software. There's a wide gap between the state of art in formal verification, and what is actually practiced in the industry. It's because of this general unreliability that AI has a large supply of vulnerabilities to find. The situation will only get better if software becomes reliable and written in solid foundations. My guess is that AI will be even more useful to verify software (something like, write Lean or Coq proofs that the software is not vulnerable, things like that), rather than finding vulnerabilities piecemeal but still letting software be written in unsuitable languages, with no formal verification to prevent bugs from sneaking through.
- cbzbc 4mo ago
  That gap explains much of the spike. Companies who never used any scanning tools on much of their codebase are suddenly having that gap closed.
- re-thc 4mo ago
  > At some point it's inevitable to just rubber stamp whatever is coming from AI. To make it worse? AI and even Fable can make things +50% and then -50% in different places. You can trade 1 bug for another. So just "rubber stamp" doesn't make it better.
- fweimer 4mo ago
  We have plenty of functions that convert one byte array to another byte array. Both arrays have specified bounds. The functions are total (an error return indicates if the input or output arrays are incomplete). Most of them do not even have state that is preserved between calls. Complete source code is available in the same build for all the functions they call. In theory, this should be very straightforward to prove correct with many of the current tools. In practice, no one has shown us how to do it. We could even rewrite the code from a macro/#include maze to proper function calls if that's a prerequisite for analysis. At this point, I would even take a one-off analysis.

realusername 4mo ago

Currently it looks like the opposite is happening haha, "oh it came from AI, let's discard it ASAP" is the trend in open source

cmxch 4mo ago
And if it’s Mythos or Claude derived (a strong hint for Mythos), discard it faster.

cmxch 4mo ago

TBH, I’d reject Mythos or similar reports and require full reproduction on a publicly available model before considering them valid.