2 ms·
If you are talking about publicly known vulns, it's a bit moot since they should be in the training sets. If not, you just burned the vulns to that inference pr
by Aissen 17d ago
If you are talking about publicly known vulns, it's a bit moot since they should be in the training sets. If not, you just burned the vulns to that inference provider's training data (and any intermediary), and future benchmarks will be meaningless.
- spider-mario 17d ago> If not, you just burned the vulns to that inference provider's training data (and any intermediary), and future benchmarks will be meaningless. Even assuming that the provider necessarily trains on what it’s used on, how does “Find all vulnerabilities in this code: <code>” and checking that it contains all the expected vulnerabilities magically make future models aware of what the expected vulnerabilities were?
- deleted 17d ago[deleted]
- networked 17d ago> If not, you just burned the vulns to that inference provider's training data (and any intermediary), and future benchmarks will be meaningless. Inference providers can credibly promise to not train on your data if they are in a position to get sued.