3 ms·

You could also do some adverserial training (basically iteratively attempt this attack and add the resulting exploits to the training set). Research in machine

You could also do some adverserial training (basically iteratively attempt this attack and add the resulting exploits to the training set). Research in machine vision suggests this is possible, and even has some positive effects, but it significantly degrades capabilities.

brucethemoose2 3y ago
A "pre filter" could classify and filter the prompts without affecting the actual generative models. But this would make APIs increasingly annoying and unreliable with false positives and such. It seems like another advantage for local LLMs.
ke88y 3y ago
> but it significantly degrades capabilities On a train/test/eval split. But the degradation is lower on OOD data. Which suggests perhaps the degradation is merely "less overfitting".
- DoctorOetker 3y ago
  I think most researchers would consider such a phenomena far from obvious and thus worthy of publication. Do you know or have any references on this? If one disregards the emphasis on alignment and "merely" considers the "less overfitting" aspect, that would seem very profound in and of itself, a capability to avoid overfitting. If you look at historical debates in the sciences, where universal truths are sought, candidate truths are intentionally and adversarially stretched to apparent or real inconsistencies in order to test the universality of a claim. Think of say the back and forths between Einstein and Bohr concerning entanglement. They are assuming adversarial roles, tricking each others belief system into expressing the absurd. Together they mapped out predictions for non-obvious or outright bizarre aspects of reality. If no-one dares to take a potentially vulnerable position, there will be nothing to attack, but also nothing to disturb the scientific mind and prod the community into settling the matter by measurements.

3 ms·

You could also do some adverserial training (basically iteratively attempt this attack and add the resulting exploits to the training set). Research in machine

by sebzim4500 3y ago

brucethemoose2 3y ago
A "pre filter" could classify and filter the prompts without affecting the actual generative models. But this would make APIs increasingly annoying and unreliable with false positives and such. It seems like another advantage for local LLMs.
ke88y 3y ago
> but it significantly degrades capabilities On a train/test/eval split. But the degradation is lower on OOD data. Which suggests perhaps the degradation is merely "less overfitting".
- DoctorOetker 3y ago
  I think most researchers would consider such a phenomena far from obvious and thus worthy of publication. Do you know or have any references on this? If one disregards the emphasis on alignment and "merely" considers the "less overfitting" aspect, that would seem very profound in and of itself, a capability to avoid overfitting. If you look at historical debates in the sciences, where universal truths are sought, candidate truths are intentionally and adversarially stretched to apparent or real inconsistencies in order to test the universality of a claim. Think of say the back and forths between Einstein and Bohr concerning entanglement. They are assuming adversarial roles, tricking each others belief system into expressing the absurd. Together they mapped out predictions for non-obvious or outright bizarre aspects of reality. If no-one dares to take a potentially vulnerable position, there will be nothing to attack, but also nothing to disturb the scientific mind and prod the community into settling the matter by measurements.