6 ms·
I think something that has concerned me with the concept of scaling to AGI is the concept of "adversarial examples". Small tweaks that can be made to cause unpr
by 3vidence 4y ago
I think something that has concerned me with the concept of scaling to AGI is the concept of "adversarial examples". Small tweaks that can be made to cause unpredictable behavior in the system. At a high level these are caused by unexpected paths in high dimensional model weight space that don't align with our intuition. This problem in general seems to get worse as the size of the weights grow.
From a value perspective a very high fidelity model with extremely unexpected behavior seems really low value since you need a human there full time to make sure that the model doesn't go haywire that 1-5% of the time
- danuker 4y agoSuppose an adversarial example is recognized correctly by a human, yet a model recognizes it wrongly. Therefore, the model uses other information than humans in order to classify, and the information it uses is wrong. Therefore, the model needs to be trained ON THE ADVERSARIAL EXAMPLES in order to gain robustness. Similarly to GANs using a classifier adversary for augmenting a generative network, one could use a generative adversary for augmenting a classifier network. You can repeat until the adversarial examples look ambiguous even to a human.