4 ms·

Any enormous model (especially created by some sort of optimization) will have a huge surface of problems that needs a second, independent layer to box it in. I

Any enormous model (especially created by some sort of optimization) will have a huge surface of problems that needs a second, independent layer to box it in. If the box cannot be described properly that will fail, of course.

svaha1728 3y ago
The problem is the training set. If the training set contains something disruptive or of value, there is a nonzero percent chance it will be the next token. Boxes can attenuate that risk, but cannot set it to zero.
- dragonwriter 3y ago
  > If the training set contains something disruptive or of value, there is a nonzero percent chance it will be the next token. Boxes can attenuate that risk, but cannot set it to zero. I mean, if it was as simple as you describe ("nonzero percent chance that it will be next token"), then boxes could easily prevent it, the framework hosting the LLM can adjust logit bias on each token, so if it there was a readily-determinable forbidden-next-token, could prevent it with no problem. Unfortunately, the reality is...more complex.