4 ms·
It's interesting that in the examples (Table 3 on page 21), the model uses the backspace token to erase the randomly-added token from the prompt, but it does no
by Imnimo 3y ago
It's interesting that in the examples (Table 3 on page 21), the model uses the backspace token to erase the randomly-added token from the prompt, but it does not seem to ever use the token to correct its own output. I'm curious how frequently the model actually uses this backspace token in practice - and if the answer is "vanishingly rarely", what is the source of the improved Mauve score and sample diversity they show? Is it just that the different training procedure gives an improvement?
- deleted 3y ago[deleted]
- nomel 3y ago> I'm curious how frequently the model actually uses this backspace token in practice For it to use the backspace, wouldn't it have to predict the wrong token with greater confidence than the corrected token? I would think this would require more examples of a wrong token + correction than the correct token, which seems a bit odd. I wonder if this could be have value for when a forward and reverse word predictors collide/fight. I naively assume reverse word predictor models will have the ability to work towards goals, by working from the "solution" back, with the forward word predictor acting as some sort of "causal resolution".