3 ms·
Masking looks interesting for sequences that can't be lossy. If an image squishes a pixel here or there, it won't be noticed, but if a sentence lacks room for "
by flowerthoughts 2y ago
Masking looks interesting for sequences that can't be lossy. If an image squishes a pixel here or there, it won't be noticed, but if a sentence lacks room for "if", that sounds bad.
Does this force the model to encode a high-level answering strategy? (AFAIU, there's no reordering during sampling.) Or does it mean a masking model of a certain size is more prone to making things up that fit the blank space?
- cheald 2y agoI'd imagine you could exploit something like the stochastic denoising approach from DDIM and descendants, where you add some noise back after each denoising step, essentially randomly remasking unmasked tokens and giving the model a second chance to unmask them "properly" as the denoised response becomes better-known.