4 ms·

Basically they cut out a part of the input and make the network predict the missing part. (edit: they actually predict the average of all features). This works

Basically they cut out a part of the input and make the network predict the missing part. (edit: they actually predict the average of all features). This works for images, audio, text. This produces high quality feature representations for data which can be used to build specialised networks on. The two main tricks are: 1. Do the cutout in feature space, not the original input space. (edit: cutout is actually in input space) 2. The above would likely just collapse the features to 0, so they use the same network that does the reconstruction to produce the features (!). In their own words: "We first encode a masked version of the training sample (model in student mode) and then construct training targets by encoding the unmasked version of the input sample with the same model but when parameterized as an exponentially moving average of the model weights (model in teacher mode)"

lopuhin 5y ago
That's not quite how I understood it > Do the cutout in feature space, not the original input space. I think they do the cutout in the original input space, based on examples they show, e.g. they mask parts of text and grey out parts of an image. > and make the network predict the missing part I think predict the latent representation (as you say in (2)), but not of the missing part, but of the whole corrupted sample, and require it to be close to latent representation of the original input done by the teacher.
- l-lousy 5y ago
  So they’re doing both masking for the inputs and knowledge distillation? Is it the combination of these two methods that’s novel?
- WithinReason 5y ago
  You're right, they cut in the original space and average-pool the features before an L1/L2 loss
dfgfhjkjlkhgjfh 5y ago
I believe that your interpretation is not correct. Based on my brief reading of the paper, the model contains 1) some known architecture for embedding the modality, and 2) the feature reconstruction transformer network, the two being trained at the same time. If I am not mistaken, the masking occurs in the input modality, not the feature-space, even though it is the feature-space that is used for the reconstruction task. Regarding how the feature space is kept uncollapsed, it seems like a hyperparameter-tweaking (ie unsolved?) problem; quoting the paper: " Representation collapse. A common issue with algorithms which create and predict their own targets is representation collapse. This occurs when the model produces very similar representations for all masked segments making the problem trivial to solve. Different strategies have been proposed to address this issue, e.g., contrastive models such as wav2vec 2.0 (Baevski et al., 2020b) use the same target representation both as a positive and a negative example, preventing collapse. Algorithms such as BYOL (Grill et al., 2020) do not optimize the teacher parameters to minimize the loss. VicReg (Bardes et al., 2021) adds an explicit loss encouraging variance among different representations. In our experiments we found that collapse is most likely to happen in the following scenarios: First, the learning rate is too large or the learning rate warmup is too short which can often be solved by tuning the respective hyper-parameters. Second, the EMA decay rate is too low which leads to student model collapse which is propagated to the teacher due to parameter tracking. This can be addressed by carefully tuning τ0, τe and τn. Third, we found collapse to be more likely for modalities where adjacent targets are very correlated and where longer spans need to be masked, such as for speech. We address this by either explicitly penalizing the lack of variance (Bardes et al., 2021), or by promoting variance through normalizing target representations over the current sequence or batch (Grill et al., 2020). The former worked well for small models but is less reliable for larger models and it also requires tuning additional hyper-parameters. In contrast, we found applying instance or batch normalization before or after averaging targets to work well while being simpler. For models where targets are less correlated such as for vision and NLP, momentum tracking is sufficient to prevent representation collapse. "
- WithinReason 5y ago
  You're right. That seems to explain why representations don't collapse into a constant, but not why they don't collapse to the same feature...

lopuhin 5y ago

That's not quite how I understood it > Do the cutout in feature space, not the original input space. I think they do the cutout in the original input space, based on examples they show, e.g. they mask parts of text and grey out parts of an image. > and make the network predict the missing part I think predict the latent representation (as you say in (2)), but not of the missing part, but of the whole corrupted sample, and require it to be close to latent representation of the original input done by the teacher.

l-lousy 5y ago
So they’re doing both masking for the inputs and knowledge distillation? Is it the combination of these two methods that’s novel?
WithinReason 5y ago
You're right, they cut in the original space and average-pool the features before an L1/L2 loss

dfgfhjkjlkhgjfh 5y ago

I believe that your interpretation is not correct. Based on my brief reading of the paper, the model contains 1) some known architecture for embedding the modality, and 2) the feature reconstruction transformer network, the two being trained at the same time. If I am not mistaken, the masking occurs in the input modality, not the feature-space, even though it is the feature-space that is used for the reconstruction task. Regarding how the feature space is kept uncollapsed, it seems like a hyperparameter-tweaking (ie unsolved?) problem; quoting the paper: " Representation collapse. A common issue with algorithms which create and predict their own targets is representation collapse. This occurs when the model produces very similar representations for all masked segments making the problem trivial to solve. Different strategies have been proposed to address this issue, e.g., contrastive models such as wav2vec 2.0 (Baevski et al., 2020b) use the same target representation both as a positive and a negative example, preventing collapse. Algorithms such as BYOL (Grill et al., 2020) do not optimize the teacher parameters to minimize the loss. VicReg (Bardes et al., 2021) adds an explicit loss encouraging variance among different representations. In our experiments we found that collapse is most likely to happen in the following scenarios: First, the learning rate is too large or the learning rate warmup is too short which can often be solved by tuning the respective hyper-parameters. Second, the EMA decay rate is too low which leads to student model collapse which is propagated to the teacher due to parameter tracking. This can be addressed by carefully tuning τ0, τe and τn. Third, we found collapse to be more likely for modalities where adjacent targets are very correlated and where longer spans need to be masked, such as for speech. We address this by either explicitly penalizing the lack of variance (Bardes et al., 2021), or by promoting variance through normalizing target representations over the current sequence or batch (Grill et al., 2020). The former worked well for small models but is less reliable for larger models and it also requires tuning additional hyper-parameters. In contrast, we found applying instance or batch normalization before or after averaging targets to work well while being simpler. For models where targets are less correlated such as for vision and NLP, momentum tracking is sufficient to prevent representation collapse. "

WithinReason 5y ago
You're right. That seems to explain why representations don't collapse into a constant, but not why they don't collapse to the same feature...