Hubert does something similar; it predicts masked spectrogram regions during its first phase, and then masked embeddings later in training. It's a bit of a pain

Hubert does something similar; it predicts masked spectrogram regions during its first phase, and then masked embeddings later in training. It's a bit of a pain to get the complicated training regimen to work well, though.