They are using CNNs to encode the input. But… Data2Vec uses a “standard” transformer architecture.[1] In Data2Vec2 the transformer architecture forms the “bul

They are using CNNs to encode the input. But… Data2Vec uses a “standard” transformer architecture.[1] In Data2Vec2 the transformer architecture forms the “bulk of the model weights”.[2] The tweaks I eluded to should have more clearly referenced that each modality uses a different encoder. [1] section 3.1 data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language. [2] section 3.2 Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Language.