CLIP has a distinct Vision Transformer and distinct Text Transformer model that are then matmul'd to create the aligned embedding space. Gato apparently just u

CLIP has a distinct Vision Transformer and distinct Text Transformer model that are then matmul'd to create the aligned embedding space. Gato apparently just uses a single model.