I haven't yet read this paper, but other multimodal models have different layers for mapping the input/output of each modality to the common embedding space. So

I haven't yet read this paper, but other multimodal models have different layers for mapping the input/output of each modality to the common embedding space. So, to simplify, there is some high-dimensional coordinate that the audio sample for the morphemes for "frog," the text word "frog" and maybe the same in multiple languages like, "rana" - they all basically have the same point in the abstract feature space. The additional layers of the model then provide "translation"