3 ms·
Are multi-format embeddings — like this one, or simpler ones like text-to-image — inherently bijective? I.e. could a model like this be configured to use any of
by derefr 4y ago
Are multi-format embeddings — like this one, or simpler ones like text-to-image — inherently bijective? I.e. could a model like this be configured to use any of its trained sense data types as either an input or output? Or is there some bias in the way the training data is ingested that makes the model better at transforming in certain directions?
- tkellogg 4y agoLike laser eyes?
- jacobr1 4y agoI haven't yet read this paper, but other multimodal models have different layers for mapping the input/output of each modality to the common embedding space. So, to simplify, there is some high-dimensional coordinate that the audio sample for the morphemes for "frog," the text word "frog" and maybe the same in multiple languages like, "rana" - they all basically have the same point in the abstract feature space. The additional layers of the model then provide "translation"