3 ms·
Does this paper actually say how they provide the image/audio data to the language model? You can't really just pipe 16khz audio into CLIP and slap that into yo
by bastawhiz 3y ago
Does this paper actually say how they provide the image/audio data to the language model? You can't really just pipe 16khz audio into CLIP and slap that into your model of choice. I skimmed the paper, then skimmed it again, and I don't have an answer for what exactly they did here.
- qayxc 3y agoThe paper actually lays it out pretty clearly. Yes, you can just pipe 16kHz audio into a tokenizer, which is exactly what they did (kind of). They just treat the input as 8-bit ASCII (which in itself is technically not a thing, since ASCII is 7-bit, but the paper doesn't seem to care about technicalities). So the input alphabet is 256 entries - 1 for each byte. Simple as that.