3 ms·
To be clear, it's not LLMs creating the captions. Whisper[0], one of the best of its kind currently, is a speech recognition model, not a large language model.
by Sophira 11mo ago
To be clear, it's not LLMs creating the captions. Whisper[0], one of the best of its kind currently, is a speech recognition model, not a large language model. It's trained on audio, not text, and it can run on your mobile phone.
It's still AI, of course. But there is distinction between it and an LLM.
[0] https://github.com/openai/whisper/blob/main/model-card.md https://github.com/openai/whisper/blob/main/model-card.md
- big_toast 11mo agoIt’s an encoder-decoder transformer trained on audio (language?) and transcription. Seems kinda weird for it not to meet the definition in a tautological way even if it’s not the typical sense or doesn’t tend to be used for autoregressive token generation?
- uoaei 11mo agoIs it Transformer-based? If not then it's a different beast architecturally. Audio models tend to be based more on convolutional layers than Transformers in my experience.
- big_toast 11mo agoThe openai/whisper repo and paper referenced by the model card seem to be saying it's transformer based.
- LtWorf 11mo agoWhisper is a great random word generator when you use it on italian!
- janalsncm 11mo agoWhisper is an encoder decoder transformer. The input is audio spectrograms, the output is text tokens. It is an improvement over old school transcription methods because it’s trained on audio transcripts, so it makes contextually plausible predictions. Idk what the definition of an LLM is but it’s indisputable that the technology behind whisper is a close cousin to text decoders like gpt. Imo the more important question is how these things are used in the UX. Decoders don’t have to be annoying, that is a product choice.