5 ms·
I don't think it's just safeguards; they really don't seem to understand pitch at all. I tried asking ChatGPT's advanced voice mode to recognize a tune I was hu
by sbrother 1y ago
I don't think it's just safeguards; they really don't seem to understand pitch at all. I tried asking ChatGPT's advanced voice mode to recognize a tune I was humming, and it insisted it was Beethoven's 5th -- multiple times. I think it must have basically tokenized my humming to "dun dun dun duuun".
- bigzyg33k 1y agoadvanced voice mode operates on audio tokens directly, it doesn't transcribe them into "text tokens" as an intermediate step like the original version of voice mode did.
- cubefox 1y agoBut they behave just like models which use text tokens internally, which is also pointed out at the end of the above article.
- bigzyg33k 1y agowe don't know if that's due to inherent limitations of the tokenisation of audio, or a byproduct of reinforcement learning. In my own usage, I noticed a significant degradation in capabilities over time from when they initially released advanced voice mode. The model used to be able to sing, whisper, imitate sounds and tone just fine, but I imagine this was not intended and has subsequently been stunted via reinforcement learning. I don't find the articles argument that this is due to tokenisation convincing.
- cubefox 1y agoThey didn't say it's due to tokenization. > This is likely because they’re trained on a lot of data generated synthetically with text-to-speech and/or because understanding the tone of the voice (apparently) doesn’t help the models make more accurate predictions.
- sbrother 1y agoright, but either whatever audio tokenization it's doing doesn't seem to encode pitch, or there's ~nothing where pitch is relevant in the training set.
- oezi 1y agoAbsolutely correct! My simple test is if it can tell American and British English Tomato and Potato apart. So far it can't.
- fragmede 1y agoWhich "it" are you referring to? There are models that can.