3 ms·

I enjoyed reading this article. I was doing research on speech recognition in the 2005 to 2010 period, and the article sums up the feeling. Everything was based

I enjoyed reading this article. I was doing research on speech recognition in the 2005 to 2010 period, and the article sums up the feeling. Everything was based on HMMs but there was no clear path to getting HMM performance up from where it capped out,or any idea of what models could work better. Deep learning came along and quickly supplanted all the old stuff. It seems like speech recognition has again plateaued but at a much higher accuracy, though its not inconceivable something better than transformers comes along and improves things some more. It seems like the next big leap may be combining the text understanding of GPT with waveform data.

unicoded 3y ago
What would be interesting is if Google took the chat GPT approach to better train itself. For instance, every time you speak to Google, it doesn't just 'guess' what you're saying, rather, it will present you with 3 options of the most closely sounding things for you to pick from. And by picking, you train the model.
- seanhunter 3y ago
  Reinforcement learning from human feedback (RLHF) as you have described here is pretty common in voice recognition.[1] I don't have any inside scoop but I'd be very surprised if there isn't any RLHF going on in these applications. edit to add: The whisper paper mentions using RLHF as a possible path to improve accuracy[2] [1] https://link.springer.com/article/10.1007/s10462-022-10224-2 https://link.springer.com/article/10.1007/s10462-022-10224-2 gives many examples [2] https://cdn.openai.com/papers/whisper.pdf https://cdn.openai.com/papers/whisper.pdf
wiz21c 3y ago
When you say "plateaued" again, do you have example ? I follow speech recognition like an average engineer and I mostly encounter "success" stories. So I'm afraid that I see things with rosy glasses. Do you have some examples where it fails in a meaningful way (that is, it fails on apparently simple, or usual, scenarios).
- hanselot 3y ago
  Google's SR running on our xiaomi spy devices in my house consistently gets things way wrong. Pretty much any SR I have ever used on any device seems incredibly bad. Obviously these devices don't use whisper etc, but even so, these are sold to consumers and 50% of the time they don't get the prompt correctly. We have "smart bulbs" that are connected to this spy network, which means that if I want to turn off a light, if I don't want to be shat on my by my partner I can't just turn off the switch, but rather stand pointlessly trying to turn off/on the lights by yelling at a fucking box in my lounge. I hate this so much more than you could ever begin to imagine. And somehow, all these "success" stories has not yet been enough for him to open his eyes. People will buy any shit and then convince themselves that they have "success" stories. Lets not even breach the topic of the fact that I have been using computer vision for years to do cool shit, but no, lets buy shit xiaomi cameras that have no API to interface with, so the absolute best case scenario for me knowing what's on the camera, is opening their piece of shit app and waiting 10-30 seconds for them to figure out how to connect over our network using their own fucking API. Since I'm on this tangent now, I would love to pay special mention to their PIECE OF SHIT KETTLE Which is supposed to be a smart kettle, but the one fucking thing you would want to be able to do remotely is not supported. Oh, you wanted to boil your kettle from your desk? Well Fuck you!
  - amelius 3y ago
    > Oh, you wanted to boil your kettle from your desk? Besides a right to repair, we should also have a right to API.
    - kybernetikos 3y ago
      Maybe it could be enforced via stricter recycling and warranty laws for devices that don't allow you to control them / update them / root them.
      - amelius 3y ago
        I really think we should have a consumer organization that grants rights to companies to mass produce items (so it would hit only large companies). That way we get a kind of democratic way to say "No" to products that work against the consumer (goodbye SmartTV) and end in a landfill within a few years. I mean, I need a permit to add a window to my home. Can we please require permits for mass-dumping semi-useless products onto the market?
BiteCode_dev 3y ago
Whisper is pretty good honestly. It gets my french accent in english and french slang, without having to tell it what language I'm going to speak first. That's impressive.
- courgette 3y ago
  Exactly, the accent bump is behind us. I get good transcripts of my meetings even with my tick accent
regularfry 3y ago
I was at a conference yesterday where they had live transcription of the speakers onto big screens by the stage. On this occasion it was a human stenographer in the back-end, but what was remarkable from is that if you didn't already know that, you'd guess from the error rate (not perfect, with difficulties that made sense in context) and the delay (bounced to another continent and back) that it was something like Whisper running locally. All the more so if you happened to know that one of the organisers has been running local AI meetups. It seems to me like the only way up from here, now, is to aim for better than the average human stenographer. You'd have to have a system capable of reasoning "ah, what I heard was Y but they must have really meant X" when X involves knowledge that couldn't possibly be predicted from the context, like a foreign name coming up for the first time. I can believe that if the stenographer had more domain-specific knowledge that they wouldn't have made some of the errors they did, but we're already off into capabilities that are as good as an extremely rare type of human today.

unicoded 3y ago

What would be interesting is if Google took the chat GPT approach to better train itself. For instance, every time you speak to Google, it doesn't just 'guess' what you're saying, rather, it will present you with 3 options of the most closely sounding things for you to pick from. And by picking, you train the model.

seanhunter 3y ago
Reinforcement learning from human feedback (RLHF) as you have described here is pretty common in voice recognition.[1] I don't have any inside scoop but I'd be very surprised if there isn't any RLHF going on in these applications. edit to add: The whisper paper mentions using RLHF as a possible path to improve accuracy[2] [1] https://link.springer.com/article/10.1007/s10462-022-10224-2 https://link.springer.com/article/10.1007/s10462-022-10224-2 gives many examples [2] https://cdn.openai.com/papers/whisper.pdf https://cdn.openai.com/papers/whisper.pdf

wiz21c 3y ago

When you say "plateaued" again, do you have example ? I follow speech recognition like an average engineer and I mostly encounter "success" stories. So I'm afraid that I see things with rosy glasses. Do you have some examples where it fails in a meaningful way (that is, it fails on apparently simple, or usual, scenarios).

hanselot 3y ago
Google's SR running on our xiaomi spy devices in my house consistently gets things way wrong. Pretty much any SR I have ever used on any device seems incredibly bad. Obviously these devices don't use whisper etc, but even so, these are sold to consumers and 50% of the time they don't get the prompt correctly. We have "smart bulbs" that are connected to this spy network, which means that if I want to turn off a light, if I don't want to be shat on my by my partner I can't just turn off the switch, but rather stand pointlessly trying to turn off/on the lights by yelling at a fucking box in my lounge. I hate this so much more than you could ever begin to imagine. And somehow, all these "success" stories has not yet been enough for him to open his eyes. People will buy any shit and then convince themselves that they have "success" stories. Lets not even breach the topic of the fact that I have been using computer vision for years to do cool shit, but no, lets buy shit xiaomi cameras that have no API to interface with, so the absolute best case scenario for me knowing what's on the camera, is opening their piece of shit app and waiting 10-30 seconds for them to figure out how to connect over our network using their own fucking API. Since I'm on this tangent now, I would love to pay special mention to their PIECE OF SHIT KETTLE Which is supposed to be a smart kettle, but the one fucking thing you would want to be able to do remotely is not supported. Oh, you wanted to boil your kettle from your desk? Well Fuck you!
- amelius 3y ago
  > Oh, you wanted to boil your kettle from your desk? Besides a right to repair, we should also have a right to API.
  - kybernetikos 3y ago
    Maybe it could be enforced via stricter recycling and warranty laws for devices that don't allow you to control them / update them / root them.
    - amelius 3y ago
      I really think we should have a consumer organization that grants rights to companies to mass produce items (so it would hit only large companies). That way we get a kind of democratic way to say "No" to products that work against the consumer (goodbye SmartTV) and end in a landfill within a few years. I mean, I need a permit to add a window to my home. Can we please require permits for mass-dumping semi-useless products onto the market?

BiteCode_dev 3y ago

Whisper is pretty good honestly. It gets my french accent in english and french slang, without having to tell it what language I'm going to speak first. That's impressive.

courgette 3y ago
Exactly, the accent bump is behind us. I get good transcripts of my meetings even with my tick accent

regularfry 3y ago

I was at a conference yesterday where they had live transcription of the speakers onto big screens by the stage. On this occasion it was a human stenographer in the back-end, but what was remarkable from is that if you didn't already know that, you'd guess from the error rate (not perfect, with difficulties that made sense in context) and the delay (bounced to another continent and back) that it was something like Whisper running locally. All the more so if you happened to know that one of the organisers has been running local AI meetups. It seems to me like the only way up from here, now, is to aim for better than the average human stenographer. You'd have to have a system capable of reasoning "ah, what I heard was Y but they must have really meant X" when X involves knowledge that couldn't possibly be predicted from the context, like a foreign name coming up for the first time. I can believe that if the stenographer had more domain-specific knowledge that they wouldn't have made some of the errors they did, but we're already off into capabilities that are as good as an extremely rare type of human today.