8 ms·

Whispers of A.I.’s Modular Future

162 points by lispybanana 4y ago

nmfisher 4y ago
Whisper is great, but I also wouldn't overlook other next-generation models (RNN-T/Zipformer/etc) trained on 50k+ hour datasets. These also perform very well. That being said, Whisper is clearly a far cry from "intelligence". This should be clear when you feed it 5 seconds of silence and get hallucinated garbage in return. It's much more akin to compressing those huge datasets into something that can feasibly be run on recent hardware. That's not to downplay how impressive that is, just to draw a clear line between "compression" and "intelligence".
- wyldfire 4y ago
  Don't state of the art commercial systems do something similar? I assume there must be some automatic gain boosting the noise at the frontend of most pipelines, I know I've gotten transcribed voicemails that really just are silence but the transcript shows lots and lots of hallucinated words. Regardless of "intelligence" it's got real utility.
  - nmfisher 4y ago
    I occasionally get a single hallucinated word (more like a mis-transcription) where the audio contains a clunk/bang/cough/etc, but I've never had full hallucinated phrases from clean silence. There are a couple of GitHub discussions on the Whisper repository with various fixes/hacks to deal with it: https://github.com/openai/whisper/discussions/679 https://github.com/openai/whisper/discussions/679 https://github.com/openai/whisper/discussions/813 https://github.com/openai/whisper/discussions/813 If you get a chance, I encourage you to try out the other newer models I mentioned, I think you'd be very impressed.
    - pixl97 4y ago
      I don't see this much different than what commonly happens with humans when we hear our named called when it was some environmental noise. As for the silence, I wonder why the the model even receives it. I would think a lot of that would be compressed out of existence to save bandwidth.
  - qwertox 4y ago
    I have an app on my phone which creates a 1 minute audio file when I press a button. I have a lavalier microphone connected to the phone and use it to record notes while riding my bike. It's always 1 minute because that is usually enough, and if I see that I need more time, I record an overlapping second file. Last week I set up a Whisper instance on my server and have been feeding it with these files. The result is pretty good. I usually can remember what I was saying when I read the transcription, which usually contains a couple of errors. Then there are those added hallucinations which are entire sentences, like: ---- 00:00.000 --> 00:05.000 Also temperaturmäßig ist es recht gut. [So temperature wise, it's pretty good.] 00:05.000 --> 00:09.000 Der eine hat 12 Grad, der andere 10. [One has 12 degrees, the other 10. (I have two temperature sensors mounted on the bike, ESP32 streaming the data to the phone via BLE)] 00:09.000 --> 00:12.000 Also sagen wir mal, 10 Grad. [So let's say 10 degrees.] 00:14.000 --> 00:19.000 Es ist bewölkt und windig. [It's cloudy and windy.] 00:20.000 --> 00:24.000 Aber irgendwie vom Wetter her gut. [But somehow from the weather it's good.] 00:24.000 --> 00:31.000 Ich habe heute überhaupt nichts gegessen und sehr wenig getrunken. [I ate nothing at all today and drank very little.] 00:54.000 --> 00:59.000 Vielen Dank für's Zuschauen! [Thanks for watching!] Transcribed in 77.2 seconds ---- The last sentence, "Thanks for watching!" is a complete hallucination. There were 30 seconds remaining which were me breathing and the wind blowing into the microphone and it came up with that comment. I usually comment on the weather because I take note of what I am wearing, and it allows me to better prepare for future rides. 77 seconds for the 60 second file because my server has no GPU, so I'm running the large model on the CPU (in a VM which has 8 cores assigned to it from a Ryzen 9 5950X). I've been considering buying a small PC with a 3060 RTX only for inferencing, but it may be too expensive. I tried Google Speech-To-Text and it is nowhere as good as Whisper under these conditions (having the wind noise and the heavy breathing). This is Google's result: ---- "Also temperaturmäßig es ist recht gut, der eine hat 12° andere 10. Es ist angemalte 10 Grad. Es ist bewölkt und windig, aber er hat sie vom Wetter her gut, ich wollte überhaupt nichts gegessen und sehr wenig getrunken." ["So temperature-wise it's pretty good, one has 12° other 10. It's painted 10 degrees. It's cloudy and windy, but he has it good from the weather, I did not want to eat anything at all and drank very little."] ---- Also, whisper.cpp doesn't seem to generate the same results, and they appear to be not so good (in this case it was almost just as good). I just tested the same file on whisper.cpp with the large model and it's even funnier: ---- [00:00:00.000 --> 00:00:05.000] also temperaturmäßig ist es recht gut [...] [00:00:05.000 --> 00:00:09.000] der eine hat 12 Grad, der andere 10 [...] [00:00:09.000 --> 00:00:12.000] also sagen wir mal so 10 Grad [...] [00:00:12.000 --> 00:00:19.000] es ist bewölkt und windig [...] [00:00:19.000 --> 00:00:24.000] aber irgendwie vom Wetter her gut [...] [00:00:27.000 --> 00:00:31.000] ich habe heute überhaupt nichts gegessen und sehr wenig getrunken [...] [00:00:31.000 --> 00:00:35.000] das ist der Grund, warum ich so viel auf dem Knie gehe [this is the reason why I go so much on the knee] [00:00:35.000 --> 00:00:39.000] das war's, bis zum nächsten Mal! [that's it, until next time!] [00:00:39.000 --> 00:00:59.000] Danke fürs Zuschauen! [Thanks for watching!] `time` yields 567.63s user 1.99s system 755% cpu 1:15.36 total ---- The first 30 seconds, where the text is clearly understood, is inferenced within ~10-15 seconds. It's the "silence" which makes the AI go crazy on the workload. The idea behind this is to set up a system which then sends me an email with a map and trail of the ride as well as the transcriptions of the notes.
  - uncanneyvalley 4y ago
    It’s not that your audio is being amplified, it’s that the VAD classifier is poorly tuned. The noise should never even reach the recognition stage. Whisper’s hallucinations are pretty severe, but are improved by adding VAD to its pipeline.
- panarky 4y ago
  It's a tool that's free as in freedom, and it is incredibly useful. How does discovering that it doesn't handle some weird use case diminish its utility as a tool? Hammers are awesome at driving nails into wood. But if you strike the wood directly without a nail, the hammer puts a dent in the wood. Is that a defect in the hammer? Does it somehow make the hammer any less useful as a tool?
  - nmfisher 4y ago
    Not sure what makes you think I'm diminishing its utility as a tool; like I said, it's an incredible tool and I lean on it very heavily for various speech-processing pipelines. I'm just pointing out that Whisper definitely hasn't "solved" speech recognition, and there's still a lot more fertile ground to cover from a research perspective.
    - pixl97 4y ago
      In my personal believe I don't think speech recognition, at least based on any human model is solvable, at best we can get 'as good as an average set of humans that speak the language' People mumble crap all the time to other humans and need to repeat what they say with proper enunciation. People have hearing problems, which would correlate to microphone quality/placement issues when dealing with computer systems. Then there are issues where people say one thing incorrectly, but the person following/listening to the directions knows the procedure and does the correct thing. If you asked the speaker what they said again, they'd say they said the 'correct' thing in the first place. And this is something that I've done before as an example. Me: "Click the X button to start the process" Person writing the notes: "Click the Y button to start the process" Person writing the notes: You meant click the Y button right. Me: Yea, that's what I said. Oops, its when we get in things like this that we run into the unsolvable speech recognition issues because we don't generally understand our own error bars on what we say. The speech quality between a public speaker and the average joe I'm sure has a very wide range.
      - panarky 4y ago
        > at best we can get as good as an average set of humans Whisper is already superhuman, more accurate than experienced human transcribers.
- jasode 4y ago
  >, just to draw a clear line between "compression" and "intelligence" Not disagreeing with you but your sentence reminded of research looking at the link between "compression" and "intelligence" : https://www.google.com/search?q=compression+is+a+form+of+intelligence https://www.google.com/search?q=compression+is+a+form+of+int...
krmbzds 4y ago
https://archive.is/FEGf1 https://archive.is/FEGf1
rileyphone 4y ago
Great article, we are only just beginning to see the impact of Whisper. I hope at least that it will trickle into my Alexa sooner rather than later, but I've been scheming other uses for it too. Dictation to notes so I can think out loud. Make transcripts of talks that I would rather read than listen to. The possibilities are endless.
wyldfire 4y ago
Aside - I'd love to see similar rust and zig implementations like whisper.cpp. I'll donate $150 USD to zig and rust foundations as a bounty for respective MIT-licensed implementations of these. Let's keep it simple - scalar instructions, no need for intrinsics/assembly. Ideally there would be some tests. whisper.cpp looks like a simple-enough-but-very practical application and I think it would help promote these modern languages to have a simple and portable demonstration like this.
- fbdab103 4y ago
  What does a Rust/Zig port buy that the current implementation cannot do?
  - wyldfire 4y ago
    Nothing. I tried to address my desire for these items at the end there - it would serve as an excellent demonstration of the power of these languages (to take on a performance critical task like audio transcription).
  - itake 4y ago
    Just taking a stab, but if your code is primarily written in rust/zig, it’s really annoying calling c/cpp libraries, because you have to build them and keeping the bindings in sync after updates.
- deleted 4y ago
  [deleted]
- deleted 4y ago
  [deleted]
ur-whale 4y ago
https://github.com/ggerganov/whisper.cpp https://github.com/ggerganov/whisper.cpp
thimm 4y ago
I love whisper. It is so easy to use. I created a small pipeline that transcribes podcasts within the domain that I'm working in. It helps me and my colleagues to revisit and find podcasts episodes without having to listen to them again. You can check it out on podcasts.farmonapp.com
- losthobbies 4y ago
  This is really interesting. I am looking at doing something similar. Do you mind me asking what the Back-end API call is written in? I had looked at Deepgram and might try putting a small project together.
  - thimm 4y ago
    It is all written in Python and it uses the original python bindings. I'm using mkdocs to convert the transcripts into a website.
aaron695 4y ago
[dead]
EZ-Cheeze 4y ago
"There could be even larger changes—we talk a lot, and almost all of it goes into the ether. What if people recorded conversations as a matter of course, made transcripts, and referred back to them the way we now look back to old texts or e-mails?" Minority Report-level awesome futurism One thing that changes everything
- cloudking 4y ago
  This exists already https://www.rewind.ai/ https://www.rewind.ai/
  - anonyfox 4y ago
    I had no idea. instantly subscribed. not joking.
braindead_in 4y ago
Whisper is awesome, but managing it in production environment is not easy. I am waiting OpenAI (or someone else) to offer a API with a Real Time Factor of < 1. RTF is inference time/duration of the file. We can really use a that.
- garblegarble 4y ago
  Doesn't whisper.cpp already get you that? It takes ~6 seconds per 30 second segment on an M1 Max with the Large model. Do you mean you want snappy appearance of words shortly after you say them, rather than having to recognise in 30 second segments?
- qwertox 4y ago
  Getting a server running is easy if you use https://github.com/ahmetoner/whisper-asr-webservice https://github.com/ahmetoner/whisper-asr-webservice as a guide. It's then a REST API which you post the file to and get the transcription in return. But I don't know what you consider being "in production". If it's for internal use then it is enough. Here are some comparisons of running it on GPU vs CPU According to https://github.com/MiscellaneousStuff/openai-whisper-cpu https://github.com/MiscellaneousStuff/openai-whisper-cpu the medium model needs 1.7 seconds to transcribe 30 seconds of audio when run on a GPU.
ilaksh 4y ago
Whisper is great, but Google's built-in speech-to-text thing in Chrome with https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_API/Using_the_Web_Speech_API https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_... has been available for quite some time and works well. Not sure if Firefox has a implementation, probably one that works locally but not nearly as well. Obviously not having to send all of the data to Google is a big deal. But practically speaking the Google recognition seems to perform extremely well. So not sure this is really a new capability for people if they were willing to use Google's servers. It seems like Firefox and Chrome should now ship with Whisper built in. Or they should work on that.. and if Chrome doesn't add it then.. that's suspicious.
- qwertox 4y ago
  You may want to read my comment which I posted to another comment, it has a comparison to Google's ASR.
- krisoft 4y ago
  > So not sure this is really a new capability for people if they were willing to use Google's servers. Of course it is. Google is providing the API at their largess. One day they can decide they won't do it anymore, or they will charge you huge piles of money, or they decide they don't like something about your data and they won't process it even for all the money. Or will do it but only send back nonsense to you. It is their endpoint and they do whatever they are pleased with it. On the other hand you can take Whisper and use it on your own computer. Or rent a computer from anyone who does that kind of service. If you don't like how it performs you can refine it on your own datasets. If you have tens of thousands hours of sound files you want to convert you can just calculate what it will cost you to convert with Whisper. With google? Good luck. The difference between running Whisper on your own hardware vs using Google's endpoint is the difference between having your own car vs the mercurial rich guy not minding if skaters hold on his taillights to hitch a ride for now.
jbotz 4y ago
The article concludes: "Eventually, though, someone will release a program that’s nearly as capable as ChatGPT, and entirely open-source. An enterprising amateur will find a way to make it run for free on your laptop." Well not in the near future... Large language models like ChatGPT are called "large" for a reason, and they're too big for your laptop for the foreseeable future. I think you'd need a computer with several high-end GPUs that each have 128+GB of RAM to be able to run one of those. Maybe a laptop from 2030 will do. Otherwise a very good article.
- sebzim4500 4y ago
  FlexGen can already run GPT-3 size models on commodity hardware, albeit with high latency and fairly slow throughput (order of 1 token/s).
- api 4y ago
  You can run giant models on a high-end laptop today... they'll just be slooooowww since you'll be doing things like swapping data in/out and leveraging the CPU. If you don't mind waiting an hour for a prompt response it can work. It's the same as it's always been. Any general purpose computer is Turing complete. Spending more gets you faster results.
- wyldfire 4y ago
  > to be able to run one of those It's very important to distinguish the different use cases between training and inference. The amount of memory required to execute the ChatGPT model, once trained, is likely much, much less than 128GB.
  - dxuh 4y ago
    Afaik you actually need much more. About 400GB just to load the trained model according to this tweet thread: https://twitter.com/tomgoldsteincs/status/1600196981955100694?lang=de https://twitter.com/tomgoldsteincs/status/160019698195510069... I am not quite sure how reliable the source is, but it makes sense that you at least need to store the 175 billion parameters that define the model in VRAM. I know that for a short-ish while GPUDirect storage is a thing, so that could help for sure, but it would definitely impact execution time as well.
tikkun 4y ago
What utilities related to Whisper do you wish existed? What have you had to build yourself? On the end user application side, I wish there was something that let me pick a podcast of my choosing, get it fully transcribed, and get an embeddings search plus answer q&a on top of that podcast or set of chosen podcasts. I've seen ones for specific podcasts, but I'd like one where I can choose the podcast. (Probably won't build it) Also on the end user side, I wish there was an Otter alternative (still paid $30/mo, but unlimited minutes per month) that had longer transcription limits. (Started building this, not much interest from users though) Things I've seen on the dev tool side: Gladia (API call version of Whisper) Whisper.cpp Whisper webservice (https://github.com/ahmetoner/whisper-asr-webservice https://github.com/ahmetoner/whisper-asr-webservice) - via this thread Live microphone demo (not real time, it still does it in chunks) https://github.com/mallorbc/whisper_mic https://github.com/mallorbc/whisper_mic Streamlit UI https://github.com/hayabhay/whisper-ui https://github.com/hayabhay/whisper-ui Whisper playground https://github.com/saharmor/whisper-playground https://github.com/saharmor/whisper-playground Real time whisper https://github.com/shirayu/whispering https://github.com/shirayu/whispering Whisper as a service https://github.com/schibsted/WAAS https://github.com/schibsted/WAAS Improved timestamps and speaker identification https://github.com/m-bain/whisperX https://github.com/m-bain/whisperX MacWhisper https://goodsnooze.gumroad.com/l/macwhisper https://goodsnooze.gumroad.com/l/macwhisper Crossplatform desktop Whisper that supports semi-realtime https://github.com/chidiwilliams/buzz https://github.com/chidiwilliams/buzz
- gtirloni 4y ago
  I think AssemblyAI has tools for that.
- causi 4y ago
  Add sponsor-skipping into that. Give me a transcript, let me select a series of words, then audio containing that series of words gets skipped on all remaining episodes.
- deleted 4y ago
  [deleted]
- thundergolfer 4y ago
mark_l_watson 4y ago
I really like the Sutton quote. Manually written AI systems show promising early results and then fail when compared to machine learning approaches.

162 points by lispybanana 4y ago

Whisper is great, but I also wouldn't overlook other next-generation models (RNN-T/Zipformer/etc) trained on 50k+ hour datasets. These also perform very well. That being said, Whisper is clearly a far cry from "intelligence". This should be clear when you feed it 5 seconds of silence and get hallucinated garbage in return. It's much more akin to compressing those huge datasets into something that can feasibly be run on recent hardware. That's not to downplay how impressive that is, just to draw a clear line between "compression" and "intelligence".

wyldfire 4y ago
Don't state of the art commercial systems do something similar? I assume there must be some automatic gain boosting the noise at the frontend of most pipelines, I know I've gotten transcribed voicemails that really just are silence but the transcript shows lots and lots of hallucinated words. Regardless of "intelligence" it's got real utility.
- nmfisher 4y ago
  I occasionally get a single hallucinated word (more like a mis-transcription) where the audio contains a clunk/bang/cough/etc, but I've never had full hallucinated phrases from clean silence. There are a couple of GitHub discussions on the Whisper repository with various fixes/hacks to deal with it: https://github.com/openai/whisper/discussions/679 https://github.com/openai/whisper/discussions/679 https://github.com/openai/whisper/discussions/813 https://github.com/openai/whisper/discussions/813 If you get a chance, I encourage you to try out the other newer models I mentioned, I think you'd be very impressed.
  - pixl97 4y ago
    I don't see this much different than what commonly happens with humans when we hear our named called when it was some environmental noise. As for the silence, I wonder why the the model even receives it. I would think a lot of that would be compressed out of existence to save bandwidth.
- qwertox 4y ago
  I have an app on my phone which creates a 1 minute audio file when I press a button. I have a lavalier microphone connected to the phone and use it to record notes while riding my bike. It's always 1 minute because that is usually enough, and if I see that I need more time, I record an overlapping second file. Last week I set up a Whisper instance on my server and have been feeding it with these files. The result is pretty good. I usually can remember what I was saying when I read the transcription, which usually contains a couple of errors. Then there are those added hallucinations which are entire sentences, like: ---- 00:00.000 --> 00:05.000 Also temperaturmäßig ist es recht gut. [So temperature wise, it's pretty good.] 00:05.000 --> 00:09.000 Der eine hat 12 Grad, der andere 10. [One has 12 degrees, the other 10. (I have two temperature sensors mounted on the bike, ESP32 streaming the data to the phone via BLE)] 00:09.000 --> 00:12.000 Also sagen wir mal, 10 Grad. [So let's say 10 degrees.] 00:14.000 --> 00:19.000 Es ist bewölkt und windig. [It's cloudy and windy.] 00:20.000 --> 00:24.000 Aber irgendwie vom Wetter her gut. [But somehow from the weather it's good.] 00:24.000 --> 00:31.000 Ich habe heute überhaupt nichts gegessen und sehr wenig getrunken. [I ate nothing at all today and drank very little.] 00:54.000 --> 00:59.000 Vielen Dank für's Zuschauen! [Thanks for watching!] Transcribed in 77.2 seconds ---- The last sentence, "Thanks for watching!" is a complete hallucination. There were 30 seconds remaining which were me breathing and the wind blowing into the microphone and it came up with that comment. I usually comment on the weather because I take note of what I am wearing, and it allows me to better prepare for future rides. 77 seconds for the 60 second file because my server has no GPU, so I'm running the large model on the CPU (in a VM which has 8 cores assigned to it from a Ryzen 9 5950X). I've been considering buying a small PC with a 3060 RTX only for inferencing, but it may be too expensive. I tried Google Speech-To-Text and it is nowhere as good as Whisper under these conditions (having the wind noise and the heavy breathing). This is Google's result: ---- "Also temperaturmäßig es ist recht gut, der eine hat 12° andere 10. Es ist angemalte 10 Grad. Es ist bewölkt und windig, aber er hat sie vom Wetter her gut, ich wollte überhaupt nichts gegessen und sehr wenig getrunken." ["So temperature-wise it's pretty good, one has 12° other 10. It's painted 10 degrees. It's cloudy and windy, but he has it good from the weather, I did not want to eat anything at all and drank very little."] ---- Also, whisper.cpp doesn't seem to generate the same results, and they appear to be not so good (in this case it was almost just as good). I just tested the same file on whisper.cpp with the large model and it's even funnier: ---- [00:00:00.000 --> 00:00:05.000] also temperaturmäßig ist es recht gut [...] [00:00:05.000 --> 00:00:09.000] der eine hat 12 Grad, der andere 10 [...] [00:00:09.000 --> 00:00:12.000] also sagen wir mal so 10 Grad [...] [00:00:12.000 --> 00:00:19.000] es ist bewölkt und windig [...] [00:00:19.000 --> 00:00:24.000] aber irgendwie vom Wetter her gut [...] [00:00:27.000 --> 00:00:31.000] ich habe heute überhaupt nichts gegessen und sehr wenig getrunken [...] [00:00:31.000 --> 00:00:35.000] das ist der Grund, warum ich so viel auf dem Knie gehe [this is the reason why I go so much on the knee] [00:00:35.000 --> 00:00:39.000] das war's, bis zum nächsten Mal! [that's it, until next time!] [00:00:39.000 --> 00:00:59.000] Danke fürs Zuschauen! [Thanks for watching!] `time` yields 567.63s user 1.99s system 755% cpu 1:15.36 total ---- The first 30 seconds, where the text is clearly understood, is inferenced within ~10-15 seconds. It's the "silence" which makes the AI go crazy on the workload. The idea behind this is to set up a system which then sends me an email with a map and trail of the ride as well as the transcriptions of the notes.
- uncanneyvalley 4y ago
  It’s not that your audio is being amplified, it’s that the VAD classifier is poorly tuned. The noise should never even reach the recognition stage. Whisper’s hallucinations are pretty severe, but are improved by adding VAD to its pipeline.
panarky 4y ago
It's a tool that's free as in freedom, and it is incredibly useful. How does discovering that it doesn't handle some weird use case diminish its utility as a tool? Hammers are awesome at driving nails into wood. But if you strike the wood directly without a nail, the hammer puts a dent in the wood. Is that a defect in the hammer? Does it somehow make the hammer any less useful as a tool?
- nmfisher 4y ago
  Not sure what makes you think I'm diminishing its utility as a tool; like I said, it's an incredible tool and I lean on it very heavily for various speech-processing pipelines. I'm just pointing out that Whisper definitely hasn't "solved" speech recognition, and there's still a lot more fertile ground to cover from a research perspective.
  - pixl97 4y ago
    In my personal believe I don't think speech recognition, at least based on any human model is solvable, at best we can get 'as good as an average set of humans that speak the language' People mumble crap all the time to other humans and need to repeat what they say with proper enunciation. People have hearing problems, which would correlate to microphone quality/placement issues when dealing with computer systems. Then there are issues where people say one thing incorrectly, but the person following/listening to the directions knows the procedure and does the correct thing. If you asked the speaker what they said again, they'd say they said the 'correct' thing in the first place. And this is something that I've done before as an example. Me: "Click the X button to start the process" Person writing the notes: "Click the Y button to start the process" Person writing the notes: You meant click the Y button right. Me: Yea, that's what I said. Oops, its when we get in things like this that we run into the unsolvable speech recognition issues because we don't generally understand our own error bars on what we say. The speech quality between a public speaker and the average joe I'm sure has a very wide range.
    - panarky 4y ago
      > at best we can get as good as an average set of humans Whisper is already superhuman, more accurate than experienced human transcribers.
jasode 4y ago
>, just to draw a clear line between "compression" and "intelligence" Not disagreeing with you but your sentence reminded of research looking at the link between "compression" and "intelligence" : https://www.google.com/search?q=compression+is+a+form+of+intelligence https://www.google.com/search?q=compression+is+a+form+of+int...

krmbzds 4y ago

https://archive.is/FEGf1 https://archive.is/FEGf1

rileyphone 4y ago

Great article, we are only just beginning to see the impact of Whisper. I hope at least that it will trickle into my Alexa sooner rather than later, but I've been scheming other uses for it too. Dictation to notes so I can think out loud. Make transcripts of talks that I would rather read than listen to. The possibilities are endless.

wyldfire 4y ago

Aside - I'd love to see similar rust and zig implementations like whisper.cpp. I'll donate $150 USD to zig and rust foundations as a bounty for respective MIT-licensed implementations of these. Let's keep it simple - scalar instructions, no need for intrinsics/assembly. Ideally there would be some tests. whisper.cpp looks like a simple-enough-but-very practical application and I think it would help promote these modern languages to have a simple and portable demonstration like this.

fbdab103 4y ago
What does a Rust/Zig port buy that the current implementation cannot do?
- wyldfire 4y ago
  Nothing. I tried to address my desire for these items at the end there - it would serve as an excellent demonstration of the power of these languages (to take on a performance critical task like audio transcription).
- itake 4y ago
  Just taking a stab, but if your code is primarily written in rust/zig, it’s really annoying calling c/cpp libraries, because you have to build them and keeping the bindings in sync after updates.
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]

ur-whale 4y ago

https://github.com/ggerganov/whisper.cpp https://github.com/ggerganov/whisper.cpp

thimm 4y ago

I love whisper. It is so easy to use. I created a small pipeline that transcribes podcasts within the domain that I'm working in. It helps me and my colleagues to revisit and find podcasts episodes without having to listen to them again. You can check it out on podcasts.farmonapp.com

losthobbies 4y ago
This is really interesting. I am looking at doing something similar. Do you mind me asking what the Back-end API call is written in? I had looked at Deepgram and might try putting a small project together.
- thimm 4y ago
  It is all written in Python and it uses the original python bindings. I'm using mkdocs to convert the transcripts into a website.

aaron695 4y ago

[dead]

EZ-Cheeze 4y ago

"There could be even larger changes—we talk a lot, and almost all of it goes into the ether. What if people recorded conversations as a matter of course, made transcripts, and referred back to them the way we now look back to old texts or e-mails?" Minority Report-level awesome futurism One thing that changes everything

cloudking 4y ago
This exists already https://www.rewind.ai/ https://www.rewind.ai/
- anonyfox 4y ago
  I had no idea. instantly subscribed. not joking.

braindead_in 4y ago

Whisper is awesome, but managing it in production environment is not easy. I am waiting OpenAI (or someone else) to offer a API with a Real Time Factor of < 1. RTF is inference time/duration of the file. We can really use a that.

garblegarble 4y ago
Doesn't whisper.cpp already get you that? It takes ~6 seconds per 30 second segment on an M1 Max with the Large model. Do you mean you want snappy appearance of words shortly after you say them, rather than having to recognise in 30 second segments?
qwertox 4y ago
Getting a server running is easy if you use https://github.com/ahmetoner/whisper-asr-webservice https://github.com/ahmetoner/whisper-asr-webservice as a guide. It's then a REST API which you post the file to and get the transcription in return. But I don't know what you consider being "in production". If it's for internal use then it is enough. Here are some comparisons of running it on GPU vs CPU According to https://github.com/MiscellaneousStuff/openai-whisper-cpu https://github.com/MiscellaneousStuff/openai-whisper-cpu the medium model needs 1.7 seconds to transcribe 30 seconds of audio when run on a GPU.

ilaksh 4y ago

Whisper is great, but Google's built-in speech-to-text thing in Chrome with https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_API/Using_the_Web_Speech_API https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_... has been available for quite some time and works well. Not sure if Firefox has a implementation, probably one that works locally but not nearly as well. Obviously not having to send all of the data to Google is a big deal. But practically speaking the Google recognition seems to perform extremely well. So not sure this is really a new capability for people if they were willing to use Google's servers. It seems like Firefox and Chrome should now ship with Whisper built in. Or they should work on that.. and if Chrome doesn't add it then.. that's suspicious.

qwertox 4y ago
You may want to read my comment which I posted to another comment, it has a comparison to Google's ASR.
krisoft 4y ago
> So not sure this is really a new capability for people if they were willing to use Google's servers. Of course it is. Google is providing the API at their largess. One day they can decide they won't do it anymore, or they will charge you huge piles of money, or they decide they don't like something about your data and they won't process it even for all the money. Or will do it but only send back nonsense to you. It is their endpoint and they do whatever they are pleased with it. On the other hand you can take Whisper and use it on your own computer. Or rent a computer from anyone who does that kind of service. If you don't like how it performs you can refine it on your own datasets. If you have tens of thousands hours of sound files you want to convert you can just calculate what it will cost you to convert with Whisper. With google? Good luck. The difference between running Whisper on your own hardware vs using Google's endpoint is the difference between having your own car vs the mercurial rich guy not minding if skaters hold on his taillights to hitch a ride for now.

jbotz 4y ago

The article concludes: "Eventually, though, someone will release a program that’s nearly as capable as ChatGPT, and entirely open-source. An enterprising amateur will find a way to make it run for free on your laptop." Well not in the near future... Large language models like ChatGPT are called "large" for a reason, and they're too big for your laptop for the foreseeable future. I think you'd need a computer with several high-end GPUs that each have 128+GB of RAM to be able to run one of those. Maybe a laptop from 2030 will do. Otherwise a very good article.

sebzim4500 4y ago
FlexGen can already run GPT-3 size models on commodity hardware, albeit with high latency and fairly slow throughput (order of 1 token/s).
api 4y ago
You can run giant models on a high-end laptop today... they'll just be slooooowww since you'll be doing things like swapping data in/out and leveraging the CPU. If you don't mind waiting an hour for a prompt response it can work. It's the same as it's always been. Any general purpose computer is Turing complete. Spending more gets you faster results.
wyldfire 4y ago
> to be able to run one of those It's very important to distinguish the different use cases between training and inference. The amount of memory required to execute the ChatGPT model, once trained, is likely much, much less than 128GB.
- dxuh 4y ago
  Afaik you actually need much more. About 400GB just to load the trained model according to this tweet thread: https://twitter.com/tomgoldsteincs/status/1600196981955100694?lang=de https://twitter.com/tomgoldsteincs/status/160019698195510069... I am not quite sure how reliable the source is, but it makes sense that you at least need to store the 175 billion parameters that define the model in VRAM. I know that for a short-ish while GPUDirect storage is a thing, so that could help for sure, but it would definitely impact execution time as well.

tikkun 4y ago

What utilities related to Whisper do you wish existed? What have you had to build yourself? On the end user application side, I wish there was something that let me pick a podcast of my choosing, get it fully transcribed, and get an embeddings search plus answer q&a on top of that podcast or set of chosen podcasts. I've seen ones for specific podcasts, but I'd like one where I can choose the podcast. (Probably won't build it) Also on the end user side, I wish there was an Otter alternative (still paid $30/mo, but unlimited minutes per month) that had longer transcription limits. (Started building this, not much interest from users though) Things I've seen on the dev tool side: Gladia (API call version of Whisper) Whisper.cpp Whisper webservice (https://github.com/ahmetoner/whisper-asr-webservice https://github.com/ahmetoner/whisper-asr-webservice) - via this thread Live microphone demo (not real time, it still does it in chunks) https://github.com/mallorbc/whisper_mic https://github.com/mallorbc/whisper_mic Streamlit UI https://github.com/hayabhay/whisper-ui https://github.com/hayabhay/whisper-ui Whisper playground https://github.com/saharmor/whisper-playground https://github.com/saharmor/whisper-playground Real time whisper https://github.com/shirayu/whispering https://github.com/shirayu/whispering Whisper as a service https://github.com/schibsted/WAAS https://github.com/schibsted/WAAS Improved timestamps and speaker identification https://github.com/m-bain/whisperX https://github.com/m-bain/whisperX MacWhisper https://goodsnooze.gumroad.com/l/macwhisper https://goodsnooze.gumroad.com/l/macwhisper Crossplatform desktop Whisper that supports semi-realtime https://github.com/chidiwilliams/buzz https://github.com/chidiwilliams/buzz

gtirloni 4y ago
I think AssemblyAI has tools for that.
causi 4y ago
Add sponsor-skipping into that. Give me a transcript, let me select a series of words, then audio containing that series of words gets skipped on all remaining episodes.
deleted 4y ago
[deleted]
thundergolfer 4y ago

mark_l_watson 4y ago

I really like the Sutton quote. Manually written AI systems show promising early results and then fail when compared to machine learning approaches.