8 ms·
Voice Isolator: Strip background noise for film, podcast, interview production
- IncreasePosts 2y agoWhat is the current SOTA for voice->text? I have a recording I've been sitting on for 2 years(a guest lecture which a friend recorded) which contains a very heavy amount of background noise, where you can just barely make out what is being said by the lecturer. I wonder if there is any hope I will ever be able to read a transcript from it. I can figure out what the lecturer is saying (maybe only because I have some context about what he is talking about), but it is too painful to sit through 2 hours of it and try to transcribe it. I tried uploading the audio file to this service, but basically get nothing useful returned to me.
- bequanna 2y agoI’ve had good results with Whisper. You can use the OpenAI API and it is also open source: https://github.com/openai/whisper https://github.com/openai/whisper
- geerlingguy 2y agoAnd if you have a Mac, get MacWhisper. It's been a godsend for transcribing almost anything. Usually pretty good if the main voice is discernible at all—though in OP's case, if the main voice is almost indistinguishable it might not do amazing.
- MengerSponge 2y agoI've had good luck with Buzz: https://github.com/chidiwilliams/buzz https://github.com/chidiwilliams/buzz
- pants2 2y agoDeepgram Nova 2 is among the best right now, more accurate than Whisper in my testing.
- xan_ps007 2y agoWe have built a dockerized open source stack for Whisper + Llama3 + MeloTTS. Whisper and MeloTTS for now works fine for our use cases. https://github.com/bolna-ai/bolna/tree/master/examples/whisper-melo-llama3 https://github.com/bolna-ai/bolna/tree/master/examples/whisp...
- Prompter9856 2y agoYou can upload your file and try out Deepgram for free, just to see what the results look like for your audio. No harm in trying: https://deepgram.com/free-transcription https://deepgram.com/free-transcription Disclosure: I work for Deepgram
- ProfessorLayton 2y agoTry giving Audacity a shot to cleanup the audio, it has a built-in noise reduction feature that's configurable. I've used it to varying degrees of success, but works especially well with the same sounds ANC headphones are good at blocking.
- duped 2y agofwiw, the commercial services that do this are called "audio forensics" (unsurprisingly, they're usually hired by cops and lawyers). You pay them to use their (often expensive) software tools to clean up audio and provide a transcription. I get the appeal of automating this task but the SOTA is not to automate it at all.
- willsmith72 2y agoyou can pay people online trivial amounts of money for this, it will be far cheaper and quicker than waiting for the right AI by the way, even once we get to a sufficient AI, how do you verify it without listening to the whole thing anyway? it's only 2 hours, if you're a fast typer at max it would take you 1 work day to transcribe yourself, or <$200 by a professional
- IncreasePosts 2y agoIt's not an issue of typing speed, it's that puzzling out what was said takes a number of re-listens, at high gain which hurts my ears after a while since the voice is just barely above the noise floor.
- dbspin 2y agoI don't know about SOTA, but 'Adobe Podcast Studio' a web app that I believe is still free / in beta offers excellent sound cleanup. So much so that many podcast / radio producers I know no longer frequently use Izotope RX - one of the industry standard tools. Adobe are obviously horrendous, but if its for a one time use I'd give it a go. The feature you want is the 'enhance speech filter'. https://podcast.adobe.com/enhance https://podcast.adobe.com/enhance
- wkcheng 2y agoThis is really helpful, thanks! I have a bunch of audio that I need to clean up and this looks like it could fit the bill. Do you know if there are any license issues with this? I don't see any license page--will they train/retain the recording?
- dbspin 2y agoI'm not sure - that's a really good question. I'd assume anything uploaded to a deep learning system retains the data for future training. But I have no information on the licensing of this tool specifically. Especially given the recent furore over Adobe's licensing terms.
- echelon 2y agoSTT: Whisper TTS: GPTSOVITS / StyleTTS2 VTV: RVCv2 Open source isn't really doing a great job at voice, music, or video. It's managing to keep up in LLM and image spaces, but it's falling far behind in the multimedia department.
- mbrock 2y agoGemini 1.5 Pro multimodal with audio file input plus a text prompt that asks for a transcription. You can add all the relevant context you know to the text prompt. You can also chat with it before asking for the full transcription and clear up any confusions. There are no products for this yet, it's a new capability. I just started trying this today with my wife's hour long interviews in Latvian language and it is extremely good, far better than any transcription model. This is a huge SoTA LLM with audio tokens, so it just has vastly more capability than Whisper or whatever. In my case it nails all kinds of brand names, weird neologisms and loan words, it writes inline quotation marks when the speaker is quoting someone, and so on. This is what GPT-4o supposedly can do in the version OpenAI has postponed rolling out. If you want, I can try to do it for you if you send the audio.
- mbrock 2y agoI've noticed it's necessary to cut long audio into sections, otherwise it starts getting confused and repeating itself. The output token limit only lets you transcribe a few pages at a time anyway.
- thatsadude 2y agoRecent ASR models are already robust to noise due to Spec augment and large-scale data. If you use these noise reduction services to remove noise, ASR models will have harder time to recognize denoised audio. The reason is that noise reduction will create distortion which ASR didn't see during training.
- sweetdreamerit 2y agoI would try, for 10 minutes, the following solution: you listen to the lecture and repeat every word the lecturer said. Then use the record of your voice for the voice 2 text process. If it works, do it for the whole talk.
- dougdonohoe 2y agoHow much does it cost in the FAQ: Voice Isolator costs 1000 characters for every minute of audio. Since when are characters a currency?
- icepat 2y agoThis confused me as well, and made me lose interest. Intentional non-answers like this are rather grating.
- samspenc 2y agoThis is par for the course for any text-to-speech or speech-to-text service these days, check out ElevenLabs' other service pricing and it is similar - they have monthly pricing but the character usage is capped at each level. Actually other major cloud providers, including AWS, Azure and GCP, have similar character / token / word count based pricing as well.
- recursive 2y agoText-to-speech and speech-to-text, both involve text, which has characters. Notably, this does not.
- chankstein38 2y agoI'm not trying to argue with you but just want to point out, this is why they made an equivalence between time and characters. We know 1min of audio == 1000 characters so now we know how to translate characters to time.
- recursive 2y agoIf that's really it, then why the obtuse intermediate conversion unit? Just bill by the minute.
- Rebelgecko 2y agoThey charge in characters per time, not money per character
- throwup238 2y agoThey also just announced licensed celebrity voices in their Reader app this past week. Judy Garland, Burt Reynolds, Laurence Olivier, and James Dean are the first ones.
- simonw 2y agoAll four of whom are deceased. I guess they licensed from their estates?
- deleted 2y ago[deleted]
- pfdietz 2y agoIn California, such rights last 70 years, so Dean loses protection next year, Garland in 2039, and the others much later as they died fairly recently.
- vincenzothgreat 2y agoYikes, let the dead rest in peace
- dmix 2y agoTell that to their descendants bank accounts.
- taraparo 2y agoI prefer https://product.supertone.ai/clear https://product.supertone.ai/clear which is one time payment and not subscription based.
- ukuina 2y agoUltimate Vocal Remover 5 is free, though.
- IvyMike 2y agoThe one thing I hate about this: There are so-called "first amendment auditors", who professionally annoy people on the street, trying to provoke a reaction. They monetize the resulting video on youtube. You used to be able to pull out your phone and play Disney soundtracks or Taylor Swift music which would result in the video being non-monetizable. But improvements in audio isolation techniques have now defeated this countermeasure. Being a professional annoyance is once again a career choice. Edit: this is one instance I've personally seen: https://www.instagram.com/p/C7IEFxQSJQw/?hl=en&img_index=1 https://www.instagram.com/p/C7IEFxQSJQw/?hl=en&img_index=1
- tallytarik 2y agoI know the sorts of “creators” you’re talking about, but I’ve never heard of this as a response before. Are there really that many people who 1) are aware that this could be effective, and 2) are quick witted enough to pull their phone out and play music in response to being harassed?
- thenewwazoo 2y agoYes, police do it, in fact: https://www.vice.com/en/article/z3n75x/police-disney-music-censor-video https://www.vice.com/en/article/z3n75x/police-disney-music-c...
- dmix 2y agoA single police car did*
- TechDebtDevin 2y agoWhile 1st ammendment auditors are cringe, they only annoy police. This comment is definately pro LE coded and the irony does not escape me with its criticism of people expressing their 1st amendment (regardless of how annoying their method). Perhaps we should demonitize every form of journalism and media that annoys this guy!
- 2y ago
- tomaskafka 2y agoAre there actual before/after samples? I’m sure as hell not sending samples of my voice to AI voice cloning company.
- ygjb 2y agoI mean, you don't have to? Set up an audio source, for example, your phone, playing a reasonable length of talking, for example a youtube video, or a podcast on spotify. Then record from your computer or other recording device, and test with that?
- Murky3515 2y agoPlease think twice before sharing your personal voice samples with a random online website just because they offer a cool demo.
- Workaccount2 2y agoI suspect in the near future people are going to report randomly hearing themselves in advertisements.
- recursive 2y agoFriends and loved ones seems like a better method.
- Eisenstein 2y agoMost people either wouldn't recognize their own voice or would hate hearing it.
- latentsea 2y agoImagine if they were able to make a model that's your voice but the way that you hear it. That'd be so neat. You could hear how other people hear their own voice and have fun playing with it for an afternoon before moving onto the next shiny new toy.
- jdprgm 2y agoElevenlabs has some pretty cool stuff but I really despise how it's all cloud based. Wish there was an audio ai company following a path similar to what topaz has been doing for video/photo ai with desktop software. Open source has been lagging more than I expected in this area too.
- echelon 2y agoGPTSOVITS, StyleTTS2, and RVCv2 are still the open source SOTA for TTS and voice conversion. These models are unfortunately really far behind Elevenlabs' offerings. We're not much further along than the Tacotron2 (2018) days. Elevenlabs is the only model company I can think of that is ahead of everyone else in their category. Video and LLMs are hyper competitive, but voice is a one-company game. Elevenlabs hired up everyone in the space and utterly dominates. I'm hoping this changes. They've been in pole position for over a year and a half now with nobody even coming close. There's probably a reason why they're so research-oriented. The minute an open source model is released that rivals Elevenlabs in quality, they're in big trouble. There's absolutely zero moat for their current products and there are fifty companies nipping at their heels that want to be in the same spot. Elevenlabs' current margins are juicy.
- chmars 2y agoHow is this different from Auphonic? https://auphonic.com/features https://auphonic.com/features
- SkyPuncher 2y agoWhy does it have to be different?
- ukuina 2y ago... because this is by the market leader in AI speech synthesis?
- andrewstuart 2y agoTried it with several files. It didn't seem to do much better than audio filters for ffmpeg that have been tuned for removing background noise and enhancing voice. Maybe I'm missing something or using the wrong source data.
- ec109685 2y agoI had very loud background music playing, and while it could completely eliminate that (impressive!), the voice was much more garbled then when there wasn’t any background noise playing.
- almog 2y agoI'd like to have something else but for live calls: a process that takes two audio inputs and "subtracts" the noise from one input from the other. My use case would be to have two dynamic microphones, one directed at the window and one that I'm using for a conference call. I'm assuming having two inputs should make the process easier for real time (20ms?) processing and might require less compute. If such process can output a clear sound, I could chain it with Blackhole and have it and use the processed clear signal as an input for the call.
- chankstein38 2y agoAssuming that this is setup so that the same sound is coming through both microphones just one with your voice on top, you could theoretically do this just by feeding it through something that inverts the polarity of the "to be cancelled out" sound and overlays the two sounds. I'm sure it wouldn't be perfect but you might be able to tune it to properly do it. This is how a active noise cancelling works!
- almog 2y agoThank you for the idea. I've tried in the past to do something similar but couldn't get it right. I did try to rely on ideas from ANC but my domain knowledge is very lacking. It's been over 2 years since so I might need to give it another chance and see if any off the shelf library/product has been released since then.
- CPLX 2y agoThat doesn’t work though, the requirement for the timing to align waveform by waveform is too high, and the speed of sound is too slow. Also the frequency response isn’t going to match exactly. To do it right you really want digital analysis.
- inhumantsar 2y agoDeepFilterNet doesn't use a second microphone but it does do an absurdly good job of removing not-speech from inputs in realtime. Check out the demo video linked in the README. iirc they demonstrate removing guitar sounds and even a vacuum cleaner. It does take some technical elbow grease to integrate but I've used it in calls and while gaming on Linux via Pipewire to great effect. https://github.com/Rikorose/DeepFilterNet https://github.com/Rikorose/DeepFilterNet
- dayjah 2y agoMy test sample, me talking with my baby babbling in the background, returned a silent audio track. I guess I nor the baby are considered signal ~_~
- CSSer 2y agoI’m sorry you had to find out this way, Deckard (Rachel?).
- fbnspl 2y ago[dead]
- simshay 2y agoI have used ai|coustics previously and I think their output quality is way better than Eleven Labs or Auphonic. They really do a good job there.
- gtvwill 2y agoOr I could just download virtual dj and run it for free on a computer and just do this locally, right now, with zero fancy hardware and arguably some of the best stems algorithms on the market.
- dc3k 2y agoi think i’ll stick to nvidia broadcast for this
- localfirst 2y agojust tried it and its not that great many many ppl are complaining that they have to spend quite a bit of credit to get the desired effect so likely this is just another "pay-to-fine-tune" not unlike "pay-to-play" schemes in online games--the hook is to get you in to buy credits which you will use to chase the desired quality. besides there are local TTS models now that rivals Elevenlabs. Their pricing is ridiculous $200/1M is way too expensive.
- Animats 2y agoWhat's it going to take to do this locally?
- terrycody 2y agoSorry a noob here, is there sth can strip the Youtube videos voice talking, but leave everything else 100% untouched? I dunnon what this thing called, or is it exactly the same thing I was looking for?
- Ylpertnodi 2y agoSeveral devices can do this, but I would suggest the audio program 'audacity' to start with. Ymmv as it's always a trade off between not removing enough voice, and having too much of the background erased at the same time. Other programs often call it 'karaoke mode'.
- terrycody 2y agoThanks for your wiki reply.
- deleted 2y ago[deleted]
- greypowerOz 2y ago"Voice Isolator costs 1000 characters for every minute of audio." - can someone expand on this currency for someone out of the loop?
- hnsdmpf 2y agoThe video is impressive, but for the files I uploaded is hallucinated or changed the voices quiet a bit. IMO ai-coustics.com or auphonic is a better alternative. ai-coustics even offers video upload and let's you choose the enhancement level.
- ukuina 2y agoAre there any open source STT solutions that also handle speaker diarization? MacWhisper has promised this for a long time without delivering.