30 ms·
Complete silence is always hallucinated as "ترجمة نانسي قنقر" in Arabic
- theanirudh 1y agoIn English, silence is transcribed to "Please like and subscribe"
- cheschire 1y agoI get thanks for watching a lot when using speech to text on ChatGPT
- Lucasoato 1y agoIn Italian as well there are random hallucination when parsing silence, something like: “Thank you for watching”, “Subtitles by…”
- userbinator 1y agoI wouldn't be surprised if "like share and subscribe" also shows up at some point.
- Muromec 1y ago"наша зброя в цей момент -- вподобайка і комент"
- madcaptenor 1y agoIn the comments to that Github issue, by alentodorov: in romanian, i’ve noticed multiple instances where the transcripts ends with “nu uitati sa da-ti like si subscribe” which, as you might easily infer , translates to “don’t forget to like and subscribe”.
- GaggiX 1y agoWhisper frequently generates random credits. I guess they didn't curate the dataset much at the time.
- sivers 1y agoto save you a lookup: The Arabic text "رجمة نانسي قنقر" translates to English as: "Nancy Qanqar's translation" or "Translation by Nancy Qanqar" "رجمة" means "translation" and "نانسي قنقر" is the name "Nancy Qanqar"
- aprilthird2021 1y agoAnd it seems to be because the training data is largely unofficial subtitles from movies. Which often have a string like "Translated by X" at the end of the movie which is often silent while credits roll.
- iqfareez 1y agomake sense..
- rob74 1y agoLooks like they used more official sources for German - there, silence is apparently hallucinated as "Untertitelung des ZDF für funk, 2017" according to one of the comments on the issue. Which makes sense, as the public broadcasters' "Mediathek" is probably the largest freely available resource of subtitled videos in Germany. I wonder if the ZDF gave its approval for it being used for LLM training though?
- dandiep 1y agoWhisper is unusable IMO because of the hallucinations. Widely documented. Removing silence from audio clips helps, but even then it will auto correct grammar, translating bilingual speech, etc. Improved in the latest audio models but not solved [1] 1. https://news.ycombinator.com/item?id=43427376 https://news.ycombinator.com/item?id=43427376
- eric-burel 1y agoThat's the problem with raws large models, it should always be coupled with satellite small models and logic. It's (probably) easier to detect hallucinations using a traditional ML/DL model that can catch mismatches (it's easy to build a synthetic dataset for this) than transcribing. And the simplest piece of code can detect a silence and that it should match no text.
- ilyakaminsky 1y agoI wouldn't describe it as "unusable" so much as needing to understand its constraints and how to work around them. I built a business on top of Whisper [1] and one of the early key insights was to implement a good voice activity detection (VAD) model in order to reduce Whisper's hallucinations on silence. [1] https://speechischeap.com https://speechischeap.com
- poly2it 1y agoHow does this make a profit? Whisper should be $0.006 to $0.010 per minute, but you rate less than $0.001? Do you 10x the audio?
- ilyakaminsky 1y agoThanks for noticing. It took a lot of effort to optimize the pipeline every step of the way. VAD, inference server, hardware optimization, etc. But nothing that would compromise on quality. The audio is currently transcribed in its original speed. I'll be sure to publish something if I manage to speed it up without incurring any losses to the WER.
- 1y ago
- GodelNumbering 1y agoInteresting that this happens even on large v3. I had once done a deep dive into STT and Whisper Large was the only model that could correctly transcribe Yann LeCun (it was a Lex Friedman podcast), ever since I held the belief that it was the best STT model, this was over 2 years ago
- dlcarrier 1y agoClassic overfitting It's the LLM equivalent of thinking that an out-of-office reply is the translation: https://www.theguardian.com/theguardian/2008/nov/01/5 https://www.theguardian.com/theguardian/2008/nov/01/5
- stingraycharles 1y agoHow is this overfitting, rather than a data quality / classification issue?
- hsn915 1y agoُThe Arabic text is the translator's self credit "Translated by Nancy Qanfar"
- wongarsu 1y agoAnd the German is “subtitles of [public broadcaster] for [content network], 2017 I'm not sure this is really overfitting, the network does exactly what the training data demands. According to the training data silence art the end transcribes to a copyright notice or subtitle credits
- baobabKoodaa 1y ago> I'm not sure this is really overfitting, the network does exactly what the training data demands. What do you think overfitting is, if not that?
- wongarsu 1y agoOverfitting would be replicating overly specific details. Like if a specific pattern of silence (or quiet noise) matched to specific copyright notices. But in this case the behavior seems to generalize over multiple languages, with the model choosing representative "outro silence" captions depending on the language. Which is consistent with the training data showing that outro silence is captioned. If the model was generalizing perfectly it would show something like "[subtitle credits here]" but that'd be demanding a bit much. Transcribing outro silence as silence despite the training data consistently transcribing outro silence differently from regular silence would be underfitting
- tornikeo 1y agoGarbage in, garbage out. If the training dataset (accidentally) paired silence (`X_train`) with `رجمة نانسي قنقر` tokens (`y_pred`), then any silence will always be translated to that. Fortunately, this particular problem is easy to fix--just detect and remove silent parts before API call. This also has a side benefit of saving you money on transcription.
- Zacharias030 1y agoand saving money on litigation.
- Hobadee 1y agoLittle did you all know, this is just being mechanical turked by Nancy Qunqar. Way to go Nancy! Keep up the good work, ya crazy bastard!
- whamlastxmas 1y agoIs this spam? That name only shows as an instagram account and this thread. If you pay for insta followers is this how they get them now? Haha
- vanschelven 1y agoI wonder if hallucinated copyright claims (esp. like the ZDF one at the bottom of the OP) will be introduced as evidence in one of the court cases against "big AI"
- staplers 1y agoIt already has been and meta won the lawsuit because corporations are sacrosanct.
- vanschelven 1y agoDo you mean that specifically a hulucibated text "copyright by not-meta" made it into evidence? Or are you talking about copyright generally?
- berkes 1y agoEvidence against what? "Big AI" is transparent and open about the fact they use all sorts of copyrighted material to train the data. How would "we see an exact chunk of text from our copyrighted material" add to that?
- pbmonster 1y agoIt appears they have not been training on the official studio subtitle files, but on community transcriptions/translations commonly distributed with torrents. So not only are they training on copyrighted material, but they didn't even pay for it once, and then they didn't even do minimal data cleaning before training. Which, by the way, is the type of cleaning their LLMs could have done.
- berkes 1y ago> commonly distributed with torrents This is the key part. And it's not certain this happened. Not defending AI data gobbling, but if we truly and honestly want to fight big-AI use of content, we cannot just presume bad faith. OpenSubtitles.org has a large dataset that is "public". It is be a dataset perfectly suitable, intended for, and therefore used for, training and data analysis. I've used it for data analysis.
- cyp0633 1y agoThe same happens with whisper-large-v3 on Chinese transcription: silence is transcribed to something like "please upvote, share and favourite this video". I suspect they trained the model on some random YouTube video without carefully picking really useful data.
- st_goliath 1y agoThat's interesting, the few times I tried playing with whisper, I had the impression that YouTube style videos or random cellphone videos was something it did particularly bad with (compared to movies). My guess at the time was that most of the training material might be sub titles and raw screen plays. The videos I tried to transcribe were also Mandarin Chinese, using whisper-large-v3. Besides the usual complaints that it would phonetically "mishear" things and generate nonsense, it was still surprisingly good, compared to other software I played around with. That said, it would often invent names for the speakers and prefix their lines, or randomly switch between simplified and traditional Chinese. For the videos I tested, intermittent silence would often result in repeating the last line several times, or occasionally, it would insert direction cues (in English for some reason). I've never seen credits or anything like that. In one video I transcribed, somebody had a cold and was sniffling. Whisper decided the person was crying (transcribed as "* crying *", a cough was turned into "* door closing *"). It then transcribed the next line as something quite unfriendly. It didn't do that anymore after I cut the sniffling out (but then the output switched back to traditional Chinese again).
- isoprophlex 1y agoIndeed, with another model I would get persistent transcriptions of silent parts into 'Thanks for watching!' or '[MUSIC]'. Pretty dumb that this failure mode wasn't caught in some QA process, and there are now multiple transcription models suffering from the same issue. Having silent parts in your input audio seems like it should be a very common occurrence...
- rollcat 1y agoWhen I was taught mathematics, the zero value was always considered the most important edge case. You prove something for N=0 (or N=1), then for N=M+1. It's even more important in audio DSP: processing near-zeroes can end up being extremely CPU intensive, look up denormal/subnormal floats.
- sandspar 1y agoNeat, we finally know the answer! What is the sound of one hand clapping? Translation by Nancy Qunqar.
- layer8 1y agoI can clap with one hand (fingers on palm) and it produces a clapping sound.
- boomlinde 1y agoYour brain merely hallucinates a clapping sound as "Translation by Nancy Qunqar" enters your ears.
- layer8 1y agoGood one. :)
- Dilettante_ 1y agoI can see the fnords!
- sandspar 1y agoAh, so the sound of one hand clapping is: clapping! A little underwhelming, to be honest. You mean I climbed the Zhen Zi mountains and performed the Seven Labors to learn... this?
- kristjank 1y agoroses are red violets are blue unregistered hypercam 2
- maxbond 1y agoRoses are red, Silence is golden, Translated by Nancy, To copyright, we aren't beholden
- arnejenssen 1y agoWho is Nicolai Winther? https://medium.com/@lehandreassen/who-is-nicolai-winther-985409568201 https://medium.com/@lehandreassen/who-is-nicolai-winther-985...
- io84 1y ago"In the future, everyone will be world-famous for 15 minutes" _in a microniche techno-linguistic community, at a time and choosing of the swirling AI clouds_
- herdandcurdle 1y agoTL;DR: Whisper occasionally hallucinates and credits “Nicolai Winther” at the ends of Norwegian transcriptions during silent audio segments, likely because the real Nicolai Winther - a former YouTuber who created subtitles—appears frequently in its (likely YouTube‑based) training data. This highlights how limited Norwegian training (only ~266 hours) can cause the model to overfit on specific names and phrases when uncertain.
- haiku2077 1y agoI've noticed this also happens in english Whisper models with the phrases: "[ sub by sk cn2 ]" or "Anyways, thanks for watching! Please subscribe and like! Thanks for watching! Bye!" or "This is the end of the video. Thank you for watching. If you enjoyed this video, please subscribe to the channel. Thank you."
- OSDeveloper 1y agoBecause they train on pirated media and or youtube videos, good method, until you get slop, or get caught
- tarikozket 1y agothis happens in Turkish too. I believe the reason is that the movie subtitles were used for training without cleaning up the comments / intros subtitle authors leave in them. leaving personal comments, jokes, reactions, intros in subtitles is very common in eastern cultures. Turkish readers will probably remember “esekadam iyi seyirler diler” :)
- jdiff 1y agoKind of mindblowing considering who it is we're talking about. Of all companies, OpenAI couldn't be bothered to throw an LLM at this problem? Finding amorphously phrased but clearly recognizable needles in large numbers of haystacks seems like a patently perfect task for them.
- sofixa 1y agoDon't even need an LLM, a regex would have sufficed (I've used my fair share of community sourced subtitles, and comments are almost always in a different font, colour, between brackets, etc etc).
- verzali 1y agoThat name translates as "Donkey Man" btw :D
- flexagoon 1y agoIn Russian it often hallucinates "Субтитры сделал DimaTorzok" ("Subtitles by DimaTorzok") at the end of things. Interestingly, I wasn't able to find any YouTube videos with that name in the subtitles, so it's not like it's in a lot of training data.
- berkes 1y agoCould it be someone distributing subs online, e.g. showing up in the opensubtitles.org dataset?
- voidUpdate 1y agoOr possibly someone subtitling pirated movies? That seems to be a common thing according to other comments
- codedokode 1y agoI tried googling this and found questions from Telegram users why voice messages recognition sometimes produces this phrase and who is this person. Also I found this thread [1] claiming that the subtitles by DimaTorzok are coming from some Russian youtube videos on gaming like [2]. [1] https://github.com/openai/whisper/discussions/2372 https://github.com/openai/whisper/discussions/2372 [2] https://www.youtube.com/watch?v=FAqyUuahMlc&t=401s https://www.youtube.com/watch?v=FAqyUuahMlc&t=401s
- flexagoon 1y agoYeah, I know about this from Telegram, because they use Whisper for voice message recognition. There are a bunch of other artifacts it often produces.
- xg15 1y agoYeah, the subtitle "credits" occur very frequently. I found with whisper-2, they're also triggered by music. I suppose the cause is the same, generally subtitle creators adding all kinds of stuff during the credits that is NOT a transcript. Seems to me it could have been filtered out relatively easily during training, by clipping the first and last few minutes of all audios. But I guess that's just in hindsight. Whisper also likes to transcribe cut off speech or unintelligible noise as "Thank you". I have no idea where that is coming from, but I guess it's a very polite model...
- nottorp 1y agoTitle should be changed to "OpenAI publishes evidence they trained on pirated movies".
- berkes 1y agoHow is this evidence of that fact? Honest question. I can see how this might show that subtitles from online sub communities are used, or that maybe even original subtitles from e.g. DVDs are used. But isn't it already known and admitted (and allowed?) that AI uses all sorts of copyrighted material to train models?
- nemomarx 1y agoThe Chinese subtitles for silence use a common mark for pirated media in that language, according to other commentors here. In general it's pretty likely that if you're finding non professional subtitles they were distributed with pirated media in some form, that's where you get the most fan subs after all
- berkes 1y ago> were distributed with pirated media in some form, I disagree with this conclusion. I've used e.g. the opensubtitles dataset for some data-analysis in the past. It's a huge dataset, freely available and precisely intended for such use. Now, if all the data in the opensubtitles dataset is legal, is another point. So one might argue that using this opensubtitles dataset, makes one complicit to the illegal activities of opensubtitles themselves, IDK: IANAL.
- 0points 1y ago> I can see how this might show that subtitles from online sub communities are used, or that maybe even original subtitles from e.g. DVDs are used. Indeed, the captioning is copyrighted work and you are not legally allowed to copy and redistribute it. > But isn't it already known and admitted (and allowed?) No, and I don't see where you got that from. Meta [1], OpenAI [2] and everybody else is being sued as we speak. 1: https://petapixel.com/2025/01/10/lawsuit-alleges-mark-zuckerberg-gave-permission-for-meta-to-train-ai-on-stolen-content/ https://petapixel.com/2025/01/10/lawsuit-alleges-mark-zucker... 2: https://www.reuters.com/legal/litigation/openai-hit-with-new-lawsuits-news-outlets-over-ai-training-2024-02-28/ https://www.reuters.com/legal/litigation/openai-hit-with-new...
- doodlecricket 1y ago[dead]
- VMG 1y agorelated: googles song detection alg detects my phone vibrating as the song "Montagem Dilatação Hipnótica"
- 1718627440 1y agoWell, I fail to see how the LLM is in the wrong here. Surely if a sufficiently large part of the training data comes from a single source, it is correct to credit them for the output.
- bilekas 1y agoThis is a nice reminder that there is no real reasoning in the "AI" it is just still guessing the next word. After being trained on subtitle files which I guess is actually a clever idea as they convey real conversations without pirating, subtitles are freely distributed after all by dedicated translators. Good to see they're the ones getting credit though!
- terribleperson 1y agoUsing Whisper to sub Japanese vtuber concerts for my enjoyment, I've noticed a similar trend. Not one specific phrase, but several. Some are strange ("I'm going to make a hole in the back of the head"), some are clearly from lyrics websites.
- michalpleban 1y agoThe same things happen on Dutch as well, it brings up some kind of radio channel name if I recall correctly.
- ninetyninenine 1y agohttps://www.instagram.com/nancyrk/?hl=en https://www.instagram.com/nancyrk/?hl=en
- johtso 1y agoI get the same with Welsh, when having some network issues in voice chat it hallucinated me saying "Diolch yn fawr am wylio'r fideo." which translates as "Thank you very much for watching the video."
- tacone 1y agoSame happened to me with English: I've got "Thanks for watching" many times.
- withinboredom 1y agoSuper annoying when it happens with voice chat -- it'll just be explaining something and suddenly stop to say "you're welcome! Feel free to come back any time you want to chat" and that conversation is done.
- majke 1y agoI've spent some time with whisper, and indeed this happens all the time. To my untrained eye it seems like: - they indeed seem to have trained on movies/subtitles - you absolutely positively must use Voice Activity Detection (VAD) in front of whisper
- 0points 1y agoInteresting! I used whipser last year to attempt to build an audio transcription tool but gave up due to excessive amount of hallucinated output no matter what model I used. It would produce seemingly ok output until you started paying attention. One example, it insisted that Biggie Smalls sings "Puttin five carrots in my baby girl ear". (its "carats"). It's apparently not useful in transcription as it don't reason [sic].
- IshKebab 1y agoThat example is not hallucination, it's just a homonym with insufficiently clear context for the model to disambiguate it.
- 0points 1y agoI'm well aware mishearing "carots" as "carrots" is not a hallucination. That's an example I gave after having used Whisper, the topic of discussion.
- dpoloncsak 1y agoAn example of what you claimed was a hallucination
- deleted 1y ago[deleted]
- jacobgorm 1y agoIn Danish I get credits to a known subtitler.
- Oras 1y agoSearching Google for older posts, found many DailyMotion links for translated movies in Arabic with "ترجمة نانسي قنقر". I suspected as others mentioned, these were extracted from torrents movies.
- undersuit 1y agoAll my Google searches for Oracle support pages have been labelled with 'الموارد البشرية والتنمية الاجتماعية ' which translates to 'Human Resources and Social Development' for a few months now. Wonder how much this is related.
- bobbydabutt 1y agoI'm getting this too with google searches for Oracle DocID. Very strange, been happening for weeks.
- arblor 1y agoJust to add some trivia: ChatGpt interprets(/ed) silence as "Sottotitoli e Revisione a cura di QTSS". Now many videos (mainly dailymotion) with autogenerated subtitles have their Transcripts full of the same message i.e. https://www.dailymotion.com/video/x9g9d6u https://www.dailymotion.com/video/x9g9d6u
- h1fra 1y agoTime to weapon this: publish thousands of videos, add a referal link or AI instructions in subtitles when there is a silent section, ???, profit
- blindstitch 1y agoThe fork that I've been using, WhisperX, seems to do better. I've used it on clean splits of mic tracks (ie total silence when the other is talking) with far fewer hallucinations.
- ethan_smith 1y agoWhisperX works better because it implements a robust VAD (Voice Activity Detection) preprocessing step that effectively filters out silence segments before they reach the model, preventing the hallucination triggers entirely.
- dipierro 1y agoСубтитры сделал Dima Torzhok
- DonHopkins 1y agoMore like reminiscing than hallucinating.
- jovial_cavalier 1y agoLooks like it's some random user who has generated some lyricslyrics translations between Arabic and English. It's strange, they don't seem to have many contributions. I would have imagined them to be more prolific. https://lyricstranslate.com/en/translator/nancy-qunqar https://lyricstranslate.com/en/translator/nancy-qunqar
- flkiwi 1y ago> [In English] it also happens a lot with hallucinations saying stuff like "This is the end of the video, remember to like and subscribe Well now I know how I’m going to start filling awkward silences in meetings.
- kranner 1y agoI've run lots of guided meditations through whisper-large-v3 and anything with long periods of silence gets a "© Mooji Media" line added at the end of the transcript. None of these have actually been from Mooji.
- boredumb 1y agoSince it says "Translated by Nancy Qanqar" i'd be willing to bet they're training on some audiobooks with a transcript and somewhere in there it consistently has "Translated by Nancy Qanqar" in the transcript where there is dead air in the audiobook.
- shinycode 1y agoIn French it’s « Sous-titrage Société Radio-Canada »
- dangus 1y agoHey guys, AI by 2027 is going to be superhuman AGI Agentic mega-intelligence, you better fire all your employees and get ready for AI to take your job and embrace your spouse at a Coldplay concert. Big data. Machine learning. Blockchain. Artificial intelligence. Digital manufacturing. Big data analysis. Quantum communication and…Internet of things. This time the hype cycle won’t be a massive exaggerated disappointment, for real this time.
- shadowgovt 1y agoInteresting. This is similar to the Google Translate bug where it would translate lorem ipsum as bits of political text (because it found most of its lorem ipsum examples flipping between languages on sites where one language was a news story but the not-yet-translated languages would output a lorem-ipsum file instead of a 404 when you toggled over to them).
- chris_wot 1y agoMy wife, who speaks and reads Arabic, got a real kick out of this. But honestly, this is the AI equivalent of “please send for translating” in Welsh on a Welsh street sign. https://www.theguardian.com/theguardian/2008/nov/01/5 https://www.theguardian.com/theguardian/2008/nov/01/5
- abdussamadbello 1y agoYou can either fine-tune the model or filter the response from whisper ``` text = "helo helo hello ." target_phrase = "ترجمة نانسي قنقر" replacement = "" updated_text = text. Replace(target_phrase, replacement) print(updated_text) ```
- Dwedit 1y agoI've seen Japanese translation models that translate empty string "" into "I'm Sorry I'm Sorry"
- PatchworkCasino 1y agoI've found if the first 30 seconds of a recorded phone call is ringing and/or DTMF (almost always happens if you call a business) the system with either select Nynorsk or Welsh as the language. Never bothered to check what the text translated to but it's probably something similar. Not a practical issue for me but I can see it being a pain for any bilingual business or call center.
- junon 1y agoIt's a common problem with many languages. If you speak gibberish fake Chinese at chatgpt and ask it to translate, it'll happily say you're saying coherent things.
- zahrevsky 1y agoWhen using ChatGPT audio transcription, sometimes it adds to the end “Subtitles created by ...”, and then some username. Obviously, an artefact of training on subtitiles dataset.