4 ms·

Cleaning Up Speech Recognition with GPT

28 points by BeetleB 3y ago

simonw 3y ago
"I’ll probably write an elisp function to send the output to GPT and insert the results into the buffer" I suggest taking a look at my LLM command-line tool. It's great for cobbling together these kinds of things, because you can use whatever shell integration your environment has to pipe things to it. cat bad-dictation.txt | \ llm -m gpt-4-turbo --system ' You are going to correct for text that has been produced by voice recognition software. Rewrite any text provided to you. The text will not have punctuation, so please add where needed. If the text has the word period, then insert a period, and do not insert another one just before or after. If the text has “new line” or “newline”, insert a newline character and start a new paragraph. If the text says “comma”, insert a comma. If the text has a number spelled out, replace it with the actual number. So “five thousand” becomes “5000”. Also, if something seems off in the text, it was probably due to a misrecognized word. Please correct for it.' > out.txt Should be easy to call that from elisp, and you can then install plugins to have it talk to other models like Claude or Llama 3: https://llm.datasette.io/en/stable/plugins/directory.html https://llm.datasette.io/en/stable/plugins/directory.html
- fzzzy 3y ago
  Thanks a lot for llm, Simon. I've started to integrate it more and more in what I do as an abstraction layer over llm apis. By the way, do you know of any models that support streaming input tokens to reduce latency to output tokens?
  - simonw 3y ago
    I don't think I've seen one that does that. Groq is pretty amazing to play with to see what fast models are like (llm-groq plugin).
    - fzzzy 3y ago
      Thanks. The problem with groq is time to first token (being in a queue, presumably).
- BeetleB 3y ago
  I wrote it in December, when I was none the wiser. I've already got an elisp function (using whisper now for the backend - which has some cons to it). In fact, it's fairly common for me to use the combination of Whisper and GPT to compose emails, etc. I should do a followup post on my setup. Interestingly, Simon, I also submitted another story[1] where I refer to your shot-scraper. You're the gift that keeps on giving (especially given that my first programming job was with Django in 2008!) [1] https://news.ycombinator.com/item?id=40174802 https://news.ycombinator.com/item?id=40174802
- jilijeanlouis 3y ago
  BTW I’ve heard from researchers from a big GAFAM (can’t name) that they actually use gpt to do this on the ground truth and spot that correction and do a second pass of human labelling on the ground truth to have better dataset.
- tedmiston 3y ago
  Thanks for sharing this! Direct link to the repo for those curious - https://github.com/simonw/llm https://github.com/simonw/llm
psadri 3y ago
Use whisper (or whisperx)
- rahimnathwani 3y ago
  Whisperx is great for English, and there's minimal cleanup to do at the end. (Diarization makes it take much longer, and sometimes you get a bunch of repeated phrases, but transcript-only is good.) But for other languages, especially those with a variety of dialects (like Arabic), the output can be really hard to read. Interestingly, gpt-3.5-turbo can turn a garbled Arabic call transcript into a decent English summary. I guess this is due to the amount of redundancy/repetition in phone calls. I wonder whether anyone knows of something better than Whisper Large v2/v3 for Arabic ASR? I had a look recently, and what I found was: - Among published multilingual models (i.e. ignoring English-only models), it seems that Whisper Large has the lowest (best) overall WER (word error rate). - Although Facebook published a multilingual model (mms-1b-all) that outperforms Whisper on many less common languages, its Arabic WER on standard benchmarks is much worse than that of Whisper, so I don't think it's worth trying on my data. - There's a paper from ~6 months ago by some researchers who claimed to get slightly better performance than Whisper vanilla, but their code is not public, and I can't find any blog posts or articles talking about using their work. - Googling turned up some attempts to finetune mms-1b-all with Arabic, but nothing I found included any WER data, so I assume these attempts didn't work out.
  - jilijeanlouis 3y ago
    That’s a bit problem with Arabic especially because of dialects and the lack of good datasets (when I say I mean robust - including noisy dirty audios).Would you mind testing gladia.io and give feedbacks ?
    - rahimnathwani 3y ago
      Before I try it out, would you mind sharing any info about your models' WER on Arabic, compared with Whisper Large V2?
  - 3y ago
deleted 3y ago
[deleted]
aragonite 3y ago
I take a lot of long voice notes thinking out loud during my walks. I use this prompt when I give the Whisper output to GPT-4. I've found it to be pretty reliable: > Punctuate the following transcript of a voice note I took. Insert periods, commas, and paragraph breaks where appropriate. Remove filler words such as 'right?' 'you know', and 'uh'. But do retain my original wording! Do not paraphrase my sentences beyond recognition: this is not a rewriting task! The transcript now follows:
- BeetleB 3y ago
  Thanks! I'll give it a try. My current prompt is much longer (lists many things Whisper misrecognizes so GPT can replace them). Did not think of telling it to remove filler words, but I find it usually removes them anyway.
ndr_ 3y ago
If you prefer an existing webapp over an elisp function, https://huggingface.co/spaces/ndurner/oai_chat https://huggingface.co/spaces/ndurner/oai_chat. Choose Whisper as the model, upload your 25 MB chunks, hit Send, choose GPT-4 Turbo, ask it to clean up, hit Send. Then, hit the Download button (hidden away on the very bottom). Helpful fact: Whisper works on 16 KHz sampling behind the scenes, so can make your recording smaller by downsampling to 22 KHz, mono. AAC is supported, and commenters to the web say Whisper is pretty robust so it doesn‘t have do be hi-fi - just so that you can make a split at the beginning of the QA session perhaps, if you can‘t fit it into one 25 MB chunk right away.
ldenoue 3y ago
I've tried using LLMs to restore and clean up raw unpunctuated transcripts, but they tend to hallucinate new words. And chunking is an issue since long transcripts need to be split according to the LLM input context size. But where do we split if we don't have the punctuation yet? In Scribe I chose instead to restore punctuation marks using a token-classifier (here a DistilBert running in your browser) https://www.appblit.com/scribe https://www.appblit.com/scribe Laurent

28 points by BeetleB 3y ago

"I’ll probably write an elisp function to send the output to GPT and insert the results into the buffer" I suggest taking a look at my LLM command-line tool. It's great for cobbling together these kinds of things, because you can use whatever shell integration your environment has to pipe things to it. cat bad-dictation.txt | \ llm -m gpt-4-turbo --system ' You are going to correct for text that has been produced by voice recognition software. Rewrite any text provided to you. The text will not have punctuation, so please add where needed. If the text has the word period, then insert a period, and do not insert another one just before or after. If the text has “new line” or “newline”, insert a newline character and start a new paragraph. If the text says “comma”, insert a comma. If the text has a number spelled out, replace it with the actual number. So “five thousand” becomes “5000”. Also, if something seems off in the text, it was probably due to a misrecognized word. Please correct for it.' > out.txt Should be easy to call that from elisp, and you can then install plugins to have it talk to other models like Claude or Llama 3: https://llm.datasette.io/en/stable/plugins/directory.html https://llm.datasette.io/en/stable/plugins/directory.html

fzzzy 3y ago
Thanks a lot for llm, Simon. I've started to integrate it more and more in what I do as an abstraction layer over llm apis. By the way, do you know of any models that support streaming input tokens to reduce latency to output tokens?
- simonw 3y ago
  I don't think I've seen one that does that. Groq is pretty amazing to play with to see what fast models are like (llm-groq plugin).
  - fzzzy 3y ago
    Thanks. The problem with groq is time to first token (being in a queue, presumably).
BeetleB 3y ago
I wrote it in December, when I was none the wiser. I've already got an elisp function (using whisper now for the backend - which has some cons to it). In fact, it's fairly common for me to use the combination of Whisper and GPT to compose emails, etc. I should do a followup post on my setup. Interestingly, Simon, I also submitted another story[1] where I refer to your shot-scraper. You're the gift that keeps on giving (especially given that my first programming job was with Django in 2008!) [1] https://news.ycombinator.com/item?id=40174802 https://news.ycombinator.com/item?id=40174802
jilijeanlouis 3y ago
BTW I’ve heard from researchers from a big GAFAM (can’t name) that they actually use gpt to do this on the ground truth and spot that correction and do a second pass of human labelling on the ground truth to have better dataset.
tedmiston 3y ago
Thanks for sharing this! Direct link to the repo for those curious - https://github.com/simonw/llm https://github.com/simonw/llm

psadri 3y ago

Use whisper (or whisperx)

rahimnathwani 3y ago
Whisperx is great for English, and there's minimal cleanup to do at the end. (Diarization makes it take much longer, and sometimes you get a bunch of repeated phrases, but transcript-only is good.) But for other languages, especially those with a variety of dialects (like Arabic), the output can be really hard to read. Interestingly, gpt-3.5-turbo can turn a garbled Arabic call transcript into a decent English summary. I guess this is due to the amount of redundancy/repetition in phone calls. I wonder whether anyone knows of something better than Whisper Large v2/v3 for Arabic ASR? I had a look recently, and what I found was: - Among published multilingual models (i.e. ignoring English-only models), it seems that Whisper Large has the lowest (best) overall WER (word error rate). - Although Facebook published a multilingual model (mms-1b-all) that outperforms Whisper on many less common languages, its Arabic WER on standard benchmarks is much worse than that of Whisper, so I don't think it's worth trying on my data. - There's a paper from ~6 months ago by some researchers who claimed to get slightly better performance than Whisper vanilla, but their code is not public, and I can't find any blog posts or articles talking about using their work. - Googling turned up some attempts to finetune mms-1b-all with Arabic, but nothing I found included any WER data, so I assume these attempts didn't work out.
- jilijeanlouis 3y ago
  That’s a bit problem with Arabic especially because of dialects and the lack of good datasets (when I say I mean robust - including noisy dirty audios).Would you mind testing gladia.io and give feedbacks ?
  - rahimnathwani 3y ago
    Before I try it out, would you mind sharing any info about your models' WER on Arabic, compared with Whisper Large V2?
- 3y ago

deleted 3y ago

[deleted]

aragonite 3y ago

I take a lot of long voice notes thinking out loud during my walks. I use this prompt when I give the Whisper output to GPT-4. I've found it to be pretty reliable: > Punctuate the following transcript of a voice note I took. Insert periods, commas, and paragraph breaks where appropriate. Remove filler words such as 'right?' 'you know', and 'uh'. But do retain my original wording! Do not paraphrase my sentences beyond recognition: this is not a rewriting task! The transcript now follows:

BeetleB 3y ago
Thanks! I'll give it a try. My current prompt is much longer (lists many things Whisper misrecognizes so GPT can replace them). Did not think of telling it to remove filler words, but I find it usually removes them anyway.

ndr_ 3y ago

If you prefer an existing webapp over an elisp function, https://huggingface.co/spaces/ndurner/oai_chat https://huggingface.co/spaces/ndurner/oai_chat. Choose Whisper as the model, upload your 25 MB chunks, hit Send, choose GPT-4 Turbo, ask it to clean up, hit Send. Then, hit the Download button (hidden away on the very bottom). Helpful fact: Whisper works on 16 KHz sampling behind the scenes, so can make your recording smaller by downsampling to 22 KHz, mono. AAC is supported, and commenters to the web say Whisper is pretty robust so it doesn‘t have do be hi-fi - just so that you can make a split at the beginning of the QA session perhaps, if you can‘t fit it into one 25 MB chunk right away.

ldenoue 3y ago

I've tried using LLMs to restore and clean up raw unpunctuated transcripts, but they tend to hallucinate new words. And chunking is an issue since long transcripts need to be split according to the LLM input context size. But where do we split if we don't have the punctuation yet? In Scribe I chose instead to restore punctuation marks using a token-classifier (here a DistilBert running in your browser) https://www.appblit.com/scribe https://www.appblit.com/scribe Laurent