3 ms·
Youtube transcripts. It only works with single-person channels at the moment, as I haven't worked on disambiguating multiple speakers. They are very messy if th
by Tostino 3y ago
Youtube transcripts. It only works with single-person channels at the moment, as I haven't worked on disambiguating multiple speakers. They are very messy if they are just an auto transcription from Google. Practically unusable in most cases.
So first step in the pipeline is cleaning up the transcripts for incorrectly transcribed words or sentences. Using the context of the rest of the transcript, it is able to fix the vast majority of them.
Then we add punctuation and format it with paragraphs.
Then I have another check over the whole transcript for any remaining issues.
After all of that, I have a relatively clean transcript that represents the original audio very closely. From there, I am doing things like:
1. creating question/answer pairs from the transcript
2. creating a document of additional context that fills in details about what the speaker is talking about but may not have explicitly said
3. creating a summary of the transcript identifying the main purpose
4. creating a knowledge graph from the transcript with nodes and edges
5. creating an annotated version of the transcript using that knowledge graph
I plan on putting some of this data into a vector database, and some of it will be used for fine tuning LLaMA2 models on specific tasks (like knowledge graph creation, annotation using a knowledge graph, and writing using a knowledge graph to keep track of events)
- dr_kiszonka 3y agoThat's very informative - thanks so much for the explanation! My only experience with transcripts is in the context of transcribing short interviews. I used Whisper and it was pretty good. I mostly work with quantitative data, though. In terms of the disambiguation of speakers, I haven't done it, but I remember blind signal separation discussed in a signal processing seminar I attended. There is also this paper, in case you haven't seen it already: https://enk100.github.io/speaker_separation/ https://enk100.github.io/speaker_separation/ Thanks again!
- Tostino 3y agoI have not read that yet, thank you! Also, I haven't tried using Whisper for getting a transcription from the audio. I went the route of downloading the automatically generated transcripts from Youtube for a set of videos. An audio processing pipeline is definitely something I could add later though as an additional input channel for the overall pipeline.
- agravier 3y agoHow do you represent knowledge in your knowledge graph? Do you use an existing open source ontology?
- Tostino 3y agoNo, I just used YAML to represent it. Here is an example: https://gist.github.com/Tostino/f6f19e88e39176452c1a765cb7c2caff https://gist.github.com/Tostino/f6f19e88e39176452c1a765cb7c2... Here is the transcript that I created that knowledge graph from, and then annotated with the knowledge graph for training purposes: https://gist.github.com/Tostino/e64524437848fbb3aebe52056df8cc58 https://gist.github.com/Tostino/e64524437848fbb3aebe52056df8... Edit: I am using symbolic IDs intentionally. Reason for that was this paper: https://ai.googleblog.com/2023/07/symbol-tuning-improves-in-context.html https://ai.googleblog.com/2023/07/symbol-tuning-improves-in-...