3 ms·
> recording, transcribing and taking minutes of all meetings with AI. Then that gets fed into a RAG system and we can see what's been said when. Could you plea
by ponector 3y ago
> recording, transcribing and taking minutes of all meetings with AI. Then that gets fed into a RAG system and we can see what's been said when.
Could you please share details on tooling for this?
- llm_trw 3y agoNot much tooling I do it all on Linux. I use audacity to capture the audio stream live, whisper.cpp with person segmentation to extract the text with notations on who was talking and a fine tuned llama model which is trained on data specific to the companies market to extract the minutes. It's a couple of passes for the model to write the prompts to extract the data into minutes. The RAG system is just untrained sbert weights of every sentence in the recording and minutes shoved into the vector postgres. Queries are @mentions in the company chat. Basically a bunch of python scripts held together by duct tape and chewing gum.
- liotier 3y agoWhen querying, can the system answer with link to the minutes and the recording ? Lack of source citation for facts is a problem with what I've seen of LLMs.
- llm_trw 3y agoYes, part of the metadata is the date, person speaking and line number where the hit happens for each sentence if it's a hit in the transcript. If it's a hit of the minutes it's just a link to the date on which the minutes happened. That's got nothing to do with the LLM used to do the summary but part of the RAG system.
- yuppiepuppie 3y agoVery cool. I tried to do this a couple months ago, but really had trouble recording the audio from the call. Are you recording the audio from the call on an external device ie. your mic? To be clear, my issue was I could only capture the streaming audio of my mic or the people on the call. Never both at the same time.
- llm_trw 3y agoI have no idea. I just have a dummy user that joins the meetings so people known when they are being recorded. I imagine I accidentally dodged a bullet there since that user isn't supposed to have audio in and the audio out is directed to audacity and muted otherwise.
- bavell 3y agoSounds like a fun project! > whisper.cpp with person segmentation Super curious about this bit. Did you use the --diarize or the --tinydiarize flag? How accurate/reliable was the segmentation? Sounds like it was good enough to use at the very least. I had looked into this a few months ago but thought it wasn't good enough yet.
- wrs 3y agoMe too! In particular, assuming whisper segmentation works, how did you recognize and label the individual speakers?
- pnut 3y agoMS teams has this, don't bother cobbling together and maintaining your own solution.