3 ms·
I've been trying to solve a problem with implementing semantic search on my YouTube search engine yt-fts (https://github.com/NotJoeMartinez/yt-fts https://githu
by notjoemartinez 3y ago
I've been trying to solve a problem with implementing semantic search on my YouTube search engine yt-fts (https://github.com/NotJoeMartinez/yt-fts https://github.com/NotJoeMartinez/yt-fts). I've managed to substantially speed up search results by storing subtitle embeddings in Chroma. But a bigger problem has been with how to properly segment the text in a way that accounts for the duration and context of word embeddings while returning precise time stamps. This a blog post exploring what I've tried so far.
- dceddia 3y agoInteresting problem and a good write up! This reminds me a little of audio processing where there are 2 representations, time domain and frequency domain. I’m no expert at this but my understanding is if you want to search for “when” some chunk of audio happened, you first need to convert to the frequency domain via Fourier transfer. But then you lose the time info. So you can’t just take the Fourier transform of the whole file, or even 10 second chunks… you have to take a bunch of short overlapping Fourier transforms – overlapping so you get the nearby context, and short so that you have a higher resolution idea of when something occurred. I wonder if a similar idea would work here, where you could search at various “zoom levels” - first search for an entire video that’s nearby in terms of embedding, then search within 50%-overlapped 60-second chunks, then within 50%-overlapped 1-second chunks.