6 ms·
What sort of metadata are you extracting with machine learning? Doing more than simple keyword indexing? I'll admit, I don't know very much about the particul
by HanClinto 5y ago
What sort of metadata are you extracting with machine learning? Doing more than simple keyword indexing? I'll admit, I don't know very much about the particulars of building a large search engine.
How large is the ML model that you're using to extract the metadata? If that can be done in a distributed way, then it might let people pre-process their own feeds and share them with others -- either on-demand, like you mentioned -- or caching them centrally so that your servers are still the authoritative repository.
Anyways, everything you've built here is wonderful, and I definitely want to keep tabs on your progress and support you however I can! Thank you for sharing!
- prohobo 5y agoHey sorry I didn't see this reply until now! We take the transcripts and run natural language processing models on them to find keywords, n-grams, and pronouns. We then categorize the pronouns according to type (person, organization, media, location, event, etc.) The search engine itself is just transcripts in Elasticsearch. From the feedback I've gotten so far, I think the best thing to do is actually integrate this with YouTube through a browser extension, and have users call the backend to do this processing on-demand (with caching of course). It seems the video breakdowns are more important than searching for most people, since with YouTube you can do some kind of content searching, just not very transparently.