7 ms·
Show HN: Gemini can now natively embed video, so I built sub-second video search
Gemini Embedding 2 can project raw video directly into a 768-dimensional vector space alongside text. No transcription, no frame captioning, no intermediate text. A query like "green car cutting me off" is directly comparable to a 30-second video clip at the vector level.
I used this to build a CLI that indexes hours of footage into ChromaDB, then searches it with natural language and auto-trims the matching clip. Demo video on the GitHub README.
Indexing costs ~$2.50/hr of footage. Still-frame detection skips idle chunks, so security camera / sentry mode footage is much cheaper.
- ygouzerh 6mo agoThat's quite interesting, well done! I haven't thought of this use case for embeddings. It open the door to quite many potential applications!
- stavros 6mo agoMan, the surveillance applications for this are staggering.
- dev_tools_lab 6mo agoNice use of native video embedding. How do you handle cases where Gemini's response confidence is low? Do you have a fallback or threshold?
- sohamrj 6mo agoas of now, no threshold but that is planned in the future. for example, for now if i search "cybertruck" in my indexed dashcam footage, i don't have any cybertrucks in my footage, so it'll return a clip of the next best match which is a big truck, but not a cybertruck
- dev_tools_lab 6mo agoMakes sense for now. Thresholding becomes critical at scale though — good luck with the next iteration!
- mdrzn 6mo agoVery interesting (not for a dashcam, but for home monitoring).
- fhe 6mo agothis function will be a must-have for all home security systems. I used to spend hours going through home security cameras to check if our cat went out the house when the door was accidentally left open (turned out it was just really good at hiding within the house).
- SoftTalker 6mo agoMost home monitoring only records when there is movement though? So that already compresses the search space a lot. And just zipping forward and back it's pretty easy to quickly find the 30 seconds where there is a figure wallking up to your front door.
- klntsky 6mo agowhy not skip the text conversion? is it usable at all?
- sohamrj 6mo agogemini embedding 2 converts straight video to vectors. in this case, dashcam clips don't have audio to transcribe and even if they did, it would be useless in the search
- password4321 6mo agoWhat are the SoA audio models right now?
- Aeroi 6mo agovery cool, anybody have apparent use cases for this?
- sohamrj 6mo agodashcam and home security footage are the 2 main ones i can think of. a bit expensive right now so it's not as practical at scale. but once the embedding model comes out of public preview, and we hopefully get a local equivalent, this will be a lot more practical.
- hebelehubele 6mo agoState surveillance
- wahnfrieden 6mo agoWorker surveillance
- giozaarour 6mo agoI think a good use case would be searching for certain products or videos across social media (TikTok and Instagram). especially useful for shopping, maybe
- vidarh 6mo agoBranding/marketing monitoring companies would be all over this.
- mannyv 6mo agoIndexing all your porn and skipping all the filler.
- iso1631 6mo agoisn't the "fill her" the point of porn?
- CamperBob2 6mo ago
- emsign 6mo agoWhere is the Exit to this dystopia?
- RobotToaster 6mo agoIn the matrix the exit was pay phones, which perhaps explains why our overlords are removing them
- greesil 6mo agoSuicide booths a la Futurama
- BrokenCogs 6mo agoThe Matrix style human pods: we live in blissful ignorance in the Matrix, while the LLMs extract more and more compute power from us so some CEO somewhere can claim they have now replaced all humans with machines in their business.
- throwup238 6mo agoI was thinking more of the season 3 episode of Doctor Who titled Gridlock where everyone lives in flying cars circling a giant expressway underground, while all the upper class people on the surface died years ago from a pandemic.
- ting0 6mo agoEver get the feeling that the universe is reading your mind? Maybe there's some truth to that after all.
- jama211 6mo agoI don’t think this means we’re in a dystopia
- zwirbl 6mo agoYou might not have been paying attention
- 7777777phil 6mo agoToday I learned that Gemini can now natively embed video.. Cool Project, thanks for sharing!
- kamranjon 6mo agoDoes anyone know of an open weights models that can embed video? Would love to experiment locally with this.
- sohamrj 6mo agoNot aware of any that do native video-to-vector embedding the way Gemini Embedding 2 does. There are CLIP-based models (like VideoCLIP) that embed frames individually, but they don't process temporal video. you'd need to average frame embeddings which loses a lot. Would love to see open-weight models with this capability since it would eliminate the API cost and the privacy concern of uploading footage.
- CamperBob2 6mo agoA quick search brought up https://qwen.ai/blog?id=qwen3-vl-embedding https://qwen.ai/blog?id=qwen3-vl-embedding but I have no idea if it does what Gemini is doing here.
- jakejmnz 6mo agomore or less works similarly, made a proof of concept for it: https://github.com/jakejimenez/sentinelsearch https://github.com/jakejimenez/sentinelsearch
- CamperBob2 6mo agoVery cool, thanks. Will check it out.
- jakejmnz 6mo ago[dead]
- SpaceManNabs 6mo ago> No transcription, no frame captioning, no intermediate text. If there is text on the video (like a caption or wtv), will the embedding capture that? Never thought about this before. If the video has audio, does the embedding capture that too?
- sohamrj 6mo agoYes to both. The embedding is over raw video frames, so anything visible (text, signs, captions) gets captured in the vector. And Gemini Embedding 2 extracts the audio track and embeds it alongside the visual frames. So a query like 'someone yelling' would theoretically match on audio. My dashcam footage doesn't have audio though, so I haven't tested that side yet.
- nullbyte 6mo agoWhat a brilliant idea! is this all done locally? That's incredible.
- apwheele 6mo agoWhile the vector store is local, it is sending the data to Gemini's API for embedding. (Which if using a paid API key is probably fine for most use cases, no long term retention/training etc.)
- jakejmnz 6mo agoworks completely locally with a decent model: https://github.com/jakejimenez/sentinelsearch https://github.com/jakejimenez/sentinelsearch
- jakejmnz 6mo agoMake a proof of concept, honestly worked fairly well: https://github.com/jakejimenez/sentinelsearch https://github.com/jakejimenez/sentinelsearch
- simonreiff 6mo agoVery impressive! A webhook could be configured to trigger an alarm if a semantic match to any category of activities is detected, and then you basically have a virtual security guard and private investigator. Well played.
- sohamrj 6mo agoThanks! Yeah that would be pretty cool, but continuous indexing would be pretty expensive now, because the model's in public preview and there are no local alternatives afaik. This very well might be a reality in a couple years though!
- CamperBob2 6mo agoCould https://qwen.ai/blog?id=qwen3-vl-embedding https://qwen.ai/blog?id=qwen3-vl-embedding be a possible local alternative?
- jakejmnz 6mo agoVery cool stuff, gave me the inspiration to try it locally. Works fairly well I think: https://github.com/jakejimenez/sentinelsearch https://github.com/jakejimenez/sentinelsearch
- macNchz 6mo agoThis is a really cool implementation—embeddings still often feel like magic to me. That said, this exact use case is sort of also my biggest point of concern with where AI takes us, much more so than most of the common AI risks you hear lots of chatter about. We live in a world absolutely loaded with cameras now but ultimately retain some semblance of semi-anonymity/privacy in public by virtue of the fact that nobody can actually watch or review all of the video from those cameras except when there is a compelling reason to do so, but these technologies are making that a much more realistic proposition. The presence of cameras everywhere is considerably more concerning than the status quo, to me at least, when there is an AI watching and indexing every second of every feed—where camera owners or manufacturers or governments could set simple natural language parameters for highly specific people or activities notify about. There are obviously compelling and easy-to-sell cases here that will surely drive adoption as it becomes cost effective: get an alert to crime in progress, get an alert when a neighbor who doesn't clean up after his dog, get an alert when someone has fallen...but the potential implications of living in a panopticon like this if not well regulated are pretty ugly.
- sohamrj 6mo agoTotally valid concern. Right now the cost ($2.50/hr) and latency make continuous real-time indexing impractical, but that won't always be the case. This is one of the reasons I'd want to see open-weight local models for this, keeps the indexing on your own hardware with no footage leaving your machine. But you're right that the broader trajectory here is worth thinking carefully about.
- mpalmer 6mo agoIt's 2.50 an hour because Google has margins. A nation state could do it at cost, and even if it's not a huge difference, the price of a year's worth of embeddings is just $21,900. That's a rounding error, especially considering it's a one time cost for footage.
- wholinator2 6mo agoRight? $2.50 an hour is trivial to a Government that can vote to invent a trillion dollars. Even just 1 million dollars is the cost of monitoring 45 real time feeds for a year. I'm sure just many very rich people would pay that for the safety of their compound.
- danbrooks 6mo agoI work in content/video intelligence. Gemini is great for this type of use case out of the box.
- cloogshicer 6mo agoCould this be used for creating video editing software? Imagine a Premiere plugin where you could say "remove all scenes containing cats" and it'll spit out an EDL (Edit Decision List) that you can still manually adjust.
- sohamrj 6mo agoYeah, this is a great idea, I’ve actually been thinking about exactly this as the next logical step. SentrySearch already returns precise in/out timestamps for any natural-language query and uses ffmpeg to auto-trim clips. Turning that into an EDL (or even a direct Premiere plugin that exports an editable cut list) feels natural. I’m not a Premiere expert myself, but I’d love to see this happen. If you (or anyone) wants to sketch out a quick EDL exporter or plugin, I’ll happily review + merge a PR and help wherever I can. Just drop a GitHub issue if you start something!
- rigrassm 6mo agoI picked up a Rexing dash cam a few months back and after getting frustrated with how clunky it is to get footage of it, I decided to look into building something out myself to browse and download the recordings without having to pull the SD card. While scrolling through the recordings, I explicitly remember thinking it would be nice to just describe what I was looking for and run a search. Looking forward to incorporating this into my project. Thanks for sharing!
- totisjosema 6mo agoWhat is your experience so far with the quality of the retrieved pieces?
- sohamrj 6mo agoI've found I have to be very specific to get the clip I'm searching for. For example, "car cuts me off" just returned a clip of a car driving past my blindspot. But, "car with bike rack on back cuts me off at night" gave me exactly the clip I was looking for.
- matzalazar 6mo ago[dead]
- bobafett-9902 6mo agoI wonder if the underlying improvements in visual language learning will allow for even more efficient search. The First Fully General Computer Action Model -> https://si.inc/posts/fdm1/ https://si.inc/posts/fdm1/
- QubridAI 6mo ago[flagged]
- WatchDog 6mo agoI don't quite understand the 5 second overlap. I assume it's so that events that occur over the chunk boundary don't get missed, but is there any examples or benchmarking to examine how useful this is?
- sohamrj 6mo agoyea, it's so events on a chunk boundary still get captured in at least one chunk. i haven't had the chance to do formal benchmarks on overlap vs. no-overlap yet. the 5s default is a pragmatic choice, long enough to catch most events that would otherwise be split, short enough to not add much cost (120 chunks/hr to ~138). also it's configurable via the --overlap flag.
- thegabriele 6mo agoWhy just the dash cam?
- sohamrj 6mo agodashcam is just one of the use cases and the one i tested on. but this could theoretically work with any kind of video footage like home security footage
- lwarfield 6mo agoDamn, I need to going with my embeddings project. I've currently got a prototype for using embeddings (not gemini in my case) for making a game that's kinda reverse connections: collections.lwarfield.dev
- cat-turner 6mo agoThis is great, thanks for sharing
- rkaliupin 6mo ago[dead]
- rao-v 6mo agoIs there a decent open video embedding model out there? I’d love to play with this without uploading video.
- crashabr 6mo agoI wonder how well this would work with dance videos.
- sans_souse 6mo agoTotal aside here but is that you driving the pickup I assume?
- sohamrj 6mo agohaha no i'm driving the tesla and that clip is from the left repeater camera (teslas record from all around the car)
- npilk 6mo agoMultimodal AI will lead to an interesting arms race in ad detection vs ad insertion. I played around with AI ad removal with older Gemini models, but it seems like this would be even more powerful to instantly identify ads (and potentially mute or strip them out). https://notes.npilk.com/experiments-with-ai-adblock https://notes.npilk.com/experiments-with-ai-adblock
- sbinnee 6mo agoNice article. I saw someone depicting the future of web search with AI. The conclusion was not the bright future. Simply put, ads will never go away. Either AI providers will get paid for whitelisting ads, or even worse these AI will directly promote advertised products.
- WarmWash 6mo agoPeople could collectively decide to start paying for stuff and most of our gripes could at least switch to providers not accommodating their customers.
- greesil 6mo agoCollective action is not our strong suit.
- CamperBob2 6mo agoTo which I'd say to the advertiser, "Good luck paying off the AI adblocker running in my closet at home." Then again, let's not be too hasty here. Let's see what you're willing to offer. I can sell you the eyeballs of the AI ad-watcher running in my closet for $10/impression. Or, for $1000/impression, you can bring your message to the attention of myself, an actual human. A bargain at any price!
- febed 6mo agoThis seems like something that would be very expensive to run. Do you have some representative figures at a particular resolution and frame rate?
- addandsubtract 6mo agoThe README on the GitHub has a section on this[0]: >Indexing 1 hour of footage costs ~$2.84 with Gemini's embedding API (default settings: 30s chunks, 5s overlap): >1 hour = 3,600 seconds of video = 3,600 frames processed by the model. 3,600 frames × $0.00079 = ~$2.84/hr >The Gemini API natively extracts and tokenizes exactly 1 frame per second from uploaded video, regardless of the file's actual frame rate. The preprocessing step (which downscales chunks to 480p at 5fps via ffmpeg) is a local/bandwidth optimization — it keeps payload sizes small so API requests are fast and don't timeout — but does not change the number of frames the API processes. [0] https://github.com/ssrajadh/sentrysearch#cost https://github.com/ssrajadh/sentrysearch#cost
- martz 6mo agothis can be done locally https://github.com/intel/openvino-ai-video-retrieval-analysis https://github.com/intel/openvino-ai-video-retrieval-analysi...
- jakejmnz 6mo ago[dead]
- hikaru_ai 6mo ago[dead]
- novoreorx 6mo agoIn the demo bro shows how to search for "a car with a bike rack on the back that cut me off at night." Given the grudge he must've held from being cut off, I strongly suspect that finding this specific car was his main motivation for building the project in the first place
- sohamrj 6mo agour not wrong
- subhashp 6mo agoCan I give it a photo of a person and ask it to search for the person in the video?
- bob1029 6mo ago> Check if a video chunk contains mostly still frames. Extracts 3 evenly-spaced frames as JPEG and compares file sizes. I believe you could use a combination of select and scene parameters in ffmpeg to do this automatically when a chunk of video is created each time.
- ideashower 6mo agoIs there a local model that this would work with?
- QubridAI 6mo ago[flagged]
- emmitska 6mo agoThe cost structure here is what stands out to me. $2.50/hr makes this viable for personal or small-business use cases that would have been unthinkable a year ago — security footage, home cameras, dashcam archives. The interesting design question is what happens when that cost drops another 10x and this is just a default feature of consumer cameras. Most people won't opt out of something they didn't know they opted into.
- giversoftheint 6mo agoOh I'm getting this for my website. Cool. Thanks.
- shivang2607 6mo agoBut don't you think 768 dimensional space is too less for comparing a video ? I build a similarity search for Anime and it worked Okayish. Really interested to know how it will work for video similarity search.
- wuliwong 6mo agoI am working on a really similar thing. My buddy just sent me this. Hit me up if you wanna chat!