10 ms·
Google Bard AI Now Has the Ability to Understand YouTube Videos
- ilaksh 3y agoI assume it actually does not understand video, but reads the transcript?
- mattnewton 3y agoBard can read images and, being made by the same company as YouTube, probably has access to high quality video embeddings they use for YouTube search, maybe the most sophisticated video search engine on the planet. I don’t know from the article but it could definitely be using the video content directly without a text representation. For an open source project that actually “sees” videos you can check out https://github.com/PKU-YuanGroup/Video-LLaVA https://github.com/PKU-YuanGroup/Video-LLaVA
- VHRanger 3y ago> maybe the most sophisticated video search engine on the planet And yet youtube search is such garbage most normal users still use google search to search for videos.
- xanderlewis 3y agoReally? What’s particularly bad about it?
- readyplayernull 3y agoFinding only what it's asked for.
- okdood64 3y agoWorks fine for me: https://i.imgur.com/JdxFDfX.jpg https://i.imgur.com/JdxFDfX.jpg https://i.imgur.com/N7Dpb4U.jpg https://i.imgur.com/N7Dpb4U.jpg
- dvngnt_ 3y agoit's okay. I found tiktok to be better as they search brings your current context for suggestions, user's comments are smartly turned into search links, and they do a better job of letting the user search by transcript
- spiderice 3y ago> most normal users still use google search to search for videos. Citation? This seems unlikely to me. Maybe a lot of users end up watching a video given to them by a Google search. But if someone is specifically looking for a video, I think most probably go to YouTube and use YouTube search (which is more than fine for “normal users”)
- Jensson 3y ago> And yet youtube search is such garbage most normal users still use google search to search for videos. They are the same thing at their core, biggest difference is that Youtube started limiting the number of results but has better filtering options.
- mattsan 3y agoYou could test this by asking about an uncaptioned video where the generated captions for any language would be trash (e.g. jiberrish) and using powerpoint slides to convey stuff.
- notatoad 3y agoGoogle search certainly seems to have the ability to understand video beyond a simple caption search, so I don't see why bard wouldn't have that same ability.
- naktinis 3y agoIt seems it's only based on the transcript for now. For example when I asked to comment on visual content it mentioned requiring a captions file: > I'm sorry, but I'm unable to access this YouTube content. This is possible for a number of reasons, but the most common are: the content isn't a valid YouTube link, potentially unsafe content, or the content does not have a captions file that I can read.
- iso8859-1 3y agoNo. It just hallucinated[0] about parallel evaluation while this talk doesn't mention parallel evaluation at all. So it has to be more than just the transcript. [0]: https://g.co/bard/share/9854794a5dc0 https://g.co/bard/share/9854794a5dc0
- johnea 3y agoThat is an accomplishment! I'll have to get it to explain, since most of that shit is incomprehensible to me...
- BoorishBears 3y agoI actually worked on this for a toy app: you can create tokens for images, and overlay those in the embedding space you do words That allows LLMs to treat images like "words" and speak on their contents. - That being said looking at the demo I'm not sure Bard isn't just using the video transcripts. I was using them to guide seeking, but you could answer all the test questions shown in the video without image understanding
- dpflan 3y agoHere is go now. The AI revolution begins with learning how-to videos. Just create the latent space for video/visual understanding, it's going to be very interesting to explore that.
- blowski 3y agoWould Bard understand if, say, a person in the video smiled or there was a sarcastic tone to a bit of audio?
- blibble 3y agocan we use it to detect and skip parts of the video that contain ads?
- thrtythreeforty 3y agoI'm sorry, it is not ethical or responsible for me to help you skip ads. In fact, the very suggestion is thoughtcrime. Is there anything else I can help you with?
- deleted 3y ago[deleted]
- robotnikman 3y agoThis really is the way things are heading it seems.
- _lvbh 3y agoI’m sorry, this response contains references to 1984, a film banned for inciting unrest. Thoughtcrime does not exist, you have freedom of expression unless it harms our profits.
- ratg13 3y agoThe supreme court recently made ad skipping legal, and have also removed all book bans. A recent study was done that also proved conclusively that skipping ads saves your company money by saving bandwidth.
- asddubs 3y agolook up sponsorblock
- deleted 3y ago[deleted]
- okdood64 3y agoI'd argue that the Bard YouTube search is worse off right now than just searching on YouTube itself. I assume it'll improve over time.
- seanhunter 3y agoUnderstanding the videos is all very well but can it understand: 1- the popularity of "tier list" videos? 2- why those douchetuber "prank" videos exist? 3- Logan and/or Jake Paul?
- deleted 3y ago[deleted]
- andrewmunsell 3y agoSince I had the same question as everyone else, it seems like it must be using just the transcript. When asking about one of those "8k HDR" showcase videos (with no speech), Bard responds with: > I'm sorry, but I'm unable to access this YouTube content. This is possible for a number of reasons, but the most common are: the content isn't a valid YouTube link, potentially unsafe content, or the content does not have a captions file that I can read.
- justrealist 3y agoMost captioning is automatic now though. So it's still true in a meaningful way.
- naet 3y agoIf it gets very good at "understanding" YouTube and other video content, Google could maybe find some kind of training data advantage not available to a pure text based model.
- echelon 3y agoEarly on it felt like Google was asleep at the wheel and would lose AI. Now I think they're the company best positioned to win it. YouTube is a huge component of that. Their recent YouTube music generation stuff just killed about a dozen music startups I've been tracking. And they've already got the users.
- tigershark 3y agoHow reading the captions of a video make google the front runner?
- echelon 3y agoThey just launched tools to make your own vocal covers of Frank Sinatra and thousands of other artists. And generate beats.
- tessierashpool 3y agothose quotation marks around "understanding" are pretty important. to save people a click, and in the hopes that the comments might stay relevant to the OP, the OP means this by "understanding": if the video is about a recipe, you can ask how many eggs are used in the recipe, and Bard can give you an answer. it seems pretty likely that this may mean Bard simply consumes video transcripts.
- pixl97 3y agoTest on a video without transcripts
- SeanAnderson 3y agoIs there any reason to believe YouTube content will only be trained on by Bard? Stuff like YouTubeDL exists and works fine. I would assume that others could scrape and train on it, too? Or does that sound outlandishly expensive?
- kredd 3y agoIt would be outlandishly expensive, compared to already having infrastructure that processes the videos anyways.
- SeanAnderson 3y agoOkay, so Bytedance, Amazon, Google, Facebook have an advantage in this area, then - assuming they each process TikTok, Twitch, YouTube, and Facebook/Insta video content? Are there any other big players in the video data space?
- Jensson 3y agoThe entity with the most data is the US government, or maybe the Chinese government. If data becomes the main bottleneck and everything else is well understood at some point expect those to win it.
- ben_w 3y agoPresumably Tesla, though I don’t know how much of that video is going to have useful audio if they want to specifically train a language model with the their self driving AI. Quite a lot of “would you like fries with that”, or whatever is actually said on drive-throughs.
- kredd 3y agoPurely user submitted content wise who also owns the content in the west? I would guess so, yeah. I am sure there are much more though, think of other companies who also own large data centers like Nvidia. Microsoft also does a lot from what I’ve heard.
- lossolo 3y agoOpen source version of something similar: https://github.com/PKU-YuanGroup/Video-LLaVA https://github.com/PKU-YuanGroup/Video-LLaVA
- theptip 3y agoWhisper (OpenAI speech-to-text) is already trained on YT content; amusingly, if you mumble incoherently, its most-probable completion for noise is “thanks for watching!”
- prox 3y agoThat’s hilarious :D
- metadat 3y agoI wonder how they got the model to avoid the viral "don't forget to like and subscribe".
- cavisne 3y agoSuper interesting. I’ve seen that as a rumor but never confirmed. It’s an example of the sort of grey area that would’ve made “openai 2.0” tricky at Microsoft. A small startup can ignore ToS and scrape YouTube but Microsoft cannot.
- Kye 3y agoI see "thanks" a lot in captions where it's noise or unintelligible speech. Maybe that's why.
- Passport4-Lurk 3y agoThat is hilarious! Every time I talk to ChatGPT Voice through the mobile app, and stop speaking for a few moments, it always seems to think the background noise is me saying "Thank you."
- readyplayernull 3y agoIt can't find the synthwave videos with least views, so Bard is being blinded by the recommendation algorithm.
- Jensson 3y agoWhy would it be able to find that? You are looking for a database query engine, not a language model.
- readyplayernull 3y agoWhy wouldn't they give it the ability to correctly search in the database they own?
- SeanAnderson 3y agoI wonder how this works. It sounds like it's transcript driven, but then the next question is - were the transcripts automatically created or user-defined? If the former, is this not going to run into the same issue as training AI on datasets created by AI? I experience so many mistranslated words when using automatic transcripts that I can't imagine the quality of data is excellent without supporting the transcripts with video inference.
- josephrmartinez 3y agoI made a simple Chrome extension that similarly pulls down the video transcript and sends this to the openai chat completions endpoint: https://github.com/josephrmartinez/AskYouTube https://github.com/josephrmartinez/AskYouTube This extension allows me to "ask" the model to perform a task on the video content: - "Give me the materials list" (for a diy video) - "What was the recommended book?" (for a 2+ hour podcast where they made a reference I can't find again easily) - "Extract the recommended protocol" (for 3+ hour health videos) - "Provide a counter argument" (for when I'm getting bored...) Big plus is that you DO NOT need to wait for the ad to play through. I can just navigate to the video and send in a query without having to watch any ads. Youtube transcripts are pretty rough. At first, I used Whisper to create a better transcript. But my primary use is to ask something of the youtube video - I found that slinging the so-so transcript along with my task was totally fine. Really simple project: Chrome extension in just html, css, and js. FastAPI server for the openai endpoint. Server function does a quick tokenization on the transcript to determine if I need to use the gpt4 model for the 128k context window or if the gptt3.5 16k context window is okay. Naturally, here is a short youtube demo of the extension: https://www.youtube.com/watch?v=M1zq9NKIcbw&t=54s https://www.youtube.com/watch?v=M1zq9NKIcbw&t=54s
- great_psy 3y agoCan it do something more intensive like asking … Give me a list of the video/recipes that use 3 eggs?
- Kye 3y agoFinally, no more sitting through 15 minutes of an extended "hey guys" intro just to find out how to make a Redstone machine.
- RadixDLT 3y agobaby steps