4 ms·

I assume it actually does not understand video, but reads the transcript?

by ilaksh 3y ago

mattnewton 3y ago
Bard can read images and, being made by the same company as YouTube, probably has access to high quality video embeddings they use for YouTube search, maybe the most sophisticated video search engine on the planet. I don’t know from the article but it could definitely be using the video content directly without a text representation. For an open source project that actually “sees” videos you can check out https://github.com/PKU-YuanGroup/Video-LLaVA https://github.com/PKU-YuanGroup/Video-LLaVA
- VHRanger 3y ago
  > maybe the most sophisticated video search engine on the planet And yet youtube search is such garbage most normal users still use google search to search for videos.
  - xanderlewis 3y ago
    Really? What’s particularly bad about it?
    - readyplayernull 3y ago
      Finding only what it's asked for.
      - okdood64 3y ago
        Works fine for me: https://i.imgur.com/JdxFDfX.jpg https://i.imgur.com/JdxFDfX.jpg https://i.imgur.com/N7Dpb4U.jpg https://i.imgur.com/N7Dpb4U.jpg
        dvngnt_ 3y ago
        it's okay. I found tiktok to be better as they search brings your current context for suggestions, user's comments are smartly turned into search links, and they do a better job of letting the user search by transcript
  - spiderice 3y ago
    > most normal users still use google search to search for videos. Citation? This seems unlikely to me. Maybe a lot of users end up watching a video given to them by a Google search. But if someone is specifically looking for a video, I think most probably go to YouTube and use YouTube search (which is more than fine for “normal users”)
  - Jensson 3y ago
    > And yet youtube search is such garbage most normal users still use google search to search for videos. They are the same thing at their core, biggest difference is that Youtube started limiting the number of results but has better filtering options.
mattsan 3y ago
You could test this by asking about an uncaptioned video where the generated captions for any language would be trash (e.g. jiberrish) and using powerpoint slides to convey stuff.
notatoad 3y ago
Google search certainly seems to have the ability to understand video beyond a simple caption search, so I don't see why bard wouldn't have that same ability.
naktinis 3y ago
It seems it's only based on the transcript for now. For example when I asked to comment on visual content it mentioned requiring a captions file: > I'm sorry, but I'm unable to access this YouTube content. This is possible for a number of reasons, but the most common are: the content isn't a valid YouTube link, potentially unsafe content, or the content does not have a captions file that I can read.
- iso8859-1 3y ago
  No. It just hallucinated[0] about parallel evaluation while this talk doesn't mention parallel evaluation at all. So it has to be more than just the transcript. [0]: https://g.co/bard/share/9854794a5dc0 https://g.co/bard/share/9854794a5dc0

I assume it actually does not understand video, but reads the transcript?

by ilaksh 3y ago

I assume it actually does not understand video, but reads the transcript?

Bard can read images and, being made by the same company as YouTube, probably has access to high quality video embeddings they use for YouTube search, maybe the most sophisticated video search engine on the planet. I don’t know from the article but it could definitely be using the video content directly without a text representation. For an open source project that actually “sees” videos you can check out https://github.com/PKU-YuanGroup/Video-LLaVA https://github.com/PKU-YuanGroup/Video-LLaVA

VHRanger 3y ago
> maybe the most sophisticated video search engine on the planet And yet youtube search is such garbage most normal users still use google search to search for videos.
- xanderlewis 3y ago
  Really? What’s particularly bad about it?
  - readyplayernull 3y ago
    Finding only what it's asked for.
    - okdood64 3y ago
      Works fine for me: https://i.imgur.com/JdxFDfX.jpg https://i.imgur.com/JdxFDfX.jpg https://i.imgur.com/N7Dpb4U.jpg https://i.imgur.com/N7Dpb4U.jpg
      - dvngnt_ 3y ago
        it's okay. I found tiktok to be better as they search brings your current context for suggestions, user's comments are smartly turned into search links, and they do a better job of letting the user search by transcript
- spiderice 3y ago
  > most normal users still use google search to search for videos. Citation? This seems unlikely to me. Maybe a lot of users end up watching a video given to them by a Google search. But if someone is specifically looking for a video, I think most probably go to YouTube and use YouTube search (which is more than fine for “normal users”)
- Jensson 3y ago
  > And yet youtube search is such garbage most normal users still use google search to search for videos. They are the same thing at their core, biggest difference is that Youtube started limiting the number of results but has better filtering options.

mattsan 3y ago

You could test this by asking about an uncaptioned video where the generated captions for any language would be trash (e.g. jiberrish) and using powerpoint slides to convey stuff.

notatoad 3y ago

Google search certainly seems to have the ability to understand video beyond a simple caption search, so I don't see why bard wouldn't have that same ability.

naktinis 3y ago

It seems it's only based on the transcript for now. For example when I asked to comment on visual content it mentioned requiring a captions file: > I'm sorry, but I'm unable to access this YouTube content. This is possible for a number of reasons, but the most common are: the content isn't a valid YouTube link, potentially unsafe content, or the content does not have a captions file that I can read.

iso8859-1 3y ago
No. It just hallucinated[0] about parallel evaluation while this talk doesn't mention parallel evaluation at all. So it has to be more than just the transcript. [0]: https://g.co/bard/share/9854794a5dc0 https://g.co/bard/share/9854794a5dc0