18 ms·
The killer app of Gemini Pro 1.5 is using video as an input
- Havoc 3y ago> 7 second video consumed just 1,841 tokens How? Video is a massive amount of data
- miroljub 3y agoIt's Google. I'd rather avoid sharing my thoughts and interests with this Borg-like entity.
- RamblingCTO 3y agoYeah, I really hope open sources catches up quickly. Why on earth would I want to create a Google account just to use this, especially in work settings?
- tarruda 3y agoI think it is only a matter of time before open source vision LLMs have the ability to process videos. The tricky part might be getting to 1M token context length, which even proprietary LLMs (other than Gemini) are struggling with.
- nolok 3y agoEither you run it fully locally, or you accept that whoever runs it has access to your thoughts and interests. Whether you go with microsoft, google, meta, or whatever apple will come up with, it feels like a case of "stay out, or make a pick and stick to it". I know some have different feelings regarding this or that company that is "better" or "worse", but the reality of it is they're not, and even if they were you don't know where they will be in ten years, and they will still have your data then.
- crooked-v 3y agoI think Apple may do interesting things here with their rumored focus in purely on-device LLM functionality across the OS, taking advantage of all the hardware work they've put into efficiency and 'Neural Engine' cores. This year's WWDC may be quite interesting.
- knowriju 3y agoI am interested to see how Apple's insistence on privacy will square with their GenAI products. If they don't collect feedback and usage data how will they use RLHF to make their suit better ? I understand that have been cutting deals with few publication companies, but will that suffice?
- jaimex2 3y agoYou can do this locally by combining CLIP with an LLM quite easily.
- minimaxir 3y agoNote that a video is just a sequence of images: OpenAI has a demo with GPT-4-Vision that sends a list of frames to the model with a similar effect: https://cookbook.openai.com/examples/gpt_with_vision_for_video_understanding https://cookbook.openai.com/examples/gpt_with_vision_for_vid... If GPT-4-Vision supported function calling/structured data for guaranteed JSON output, that would be nice though. There's shenanigans you can do with ffmpeg to output every-other-frame to halve the costs too. The OpenAI demo passes every 50th frame of a ~600 frame video (20s at 30fps). EDIT: As noted in discussions below, Gemini 1.5 appears to take 1 frame every second as input.
- simonw 3y agoThe number of tokens used for videos - 1,841 for my 7s video, 6,049 for 22s - suggests to me that this is a much more efficient way of processing content than individual frames. For structured data extraction I also like not having to run pseudo-OCR on hundreds of frames and then combine the results myself.
- famouswaffles 3y agoNo it's individual frames https://developers.googleblog.com/2024/02/gemini-15-available-for-private-preview-in-google-ai-studio https://developers.googleblog.com/2024/02/gemini-15-availabl... "Gemini 1.5 Pro can also reason across up to 1 hour of video. When you attach a video, Google AI Studio breaks it down into thousands of frames (without audio),..." But it's very likely individual frames at 1 frame/s https://storage.googleapis.com/deepmind-media/gemini/gemini_v1_5_report.pdf https://storage.googleapis.com/deepmind-media/gemini/gemini_... "Figure 5 | When prompted with a 45 minute Buster Keaton movie “Sherlock Jr." (1924) (2,674 frames at 1FPS, 684k tokens), Gemini 1.5 Pro retrieves and extracts textual information from a specific frame in and provides the corresponding timestamp. At bottom right, the model identifies a scene in the movie from a hand-drawn sketch."
- simonw 3y agoDespite that being in their blog post, I'm skeptical. I tried uploading a single frame of the video as an image and it consumed 258 tokens. The 7s video was 1,841 tokens. I think it's more complicated than just "split the video into frames and process those" - otherwise I would expect the token count for the video to be much higher than that. UPDATE ... posted that before you edited your post to link to the Gemini 1.5 report. 684,000 (total tokens for the movie) / 2,674 (their frame count for that movie) = 256 tokens - which is about the same as my 258 tokens for a single image. So I think you're right - it really does just split the video into frames and process them as separate images.
- rpastuszak 3y agohehe, this is great, I was just (2 days ago) playing with a similar problem in a web app form: browsing books in the foreign literature section of a Portuguese bookstore! My (less serious) ultimate goal is a universal sock pairing app: never fold your socks together again, just dump them in the drawer and ask the phone to find a match when you need them! This seems more like a visual segmentation problem though and segmentation has failed me so far.
- heckelson 3y agoI employ a different strategy: I own 25 pairs of the same gray socks (gray was chosen so that it matches most outfits) and I just wear those all the time. Obviously I do own other socks (for suits etc.) but it has cumulatively saved me hours of sock searching.
- MarcoZavala 3y ago[dead]
- mewpmewp2 3y agoYes, I tried to employ this same strategy, but maybe it's because of my ADD or something, but I never manage to buy the same bulk socks, and eventually I run out and try to buy another bulk of socks which starts to get mixed with the last ones. I need a robot that can physically sort and organize absolutely everything in my living space. I have ideas for different strategies, but I am never able to actually implement those, so it ends up that I panic search for good pair of socks when there's an important event or just any scenario where someone would see me in socks and it would be good if socks looked similar enough.
- Narciss 3y agoIf you build a solution out, you could stand to make millions
- ta8645 3y agoI'd prefer an app that can find the missing socks for all the singletons that emerge from each load of laundry. We'll probably have to wait for a super AGI though.
- jgalt212 3y agotext prompt -> LLM -> unity -> video bim, bam, boom!
- sotasota 3y agoHow does this particular use case stack up against OCR?
- rmbyrro 3y agoI think OCR would fair pretty poorly on such messy visuals. Not to mention the partially obscured titles that Gemini guessed well, which would be impossible for an OCR.
- GaggiX 3y ago>GPT-4 Video and LLaVA expanded that to images. A little error in the page: GPT-4V stands for vision, not video.
- simonw 3y agoThanks, fixed.
- ilaksh 3y agoSo you have to get invited to use Gemini Pro 1.5 right? EDIT: there is a waitlist here https://aistudio.google.com/app/waitlist/97445851 https://aistudio.google.com/app/waitlist/97445851
- it_learnses 3y agoIt’s sad that Google ai studio is not available in Canada.
- djmips 3y agoVPN?
- TheCaptain4815 3y agoI wonder if the real killer app is Googles hardware scale verses OpenAi' s(or what Microsoft gives them). Seems like nothing Google's done has been particular surprising to OpenAi's team, it's just they have such huge scale maybe they can iterate faster.
- danpalmer 3y agoAnd the fact that Google are on their own hardware platform, not dependent on Nvidia for supply or hardware features.
- dist-epoch 3y agoThe real moat is that Google has access to all the video content from YouTube to train the AI on, unlike anyone else.
- sarreph 3y agoI’m not sure I would necessarily call YouTube a moat-creator for Google, since the content on YouTube is for all intents and purposes public data.
- dist-epoch 3y agoThere is a difference between downloading a few videos and having access to ALL of them.
- samstave 3y agoEveryone is missing the point, it seems (please BOFH me when wrong); Its not going to be all about "llms" and this app or that app... They all will talk, just like any other ecosystem, but this one is going to be different... it can ferret out connections as BGP will route. Gimme an AI from here, with this context, and that one and yes, please Id like another... and it will create soft LLMs - temporal ones dedicated to their prompt and will pull from the tentriles of knowledge it can grasp and give you the result. AI creates IRL Human Ephemeral Storage.
- samstave 3y agoPre-emptive temporal curated LLMs in ..x0x Meatbag translation: The pre-emptive is the cancer that will kill us. Fuck you: * insurance * taxes * health... (what MAY this body-populous do, based on LLM-x trained on accuarial q and reduce from Human to cellular. How fucking cyberpunk dystopian would one like to get. The scariest wave of intellect is those that create technology before we had such technology "well, weve always been that way... Robots (AI) have no such "I would like to play in the yard"
- technics256 3y agoThe real frustrating thing about this is how Gemini 1.5 is a marketing ploy only. Not even 1.0 Ultra is available in the GCP API. only for their "allowlist" clients.
- daft_pink 3y agoI feel that while youtubers and influencers are heavily interested in video tools, most average users aren’t that interested in creating video. I write a lot more email than sending out videos and the value of those videos is mostly just for sharing my life with friends and family, but my emails are often related to important professional communications. I don’t think video tools will ever reach the level of usefulness to everyday consumers that generative writing tools create.
- simonw 3y agoThat's why I'm excited about this particular example: indexing your bookshelf by shooting a 30s video of it isn't producing video for publication, it's using your phone as an absurdly fast personal data entry device.
- valleyer 3y agoRecall that TFA discusses analyzing video, not generating video.
- acid__ 3y agoWow, only 256 tokens per frame? I guess a picture isn’t worth a thousand words, just ~192.
- deleted 3y ago[deleted]
- aantix 3y agoIt’d be interesting to feed it several comedies, and see what it would calculate as "laughs per minute". https://www.forbes.com/sites/andrewbender/2012/09/21/top-10-funniest-movies-ever-as-measured-in-laughs-per-minute/?sh=3b4bac433c40 https://www.forbes.com/sites/andrewbender/2012/09/21/top-10-...
- ugh123 3y agoTitle should have input added to the end "The killer app of Gemini Pro 1.5 is video input" Seems like a good way to do video moderation (YouTube) at scale, if they can keep costs down...
- CobrastanJorji 3y agoProbably overkill for content moderation, I'd think. You can identify bad words looking only at audio, and you can probably do nearly as good a job of identifying violence and nudity examining still images. And at YouTube scale, I imagine the main problem with moderation isn't so much as being correct, but of scaling. statista.com (what's up with that site, anyway?) suggests that YouTube adds something like 8 hours of video per second. I didn't run the numbers, but I'm pretty sure that's way too much to cost effectively throw something like Gemini Pro at.
- CamelCaseName 3y agoFor now, but in a year? You could also stagger the moderation to reduce costs. E.g. Text analysis: 2 views Audio analysis: 300 views Frame analysis: 5,000 views I would be very surprised if even 20% of content uploaded to YouTube passes 300 views.
- elzbardico 3y agoPeople assume that we can scale the capabilities of LLMs indefinitely, I on the other side strongly suspect we are probably getting close to diminishing returns territory. There's only so much you can do by guessing the next probably token in a stream. We will probably need something else to achieve what people think that will soon be done with LLMs. Like Elon Musk probably realizing that computer vision is not enough for full self-driving, I expect we will soon reach the limits of what can be done with LLMs.
- ugh123 3y agoOr.. google supplies some kind of local LLM tool which processes your videos before uploaded. You pay for the gpu/electricity costs. Obviously this would need to be done in a way that can't be hacked/manipulated. Might need to be highly integrated with a backend service that manages the analyzed frames from the local machine and verifies hashes/tokens after the video is fully uploaded to YouTube.
- jpeter 3y agoNext step is to use all of YouTube to train Gemini 2.0.
- karmasimida 3y agoAs long as it doesn't regenerate (I don't think google will allow it), for video analysis, it is totally within google's rights to do it.
- whiterknight 3y agoSafety is becoming an orwellian word to refer to things that can’t actually harm you.
- JieJie 3y agoSure, but does it pass the Selective Attention Test? https://www.youtube.com/watch?v=vJG698U2Mvo https://www.youtube.com/watch?v=vJG698U2Mvo (I don't know, I don't have access.)
- leumassuehtam 3y agoYes, it was tested in 8:08 of https://youtu.be/D5u7trVY5Ho?si=n1N_sm3iy0dTx4b5 https://youtu.be/D5u7trVY5Ho?si=n1N_sm3iy0dTx4b5
- smartmic 3y agoThat is impressive at first glance, no question. To stay with the example of the bookshelf, you would only follow this path for several or very many books, as in the example with the cookbooks. I have no idea how good the Geminis or GPTs of this world currently are, but let's optimistically assume a 3% error rate due to hallucinations or something. If I want to be sure that the results are correct, then I have to go through and check each entry manually. I want to rule out the possibility that there are titles listed in the 3% that would completely turn an outsider's world view of me upside down. So, even if data entry is incredibly fast, curation is still time-consuming. On balance, would it even be faster to capture the ISBN code of 100 books with a scanner app, assuming that the index lookup is correct, or to compare 100 JSON objects with title and author for correctness? The example is only partly serious. I just think that as long as hallucinations occur, Generative AI will only get part of my trust - and I don't know about you, but if I knew that a person was outright lying to me in 3% of all his statements, I wouldn't necessarily seek his proximity in things that are important to me...
- jpc0 3y agoThis right here. I'm currently building out some code that should go in production in the next week or two and simply because of this we are using LLM to prefill data and then have a human look over it. For our use case the LLM prefilling the data is significantly faster but if it ever gets to the point of that not needing to happen it would take a task whichtakes about 3 hours ( now down to one hour ) and make it a task that takes 3 minutes. Will LLMs ever get to the point where it is perfectly reliable ( or at least with an error margin low enough for our use case ), I don't think so. It does make for a very cheap accelerator though.
- simonw 3y agoThis isn't a problem that's unique to LLMs though. Pay a bunch of people to go through and index your book collection and you'll get some errors too. What's interesting about LLMs is they take tasks that were previously impossible - I'm not going to index my book collection, I do not have the time or willpower to do that - and turned them into things that I can get done to a high but not perfect standard of accuracy. I'll take a searchable index of my books that's 95% accurate over no searchable index at all.
- darkwater 3y agoI find really hard to understand how a system like this can STILL be fooled by the Scunthorpe issue (this time with "cocktail"). Aren't LLM supposed to be good at context?
- keefle 3y agoHow would the results compare to: 1. Video frames are sampled (based on frame clarity) 2. The images are fed to OCR, with their content outputed as: Frame X: <content of the frame> 3. The accomulated text is given to an average LLM (Mistral) and asked the same request mentioned by the author (creating a JSON file containing book information) Wouldn't we get something similar? maybe if a more sophisticed AI is used? So the monopoly on Gemini Pro for video processing (specifically when it comes to handling text present inside the video) is not really a sustainable advantage? or am I missing something (as this is something beyond just a fancy OCR hooked into a LLM? as the model would be able to tell that this text is on a book for instance?)
- simonw 3y agoSure, you can slice a video up into images and process them separately - that's apparently how Gemini Pro works, it uses one frame from every second of video. But you still need a REALLY long context length to work with that information - the magic combination here is 1,000,000 tokens combined with good multi-model image inputs.
- keefle 3y agoI see, but I was wondering about the partial transferability of this feature to other LLMs But fair enough, context length is key in this scenario
- phaser 3y agoThis is nice. but since google is probably training on it's vast google books data set, i'm not extremely surprised.
- barrkel 3y agoI wonder if it could identify new books with titles it's never seen before.
- blueblimp 3y agoThe tech is legitimately impressive and exciting, but I couldn't help but chuckle at the revenge of the Scunthorpe problem: > It looks like the safety filter may have taken offense to the word “Cocktail”!
- 1oooqooq 3y ago> > It looks like the safety filter may have taken offense to the word “Cocktail”! It's almost as if they got some intern to "code" the correctness filter using some AI coding assistant!
- IshKebab 3y agoWhat do the tokens for an image even look like? I understand that tokens for text are just fragments of text... but that obviously doesn't make sense for images.
- bonoboTP 3y agoThe image is subdivided by a grid and the resulting patches are fed through a linear encoder to get the token embeddings.
- luke-stanley 3y agoI can't access that Google AI Studio link because I'm in some strange place called the UK so I'm unable to verify or prototype with it currently. People at Deepmind, what's with that?
- evanmoran 3y agoI can’t either because I’m using a strange new device called a phone and it says my device width is too small to support.
- tekni5 3y agoI was thinking about this a while back, once AI is able to analyze video, images and text and do so cheap & efficiently. It's game over for privacy, like completely. Right now massive corps have tons of data on us, but they can't really piece it together and understand everything. With powerful AI every aspect of your digital life can be understood. The potential here is insane, it can be used for so many different things good and bad. But I bet it will be used to sell more targeted goods and services.
- worldsayshi 3y agoUnless you live in the EU and have laws that should protect you from that.
- seniorivn 3y agoincentives cannot be fixed with just prohibitive laws, war on drags should've taught you something
- garbagewoman 3y agoWar on drags? I thought that was just in Florida
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- chefandy 3y agoThese things seem great for casual use, but not trustworthy enough for archival work, for example. The world needs casual-use tools, too, but there are bigger impact use cases in the pipeline. I'd love for these things to communicate when they're shaky on an interpretation, for example. Maybe pairing it with a different model and using an adversarial approach? Getting a confidence rating on existing messy data where the source is available for a second pass could be a good use case. Looking at this, however, my hope is soured by the exponentially growing power of our law enforcement's panopticon. The existing shitty, buggy facial recognition system is already bad, but making automated fingerprints of people's movements based on their face combined with text on clothing and bags, the logos on your shoes, protest signs, alerting authorities if people have certain bumper stickers or books, recording the data on every card made visible when people open their wallets at public transit hubs or to pay for coffee or groceries, or set up a cheap remote camera across the street from a library to make a big list of every book checked out correlated with facial recognition... I mean, damn. Even in the private sector affording retailers the ability to make mass databases of any logo you've had on you when walking into their stores... or any stores considering it will be data brokers who keep it. Considering how much privacy our society has killed with the data we have, I'm genuinely concerned about what they will make next. Attempts to limit Facebook, et al may well seem quaint pretty soon. How about criminal applications? You can get a zoom camera with incredible range for short money, and surely it wouldn't be that hard to find a counter in front of a window where people show sensitive documents. Even just putting a phone with the camera facing out in your shirt pocket and walking around a target rich environment could be useful when you can comb through that gathered data looking for patterns, too. That said, I'm not in security, law enforcement, crime, or marketing data collection so maybe I'm full of beans and just being neurotic. Edit: if you're going to downvote me, surely you're capable of articulating your opposition in a comment, no?
- nox101 3y agohonest question: Why is it bad? I see that posted over and over. Right now I watch SF and LA feel like 3rd world countries. Nothing appears to be enforced. Traffic laws, car break-ins, car theft, garage break-ins, house break-ins. I'd personally choose a little less privacy if it meant less people were getting injured by drivers ignoring the traffic laws and, less people were having to shell out for all the costs associated with theft including replacing or repairing the damaged/stolen item as well as the increased insurance costs, cost that get added to everyone's insurance regardless of income level. Note: car break-in, garage break-in has both costs for the items stolen and costs to repair the car/garage/house. I don't know where to draw the line. I certainly don't want cameras in my house or looking through my windows. Nor do I want it on my computer or TV looking at what I do/view. For traffic, I kind of feel like at a minimum, if they can move the detection to the cameras and only save/transmit the violations that would be okay with me. You violated the law in a public space that affected others, your right to not be observed ends for that moment in time. Also, if I could personally send in violations I would have sent 100s by now. I see 3-8 violations every time I go out for a 30-60 minute drive. https://www.latimes.com/california/story/2024-01-25/traffic-deaths-surpass-homicides-in-los-angeles https://www.latimes.com/california/story/2024-01-25/traffic-... There are similar articles for SF.
- plastic3169 3y agoI was just today thinking that AI assisted editing could be a nice interface. You could watch the image and work mostly by speaking. Computer could pull the images based on description. Make first assembly edit and give alternatives. Ok drop that shot, cut from this shot when the characters eyes leave the frame, replace this take etc. There is something in editing that feels contained enough that in can be described with language.
- hendry 3y agoCan it work on traffic I wonder? Automatic number-plate recognition (ALPR)
- djmips 3y agoYes.
- 7357 3y agoTo me the 'It didn’t get all of them' is what makes me think this AI thing is just a toy. Don't get me wrong, it's marvelous as it is, but it only is useful (I use ollama + mistral 7B) when I know nothing, if I do have some understanding of the topic at hand it just becomes plain wrong. Hopefully I will be corrected.
- simonw 3y agoHave you spent much time with GPT-4? I like experimenting with Mistral 7B and Mixtral, but the quality of output from those is still sadly in a different league from GPT-4.
- 7357 3y agoNo I have not, I am not convinced I should spend money on it (yet) Using 'sadly' in your answer hints at triggering an emotional response, therefore I will ignore You are a journalist according to your profile, and please don't get me wrong, but I like to use Mistral 7B, even if it is not as good as GPT4, but it only works for me if I want to be creative, but not accurate, e.g. marketing, writing condolences :( I would not use it for anything serious PS: I checked a few other comments here, and I am not the only one who thinks the same, so pointing me at another paid version is not a proof. All I am saying is that there is too much error for it to be more than a toy
- deleted 3y ago[deleted]
- MyFirstSass 3y agoOk, crazy tangent; Where agents will potentially become extremely useful/dystopian is when they just silently watch your entire screen at all times. Isolated, encrypted and local preferably. Imagine it just watching you coding for months, planning stuff, researching things, it could potentially give you personal and professional advice from deep knowledge about you. "I noticed you code this way, may i recommend this pattern" or "i noticed you have signs of this diagnosis from the way you move your mouse and consume content, may i recommend this lifestyle change". I wonder how long before something like that is feasible, ie a model you install that is constantly updated, but also constantly merged with world data so it becomes more intelligent on two fronts, and can follow as hardware and software advances over the years. Such a model would be dangerously valuable to corporations / bad actors as it would mirror your psyche and remember so much about you - so it would have to be running with a degree of safety i can't even imagine, or you'd be cloneable or loose all privacy.
- huytersd 3y agoIf that much processing power is that cheap, this phase you’re describing is going to be fleeting because at that point I feel like it could just come up with ideas and code it itself.
- MetalGuru 3y agoIsn’t this what rewind does?
- az226 3y agoRewind.ai
- evaneykelen 3y agoI have tried Rewind and found it very disappointing. Transcripts were of very poor quality and the screen capture timeline proved useless to me.
- Falimonda 3y agoIf it wasn't for the poor transcript quality would you consider Rewind.ai to be valuable enough to use day-to-day? Could you elaborate on what was useless about the screen capture timeline?
- elzbardico 3y agoReally. I am not that impressed. It is not something radically different from doing the same thing with a still photo which by now is trivial for those models. What is being tested here doesn't require a video. It is not showing to be able to derive any meaning from a short clip. It is fucking doing very fancy OCR, that's all. What would impress me is if shown a clip of an open chest surgery it was able to comment what surgery is being done, which technique is being used, or if shown video of construction workers, be able to figure out what is the building technique, what they are actually doing, telling that the guy with the yellow shirt is not following safety regulations by not wearing a helmet.
- andiareso 3y agoBut it can do that…
- deleted 3y ago[deleted]
- z7 3y ago>What would impress me is if shown a clip of an open chest surgery it was able to comment what surgery is being done, which technique is being used, or if shown video of construction workers, be able to figure out what is the building technique, what they are actually doing, telling that the guy with the yellow shirt is not following safety regulations by not wearing a helmet. You mean like in this demo? https://www.youtube.com/watch?v=wa0MT8OwHuk https://www.youtube.com/watch?v=wa0MT8OwHuk
- AI_beffr 3y agohe calls this technology "exciting." it makes me shudder. i have been contemplating this for a decade, this specific thing, and now it really is right in front of us. what happens when the useful data within any image or video stream can be extracted into the form of text and descriptions? a model of the world or of a country will emerge that you can hold in your hand. you can know the exact whereabouts of anyone at any time. you can know anything at any time. a real-time model of a country. and AI will be able to digest this model and answer questions about it. any government that has possession of such a system will wield absolute control in a way that has never been possible before. it will have massive implications. liberal democracy will no longer be viable as an economic of political framework. jeff bezos once said that we are essentially lucky that the most efficient way for resources to be utilized is in a decentralized manner. the fact that liberty is the strongest model economically, where everyone acts independently, is a happy coincidence. centralized economies, otherwise known as communism, havent worked in the past but that will change because with the power of AI, and with the real-time model and control-loop that it will make possible, the most efficient way to manage and deploy resources will be with one central management entity. in other words, an advanced AI will do literally everything for us, human labor will be made worthless, and countries that stick to the old ways will simply be made obsolete. inevitably, the AI-driven countries, with their pathetic blobs of parasitic human enclaves hanging off their tits, will move in on the old countries and destroy them for some inane reason such as needing more space to store antimatter. whatever. even without looking all the way into the future, these AI video and image digesting tools will give birth to new and horrifying possibilities for bad actors in the government. their ability to steam roll over peoples lives in a bureaucratic stupor will be completely out of control. this seems like a sure thing but it doesnt seem likely at all that AI will be proactively and bravely used to counter-balance the negative uses by concerned citizens. people need to open their eyes to the possibility that different levels of technology are like points on a landscape -- not necessarily getting better or worse with time or "progress."
- elzbardico 3y agoMan. LLMs are basically auto-complete systems. This scenario you're painting seems too far-fetched for this technology at any timeline you could propose.
- waynesonfire 3y ago> It looks like the safety filter may have taken offense to the word “Cocktail”! how dare you!!! You are not allowed to think that. It's crazy we are witnessing modern day equivalent of book burning / freedom of speech restrictions. Kind of a bummer. I'm not smart enough to argue freedom of speech and wish someone smarter than me addressed this. Maybe I can ask chatgpt.
- cubefox 3y agoSo it is only about 256 tokens per image. I think the standard text tokenization method encodes two bytes per token, resulting in around 65.000 different tokens. If the same holds for images, given that they have the same price in the API, that would be just 512 bytes per image. Which seems impossibly low considering that the AI is still able to read those book titles. I don't understand what is going on here.
- loudmax 3y agoAt the end of the article, a single image of the bookshelf uploaded to Gemini is 258 tokens. Gemini then responds with a listing of book titles, coming to 152 tokens. Does anyone understand where the information for the response came from? That is, does Gemini hold onto the original uploaded non-tokenized image, then run an OCR on it to read those titles? Or are all those book titles somehow contained in those 258 tokens? If it's the later, it seems amazing that these tokens contain that much information.
- simonw 3y agoI would LOVE to understand that myself.
- jacobr1 3y agoI'm not sure about Gemini, but OpenAI GTP-V bills at roughly a token per 40x40px square. It isn't clear to me these actually processed as units, but rather it seems like they tried to approximate the cost structure to match text.
- zacmps 3y agoRemember, if it's using a similar tokeniser to GPT-4 (cl100k_base iirc), each token has a dimension of ~100,000. So 258x100,000 is a space of 25,800,000 floats, using f16 (a total guess) that's 51.6kB, probably enough to represent the image at ok quality with JPG.
- Animats 3y agoCan you look at the tokens generated from an image?
- andy_xor_andrew 3y ago> That 7 second video consumed just 1,841 tokens out of my 1,048,576 token limit. is this simply an approximation done by Gemini in order to add some artificial limit on the amount of video? Or do video frames actually equate directly to tokens somehow? I guess my question is, is there a real relationship between videos and tokens as we understand them (i.e. "hello" is a token) or are they just using the term "tokens" because it's easy for a user to understand, and an image is not literally handled the same way a token is?
- simonw 3y agoThere's a new section at the bottom of the article about that. It looks like an image is 258 tokens, and Gemini splits videos into one frame per second and processes those as images.
- deleted 3y ago[deleted]
- msk-lywenn 3y agoI like how the article points out the token consumption at each step. Do we have an idea of how much energy is actually used by each token?
- justinclift 3y agoAs it says the audio is stripped/removed from video before processing, wonder how well it'd do if asked to transcribe by lip reading?
- simonw 3y agoIt looks like it actually only considers one frame for every second of video, so that certainly wouldn't work.
- justinclift 3y agoYeah. If that interval isn't able to be adjusted then you're likely right. Oh well. ;)
- nostromo 3y ago> It looks like the safety filter may have taken offense to the word “Cocktail”! I opened up the safety settings, dialled them down to “low” for every category and tried again. It appeared to refuse a second time. Google really is its own worst enemy. Their risk management people have completely taken over the organization to a point where somehow the smartest computers ever created are afraid of using dangerous words like "cocktail" or creating dangerous images of people like "Abraham Lincoln."
- wepple 3y agoAt the same time, nearly daily there’s a “google did a bad thing” post on HN front page. Can’t win I guess?
- davidmurdoch 3y agoThere are things Google does itself, and then there are the things Google won't allow users to do.
- wepple 3y agoCan we trust the media and congress to distinguish the two? So many platforms have come under fire for “supporting” a theme, when all they’ve done in reality is provide media hosting services for user generated content, and %0.001 of bad content isn’t removed, thus Facebook/twitter/YouTube is held to blame FWIW I don’t think there’s a clear answer to the underlying problem. I have just learnt to expect the media to blame whoever is easiest for clicks at any given point. Right now, it’s big tech.
- baobabKoodaa 3y agoNobody would complain on HN if Google Gemini was generating pictures of Lincoln existing as a... gasp... white person. This absurd level of woke censorship is not doing them any good.
- narrator 3y ago
- veilmail 3y ago[dead]
- 2sk21 3y agoCan someone give mea reference that describe how exactly multimodal tokens are generated?
- mberning 3y agoI can’t wait for the closed source and NDA future of everything. It’s gonna suck.
- Vicinity9635 3y ago"So Google’s new Gemini chatbot is racist as fuck." https://twitter.com/JoshWalkos/status/1760423141942178037 https://twitter.com/JoshWalkos/status/1760423141942178037
- lucubratory 3y agoThis has nothing to do with the model's capabilities and isn't substantially different from the vast majority of mainstream values in content moderation on social media.
- Vicinity9635 3y ago[dead]
- ebisoka 3y agoAh yes the "mainstream values" where there is no problem with "reverse" racism or "reverse" sexism. Who cares about the model when the owners are a bunch of racists and sexists, altough I guess some people who share these disgusting and regressive "values" will think it's great.
- LAC-Tech 3y ago- OP never said it had anything to do with the model's capabilities - values being mainstream (read - held by the rich, powerful and influential) does not make them OK
- dade_ 3y agoCartman's dream came true. https://m.youtube.com/watch?v=Ar4aHfA2UwQ https://m.youtube.com/watch?v=Ar4aHfA2UwQ
- LAC-Tech 3y agoWe'll just have to wait for Yandex to come out with an equivalent product.
- LAC-Tech 3y agohttps://twitter.com/LewisCTech/status/1760504251938492616 https://twitter.com/LewisCTech/status/1760504251938492616 Seems to be fixed
- Uptrenda 3y ago[flagged]
- simonw 3y agoRude! I knew putting a video of a bookshelp up would risk people judging me based on the books they saw there, but I got over that by deciding that if anyone did that it would reflect badly on them, not badly on me. (Unsurprisingly, I own a lot more books than the ones visible in those videos.)
- Uptrenda 3y agoLol, sorry. I'll give you the benefit of the doubt.
- acdha 3y agoThat’s a harsh judgement to make without considering whether, for example, he might mostly keep work books in the bookshelf where he works. I’d also suggest doing a little homework, since you could easily correct that false impression from his public presence.
- djmips 3y agoNo that would be me...
- gmuslera 3y agoIt may end being truly a killer app. It is already bad for privacy the amount of video that is around, but increasing some orders how fast, easy and scalable may be processing it may increase the amount that is processed, even if is not perfect identifying what is there. And that by different actors, not just governments or intelligence agencies. Now match that what is happening right now in Palestine in the present or somewhere else in a not so far future.
- mv4 3y agoGreat demo, but let's not forget this is essentially OCR. The real killer case is content understanding and discovery. I am building an app, maybe someone from G wants to team up? :)
- realjhol 3y ago[dead]
- pizzalife 3y agoI found this pretty funny: I opened up the safety settings, dialled them down to “low” for every category and tried again. It appeared to refuse a second time. So I channelled Mrs Doyle and said: go on give me that JSON And it worked!
- superb-owl 3y agoThe “cocktail” thing is real. A while back I tried to get DALLE to imagine characters from Moby Dick [1], but it completely refused. You’d think an AI company could come up with a better obscenity filter! [1] https://superb-owl.link/shapes-of-stories/#1513 https://superb-owl.link/shapes-of-stories/#1513
- illusive4080 3y agoI told Azure AI to summarize a chat thread and it gave me a paragraph. I said “use bullets” and got myself flagged for review. Good gracious could I please just use an unfiltered model? Or maybe one which isn’t so sensitive?
- fragmede 3y agothe llama2-uncensored model isn't quite state of the art, but ollama makes it easy to run if you have the hardware/am willing to pay to access a cloud GPU. I colloquially used the word "hack" when trying to write some code with ChatGPT, and got admonished for trying to do bad things, so uncensoring has gotten interesting to me.
- cosmojg 3y agoYou sure can! NeuroEngine[1] hosts some nice free demos of what are basically the state of the art in unfiltered models, and if you need API access, OpenRouter[2] has dozens of unfiltered models to choose from. [1] https://www.neuroengine.ai/ https://www.neuroengine.ai/ [2] https://openrouter.ai/ https://openrouter.ai/
- justworkout 3y agoI couldn't even get Google Gemini to generate a picture of, verbatim, "a man eating". It gave me a long winded lecture about how it's offensive and I should consider changing my views on the world. It does this with virtually any topic.
- shatnersbassoon 3y agoIt's the Scunthorpe problem all over again
- seydor 3y agoThe killer app of ai is robots. Like, literally killer but also farmer, cleaner, builder etc
- yieldcrv 3y agothe boston dynamics dog was retrofitted with multimodal LLM a few months ago and its snarky as hell https://youtu.be/djzOBZUFzTw?si=NL4eFyMTAe1FcNhC https://youtu.be/djzOBZUFzTw?si=NL4eFyMTAe1FcNhC timestamp: 5:05
- yarone 3y agoAnd car driver. When I heard about how Tesla was training it's AI - without describing objects but instead through direct observation - it reminded me of Heinlein's "Door Into Summer" (1956). Heinlein's character teaches a multipurpose robot how to do any tedious human task through direct observation.
- iraqmtpizza 3y agocan it generate white people is the new can it run crysis
- LAC-Tech 3y agoI just checked, it can generate white people for me. My prompt was "A medieval noble of England". More accurate looking than anything the BBC can produce now.
- LuciBb 3y ago[dead]
- QuercusMax 3y agoGuess the author didn't bother to check that those books actually are correct? The first one I checked, "Growing Up with Lucy by April Henry" doesn't exist. The actual book is by Steve Grand, and it's very obviously so in the video used as input. So a cool demo, but sadly useless for anything more.
- flextheruler 3y agoI think this post and others reactions and then your comment this far down really encapsulates where we’re at with this technology. Nearly 90 percent of comments on posts about LLMs are people talking about how the near future is about to boggle our minds and that general intelligence is near, but all my experiences with these LLMs show they’re capable of making the most basic of mistakes and doing so confidently and that’s just the tip of the iceberg in terms of their problems. I’m having a hard time buying into the hype that these will be able to competently replace nearly any job anytime soon. They’re useful tools but they all come with a big asterisk of human hand holding.
- TrackerFF 3y agoHumans are also perfectly capable of confidently doing mistakes. The big difference here is that these models can scale the work beyond human capability. Why pay 10000 mechanical turks to extract information from vids, if you can deploy N of these models, and get the work done at a fraction of the time? Instead you can keep x% of the MTurks to check the vids where the model yields some high uncertainty score, and randomly audit other vids for quality assurance. There's crazy amounts of potential in these things. Hell, the place I work at has already replaced certain human tasks with LLM-integrated solutions, with extremely good results.
- bergenbergen 3y agoThanks for this comment. I am yet to see any “art” produced by AI that is not superficial or hollow (best case) or deeply unsettling (common case).
- 3y ago
- dEnigma 3y ago> It looks like the safety filter may have taken offense to the word “Cocktail”! I'm definitely not a fan of these severely hamstrung by default models. Especially as it seems to be based on an extremely puritan ethical system.
- riwsky 3y agoFinally, early-aughts 1337 a3s7h37ic can be cool again
- xyzelement 3y agoWe're months into this technology being available so it's not a surprise that the various "safeties" have not been perfectly tuned. Perhaps Google knew they couldn't be perfect right now and they could err on the side of the model refusing to talk about cocktails, or err on the side of it gladly spouting about cocks. They may have made a perfectly valid choice for the moment.
- slimsag 3y agoIf you want a great example of how this plays out long-term, look no further than algospeak[0] - the new lingo created by censorship algorithms like those on youtube and tiktok. [0] https://www.nytimes.com/2022/11/19/style/tiktok-avoid-moderators-words.html https://www.nytimes.com/2022/11/19/style/tiktok-avoid-modera...
- PFUguides 3y agoThere aren't much video captchas yet. But I'm pretty sure it will be able to solve a lot of those
- sdenton4 3y agoToday I learned I own basically the same set of cookbooks as Simon Willison.
- deleted 3y ago[deleted]
- tamimio 3y agoCool and all, but are we going to pass the need for prompts already? I can see big usage for video access but the prompt mechanism is making it like a toy, is there an auto processing, where I predefine what to look for and feed the video and as long as the video is running it will process based on the criteria?
- Edidiongben 3y ago[dead]
- sinuhe69 3y agoI don’t get it. The video mentioned was just about text recognition, something AI has mastered long ago. It was not about objects, movements or other complex actions (drawing or building for example). What is so impressive about it then?
- greesil 3y agoCan it do face recognition?
- Liquix 3y agoIn the same vein as "agents watching your screen" - what about "agents watching your posture"? Pages like [0] and [1] exist because people experience great benefits from becoming (even slightly) more aware of the way they are holding their bodies. Imagine this idea taken to the extreme, with a local agent intelligently reminding you to tighten your core, square your shoulders, relax your tongue, or warning of potential incoming RSI? [0] https://news.ycombinator.com/item?id=35939206 https://news.ycombinator.com/item?id=35939206 [1] https://static.virtualsaleslab.com/vsl-poc/ppp/ https://static.virtualsaleslab.com/vsl-poc/ppp/
- polygot 3y ago> He sat as still as he could on the narrow bench, with his hands crossed on his knee. He had already learned to sit still. If you made unexpected movements they yelled at you from the telescreen.
- Liquix 3y agoagreed, anything FAANG/internet facing with this capability is Orwellian, which is why > local is explicitly included in the idea.
- darepublic 3y agoI prompted gemini 1.0 on a screenshot of the hackernews home page but it could not perform OCR on it with the image I gave it.
- _kb 3y agoModelling video as a series of frames seems like such a waste; and a great point of focus for optimisation. The vast majority of video content has a lot of redundant inter-frame information. De-duping this is a key part of most compression schemes and (as an AI simpleton) seems like on obvious entry point for minimising token usage. Or is this simply a case where token windows are expected to / have already grown to a point where this sort of optimisation is not needed?
- tomas789 3y agoSomeone should do Justin.tv again but with this and people could query their life.
- deleted 3y ago[deleted]
- DerCommodore 3y agoCrazy Times
- csk111165 3y agoDon't over-hipe this feature, it was already there before as well in some other app.
- lupusreal 3y agoThings are going to get strange as soon as we have AI wearables that monitor everything a person does/sees/hears in real time and privately offers them suggestions. It will seem great at first, vigilant life-coaching for people who need help, or knowledge/memory enhancement to make effective people even more effective. But what happens when people really start to trust the voice whispering in their ear and defer all their decision making to it? They'll probably become addicted to it, then enslaved to it. They will become meat puppets for the AI.
- leumon 3y agoI wonder how gemini 1.5 compares to its open source variant (that also has video input) released about the same time as gemini: https://largeworldmodel.github.io/ https://largeworldmodel.github.io/
- DeathArrow 3y agoSo if you have a video of people at a protest, you can send it to Gemini, get JSON and send it to an API to print arrest warrants.
- Drblessing 3y agoWhy would I use an anti-white AI tool?
- Psychoshy_bc1q 3y agoWoke trash.
- rapjr9 3y agoOne thing that has held back a lot of computer automation is context. For example, an app can know your geographic location, it can know your activity level, it can know what messages you send and receive, but it doesn't know you, it doesn't know how you think, what influences you, what perspective you have, what is happening in the world around you. An LLM that can look at images of the world around you constantly and continuously extracted details could provide an enormous amount of context that could feed automation. So your personal assistant knows why now is the right time to offer particular suggestions and knows why you would be interested (I'm anthropomorphizing as a shortcut). Huge privacy issues here also. As an example, using AI to detect cognitive decline. A senior person is losing their balance more often recently as detected by an accelerometer. Are they experiencing a sudden cognitive decline? Part of the context might be that they had a visit from grandchildren recently and the children spent the day playing and left stuff scattered all over the house. Hence there is more stuff to trip over. Without the ability to extract that context the accelometer readings are difficult to interpret.