11 ms·
Show HN: I Remade the Fake Google Gemini Demo, Except Using GPT-4 and It's Real
- anon115 3y ago[flagged]
- zainhoda 3y agoWow, this is super cool! From the code it seems like the speech to text and text to speech are using the browser’s built-in features. I always forget those capabilities even exist!
- op00to 3y agoVery cool!
- swyx 3y agohaha yes it was entirely possible with gpt4v. literally just screenshot and feed in the images and text in chat format, aka “interleaved”. made something similar at a hackathon recently. (https://x.com/swyx/status/1722662234680340823 https://x.com/swyx/status/1722662234680340823). the bizarre thing is that google couldve done what you did, and we wouldve all been appropriately impressed, but instead google chose to make a misleading marketing video for the general public and leave the rest of us frustrated nerds to do the nasty work of having to explain why the technology isnt as seen on tv yet; making it seem somehow our fault i am curious about the running costs of something like this
- gregsadetsky 3y agoI made 77 requests to the GPT-vision API while developing/demo'ing this, and that resulted in a $0.47 bill. Pretty reasonable!
- jankovicsandras 3y agoHi Greg, Congratulations, great demo! The $0.47 bill seems reasonable for an experiment, but imagine someone doing a task of this complexity as a daily job - let's say 100x times, or a little more than 4 hours - the bill would be $47/day. It feels like there's still an opportunity for a cheaper solution. Have you or someone else experimented with e.g. https://localai.io/ https://localai.io/ ?
- swyx 3y agoif i did not have your comment history i'd have sworn you worked for localai.io
- iandanforth 3y agoLooks like, again, this doesn't have GPT-4 processing video as much as a stack of video frames, concatenated and sent as a single image. But much closer to real!
- ShamelessC 3y agoThe video is an actual live demo without any editing or other tricks involved and even includes reasonable mistakes and the code used. It is not close to real, it's just real.
- trescenzi 3y agoHow does a video differ from a stack of video frames? Isn’t that all a video is? A bunch of images stuck together and played back quickly?
- famouswaffles 3y agoI'd guess you'd miss any audio that way. But otherwise, yeah a video is a stack of images.
- zwily 3y agoYou could say that this demo is processing a 2.4s video that is 1.25fps.
- civilitty 3y agoIn a technical sense most video is compressed using motion prediction algorithms so the preprocessing on the data is already significantly different to static images, containing more compressed information. Only the key frames are actually full images and they only make up 1-5% of the frames. On top of that the video container usually provides synchronized audio packets.
- fortunefox 3y agoSince audio is processed separately this isn't just close to real. it is real. After all what is video if not a stack of frames! :D
- 3y ago
- 0raymond0 3y ago[flagged]
- mrdoop 3y ago[dead]
- ShamelessC 3y agoSad state of affairs for Google.
- paul7986 3y agoIndeed they've been sad starting in 2010 and on... (maybe before)... all the projects they kill.. their IP theft, them doing evil, etc
- bergen 3y agoANY indication the video was fake? I see none. The example at hand is nice and all, but if you say someone is faking you better have receipts.
- ShamelessC 3y agoI’m not making the claim, OP is. It’s not hard to arrive at this conclusion yourself though. Here is Yannick’s take on it: https://youtu.be/zut38E-BHH0?si=oj3S3qWLgvx3743I https://youtu.be/zut38E-BHH0?si=oj3S3qWLgvx3743I If you need more than that, then I’ll admit to being more cynical about big tech companies than you are.
- ric2b 3y agoGoogle themselves admitted that it was just a "visualization" of the actual prompts and responses, and some of the prompts were very different and more detailed than the voiced "visualizations".
- sheepscreek 3y agoThank you for creating this demo. This was the point I was trying to make when the Gemini launch happened. All that hoopla for no reason. Yes - GPT-4V is a beast. I’d even encourage anyone who cares about vision or multi-modality to give LLaVA a serious shot (https://github.com/haotian-liu/LLaVA https://github.com/haotian-liu/LLaVA). I have been playing with the 7B q5_k variant last couple of days and I am seriously impressed with it. Impressed enough to build a demo app/proof-of-concept for my employer (will have to check the license first or I might only use it for the internal demo to drive a point).
- ok_dad 3y agoI’ve been using llava via https://github.com/Mozilla-Ocho/llamafile https://github.com/Mozilla-Ocho/llamafile which runs on any modern system.
- jart 3y agoIt's so great. I've been this vision model to rename all the files in my Pictures folder. For example, the one-liner: llamafile --temp 0 \ --image ~/Pictures/lemurs.jpg \ -m llava-v1.5-7b-Q4_K.gguf \ --mmproj llava-v1.5-7b-mmproj-Q4_0.gguf \ --grammar 'root ::= [a-z]+ (" " [a-z]+)+' \ -p $'### User: What do you see?\n### Assistant: ' \ --silent-prompt 2>/dev/null | sed -e's/ /_/' -e's/$/.jpg/' Prints to standard output: a_baby_monkey_on_the_back_of_a_mother.jpg This is something that's coming up in the next llamafile release. You have to build from source to have the ability to use grammar and --silent-prompt on a vision model right now. Weights here: https://huggingface.co/jartine/llava-v1.5-7B-GGUF/tree/main https://huggingface.co/jartine/llava-v1.5-7B-GGUF/tree/main Sauce here: https://github.com/mozilla-Ocho/llamafile https://github.com/mozilla-Ocho/llamafile
- sheepscreek 3y agoTruly grateful for your work on cosmopolitan, cosmo libc, redbean, nudging POSIX towards realizing the unachieved dream and also for contributing to llama.cpp. It’s like wherever I look, you’ve already left your mark there! To me, you exemplify and embody the spirit of OSS, and to top that - you seem to be just an amazing human. You are an inspiration for me and many others. And even though I know I’ll never ever get close, you make me want to try. Thank you. :)
- dvaun 3y agoGreat demo, I laughed at the final GPT response too. Honestly: it would be fun to self-host some code hooked up to a mic and speakers to let kids, or whoever, play around with GPT4. I’m thinking of doing this on my own under an agency[0] I’m starting up on the side. Seems like a no-brainer as an application. [0]: https://www.divinatetech.com https://www.divinatetech.com
- jakderrida 3y agoLmao! So, presumably, they could have hired Greg to improvise almost the exact same demonstration, but with evidence it works. I don't know how much Greg costs, but I'll bet my ass it's less than the cost in investor sentiment after getting caught committing fraud. Not saying you're cheap. Just cheaper.
- phire 3y agoThe "magic" of the fake Gemini demo was the way it seemed like the LLM was continually receiving audio + video input and knew when to jump in with a response. It appeared to be able to wait until the user had finished the drawing, or even jumping in slightly before the drawing finished. At one point the LLM was halfway through a response and then saw the user was now colouring the duck in blue, and started talking about how the duck appearing to be blue. The LLM also appeared to know when a response wasn't needed because the user was just agreeing with the LLM. I'm not sure how many people noticed that on a conscious level, but I positive everyone noticed it subconsciously, and felt the interaction was much more natural, and much more advanced than current LLMs. ----------------- Checking the source code, the demo takes screenshots of the video feed every 800ms, waits until the user finishes taking and then sends the last three screenshots. While this demo is impressive, it kind of proves just how unnatural it feels to interact with an LLM in this manner when it doesn't have continuous audio-video input. It's been technically possible to do kind of thing for a while, but there is a good reason why nobody tried to present it as a product.
- famouswaffles 3y agoI think probably training on pause tokens or something similar would be the key to something like this. Maybe it's not even necessary. Maybe if you just tell GPT-4 to output something like .... every time it thinks it should wait for a response (you wouldn't need to wait for the user to finish then), things would be a lot smoother.
- phire 3y agoYes, you could probably fine-tune (or even zero-shot) a LLM to handle the "knowing when to jump in" use case. The real problem is that it's simply too computationally expensive to continually feed audio and video into it one of these massive LLMs just in case it might decide to jump in. I was wondering if you could train a lightweight monitoring model that continually watching the audio/video input and only tried to work out when the full-sized LLM might want to jump in and generate a response.
- 3y ago
- godelski 3y agoI don't get why companies lie like this. How much do they have to gain? It seems like they actually have a lot to lose. What's crazy to me is that these tools are wildly impressive without the hype. As a ML researcher, there's a lot of cool things we've done but at the same time almost everything I see is vastly over hyped from papers to products. I think there's a kinda race to the bottom we've created and it's not helpful to any of us except maybe in the short term. Playing short term games isn't very smart, especially for companies like Google. Or maybe I completely misunderstand the environment we live in. But then again, with the discussions in this thread[0] maybe there's a lot of people so ethically bankrupt that they don't even know how what they're doing is deceptive. Which is an entirely different and worse problem. [0] https://news.ycombinator.com/item?id=38559582 https://news.ycombinator.com/item?id=38559582
- nextworddev 3y agoGoogle stock rallied 5% ish the after the demo (though the stock didn’t move immediately). Then it gave back about 1% once the news broke that it was faked
- godelski 3y agoThat's not a great answer. We need to know the counterfactual question of "How much would google's stock have rallied after a realistic demo was given?" I would not have been surprised if the answer was also 5%. Almost certainly Google's stock would have risen after announcing Gemini. There are other long term questions too like about how this feeds into growing dissent against Google and trust. But what the stock did is only a small part of a much bigger and far more important question. Edit: Can someone explain the downvotes? Is there a error in my response? I'm glad to learn but I'd appreciate a bit better feedback signal so that I can do so better instead of guessing.
- nextworddev 3y agoBroad investor community was spooked by Gemini Ga being delayed to Q1 so this stunt was a good stop gap / distraction
- adtac 3y ago[tangential to this really cool demo] JPEG images being the only possible interface to GPT-4 feels wasteful. the human eye works the delta between "frames", not the image itself. I wonder if the next big step that would allow real-time video processing at high resolutions is to have the model's internal state operate on keyframes and deltas similar to how video codecs like MPEG work.
- zwily 3y agoWhen Google talks about Gemini's "multi-modal", they include "video" in the list of modes. It's totally possible they don't actually mean video, and just mean frames like in this demo. They haven't elaborated on it anywhere that I've seen.
- dannyw 3y agoTheir technical report clarifies that video is just a sequence of frames fed as images.
- sibeliuss 3y agoLol at choosing the name Sagittarius, which is exactly across from Gemini in the Zodiac
- SilasX 3y agoI remember there was speculation that Facebook named their vaporware cryptocurrency Libra (later, “Diem”) as a jab at the longtime rival Winklevoss twins, who had started a crypto exchange called Gemini. I have no idea how astrologically clever that would be.
- nostrademons 3y agoLibra is one of two other air signs - the other is Aquarius. They're 4 months offset from Gemini.
- iamleppert 3y agoI’ve recently been trying to actually use Google’s AI conversational translation app that was released awhile back and has many updates and iterations since. It’s completely unusable for real conversation. I’m actually in a situation where I could benefit from it and was excited to use it because I remember watching the demo and how natural it looked but was never able to actually try it myself. Now having used it, I went back and watched their original demo and I’m 100% convinced all or part of it was faked. There is just no way this thing ever worked. If they can’t manage to make conversational live translation work (which is a lot more useful than drawing a picture of a duck) I have high doubts about this new AI. Seems like the exact same situation to me. It’s insane to me how much nerve it must take to completely fake something like this.
- cubefox 3y agoWhat was the App called?
- razodactyl 3y agoThe latency is excusable as this is through the API. Inference on local infrastructure is almost instant so this demo would smoke everything else if this dude had access.
- razodactyl 3y agoThe part that really confuses me is the lack of a "*some sequences simulated" disclaimer.
- cubefox 3y agoThe Gemini demo had a disclaimer in the beginning, albeit not a very clear one.
- frays 3y agoThanks for sharing!
- cylinder714 3y agoSnader's Law: "Any sufficiently advanced technology is indistinguishable from a rigged demo."
- dingclancy 3y agoI am now convinced that Google Deepmind really had nothing in terms of SOTA LLMs. They were just bluffing. I remember when chatgpt was released Google was saying that they had much much better models that they are not releasing because they for AI Safety. Then theu released palm and palm 2 saying that it is time to release these models to beat ChatGPT. It was not a good model. The they hyped up Gemini, and if Gemini Ultra is the best they have, I am not convinced that they have a better model. So this is it. So in one year, we went from Google has to have the best model, they just do not want to release to they have the infrastructure and data and the talent to make the best model. Why they really had was nothing.
- dingclancy 3y agoI am now convinced that Google DeepMind really had nothing in terms of state-of-the-art language models (SOTA LLMs). They were just bluffing. I remember when ChatGPT was released; Google was saying that they had much better models they were not releasing due to AI safety. Then they released Palm and Palm 2, saying it's time to beat ChatGPT with these models. However, it was not a good model. They then hyped up Gemini, and if Gemini Ultra is the best they have, I am not convinced that they have a better model. Sundar's code red was genuinely alarming because they had to dig deep to make this Gemini model work, and they still ended up with a fake video. Even if Gemini was legitimate, it did not beat GPT-4 by leaps and bounds, and now GPT-5 is on the horizon, putting them a year behind. It makes me question if they had a secret powerful model all along
- n8fr8too 3y agoI had been working on an idea for an interface "Sorting Hat" system to help kids at schools know whether something was for trash, compost, or recycling. While I had been hacking on it for a bit, Greg's "demo" was much better integrated than what I could do, so thanks Greg! I did add ElevenLabs support to make it a little more snazzy sounding... So, here it is the "Compose/Trash/Recycle Sorting Hat, Built on Sagittarious" https://github.com/n8fr8/CompostSortingHatAI https://github.com/n8fr8/CompostSortingHatAI You can see a realtime, unedited YouTube demo video of my kid testing it out here: https://www.youtube.com/watch?v=-9Ya5rLj64Q https://www.youtube.com/watch?v=-9Ya5rLj64Q