25 ms·
Gemini "duck" demo was not done in realtime or with voice
- FartyMcFarter 3y agoUnpaywalled Bloomberg article linked in the tweet: https://archive.is/4H1fB https://archive.is/4H1fB
- retox 3y agoAI: artificial incompetence
- throwitaway222 3y agolink to duck video: https://www.youtube.com/watch?v=UIZAiXYceBI https://www.youtube.com/watch?v=UIZAiXYceBI
- Inward 3y agoYes, that was obvious as soon as I saw it wasn’t live I clicked off. You can train any LLM to perform a certain task(s) well and google engineers are not that dense. This was obvious marketing PR as open AI has completely made google basically obsolete with 90% of my queries can be answered without wading through LLM generated text for a simple answer.
- nirvael 3y ago>without wading through LLM generated text ...OpenAI solved this by generating LLM text for you to wade through?
- rose_ann_ 3y agoNo. It solved it by (most of the time) giving the OP and I the answer to our queries, without us needing to wade through spammy SERP links.
- kweingar 3y agoIf LLMs can replace 90% of your queries, then you have very different search patterns from me. When I search on Kagi, much of the time I’m looking for the website of a business, a public figure’s social media page, a restaurant’s hours of operation, a software library’s official documentation, etc. LLMs have been very useful, but regular search is still a big part of everyday life for me.
- balder1991 3y agoSure we now have options, but before LLMs, most queries relied solely on search engines, often leading to sifting through multiple paragraphs on websites to find answers — a barrier for most users. Today, LLMs excel in providing concise responses, addressing simple, curious questions like, "Do all bees live in colonies?"
- GolfPopper 3y agoHow do you tell a plausible wrong answer from a real one?
- rose_ann_ 3y agoBy testing the code it returns (I mostly use it as a coding assistant) to see if it works. 95% of the time it does. For technical questions, ChatGPT has almost completely replaced Google & Stack Overflow for me.
- kweingar 3y agoThe video itself and the video description give a disclaimer to this effect. Agreed that some will walk away with an incorrect view of how Gemini functions, though. Hopefully realtime interaction will be part of an app soon. Doesn’t seem like there would be too many technical hurdles there.
- billconan 3y agoperformance and cost are hurdles?
- kweingar 3y agoIt can be realtime while still having more latency than depicted in the video (and the video clearly stated that Gemini does not respond that quickly). A local model could send relevant still images from the camera feed to Gemini, along with the text transcript of the user’s speech. Then Gemini’s output could be read aloud with text-to-speech. Seems doable within the present cost and performance constraints.
- anigbrowl 3y agoPeople don't really pay attention to disclaimers. Google made a choice knowing people would remember the hype, not the disclaimer.
- lainga 3y ago:%s/Google/the team :%s/people/the promotion board Conway's law applied to the corporate-public interface :)
- 3pt14159 3y agoI remember watching it and I was pretty impressed, but as I was walking around thinking to myself I came to the conclusion that there was something fishy about the demo. I didn't know exactly what they fudged, but it was far too polished to explain how well their current AI demos preform. I'm not saying there have been no improvements in AI. There is and this includes Google. But the reason why ChatGPT has really taken over the world is that the demo is in your own hands and it does quite well there.
- kaoD 3y agoHow is this not false advertising?
- barbazoo 3y agoOr worse, fraud to make their stock go up edit: s/stuck/stock
- deleted 3y ago[deleted]
- drcode 3y agoI suppose it's not false advertising, since they don't even claim to have a product released yet that can do this, since Trojans Ultra won't be available until an unspecified time next year
- stephbu 3y agoYou're right, it's astroturfing a placeholder in the market in the absence of product. The difference is probably just the target audience - feels like this one is more aimed at share-holders and internal politics.
- empath-nirvana 3y agopossibly securities fraud though. Their stock popped a few percent on the back of that faked demo.
- imiric 3y agoIt's still false advertising. This is common in all industries. Take gaming, for example. Game publishers love this kind of publicity, as it creates hype, which leads to sales. There have been numerous examples of this over the years: Watch Dogs, No Man's Sky, Cyberpunk 2077, etc. There's a period of controversy once consumers realize they've been duped, the company releases some fake apology and promises or doubles down, but they still walk out of it richer, and ready to do it again next time. It's absolutely insidious, and should be heavily fined and regulated.
- drcongo 3y agoRemember when they faked that Google Assistant booking a restaurant thing too.
- miraculixx 3y agoMhm
- umeshunni 3y agoHow was that fake?
- dannyw 3y agoIt was actually a human doing the entire call.
- valine 3y agoIt’s not live, but it’s in the realm of outputs I would expect from a GPT trained on video embeddings. Implying they’ve solved single token latency, however, is very distasteful.
- zozbot234 3y agoOP says that Gemini had still images as input, not video - and the dev blog post shows it was instructed to reply to each input in relevant terms. Needless to say, that's quite different from what's implied in the demo, and at least theoretically is already within GPT's abilities.
- valine 3y agoHow do you think the cup demo works? Lots of still images?
- watusername 3y agoA few hand-picked images (search for "cup shuffling"): https://developers.googleblog.com/2023/12/how-its-made-gemini-multimodal-prompting.html https://developers.googleblog.com/2023/12/how-its-made-gemin...
- valine 3y agoHoly crap that demo is misleading. Thanks for the link.
- Animats 3y agoThe Twitter-linked Bloomberg page is now down.[1] Alternative page: [2] New page says it was partly faked. Can't find old page in archives. [1] https://www.bloomberg.com/opinion/articles/2023-12-07/google-s-gemini-ai-model-looks-remarkable-but-it-s-still-behind-openai-s-gpt-4io https://www.bloomberg.com/opinion/articles/2023-12-07/google... [2] https://www.bloomberg.com/opinion/articles/2023-12-07/google-s-gemini-ai-model-looks-remarkable-but-it-s-still-behind-openai-s-gpt-4 https://www.bloomberg.com/opinion/articles/2023-12-07/google...
- sowbug 3y agoI am similarly enraged when TV show characters respond to text messages faster than humans can type. It destroys the realism of my favorite rom-coms.
- deleted 3y ago[deleted]
- atdaemon 3y agoThe report from TechCrunch has more details - https://techcrunch.com/2023/12/07/googles-best-gemini-demo-was-faked/ https://techcrunch.com/2023/12/07/googles-best-gemini-demo-w...
- imacomputertoo 3y agoit was obviously marketing material, but if this tweet is right, then it was just blatant false advertising.
- kjkjadksj 3y agoGoogle always does fake advertising. “Unlimited” google drive accounts for example. They just have such a beastly legal team no one is going to challenge them on anything like that.
- Dylan16807 3y agoWhat was fake about unlimited google drive? There were some people using petabytes. The eventual removal of that tier and anything even close speaks to Google's general issues with cancelling services, but that doesn't mean it was less real while it existed.
- ambrose2 3y agoWhat about when gmail was released and the storage was advertised as increasing forever, but at first they just increased it infinitesimally slower and then stopped increasing it all.
- Dylan16807 3y agoOh, long before google drive existed? I don't remember the "increasing forever" ever being particularly fast. I found some results from 2007 and 2012 both saying it was 4 bytes per second, <130MB per year. So it's true that the number hasn't increased in ten years, but that last increase was +5GB all by itself. They've done a reasonable job of keeping up. Arguably they should have kept adding a gigabyte each year, based on the intermittent boosts they were giving, but by that metric they're only about 5GB behind.
- peteradio 3y agoLol, could have done without the cocky narration. "I think we're done here."
- cedws 3y agoThe whole launch is cocky. Bleh. Stick to the engineering.
- cryptoz 3y agoI'll admit I was fooled. I didn't read the description of the video. The most impressive thing they showed was the real-time responses to watching a video. Everything else was about expected. Very misleading and sad Google would so obviously fake a demo like this. Mentioning in the description that it's edited is not really in the realm of doing enough to make clear the fakery.
- LesZedCB 3y agoi too was excited and duped about the real-time implications. though i'm not surprised at all to find out it's false. mea cupla i should have looked at the bottom of the description box on youtube where it probably says "this demonstration is based on an actual interaction with an LLM"
- wvenable 3y agoI'm surprised it was false. It was made to look realistic and I wouldn't expect Google to fake this kind of thing. All they've done is completely destroy my trust in anything they present.
- zozbot234 3y agoGood, that video was mostly annoying and creepy. The AI responses as shown in the linked Google dev blogpost are a lot more reasonable and helpful. BTW I agree that the way the original video was made seems quite misleading in retrospect. But that's also par for the course for AI "demos", it's an enduring tradition in that field and part of its history. You really have to look at production systems and ignore "demos" and pointless proofs of concept.
- peteradio 3y agoWhat the Quack? I found it tasty as pâté.
- danielbln 3y agoThe GPT-4 demo early this year when it was released was a lot less.. fake, and in fact very much indicative of it's feature set. The same is true for what OpenAI showed during their dev days, so at the very least those demos don't have too much fakery going on, as far as I could tell.
- ukuina 3y agoA certain minimum level of jank always makes demos more believable. Watching Brockman wade through Discord during the napkin-to-website demo immediately made the whole thing convincing. AI is in the "hold it together with hope and duct tape" phase, and marketing videos claiming otherwise are easy to spot and debunk.
- Frost1x 3y ago>You really have to look at production systems and ignore "demos" and pointless proofs of concept. While I agree, I wouldn't call proofs or concepts and demos pointless. They often illustrate a goal or target functionality you're working towards. In some cases it's really just a matter of allotting some time and resources to go from a concept to a product, no real engineering is needed, it all exists, but there's capital needed to get there. Meanwhile some proof of concepts skip steps and show higher level function that needs some serious breakthrough work to get to, maybe multiple steps of that. Even this is useful because it illustrates a vision that may be possible so people can understand and internalize things you're trying to do or the real potential impact of something. That wasn't done here, it was embedded in a side note. That information needs to be before the demo to some degree without throwing a wet blanket on everything and needs to be in the same medium as the demo itself so it's very clear what you're seeing. I have no problem with any of that. I have a lot of problems when people don't make it explicitly clear beforehand that it's a demo and explain earnestly what's needed. Is it really something that exists today in working systems someone just needs to invest money and wire it up without new research needed? Or is it missing some breakthroughs, how many/what are they, how long have these things been pursued, how many people are working on them... what does recent progress look like and so on (in a nice summarized fashion). Any demo/poc should come up front with an earnest general feasibility assessment. When a breakthrough or two are needed then that should skyrocket. If it's just a lot of expensive engineering then that's also a challenge but tractable. I've given a lot of scientific tech demonstrations over the years and the businesses behind me obviously want me to be as vague as possible to pull money in. I of course have some of those same incentives (I need to eat and pay my mortgage like everyone else). None-the-less the draw of science to me has always been pulling the veil from deception and mystery and I'm a firm believer in being as upfront as possible. If you don't lead with disclaimers, imaginations run wild into what can be done today. Adding disclaimers helps imaginations run wild about what can be done tomorrow, which I think is great.
- borissk 3y agoGoogle did the same with Pixel 8 Pro advertising - they showed stuff like photo and video editing, that people couldn't replicate on their phones.
- pizzafeelsright 3y agoI suppose this is a great example of how trust in authentic videos, audio, images, company marketing must be questioned and, until verified, assumed to be 'generated'. I am curious, if the voice, email, chat, and shortly video can all be entirely generated in real or near real time, how can we be sure that remote employee is actually not a full or partially generated entity? Shared secrets are great when verifying but when the bodies are fully remote - what is the solution? I am traveling at the moment. How can my family validate that it is ME claiming lost luggage and requesting a Venmo request?
- takoid 3y ago>I am traveling at the moment. How can my family validate that it is ME claiming lost luggage and requesting a Venmo request? PGP
- tadfisher 3y agoNow you have two problems. (I say this in jest, as a PGP user)
- adewinter 3y agoMake up a code phrase/word for emergencies, share it with your family, then use it for these types of situations.
- mikepurvis 3y agoFair, but that also assumes the recipients ("family") are in a mindset of constantly thinking about the threat model in this type of situation and will actually insist on hearing the passphrase.
- pizzafeelsright 3y agoThis will only work once.
- vasco 3y ago
- GaggiX 3y agoI imagine the model also has some video embeddings like for the example when it needed to find where the ball was hiding.
- AndrewKemendo 3y agoI have used Swype texting since the t9 days. If I demoed swype texting as it functions in my day to day life to someone used to a querty keyboard they would never adopt it The rate at which it makes wrong assumptions about the word, or I have to fix it is probably 10% to 20% of the time However because it’s so easy to fix this is not an issue and it doesn’t slow me down at all. So within the context of the different types of text Systems out there, I t’s the best thing going for me personally, but it takes some time to learn how to use it. This is every product. If you demonstrated to people how something will actually work after 100 hours of habituation and compensation for edge cases, nobody would ever adopt anything. I’m not sure how to solve this because both are bad. (Edit: I’m keeping all my typos as meta-comment on this given that I’m posting via swype on my phone :))
- mulmen 3y agoDoes swype make editing easier somehow? iOS spellcheck has negative value. I turned it off years ago and it reduced errors but there are still typos to fix. Unfortunately iOS text editing is also completely worthless. It forces strange selections and inserts edited text in awkward ways. I’m a QWERTY texter but text entry on iOS is a complete disaster that has only gotten worse over time.
- mikepurvis 3y agoI'm an iOS user and prefer the swipe input implementation in GBoard over the one in the native keyboard. I'm not sure what the differences are, but GBoard just seems to overall make fewer mistakes and do a better job correcting itself from context.
- wlesieutre 3y agoHave you tried the native keyboard since iOS 17? It’s quite a lot better than older versions.
- nozzlegear 3y agoAs I was reading Andrew's comment to myself, I was trying to figure out when and why I stopped using swype typing on my phone. Then it hit me – I stopped after I switched from Android to iOS a few years ago. Something about the iOS implementation just doesn't feel right.
- CamperBob2 3y agoAny sufficiently-advanced technology is indistinguishable from a rigged demo.
- peteradio 3y agoFake it til you make it, then keep faking it.
- nlpfromscratch 3y ago[flagged]
- rollulus 3y agoI watched this video, impressed, and thought: what if it’s fake. But then dismissed the thought because it would come out and the damage wouldn’t be worth it. I was wrong.
- imiric 3y agoThe worst part is that there won't be any damage. They'll release a blog post with PR apologies, but the publicity they got from this stunt will push up their brand in mainstream AI conversations regardless. "There's no such thing as bad publicity."
- steego 3y agoThere’s no such thing as bad publicity only applies to people and companies that know how to spin it. Reading the comments of all these disillusioned developers, it’s already damaged them because now smart people will be extra dubious when Google starts making claims. They just made it harder for themselves to convince developers to even try their APIs, let alone bet on them. This was stupid.
- h0rv 3y ago[dead]
- Nekorosu 3y agoGemini demo looks like ChatGPT with a video feed, except it doesn't exist, like ChatGPT. I have ChatGPT on my phone right now, and it works (and it can process images, audio, and audio feed in). This means Google has shown nothing of substance. In my world, it's a classic stock price manipulation move.
- onlyrealcuzzo 3y agoGemini Pro is available on Bard now. Ultra is not yet available.
- replwoacause 3y agoYeah and have you tried it? It’s as dogshit as the original Bard.
- CrayKhoi 3y agoI've been using Gemini in Bard since the launch, with respect to coding it is outperforming GPT4 in my opinion. There is some convergence in the answers,but Bard is outputting really good code now.
- replwoacause 3y agoYou're right. I just tested it with some code prompts and it did surprisingly well.
- Nekorosu 3y agoThat's something! I'll try it too. Thanks <3
- Kim_Bruning 3y agoEven a year ago, this advert would have been obvious puffery in advertising. But right now, all the bits needed to do this already exist (just need to be assembled and -to be fair- given a LOT of polish), so it would be somewhat reasonable to think that someone had actually Put In The Work already.
- xnx 3y agoThat demo was much further on the "marketing" end of the spectrum when compared to some of their other videos from yesterday which even included debug views: https://youtu.be/v5tRc_5-8G4?t=43 https://youtu.be/v5tRc_5-8G4?t=43
- karaterobot 3y agoThis is endemic to public product demos. The thing never works as it does in the video. I'm not excusing it, I'm saying: don't trust public product demos. They are commercials, they exist to sell to you, not to document objectively and accurately, and they will always lie and mislead within the limits of the law.
- k__ 3y agoI really thought this was a realtime demo. Shame on them :(
- suriyaG 3y agoThe bloomberg article seems to have been taken down and is now going to 404. https://www.bloomberg.com/opinion/articles/2023-12-07/google-s-gemini-ai-model-looks-remarkable-but-it-s-still-behind-openai-s-gpt-4io https://www.bloomberg.com/opinion/articles/2023-12-07/google...
- dilap 3y agoJust an error in the link, here's the corrected version: https://www.bloomberg.com/opinion/articles/2023-12-07/google-s-gemini-ai-model-looks-remarkable-but-it-s-still-behind-openai-s-gpt-4 https://www.bloomberg.com/opinion/articles/2023-12-07/google...
- thrtythreeforty 3y agoand here's a readable version: https://archive.ph/ABhZi https://archive.ph/ABhZi
- dougmwne 3y agoI was fooled. The model release announcement said it could accept video and audio multi-modal input. I understood that there was a lot of editing and cutting, but I really believed I was looking at an example of video and audio input. I was completely impressed since it’s quite a leap to go from text and still images to “eyes and ears.” There’s even the segment where instruments are drown and music was generated. I thought I was looking at a model that could generate music based on language prompts, as we have seen specialized models do. This was all fake. You are taking a collection of cherry picked prompt engineered examples, then dramatizing them for maximum shareholder hype. The music example was just outputting a description of a song, not the generated music we heard in the video. It’s one thing to release a hype video with what-ifs and quite another to claim that your new multi-modal model is king of the hill then game all the benchmarks and fake all the demos. Google seems to be in an evil phase. OpenAI and MS must be quite pleased with themselves.
- skepticATX 3y agoExactly. Personally I’m fine with both: 1) Forward looking demoes that demonstrate the future of your product, where it’s clear that you’re not there yet but working in that direction or 2) Demoes that show off current capabilities, but are scripted and edited to do so in the best light possible. Both of those are standard practice and acceptable. What Google did was just wrong. They deserve to face backlash for this.
- miraculixx 3y agoDo you believe everything verbatim that companies tell you in advertising?
- iamleppert 3y agoYou can tell whoever put together that demo video gave no f*cks whatsoever. This is the quality of work you can expect under an uninspiring leader (Sundar) in a culture of constant layoff fear and bureaucracy. Literally everyone I know who works at Google hates their job and are completely checked out.
- CamperBob2 3y agoHuh? It was a GREAT demo video! If it had been real, that is.
- htk 3y agoThe whole Gemini webpage and contents felt weird to me, it's in the uncanny valley of trying to look and feel like an Apple marketing piece. The hyperbolic language, surgically precise ethnic/gender diversity, unnecessary animations and the sales pitch from the CEO felt like a small player in the field trying to pass as a big one.
- kjkjadksj 3y agoI’m imagining the project managers are patting themselves on the back for checking all the performative boxes, blind to the absolute satire of it all.
- cedws 3y agoI got the same vibes. Ultra and Pro. It feels tacky that it declares the "Gemini era" before it's even available. Google really want to be seen as level on the playing field.
- crazygringo 3y ago> surgically precise ethnic/gender diversity What does that mean and why is it bad? Diversity in marketing is used because, well, your desired market is diverse. I don't know what it means for it to be surgically precise, though.
- cheeze 3y agoAgreed with your comment. This is every marketing department on the planet right now, and it's not a bad thing IMO. Can feel a bit forced at times, but it's better than the alternative.
- ilkke 3y agoThe alternative being showing actual level of diversity in the company?
- rtsil 3y agoIt's bad if the makeup of the company doesn't reflect the diversity seen in the marketing, because it doesn't reflect any genuine value and is just for show. Now, I don't know how diverse the AI workforce is at Google, but the YT thumbnails show precisely 50% of white men. Maybe that's what the parent meant by "surgically precise".
- milofeynman 3y agoI looked at is as if it were a good aspirational target for 5 years from now. It was obvious the whole video was edited together not real time.
- sheepscreek 3y agoI don’t understand why is Gemini even considered “jaw-dropping” to begin with. GPT-4V has set the bar so high that all their demos and presentations paled in comparison. And it’s available for anyone to use. People have already build mind-blowing demos with it (like https://arstechnica.com/information-technology/2023/11/ai-powered-drawing-app-stuns-developers-by-turning-sketches-into-functional-games/ https://arstechnica.com/information-technology/2023/11/ai-po...). The entire launch felt like a concentrated effort to “appear” competitive to OpenAI. Google was splitting hairs talking about low single digit percentage improvement in benchmarks. Against a model that has been out for over 6 months. I have never been so unimpressed with them. Not only has OpenAI managed to snag this one from under Google’s nose, IMO - they seem to be defending their lead quite well. Now that is something unmistakably remarkable. Color me impressed!
- kjkjadksj 3y agoSome other commenter, a former googler, a while back alluded to figuring out the big secret and being thrown for a tizzy at the resulting cognitive dissonance they realize they’ve been buying into. Its never about making a good product. Its about keeping up with the joneses in the eyes of tech investors. And just look at the movement on the stock today as a result of this probable lemon of a product: nothing else mattered except keeping up appearances. CEOs make historic careers optimizing companies for appearances over function like this.
- sheepscreek 3y agoI get that. But I don’t get why journalists who cover the industry and are expected to get it would call this “jaw dropping”. Maybe I am reading too much into it. It was likely added to increase the shock factor.
- deleted 3y ago[deleted]
- modeless 3y agoThat's not the only thing wrong. Gemini makes a false statement in the video, serving as a great demonstration of how these models still outright lie so frequently, so casually, and so convincingly that you won't notice, even if you have a whole team of researchers and video editors reviewing the output. It's the single biggest problem with LLMs and Gemini isn't solving it. You simply can't rely on them when correctness is important. Even when the model has the knowledge it would need to answer correctly, as in this case, it will still lie. The false statement is after it says the duck floats, it continues "It is made of a material that is less dense than water." This is false; "rubber" ducks are made of vinyl polymers which are more dense than water. It floats because the hollow shape contains air, of course.
- ace2358 3y agoI totally agree with you on the confident lies. And it’s really tough. Technically the duck is made out of air and plastic right? If I pushed the model further on the composition of a rubber duck, and it failed to mention its construction, then it’d be lying. However there is this disgusting part of language where a statement can be misleading, technically true, not the whole truth, missing caveats etc. Very challenging problem. Obviously Google decided to mislead the audience and basically cover up the shortcomings. Terrible behaviour.
- mechagodzilla 3y agoNo, the density of the object is less than water, not the density of the material. The Duck is made of plastic, and it traps air. Similarly, you can make a boat that floats in water out of concrete or metal. It is an important distinction when trying to understand buoyancy.
- deleted 3y ago[deleted]
- modeless 3y agoCalling the air inside the duck (which is not sealed inside) part of its "material" would be misleading. That's not how most people would interpret the statement and I'm confident that's not the explanation for why the statement was made.
- neilv 3y agoI missed the disclaimer. So, when watching it, I started to think "Wow, so Google is releasing their best stuff". But then I soon noticed some things that were too smooth, so seemed at best to be cherry-picked interactions occasionally leaning on hand-crafted situation handlers. Or, it turns out, faked. Regardless of disclaimers, this video seems misleading to be releasing right now, in the context of OpenAI eating Google's lunch. Everyone is expecting Google to try to show they can do better. This isn't that. This isn't even an mocked-up interaction future of HCI concept video, because it's not showing a vision of what people want to do --- it's only showing a demo of technical capabilities. It's saying "This is what a contrived tech demo (not application vision concept) could look like, but we can't do it yet, so we faked it. Hopefully, the viewer will get the message that we're competitive with OpenAI." (This fake demo could just be an isolated oops of a small group, not representative of Google's ability to rise to the current disruption challenge, I don't know.)
- miraculixx 3y agoI knew immediately this was just overhyped PR when I noticed the author of the blogpost is Sundar.
- Alifatisk 3y agoThe bloomberg article gives 404 for me
- dramm 3y agoThe more Google tries to over-hype stuff the more that keeps giving me a greater impression they are well behind OpenAI. Time to STFU and focus on working on stuff.
- SheinhardtWigCo 3y agoJust how many lives does Sundar have? Where is the board?
- miraculixx 3y agoCounting their bonusses?
- miraculixx 3y ago[flagged]
- 1024core 3y agoFor more details about how the video was created, see this blog post: https://developers.googleblog.com/2023/12/how-its-made-gemini-multimodal-prompting.html https://developers.googleblog.com/2023/12/how-its-made-gemin...
- onemoresoop 3y agoIt seems like the fake video did the trick, their stock is up 5.5% today.
- eh_why_not 3y agoThere was also the cringey "niiice!", "sweeeet!", "that's greaatt", "that's actually pretty good" responses from the narrator in a few of the demo videos that gave them the feel of a cheap 1980's TV ad.
- carabiner 3y agoIt really reminds me of the Black Mirror episode Smithereens with the tech CEO talking with the shooter. Tech people really struggle with empathy, not just 1 on 1 but with the rest of the outside world which is predominantly low income relatively, with no college education. Paraphrased, Black Mirror ep was like: [Tech CEO read instructions to "show empathy" from his assistant via Slack] CEO: I hear you. It must be very hard for you. Shooter: Of course you fucking hear me, we're on the phone! Talk like a normal person!
- baobabKoodaa 3y agoI remember that conversation! Man, that was a great episode.
- seydor 3y agoI thought it was implied and obvious that the video was edited. So what?
- frozenlettuce 3y agotoo little, too late. my impression is that google is not one, but two steps behind what MS can offer (they need a larger leap if they want to get ahead)
- golly_ned 3y agoIf you've seen the video, it's very apparent it's a product video, not a tech demo. They cut out the latencies to make a compelling product video. I wasn't at all under the impression they were showcasing TTS or low latencies as product features. I don't find the marketing misleading at all, and find these criticisms don't hit the mark. https://www.youtube.com/watch?v=UIZAiXYceBI https://www.youtube.com/watch?v=UIZAiXYceBI
- DominikPeters 3y agoIt's not just cutting. The answers were obtained by taking still photos and inputting them into the model together with detailed text instructions explaining the context and the task to the model, giving some examples first and using careful chain-of-thought style prompting. (see e.g. https://developers.googleblog.com/2023/12/how-its-made-gemini-multimodal-prompting.html#:~:text=%F0%9F%94%80%20Cup-,shuffling,-Let%E2%80%99s%20try%20a https://developers.googleblog.com/2023/12/how-its-made-gemin...) My guess is that the video was fully produced after the Gemini outputs were generated by a different team, instead of while or before.
- jbverschoor 3y agoWell, google has a history for faking things.. so I'm not not surprised. I expected that.. All companies are just yelling that they're "in" the AI/LLM game.. If they don't, share prices will drop.
- hifreq 3y agoThe red flag for me was that they started that demo video with a background noise to make it seem like it's a raw video. A subtle manipulation for no reason, it's obviously not a raw video. The fact that they did not fact check the videos again makes me not particularly confident in the quality of Google's work. The bit where the model misinterpreted music notation (the circled area does not mean "piano"), and the "less dense than water" rubber duck are beyond the pale. The SVG demo where they generate a South Park looking tree looks like a parody.
- deleted 3y ago[deleted]
- crazygringo 3y agoDoes it matter at all with regards to its AI capabilities though? The video has a disclaimer that it was edited for latency. And good speech-to-text and text-to-speech already exists, so building that part is trivial. There's no deception. So then it seems like somebody is pressing a button to submit stills from a video feed, rather than live video. It's still just as useful. My main question then is about the cup game, because that absolutely requires video. Does that mean the model takes short video inputs as well? I'm assuming so, and that it generates audio outputs for the music sections as well. If those things are not real, then I think there's a problem here. The Bloomberg article doesn't mention those, though.
- beering 3y agoEven your skeptical take doesn't fully show how faked this was. > The video has a disclaimer that it was edited for latency. There was no disclaimer that the prompts were different from what's shown. > And good speech-to-text and text-to-speech already exists, so building that part is trivial. There's no deception. Look at how many people thought it can react to voice in real-time - the net result is that a lot of people (maybe most?) were deceived. And the text prompts were actually longer and more specific than what was said in the video! > somebody is pressing a button to submit stills from a video feed, rather than live video. Somebody hand-picked images to convey exactly the right amount of information to Gemini. > Does that mean the model takes short video inputs as well? I'm assuming so It was given a hand-picked series of still images with the hands still on the cups so that it was easier to understand what cup moved where. Source for the above: https://developers.googleblog.com/2023/12/how-its-made-gemini-multimodal-prompting.html https://developers.googleblog.com/2023/12/how-its-made-gemin...
- IanCal 3y agoAudio input that's not text in the middle and video input are two things they made a big deal out of. Then they called it a hands on demo and it was faked. > My main question then is about the cup game, because that absolutely requires video. They did it with carefully timed images, and provided a few examples first. > I'm assuming so, and that it generates audio outputs for the music sections as well No, it was given the ability to search for music and so it was just generating search terms. Here's more details: https://developers.googleblog.com/2023/12/how-its-made-gemini-multimodal-prompting.html?m=1 https://developers.googleblog.com/2023/12/how-its-made-gemin...
- skilled 3y agofake benchmarks, fake stitched together videos, disingenuous charts, no developer API on launch, announcements stuffed with marketing fluff. As soon as I saw that opening paragraph from Sundar and how it was written I knew that Gemini is going to be a steaming pile of shit. They should have watched the GPT-4 announcement from OpenAI again. That demo Greg Brockman did with converting a sketch on a piece of paper to a CodePen from a Discord channel, with all the error correcting and whatnot, is how you launch a product that's appealing to users. TechCrunch, Twitter and some other sites (including HN i guess) are already piling on to this and by Monday things will go back to how they were and Google will have to go back to the drawing board to figure out another way to relaunch Gemini in the future.
- taspeotis 3y agoGoogle Gemi-lie
- vjerancrnjak 3y agoThere is a possibility of dataset contamination on the competitive programming benchmark. A nice discussion on the page where AlphaCode2 was solving the problems https://codeforces.com/blog/entry/123035 https://codeforces.com/blog/entry/123035 Problem showed in the video was reused in a recent competition (so could have been available in the dataset).
- mtrovo 3y agoI guess a much better next step is to compare how GPT4V performs when asked similar prompts. Even if mostly staged this is very impressive to me, not much on the current tech but more on how much leverage Google has to win this race on the long run because of its hardware presence. The more these models improve the more we will want less friction and faster interactions, this means that in the long term having to open an app and ask a question is not gonna fly compared to just pointing your phone camera to something, asking a question and getting an answer that's tailored to everything Google knows about you in real time. Apple will most likely also roll their own in house solution for Siri instead of relying on an external company. This leaves OpenAI and the other small companies not just competing for the best models but also on how to put them in front of people in the first place and how to get access to their personal information.
- bradhe 3y ago> Even if mostly staged this is very impressive to me, not much on the current tech but more on how much leverage Google has to win this race on the long run because of its hardware presence. I think you have too much information to form a reasonable opinion on the situation. Google is using editing techniques and specific scripting to try to demonstrate they have a sufficiently powerful general AI. The magnitude of this claim is huge, and the fact that they're faking it should be a likewise enormous scandal. To sum this up "well I guess they're doing better than XYZ" discounts the absurd context of all this.
- DonnyV 3y agoThis is so crazy. Google invented transformers which is the bases for all these models. How do they keep fumbling like this over and over. Google Docs created in 2006! Microsoft is eating their lunch. Google creates the ability to change VM's in place and makes a fully automated datacenter. Amazon and Microsoft are killing them in the cloud. Google has been working on self driving longer than anyone. Tesla is catching up and will most likely beat them. The amount of fumbles is monumental.
- bradhe 3y agoMicrosoft eating Google's lunch on documents is laughable at best. Not to mention it confuses the entire timeline of office productivity software??
- hot_gril 3y agoIs paid MS Teams is more or less common than paid GSuite? It's hard to find stats on this. GSuite is the better product IMO, but MS has a stronger b2b reputation, and anecdotally I hear more about people using Teams.
- UrineSqueegee 3y agoI worked at many companies in my times and all of them used teams except from one that used slack but all used MS products, none used googles.
- abustamam 3y agoDoes anyone use paid GSuite for anything other than docs/drive/Gmail ? In all companies I've worked at, we've used GSuite exclusively for those, and used slack/discord for chat, and zoom/discord for video/meetings. I know that MS Teams is a more full-featured product suite, but even at companies that used it, we still used Zoom for meetings.
- hot_gril 3y agoGSuite for calendar makes sense too. Chat sucks, and Meet would be decent if it weren't so laggy, but those are two things you can easily not use.
- davesque 3y agoThe hype really is drowning out the simple fact that basically no one really knows what these models are doing. Why does it matter so much that we include auto-correlation of embedding vectors as the "attention" mechanism in these models? And that we do this sufficiently many times across all the layers? And that we blindly smoosh values together with addition and call it a "skip" connection? Yes, you can tell me a bunch of stuff about gradients and residual information, but tell me why any of this stuff is or isn't a good model of causality.
- stormfather 3y agoSo what? Voice to text is a solved problem. And in cases where realtime is important, just throw more compute at it. I'm missing the damning gotcha moment here.
- deleted 3y ago[deleted]
- davesque 3y agoA big red flag for me was that Sundar was prompting the model to report lots of facts that can be either true or false. We all saw the benchmark figures that they published and the results mostly showed marginal improvements. In other words, the issue of hallucination has not been solved. But the demo seemed to imply that it had. My conclusion was that they had mostly cherry picked instances in which the model happened to report correct or consistent information. They oversold its capabilities, but it does still seem that multi-modal models are going to be a requirement for AI to converge on a consistent idea of what kinds of phenomena are truly likely to be observed across modalities. So it's a good step forward. Now if they can just show us convincingly that a given architecture is actually modeling causality.
- LesZedCB 3y agoi think this was demonstrated in that mark rober promo video[1] where he asked why the paper airplane stalled by blatantly leading the witness. "do you believe that a pocket of hot air would lead to lower air pressure causing my plane to stall?" he could barely even phrase the question correctly because it was so awkward. just embarrassing. [1] https://www.youtube.com/watch?v=mHZSrtl4zX0&t=277s https://www.youtube.com/watch?v=mHZSrtl4zX0&t=277s
- calf 3y agoEver since the "stochastic parrots" and "super-autocomplete" criticisms of LLMs, the question is whether hallucinations are solvable in principle at all. And if hallucinations are solvable, it would of such basic and fundamental scientific importance that I think would be another mini-breakthrough in AI.
- bradhe 3y agoFucking. Shocking. Anyone with half a brain could see through this "demo." It was vastly too uncanny to be real, to the point that it was poorly setup. Google should be ashamed.
- hollowturtle 3y agoI didn’t believed Google presentation off-hand because I don’t care anymore, especially because it comes from them. I just use tools and adapt. Copilot helps me automating boring tasks, can’t help much at new stuff, so I actually discovered I often do “interesting” work. I use gpt 3.5/4 for everything but work, it’s been a bless, best suggestion engine for movies, books and music with just a prompt and without the need of tons of data about my watch history(looking at you youtube). In these strange times I’m actually learning a lot more, productivity is more or less the same as before llms, but annoying tasks are relieved a bit. All of that without the hype. Sometimes I laugh at Google, it must be a real shit show inside that mega corporation, but I kinda understand the need of a marketing editing, having a first class ticket on the AI train is so important for them as it seems they see it as an existential threat. At least it seems so since they decided to take the risk of lying.
- zdrummond 3y agoThis is just a tweet that makes a claim without backing, and links to an article that was pulled. Can we change the URL to the real article if it still exists?
- dilawar 3y agoBloomberg link in Xeet is 404 for me (Bangalore).
- gsuuon 3y agoWow - my first thought was I wonder what framerate they're sending video at. The whole demo seems significantly less impressive in that case.
- ElijahLynn 3y agoLink to the Bloomberg article from the Tweet is 404 now.
- nblgbg 3y agoThat's my suspicion when I first saw it. Its really an impressive demo though.
- OkGoDoIt 3y agoHere’s the detailed article: https://archive.is/20231207205555/https://www.bloomberg.com/opinion/articles/2023-12-07/google-s-gemini-ai-model-looks-remarkable-but-it-s-still-behind-openai-s-gpt-4 https://archive.is/20231207205555/https://www.bloomberg.com/...
- pacomerh 3y agoI was looking at this demo today and was wondering the same. It looked way to fast to be calculating all of that info.
- kokanee 3y agoBloomberg link not working; here's TechCrunch: https://techcrunch.com/2023/12/07/googles-best-gemini-demo-was-faked/ https://techcrunch.com/2023/12/07/googles-best-gemini-demo-w...
- lumost 3y agoAnybody remember the google duo demo?
- jonplackett 3y agoOK I get that everyone’s hype sensitive and I absolutely remain to be convinced on Gemini’s actual ability BUT The fact this wasn’t realtime or with voice is not the issue. Voice to text could absolutely capture this conversation easily. And from what I’ve seen Gemini seems quicker than GPT4 Being able to respond quicker and via voice chat is not actually a big deal. The underlying performance of the model is what we should be focussing on
- pphysch 3y agoExactly. Corporate tech demo exaggerates actual capabilities and smoothes over rough edges? Impossible, this is unprecedented!! The Apple vs Google brand war is so tiresome. Let's focus on the tech.
- jonplackett 3y agoTo be clear - I’m not saying this makes Gemini good. Just that it isn’t bad for these reasons!
- michaelt 3y agoI disagree. One of the major issues in LLMs is the economics; a lot of people suspect ChatGPT loses money on every user, or at least every heavy user, because they've got a big model and A100 GPUs are expensive and in short supply. They're kinda reluctant to have customers, with API rate limits galore, and I've heard people claiming ChatGPT has lost the performance crown having switched to a cheaper-to-run model. If google had a model that operated on video in realtime, that would imply they've got a model that performs well, and is also very fast or that their 'TPUs' outperform the A100 quite a bit, either of which would be a big step forward.
- turquoisevar 3y agoEven if you'd be inclined to shrug off the fact that this wasn't real-time voice and video-based, which you shouldn't because the underlying implications would be huge for performance, there's still the matter that the prompts used. The prompts shown are miles apart and significantly misrepresent the performance of the underlying model. It goes from a model being able to infer a great deal at the level of human intelligence to a model that needs to be fed essential details, and that doesn't do much inferring. I get the feeling that many here on HN who are just shrugging it off don't realize how much of the “demo” was faked. Here’s a decent article that goes into it some more: https://techcrunch.com/2023/12/07/googles-best-gemini-demo-was-faked/ https://techcrunch.com/2023/12/07/googles-best-gemini-demo-w...
- jaimex2 3y agoGoogle is done, they can't compete in this space.
- artifact_44 3y ago[dead]
- dvsfish 3y agoI did this at university. It was our first comp sci class ever, we were given raspberry pi's. We had no coding experience or guidance, and were asked to create "something". All we had to work with was information on how to communicate with the pi using putty. Oddly, this assignment didn't require us to submit code, but simply demonstrate it working. My group (3 of us) bought a moisture sensor to plug into the pi, and had the idea to make a "flood detection system" that would be housed under a bridge, and would send an email to relevant people when the bridge home from work is about to flood. So for our demonstration, we had a guy in the back of the class with gmail open ready to send an email saying some variation of "flood warning". Our script was literally just printing lines with wait statements in between. Running the script, it prints to the screen "awaiting moisture", and after 3 seconds it will print "moisture detected". In that 3 seconds I dip the sensor into the glass of water. Then the script would wait a few more seconds before printing "sending email to xxx@yyy.com". We then opened up our email, our mate at the back of the room hit send, and an email appeared saying flood warning, and we would get full marks.
- 10u152 3y agoRelated, I work with industrial control systems. We’d call this “smoke and mirrors”. Sometimes the client would insist on seeing a small portion of a large project working well before it was ready. They’d misunderstand that 90% of the bulk of the work is not visible to the user but they’d want to see a finished state. We’d set up a dummy HMI and have someone pressing buttons on it for the demo, and someone in the next room manually driving outputs and inputs to make it seem like it was working. Very common.
- trash_cat 3y agoI geniounly don't understand why this comes as a surprise. If it was a realtime video with realtime responses it would have been some other generational leap beyond transformers. Not only that, it would somehow deal with video + audio with smaller latency than the current SOTA model? Cmon...
- Uptrenda 3y agoImagine being google and having 100 BILLION+ in liquid cash, tens of thousands of the best engineers world-wide, and everything you could possibly need for running tech products. Yet being completely unable to launch anything new or worthwhile. Like, how tf does that even happen? Is Google the next Kodak?
- sugavaneshb 3y agoI can't find the original article anymore
- nyxtom 3y agoSilicon Valley is plagued by over promising far too early and fooling everyone we are years ahead of where we actually are. Estimates are always way off, and claims and even fake demos are almost always oversold to get investors.
- throw__away7391 3y agoWhich means anyone who isn’t engaged in the hype cycle looks bad by comparison. Still I would distinguish between genuine optimism and cynical exaggeration. If you go back and read what was being said in and around the 70s about technology and watch the demos they gave…it always makes me feel a little sad for them. People really thought so many things were right around the corner, e.g. if a computer can beat a human star chess player, surely understanding natural language or driving a car will be easy. There were so many things being made like primitive robots that could follow a line painted on the floor, that in a controlled environment seemed so promising but the realization of the vision would take decades longer than predicted. Without their efforts we would not be where we are today.
- nyxtom 3y agoTrue. I think the optimism and grandiosity of the engineers with gems in their eyes keeps the passion going to work hard - some are just are terrible at grounding ourselves in reality. Genuine science needs to take more foothold, skepticism and curiosity over certainty of accelerated outcomes or non existant (heck even fraudulent in some situations) capabilities
- squigglydonut 3y agoHard to build when you fire everyone huh
- Al-Khwarizmi 3y agoI didn't even look at the demo. And not due to lack of interest in LLMs (I'm an academic working in NLP, and I have some work on LLMs). The thing is that for all practical intents and purposes, if I can't try it, it doesn't exist. If they claim it exists, they should show it, a video or a few cherry-picked examples don't prove anything. It's easy to make a demo making even Eliza look like AGI by asking the right questions.
- pookietactics 3y agoAt the 5:35 point, the phone screen rotates before he touches it. https://youtu.be/UIZAiXYceBI?t=335 https://youtu.be/UIZAiXYceBI?t=335
- sakesun 3y agoThis is what convince me that all are hoax.
- spaceman_2020 3y agoI’ll assume Gemini is vaporware unless I can actually use it.
- teleforce 3y agoIs the link to the article broken and anyone has it archived somewhere? I wish people stop posting Twitter messages to HN and provide a link directly to the original article. What's next, post on HN on an Instagram post?
- choppaface 3y agoHere's a related article: https://www.theverge.com/2023/12/7/23992737/google-gemini-misrepresentation-ai-accusation https://www.theverge.com/2023/12/7/23992737/google-gemini-mi... And another: https://techcrunch.com/2023/12/07/googles-best-gemini-demo-was-faked/ https://techcrunch.com/2023/12/07/googles-best-gemini-demo-w...
- gloosx 3y agoI dont get it too, my browser was loading for good 15 seconds and made 141 requests fetching almost 9 MB of resources to show me exactly same content as provided in OpenGraph tags and a freaking redirect to a Bloomberg link. Feels like a slap in the face to open such phishing link at any time, just a useless redirect with nine million bytes of overhead.
- maremmano 3y agoWell, sometimes I have this "Google Duplex: A.I. Assistant Calls Local Businesses To Make Appointments" feeling. https://www.youtube.com/watch?v=D5VN56jQMWM https://www.youtube.com/watch?v=D5VN56jQMWM
- kccqzy 3y agoWhile this might just be a bit of bad PR now, it will eventually be a nothing burger. Remember the original debut of Apple's Siri for which Apple also put out a promotional demo with greatly exaggerated functionality? People even sued Apple and they lost. As much as I hate it, this is absolutely fine by our society's standards. https://www.theregister.com/AMP/2014/02/14/apple_prevails_in_siri_marketing_spat/ https://www.theregister.com/AMP/2014/02/14/apple_prevails_in...
- turquoisevar 3y agoThere's a vast difference between advertising a product, slightly shortening the sequences and, cutting out failed prompts, and completely misrepresenting the product at hand to a degree that the depiction doesn't resemble the product at all[0]. The former is considered Puffery[1] and is completely legal, and the latter is straight up lying. 0: https://techcrunch.com/2023/12/07/googles-best-gemini-demo-was-faked/ https://techcrunch.com/2023/12/07/googles-best-gemini-demo-w... 1: https://en.wikipedia.org/wiki/Puffery https://en.wikipedia.org/wiki/Puffery
- zetabyte 3y agoI wanted to see what's really going on, but the bloomberg article on the twit link seems taken down right now.
- sinuhe69 3y agoI, too, was fooled to think Gemini has seen and heard through a video/audio feed instead of showing still images and prompting though text. While it might seem not much difference between still images and a video feed, in fact it requires a lot of (changing) context understanding to not make the bot babbling like an idiot all the time. It also requires the bot to recognize the “I don’t know it yet” state to keep appropriate silence in a conversation with live video feed, which is notoriously difficult with generative AI. Certainly one can do some hacking, build in some heuristics to make it easier, but to make a bot seems like a human partner in a conversion is indeed very hard. And that has been the most impressive aspect of the showed “conversations”, which are unfortunately all faked :(
- larodi 3y agoThis must’ve been shot by one of the directors who did Apple’s new show Extrapolations. Very plausible illustration of AI in daily life though, despite the aggressive climate change claims made in it. But neither AI not climate are there yet…
- ShamelessC 3y agoThat show’s most out-there predictions all take place after 2050. Your criticism isn’t relevant.
- macilacilove 3y agoThe demo may be fake, but people genuinely love it. the top comment on youtube of that demo is still: "Absolutely mindblowing. The amount of understanding the model exhibits here is way way beyond anything else."
- L_226 3y agothe original linked article in the tweet [0] now returns 404 for me [0] - https://www.bloomberg.com/opinion/articles/2023-12-07/google-s-gemini-ai-model-looks-remarkable-but-it-s-still-behind-openai-s-gpt-4io https://www.bloomberg.com/opinion/articles/2023-12-07/google...
- jgalt212 3y agoIf the truck doesn't have a working engine, we can just roll it down the hill. Brilliant idea!
- nojvek 3y agoAnyone remember the Google IO demo where they had their “AI” call a barber to book an appointment. Turns out it was all staged. Lost a lot of trust after that. Google is stuck in innovators dilemma. They make 300B of revenue which ~90% is ads revenue. Their actual mission that management chain optimizes for is their $ growth. A superior AI model that gives the user exactly what they want would crash their market cap. Microsoft has tons of products with Billion+ profit, Google has only a handful and other than cloud they all tie to Ads. Google is addicted to ads. If chrome adds a feature that decreases ad revenue, that team gets a stick. Nothing at Google should jeopardize their ad revenue. AI is directly a threat to Google’s core business model - ads. It’s obvious they’re gonna half ass it. For OpenAI, AI is existential for them. If they don’t deliver, they’ll die.
- nsonha 3y agowhy didn't they do so? what's the challenge? I suppose they could've programmed in some prompt indicator (like "OK Google" but less obvious), then that demo could have been technically feasible.
- rattatossk 3y agoIt was in fact done by a duck. I for one, welcome our new quack overlords.
- fennecfoxy 3y agoI mean, I know people aren't gonna boycott Google (cause people aren't capable of boycotting anything), but fuck Google, man. This AI stuff is becoming super cool and the last thing we need is the usual charlatans that surround technology stuff. They should be ashamed of themselves, but I know who called for it - not the engineers, but the "management" layer and they're incapable of feeling shame.
- dingclancy 3y agoWhat is clear is that Google really has no 'God model' that they were holding back all along. Gemini Ultra is barely beating ChatGPT in their manufactured benchmarks, and this is all that they got. What this means is that those who are saying, including people at Google, that they have better models but are not releasing them in the name of AI safety, imply that, at least in the realm of LLMs, Google DeepMind had nothing all along.