16 ms·

One observation: Sundar's comments in the main video seem like he's trying to communicate "we've been doing this ai stuff since you (other AI companies) were li

by tikkun 3y ago

One observation: Sundar's comments in the main video seem like he's trying to communicate "we've been doing this ai stuff since you (other AI companies) were little babies" - to me this comes off kind of badly, like it's trying too hard to emphasize how long they've been doing AI (which is a weird look when the currently publicly available SOTA model is made by OpenAI, not Google). A better look would simply be to show instead of tell. In contrast to the main video, this video that is further down the page is really impressive and really does show - the 'which cup is the ball in is particularly cool': https://www.youtube.com/watch?v=UIZAiXYceBI https://www.youtube.com/watch?v=UIZAiXYceBI. Other key info: "Integrate Gemini models into your applications with Google AI Studio and Google Cloud Vertex AI. Available December 13th." (Unclear if all 3 models are available then, hopefully they are, and hopefully it's more like OpenAI with many people getting access, rather than Claude's API with few customers getting access)

infoseek12 3y ago
> "we've been doing this ai stuff since you (other AI companies) were little babies" Well in fairness he has a point, they are starting to look like a legacy tech company.
smoldesu 3y ago
In fairness, the performance/size ratio for models like BERT still gives GPT-3/4 and even Llama a run for it's money. Their tech isn't as product-ized as OpenAI's, but Tensorflow and it's ilk have been an essential part of driving actual AI adoption. The people I know in the robotics and manufacturing industries are forever grateful for the out-front work Google did to get the ball rolling.
- wddkcs 3y ago
  You seem to be saying the same thing- Googles best work is in the past, their current offerings are underwhelming, even if foundational to the progress of others.
cowsup 3y ago
> to me this comes off kind of badly, like it's trying too hard to emphasize how long they've been doing AI These lines are for the stakeholders as opposed to consumers. Large backers don't want to invest in a company that has to rush to the market to play catch-up, they want a company that can execute on long-term goals. Re-assuring them that this is a long-term goal is important for $GOOG.
- hinkley 3y ago
  Large backers and stakeholders are not 25 years old.
- gessha 3y ago
  It would be interesting to write a LLM query to separate speech details based on target audience: stakeholders, consumers, etc.
tikkun 3y ago
To add to my comment above: Google DeepMind put out 16 videos about Gemini today, the total watch time at 1x speed is about 45 mins. I've now watched them all (at >1x speed). In my opinion, the best ones are: * https://www.youtube.com/watch?v=UIZAiXYceBI https://www.youtube.com/watch?v=UIZAiXYceBI - variety of video/sight capabilities * https://www.youtube.com/watch?v=JPwU1FNhMOA https://www.youtube.com/watch?v=JPwU1FNhMOA - understanding direction of light and plants * https://www.youtube.com/watch?v=D64QD7Swr3s https://www.youtube.com/watch?v=D64QD7Swr3s - multimodal understanding of audio * https://www.youtube.com/watch?v=v5tRc_5-8G4 https://www.youtube.com/watch?v=v5tRc_5-8G4 - helping a user with complex requests and showing some of the 'thinking' it is doing about what context it does/doesn't have * https://www.youtube.com/watch?v=sPiOP_CB54A https://www.youtube.com/watch?v=sPiOP_CB54A - assessing the relevance of scientific papers and then extracting data from the papers My current context: API user of OpenAI, regular user of ChatGPT Plus (GPT-4-Turbo, Dall E 3, and GPT-4V), occasional user of Claude Pro (much less since GPT-4-Turbo with longer context length), paying user of Midjourney. Gemini Pro is available starting today in Bard. It's not clear to me how many of the super impressive results are from Ultra vs Pro. Overall conclusion: Gemini Ultra looks very impressive. But - the timing is disappointing: Gemini Ultra looks like it won't be widely available until ~Feb/March 2024, or possibly later. > As part of this process, we’ll make Gemini Ultra available to select customers, developers, partners and safety and responsibility experts for early experimentation and feedback before rolling it out to developers and enterprise customers early next year. > Early next year, we’ll also launch Bard Advanced, a new, cutting-edge AI experience that gives you access to our best models and capabilities, starting with Gemini Ultra. I hope that there will be a product available sooner than that without a crazy waitlist for both Bard Advanced, and Gemini Ultra API. Also fingers crossed that they have good data privacy for API usage, like OpenAI does (i.e. data isn't used to train their models when it's via API/playground requests).
- tikkun 3y ago
  My general conclusion: Gemini Ultra > GPT-4 > Gemini Pro See Table 2 and Table 7 https://storage.googleapis.com/deepmind-media/gemini/gemini_1_report.pdf https://storage.googleapis.com/deepmind-media/gemini/gemini_... (I think they're comparing against original GPT-4 rather than GPT-4-Turbo, but it's not entirely clear) What they've released today: Gemini Pro is in Bard today. Gemini Pro will be coming to API soon (Dec 13?). Gemini Ultra will be available via Bard and API "early next year" Therefore, as of Dec 6 2023: SOTA API = GPT-4, still. SOTA Chat assistant = ChatGPT Plus, still, for everything except video, where Bard has capabilities . ChatGPT plus is closely followed by Claude. (But, I tried asking Bard a question about a youtube video today, and it told me "I'm sorry, but I'm unable to access this YouTube content. This is possible for a number of reasons, but the most common are: the content isn't a valid YouTube link, potentially unsafe content, or the content does not have a captions file that I can read.") SOTA API after Gemini Ultra is out in ~Q1 2024 = Gemini Ultra, if OpenAI/Anthropic haven't released a new model by then SOTA Chat assistant after Bard Advanced is out in ~Q1 2024 = Bard Advanced, probably, assuming that OpenAI/Anthropic haven't released new models by then
lkbm 3y ago
It's worth remembering that AI is more than LLMs. DeepMind is still doing big stuff: https://deepmind.google/discover/blog/millions-of-new-materials-discovered-with-deep-learning/ https://deepmind.google/discover/blog/millions-of-new-materi...
- dpflan 3y ago
  Indeed, I would think the core search product as another example of ai/ml...
  - mpweiher 3y ago
    The question is whether greater use of AI correlates with the declining quality of search results.
    - visarga 3y ago
      I think the bottom line (profit) inversely correlates with the quality of search results. I've been using phind.com lately and it seems there can be search without junk even in this age. Google has lots of people tagging search rankings, which is very similar with RLHF ranking responses from LLMs. It's interesting that using LLMs with RLHF it is possible to de-junk the search results. RLHF is great for this task, as evidenced by its effect on LLMs.
      - mattmaroon 3y ago
        Right. It’s less that their declining quality of search results is due to AI and more that the AI got really good at monetizing and monetizing and quality search results are sometimes in opposition.
        avereveard 3y ago
        This entire thread kinda ignore that they are also selling ad space on many sites and their objective function in ordering search is not just the best possible result. Case in point the many sites stealing stack overflow content and filling it with adverts ranking higher than the source, that committed the cardinal sin of running their own and network.
willsmith72 3y ago
they have to try something, otherwise it looks like they've been completely destroyed by a company of 1000 people
hinkley 3y ago
Sundar studied material science in school and is only slightly older than me. Google is a little over 25 years old. I guarantee you they have not been doing AI since I was a baby. And how many financial people worth reconning with are under 30 years old? Not many.
- crossroadsguy 3y ago
  Unless you are OpenAI, the company, I doubt OP implied it was aimed at you. But then I wouldn't know as I am much younger than Sundar Pichai and I am not on first name basis with him either ;-)
dist-epoch 3y ago
> show instead of tell They showed AlphaGo, they showed Transformers. Pretty good track record.
- visarga 3y ago
  That was ages ago. In AI even a week feels like a whole year in other fields. And many/most of those researchers have fled to startups, so those startups also have a right to brag. But not too much - only immediate access to a model beating GPT4 is worth bragging today (cloud), or getting GPT3.5 quality from a model running on a phone (edge). So it's either free-private-gpt3.5 or cloud-better-than-gpt4v. Nothing else matters now. I think we have reached an extreme point of temporal discounting (https://en.wikipedia.org/wiki/Time_preference https://en.wikipedia.org/wiki/Time_preference).
  - jvolkman 3y ago
    > That was ages ago. I think that was the point.
  - thefourthchime 3y ago
    The Transformer paper “Attention is All You Need” came out in 2017. Sundar got the CEO job two years earlier, so he was in CEO diapers at the time if you will. I would argue Google has done almost nothing interesting since then (at least not things they haven't killed)
vinniepukh 3y ago
No surprises here. Google DeepMind squandered their lead in AI so much that they now have to have “Google” prepended to their name to show that adults are now in charge.
- password54321 3y ago
  What an ugly statement. DeepMind has been very open with their research since the beginning because their objective was much more on making breakthroughs with moonshot projects than near term profit.
  - vinniepukh 3y ago
    [dead]
misterbwong 3y ago
I also find that tone a bit annoying but I'm OK with it because it highlights how these types of bets, without an immediate benefit, can pay off very well in the long term, even for huge companies like Google. AI, as we currently know it, wasn't really a "thing" when Google started with it and the payoff wasn't clear. They've long had to defend their use of their own money for big R&D bets like this and only now is it really clearly "adding shareholder value". Yes, I know it was a field of interest and research long before Google invested, but the fact remains that they _did_ invest deeply in it very early on for a very long time before we got to this point. Their continued investment has helped push the industry forward, for better or worse. In light of this context, I'm ok with them taking a small victory lap and saying "we've been here, I told you it was important".
- jeffbee 3y ago
  > only now is it really clearly "adding shareholder value". AI has been adding a huge proportion of the shareholder value at Google for many years. The fact that their inference systems are internal and not user products might have hidden this from you.
- deleted 3y ago
  [deleted]
corethree 3y ago
Didn't Google invent LLMs and didn't Google have an internal LLm with similar capabilities long before openai released the gpts? Remember when that guy got fired for making a claim it was conscious ? The look isn't good. But it's not dishonest.
- ma2rten 3y ago
  No this is not correct. Arguably OpenAI invented LLMs with GPT3 and the preceding scaling laws paper. I worked on LAMDA, it came after GPT4 and was not as capable. Google did invent the transformer, but all the authors of the paper have left since.
  - visarga 3y ago
    This is like the Ship of Theseus. If all transformer authors reunited under one startup rival of Google, who would have the bragging rights?
    - deleted 3y ago
      [deleted]
    - ma2rten 3y ago
      Noam.
  - corethree 3y ago
    Isn't a transformer an LLM?
    - avarun 3y ago
      No. You’re confused. Most LLMs in production today use transformers under the hood but a transformer is most certainly not a type of LLM.
  - deleted 3y ago
    [deleted]
dontupvoteme 3y ago
He's not wrong. DeepMind spends time solving big scientific / large-scale problems such as those in genetics, material science or weather forecasting, and Google has untouchable resources such as all the books they've scanned (and already won court cases about) They do make OpenAI look like kids in that regard. There is far more to technology than public facing goods/products. It's probably in part due to the cultural differences between London/UK/Europe and SiliconValley/California/USA.
- deleted 3y ago
  [deleted]
- jahsome 3y ago
  That statement isn't really directed at the people who care about the scientific or tech-focused capabilities. I'd argue the majority of those folks interested in those things already know about DeepMind. This statement is for the mass market MBA-types. More specifically, middle managers and dinosaur executives who barely comprehend what generative AI is, and value perceived stability and brand recognition over bleeding edge, for better or worse. I think the sad truth is an enormous chunk of paying customers, at least for the "enterprise" accounts, will be generating marketing copy and similar "biz dev" use cases.
- roguas 3y ago
  Oh it's good they working on important problems with their ai. Its just openai was working on my/our problems (or providing tools to do so) and that's why people are more excited about them. Not because of cultural differences. If you are more into weather forecasting, yeah it sure may be reasonable to prefer google more.
  - faramarz 3y ago
    That’s what makes Altman a great leader. He understands marketing better than many of these giants. Google got caught being too big. Sure they will argue that AI mass release is a dangerous proposition, but Sam had to make a big splash otherwise he would be competing with incumbent marketing spendings far greater than OpenAI could afford. It was a genius move to go public with a simple UI. No matter how stunning the tech side is, if human interaction is not simple, the big stuff doesn’t even matter.
mattmaroon 3y ago
I do think that’s a backfire. Telling me how long you’ve been doing something isn’t that impressive if the other guy has been doing it for much less time and is better at it. It’s in fact the opposite.
- pb7 3y ago
  Not if the little guy leveraged your inventions/research.
  - dragonwriter 3y ago
    That's even worse: what it says is that you are getting beat at product even where you create the tech. Which is definitely where Google is in the generative AI space.
  - fennecbutt 3y ago
    Weird for us to personify a corporation like that tbh. Google didn't invent transformers, researchers working at Google did. Sure Google paid em money/employed em, but the smarts behind it isn't the entity Google or the execs at the top, Sundar etc; it's those researchers. I like to appreciate individualism in a world where those at the top have lobbied their way into a 1% monopoly lmao.
    - pb7 3y ago
      This whole conversation is weird. First, OpenAI is not some "little guy". It's a bigger corporation than 99.99% of companies that have ever existed. It's like calling Airbnb or Uber "little guys". Second, yes, the researchers did that but the company funded it with no guarantee of return. Your argument can be applied to any company or organization; it's needless pedantry.
      - Solvency 3y ago
        770 employees is bigger than 99.9% off all companies ever? Because we saw how easily those employees would have walked out the front door only too recently.
bogwog 3y ago
> Sundar's comments in the main video seem like he's trying to communicate "we've been doing this ai stuff since you (other AI companies) were little babies" - to me this comes off kind of badly Reminds me of the Stadia reveal, where the first words out of his mouth were along the lines of "I'll admit, I'm not much of a gamer" This dude needs a new speech writer.
- cmrdporcupine 3y ago
  This dude needs a new speech writer. How about we go further and just state what everyone (other than Wall St) thinks: Google needs a new CEO. One more interested in Google's supposed mission ("to organize the world's information and make it universally accessible and useful"), than in Google's stock price.
  - whatshisface 3y ago
    "The shareholders are too well-represented" is one of the complaints least likely to be acted on, second only to "I'm getting old."
    - cmrdporcupine 3y ago
      Sure, anti-capitalist-impulse Me 100% agrees. It's just doubly icky in the context of Google's original "letter to shareholders": https://www.nytimes.com/2004/04/29/business/letter-from-the-founders.html https://www.nytimes.com/2004/04/29/business/letter-from-the-... ("Google is not a conventional company. We do not intend to become one")
      - avarun 3y ago
        Haha that’s a great find. Google may be one of the most conventional companies out there now.
        whatshisface 3y ago
        "One of the most conventional companies" sounds like some kind of paradox.
FrustratedMonky 3y ago
Well, deepmind was doing amazing stuff before OpenAI. AlphaGo, AlphaFold, AlphaStar. They were groundbreaking a long time ago. They just happened to miss the LLM surge.
headcanon 3y ago
Its a conceit but not unjustified, they have been doing "AI" since their inception. And yeah, Sundar's term up until recently seems to me to be milking existing products instead of creating new ones, so it is a bit annoying when they act like this was their plan the whole time. Google's weakness is on the product side, their research arm puts out incredible stuff as other commenters have pointed out. GPT essentially came out from Google researchers that were impatient with Google's reluctance to ship a product that could jeopardize ad revenue on search.
- radicaldreamer 3y ago
  It's such a short sighted approach too because I'm sure someone will develop a GPT with native advertising and it'll be a blockbuster because it'll be free to use but also have strong revenue generating potential.
  - TerrifiedMouse 3y ago
    One problem I see with ChatGPT replacing search is, who is going to fund the content creators that generate the training data?
  - vitorgrs 3y ago
    Bing Chat/Copilot already have ads and is free.
- nonethewiser 3y ago
  The point is if you have to remind people then you’re doing something wrong. The insight to draw from this is not that everyone else is misinformed about googles abilities (the implication), its that Google has not capitalized on their resources.
StevenNunez 3y ago
This demo is blowing my mind! It's really incredible. Can't wait to play around with them.
risyachka 3y ago
Google literally invented transformers that are at the core of all current AI/LLMs so Sundar's comment is very accurate.
- tempnow987 3y ago
  But their first efforts in BARD were really not great. I'd just have left the bragging out in terms of how long. OpenAI and others have no doubt sent a big wakeup call to google. For a while it seemed like they had turned to focus an AI "safety" (remembering some big blowups on those teams as well) with papers about how AI might develop negative stereotypes (ie, men commit more violent crime then women?). That seems to have changed - this is very product focused, and I asked it some questions that in many models are screened out for "safety" and it responded which is almost even more surprising (ie. Statistically who commits more violent crime, men or women).
  - hughesjj 3y ago
    The big concern was biased datasets iirc and shit fits for people of color. Like clearly mislabeling feminine looking women as men, and a stupid high false positive rate for face detection. That was relevant given they were selling their models to law enforcement.
- dekhn 3y ago
  Sundar's comments about Google doing AI (really ML) are based more on things that people externally know very little about. Systems like SETI, Sibyl, RePhil, SmartASS. These were all production ML systems that used fairly straightforward and conventional ML combined with innovative distributed computing and large-scale infrastructure to grow Google's product usage significantly over the past 20 years. For example here's a paper 10 years old now: https://static.googleusercontent.com/media/research.google.com/en/us/pubs/archive/41159.pdf https://static.googleusercontent.com/media/research.google.c... and another close to 10 years old now: https://research.google/pubs/pub43146/ https://research.google/pubs/pub43146/ The learning they expose in those papers came from the previous 10 years of operating SmartASS. However, SmartASS and sibyl weren't really what external ML people wanted- it was just fairly boring "increase watch time by identifying what videos people wioll click on" and "increase mobile app installs" or "show the ads people are likely to click on". It really wasn't until vincent vanhoucke stuffed a bunch of GPUs into a desktop and demonstrated scalable and dean/ng built their cat detector NN that google started being really active in deep learning. That was around 2010-2012.
lossolo 3y ago
> we've been doing this ai stuff since you (other AI companies) were little babies Actually, they kind of did. What's interesting is that they still only match GPT-4's version but don't propose any architectural breakthroughs. From an architectural standpoint, not much has changed since 2017. The 'breakthroughs', in terms of moving from GPT to GPT-4, included: adding more parameters (GPT-2/3/4), fine-tuning base models following instructions (RLHF), which is essentially structured training (GPT-3.5), and multi-modality, which involves using embeddings from different sources in the same latent space, along with some optimizations that allowed for faster inference and training. Increasing evidence suggests that AGI will not be attainable solely using LLMs/transformers/current architecture, as LLMs can't extrapolate beyond the patterns in their training data (according to a paper from DeepMind last month): "Together our results highlight that the impressive ICL abilities of high-capacity sequence models may be more closely tied to the coverage of their pretraining data mixtures than inductive biases that create fundamental generalization capabilities."[1] 1. https://arxiv.org/abs/2311.00871 https://arxiv.org/abs/2311.00871
- alaskamiller 3y ago
  In short: a chat bot is not AI.
schleck8 3y ago
They always do this, every time they get to mention AI. It appears somewhat desperate imo.
ugh123 3y ago
> One observation: Sundar's comments in the main video seem like he's trying to communicate "we've been doing this ai stuff since you (other AI companies) Sundar has been saying this repeatedly since Day 0 of the current AI wave. It's almost cliche for him at this point.
- dragonwriter 3y ago
  And he's going to keep saying it to tell investors why they should believe Google will eventually catch up in product until Google does catch up in product and he doesn't need to say it anymore. Or until Google gives up on the space, or he isn't CEO, if either of those come first, which I wouldn't rule out.
- xnx 3y ago
  Sundar announced his intentions to lead Google as an "AI first" company in May 2017: https://blog.google/technology/ai/making-ai-work-for-everyone/ https://blog.google/technology/ai/making-ai-work-for-everyon...
ac1spkrbox 3y ago
“Any man who must say ‘I am the king’ is no true King”
- DonHopkins 3y ago
  Any man who must say "I won't be a dictator, except for day one" will be a permanent dictator. https://eu.usatoday.com/story/news/politics/elections/2023/12/05/trump-dictator-day-one/71816204007/ https://eu.usatoday.com/story/news/politics/elections/2023/1...
irthomasthomas 3y ago
They played the same tune at that panel with Sam Altman the night before he was fired. https://youtu.be/ZFFvqRemDv8 https://youtu.be/ZFFvqRemDv8 He mentions Transformers - fine. Then he says that we've all been using Google AI for so long with Google Translate.
jonplackett 3y ago
I find this video really freaky. It’s like Gemini is a baby or very young child and also a massively know it all adult that just can’t help telling how clever it is and showing off its knowledge. People speak of the uncanny valley in terms of appearance. I am getting this from Gemini. It’s sort of impressive but feels freaky at the same time. Is it just me?
- nolist_policy 3y ago
  https://www.youtube.com/watch?v=PJgo3BBgWDA https://www.youtube.com/watch?v=PJgo3BBgWDA
- kromem 3y ago
  No, there's an odd disconnect between the impressiveness of the multimodal capabilities vs the juvenile tone and insights compared to something like GPT-4 that's very bizarre in application. It is a great example of what I've been finding a growing concern as we double down on Goodhart's Law with the "beats 30 out of 32 tests compared to existing models." My guess is those tests are very specific to evaluations of what we've historically imagined AI to be good at vs comprehensive tests of human ability and competencies. So a broad general pretrained model might actually be great at sounding 'human' but not as good at logic puzzles, so you hit it with extensive fine tuning aimed at improving test scores on logic but no longer target "sounding human" and you end up with a model that is extremely good at what you targeted as measurements but sounds like a creepy toddler. We really need to stop being so afraid of anthropomorphic evaluation of LLMs. Even if the underlying processes shouldn't be anthropomorphized, the expressed results really should be given the whole point was modeling and predicting anthropomorphic training data. "Don't sound like a creepy soulless toddler and sound more like a fellow human" is a perfectly appropriate goal for an enterprise scale LLM, and we shouldn't be afraid of openly setting that as a goal.
jiggawatts 3y ago
That was pretty impressive… but do I have to be “that guy” and point out the error it made? It said rubber ducks float because they’re made of a material less dense than water — but that’s not true! Rubber is more dense than water. The ducky floats because it’s filled with air. If you fill it with water it’ll sink. Interestingly, ChatGPT 3.5 makes the same error, but GPT 4 nails it and explains the it’s the air that provides buoyancy. I had the same impression with Google’s other AI demos: cute but missing something essential that GPT 4 has.
- zyxin 3y ago
  I would've liked to see an explanation that includes the weight of water being displaced. That would also explain how a steel ship with an open top is also able to float.
- scoot 3y ago
  I spotted that too, but also, it didn't recognise the "bird" until it had feet, when it is supposedly better than a human expert. I don't doubt that the examples were cherry-picked, so if this is the best it can do, it's not very convincing.
choppaface 3y ago
> A better look would simply be to show instead of tell. Completely! Just tried Bard. No images and the responses it gave me were pretty poor. Today's launch is a weak poor product launch, looks mostly like a push to close out stuff for Perf and before everybody leaves for the rest of the December for vacation.
neop1x 3y ago
A simple REST API with a static token auth like OpenAI API would help. Previously when I tried Bard API it was refusing to accept token auth, requiring that terrible oauth flow so I gave up.
OJFord 3y ago
Incredible stuff, and yet TTS is still so robotic. Frankly I assume it must be deliberate at this point, or at least deliberate that nobody's worked on it because it's comparatively easy and dull? (The context awareness of the current breed of generative AI seems to be exactly what TTS always lacks, awkward syllables and emphasis, pronunciation that would be correct sometimes but not after that word, etc.)
jongjong 3y ago
Yes it sounds like a conspiracy theory about government and big tech working on advanced tech which has existed for decades but kept secret.
nothrowaways 3y ago
SOTA is made by an ex Google employee. So their argument still holds.

by tikkun 3y ago

infoseek12 3y ago

> "we've been doing this ai stuff since you (other AI companies) were little babies" Well in fairness he has a point, they are starting to look like a legacy tech company.

smoldesu 3y ago

In fairness, the performance/size ratio for models like BERT still gives GPT-3/4 and even Llama a run for it's money. Their tech isn't as product-ized as OpenAI's, but Tensorflow and it's ilk have been an essential part of driving actual AI adoption. The people I know in the robotics and manufacturing industries are forever grateful for the out-front work Google did to get the ball rolling.

wddkcs 3y ago
You seem to be saying the same thing- Googles best work is in the past, their current offerings are underwhelming, even if foundational to the progress of others.

cowsup 3y ago

> to me this comes off kind of badly, like it's trying too hard to emphasize how long they've been doing AI These lines are for the stakeholders as opposed to consumers. Large backers don't want to invest in a company that has to rush to the market to play catch-up, they want a company that can execute on long-term goals. Re-assuring them that this is a long-term goal is important for $GOOG.

hinkley 3y ago
Large backers and stakeholders are not 25 years old.
gessha 3y ago
It would be interesting to write a LLM query to separate speech details based on target audience: stakeholders, consumers, etc.

tikkun 3y ago

To add to my comment above: Google DeepMind put out 16 videos about Gemini today, the total watch time at 1x speed is about 45 mins. I've now watched them all (at >1x speed). In my opinion, the best ones are: * https://www.youtube.com/watch?v=UIZAiXYceBI https://www.youtube.com/watch?v=UIZAiXYceBI - variety of video/sight capabilities * https://www.youtube.com/watch?v=JPwU1FNhMOA https://www.youtube.com/watch?v=JPwU1FNhMOA - understanding direction of light and plants * https://www.youtube.com/watch?v=D64QD7Swr3s https://www.youtube.com/watch?v=D64QD7Swr3s - multimodal understanding of audio * https://www.youtube.com/watch?v=v5tRc_5-8G4 https://www.youtube.com/watch?v=v5tRc_5-8G4 - helping a user with complex requests and showing some of the 'thinking' it is doing about what context it does/doesn't have * https://www.youtube.com/watch?v=sPiOP_CB54A https://www.youtube.com/watch?v=sPiOP_CB54A - assessing the relevance of scientific papers and then extracting data from the papers My current context: API user of OpenAI, regular user of ChatGPT Plus (GPT-4-Turbo, Dall E 3, and GPT-4V), occasional user of Claude Pro (much less since GPT-4-Turbo with longer context length), paying user of Midjourney. Gemini Pro is available starting today in Bard. It's not clear to me how many of the super impressive results are from Ultra vs Pro. Overall conclusion: Gemini Ultra looks very impressive. But - the timing is disappointing: Gemini Ultra looks like it won't be widely available until ~Feb/March 2024, or possibly later. > As part of this process, we’ll make Gemini Ultra available to select customers, developers, partners and safety and responsibility experts for early experimentation and feedback before rolling it out to developers and enterprise customers early next year. > Early next year, we’ll also launch Bard Advanced, a new, cutting-edge AI experience that gives you access to our best models and capabilities, starting with Gemini Ultra. I hope that there will be a product available sooner than that without a crazy waitlist for both Bard Advanced, and Gemini Ultra API. Also fingers crossed that they have good data privacy for API usage, like OpenAI does (i.e. data isn't used to train their models when it's via API/playground requests).

tikkun 3y ago
My general conclusion: Gemini Ultra > GPT-4 > Gemini Pro See Table 2 and Table 7 https://storage.googleapis.com/deepmind-media/gemini/gemini_1_report.pdf https://storage.googleapis.com/deepmind-media/gemini/gemini_... (I think they're comparing against original GPT-4 rather than GPT-4-Turbo, but it's not entirely clear) What they've released today: Gemini Pro is in Bard today. Gemini Pro will be coming to API soon (Dec 13?). Gemini Ultra will be available via Bard and API "early next year" Therefore, as of Dec 6 2023: SOTA API = GPT-4, still. SOTA Chat assistant = ChatGPT Plus, still, for everything except video, where Bard has capabilities . ChatGPT plus is closely followed by Claude. (But, I tried asking Bard a question about a youtube video today, and it told me "I'm sorry, but I'm unable to access this YouTube content. This is possible for a number of reasons, but the most common are: the content isn't a valid YouTube link, potentially unsafe content, or the content does not have a captions file that I can read.") SOTA API after Gemini Ultra is out in ~Q1 2024 = Gemini Ultra, if OpenAI/Anthropic haven't released a new model by then SOTA Chat assistant after Bard Advanced is out in ~Q1 2024 = Bard Advanced, probably, assuming that OpenAI/Anthropic haven't released new models by then

lkbm 3y ago

It's worth remembering that AI is more than LLMs. DeepMind is still doing big stuff: https://deepmind.google/discover/blog/millions-of-new-materials-discovered-with-deep-learning/ https://deepmind.google/discover/blog/millions-of-new-materi...

dpflan 3y ago
Indeed, I would think the core search product as another example of ai/ml...
- mpweiher 3y ago
  The question is whether greater use of AI correlates with the declining quality of search results.
  - visarga 3y ago
    I think the bottom line (profit) inversely correlates with the quality of search results. I've been using phind.com lately and it seems there can be search without junk even in this age. Google has lots of people tagging search rankings, which is very similar with RLHF ranking responses from LLMs. It's interesting that using LLMs with RLHF it is possible to de-junk the search results. RLHF is great for this task, as evidenced by its effect on LLMs.
    - mattmaroon 3y ago
      Right. It’s less that their declining quality of search results is due to AI and more that the AI got really good at monetizing and monetizing and quality search results are sometimes in opposition.
      - avereveard 3y ago
        This entire thread kinda ignore that they are also selling ad space on many sites and their objective function in ordering search is not just the best possible result. Case in point the many sites stealing stack overflow content and filling it with adverts ranking higher than the source, that committed the cardinal sin of running their own and network.

willsmith72 3y ago

they have to try something, otherwise it looks like they've been completely destroyed by a company of 1000 people

hinkley 3y ago

Sundar studied material science in school and is only slightly older than me. Google is a little over 25 years old. I guarantee you they have not been doing AI since I was a baby. And how many financial people worth reconning with are under 30 years old? Not many.

crossroadsguy 3y ago
Unless you are OpenAI, the company, I doubt OP implied it was aimed at you. But then I wouldn't know as I am much younger than Sundar Pichai and I am not on first name basis with him either ;-)

dist-epoch 3y ago

> show instead of tell They showed AlphaGo, they showed Transformers. Pretty good track record.

visarga 3y ago
That was ages ago. In AI even a week feels like a whole year in other fields. And many/most of those researchers have fled to startups, so those startups also have a right to brag. But not too much - only immediate access to a model beating GPT4 is worth bragging today (cloud), or getting GPT3.5 quality from a model running on a phone (edge). So it's either free-private-gpt3.5 or cloud-better-than-gpt4v. Nothing else matters now. I think we have reached an extreme point of temporal discounting (https://en.wikipedia.org/wiki/Time_preference https://en.wikipedia.org/wiki/Time_preference).
- jvolkman 3y ago
  > That was ages ago. I think that was the point.
- thefourthchime 3y ago
  The Transformer paper “Attention is All You Need” came out in 2017. Sundar got the CEO job two years earlier, so he was in CEO diapers at the time if you will. I would argue Google has done almost nothing interesting since then (at least not things they haven't killed)

vinniepukh 3y ago

No surprises here. Google DeepMind squandered their lead in AI so much that they now have to have “Google” prepended to their name to show that adults are now in charge.

password54321 3y ago
What an ugly statement. DeepMind has been very open with their research since the beginning because their objective was much more on making breakthroughs with moonshot projects than near term profit.
- vinniepukh 3y ago
  [dead]

misterbwong 3y ago

I also find that tone a bit annoying but I'm OK with it because it highlights how these types of bets, without an immediate benefit, can pay off very well in the long term, even for huge companies like Google. AI, as we currently know it, wasn't really a "thing" when Google started with it and the payoff wasn't clear. They've long had to defend their use of their own money for big R&D bets like this and only now is it really clearly "adding shareholder value". Yes, I know it was a field of interest and research long before Google invested, but the fact remains that they _did_ invest deeply in it very early on for a very long time before we got to this point. Their continued investment has helped push the industry forward, for better or worse. In light of this context, I'm ok with them taking a small victory lap and saying "we've been here, I told you it was important".

jeffbee 3y ago
> only now is it really clearly "adding shareholder value". AI has been adding a huge proportion of the shareholder value at Google for many years. The fact that their inference systems are internal and not user products might have hidden this from you.
deleted 3y ago
[deleted]

corethree 3y ago

Didn't Google invent LLMs and didn't Google have an internal LLm with similar capabilities long before openai released the gpts? Remember when that guy got fired for making a claim it was conscious ? The look isn't good. But it's not dishonest.

ma2rten 3y ago
No this is not correct. Arguably OpenAI invented LLMs with GPT3 and the preceding scaling laws paper. I worked on LAMDA, it came after GPT4 and was not as capable. Google did invent the transformer, but all the authors of the paper have left since.
- visarga 3y ago
  This is like the Ship of Theseus. If all transformer authors reunited under one startup rival of Google, who would have the bragging rights?
  - deleted 3y ago
    [deleted]
  - ma2rten 3y ago
    Noam.
- corethree 3y ago
  Isn't a transformer an LLM?
  - avarun 3y ago
    No. You’re confused. Most LLMs in production today use transformers under the hood but a transformer is most certainly not a type of LLM.
- deleted 3y ago
  [deleted]

dontupvoteme 3y ago

He's not wrong. DeepMind spends time solving big scientific / large-scale problems such as those in genetics, material science or weather forecasting, and Google has untouchable resources such as all the books they've scanned (and already won court cases about) They do make OpenAI look like kids in that regard. There is far more to technology than public facing goods/products. It's probably in part due to the cultural differences between London/UK/Europe and SiliconValley/California/USA.

deleted 3y ago
[deleted]
jahsome 3y ago
That statement isn't really directed at the people who care about the scientific or tech-focused capabilities. I'd argue the majority of those folks interested in those things already know about DeepMind. This statement is for the mass market MBA-types. More specifically, middle managers and dinosaur executives who barely comprehend what generative AI is, and value perceived stability and brand recognition over bleeding edge, for better or worse. I think the sad truth is an enormous chunk of paying customers, at least for the "enterprise" accounts, will be generating marketing copy and similar "biz dev" use cases.
roguas 3y ago
Oh it's good they working on important problems with their ai. Its just openai was working on my/our problems (or providing tools to do so) and that's why people are more excited about them. Not because of cultural differences. If you are more into weather forecasting, yeah it sure may be reasonable to prefer google more.
- faramarz 3y ago
  That’s what makes Altman a great leader. He understands marketing better than many of these giants. Google got caught being too big. Sure they will argue that AI mass release is a dangerous proposition, but Sam had to make a big splash otherwise he would be competing with incumbent marketing spendings far greater than OpenAI could afford. It was a genius move to go public with a simple UI. No matter how stunning the tech side is, if human interaction is not simple, the big stuff doesn’t even matter.

mattmaroon 3y ago

I do think that’s a backfire. Telling me how long you’ve been doing something isn’t that impressive if the other guy has been doing it for much less time and is better at it. It’s in fact the opposite.

pb7 3y ago
Not if the little guy leveraged your inventions/research.
- dragonwriter 3y ago
  That's even worse: what it says is that you are getting beat at product even where you create the tech. Which is definitely where Google is in the generative AI space.
- fennecbutt 3y ago
  Weird for us to personify a corporation like that tbh. Google didn't invent transformers, researchers working at Google did. Sure Google paid em money/employed em, but the smarts behind it isn't the entity Google or the execs at the top, Sundar etc; it's those researchers. I like to appreciate individualism in a world where those at the top have lobbied their way into a 1% monopoly lmao.
  - pb7 3y ago
    This whole conversation is weird. First, OpenAI is not some "little guy". It's a bigger corporation than 99.99% of companies that have ever existed. It's like calling Airbnb or Uber "little guys". Second, yes, the researchers did that but the company funded it with no guarantee of return. Your argument can be applied to any company or organization; it's needless pedantry.
    - Solvency 3y ago
      770 employees is bigger than 99.9% off all companies ever? Because we saw how easily those employees would have walked out the front door only too recently.

bogwog 3y ago

> Sundar's comments in the main video seem like he's trying to communicate "we've been doing this ai stuff since you (other AI companies) were little babies" - to me this comes off kind of badly Reminds me of the Stadia reveal, where the first words out of his mouth were along the lines of "I'll admit, I'm not much of a gamer" This dude needs a new speech writer.

cmrdporcupine 3y ago
This dude needs a new speech writer. How about we go further and just state what everyone (other than Wall St) thinks: Google needs a new CEO. One more interested in Google's supposed mission ("to organize the world's information and make it universally accessible and useful"), than in Google's stock price.
- whatshisface 3y ago
  "The shareholders are too well-represented" is one of the complaints least likely to be acted on, second only to "I'm getting old."
  - cmrdporcupine 3y ago
    Sure, anti-capitalist-impulse Me 100% agrees. It's just doubly icky in the context of Google's original "letter to shareholders": https://www.nytimes.com/2004/04/29/business/letter-from-the-founders.html https://www.nytimes.com/2004/04/29/business/letter-from-the-... ("Google is not a conventional company. We do not intend to become one")
    - avarun 3y ago
      Haha that’s a great find. Google may be one of the most conventional companies out there now.
      - whatshisface 3y ago
        "One of the most conventional companies" sounds like some kind of paradox.

FrustratedMonky 3y ago

Well, deepmind was doing amazing stuff before OpenAI. AlphaGo, AlphaFold, AlphaStar. They were groundbreaking a long time ago. They just happened to miss the LLM surge.

headcanon 3y ago

Its a conceit but not unjustified, they have been doing "AI" since their inception. And yeah, Sundar's term up until recently seems to me to be milking existing products instead of creating new ones, so it is a bit annoying when they act like this was their plan the whole time. Google's weakness is on the product side, their research arm puts out incredible stuff as other commenters have pointed out. GPT essentially came out from Google researchers that were impatient with Google's reluctance to ship a product that could jeopardize ad revenue on search.

radicaldreamer 3y ago
It's such a short sighted approach too because I'm sure someone will develop a GPT with native advertising and it'll be a blockbuster because it'll be free to use but also have strong revenue generating potential.
- TerrifiedMouse 3y ago
  One problem I see with ChatGPT replacing search is, who is going to fund the content creators that generate the training data?
- vitorgrs 3y ago
  Bing Chat/Copilot already have ads and is free.
nonethewiser 3y ago
The point is if you have to remind people then you’re doing something wrong. The insight to draw from this is not that everyone else is misinformed about googles abilities (the implication), its that Google has not capitalized on their resources.

StevenNunez 3y ago

This demo is blowing my mind! It's really incredible. Can't wait to play around with them.

risyachka 3y ago

Google literally invented transformers that are at the core of all current AI/LLMs so Sundar's comment is very accurate.

tempnow987 3y ago
But their first efforts in BARD were really not great. I'd just have left the bragging out in terms of how long. OpenAI and others have no doubt sent a big wakeup call to google. For a while it seemed like they had turned to focus an AI "safety" (remembering some big blowups on those teams as well) with papers about how AI might develop negative stereotypes (ie, men commit more violent crime then women?). That seems to have changed - this is very product focused, and I asked it some questions that in many models are screened out for "safety" and it responded which is almost even more surprising (ie. Statistically who commits more violent crime, men or women).
- hughesjj 3y ago
  The big concern was biased datasets iirc and shit fits for people of color. Like clearly mislabeling feminine looking women as men, and a stupid high false positive rate for face detection. That was relevant given they were selling their models to law enforcement.
dekhn 3y ago
Sundar's comments about Google doing AI (really ML) are based more on things that people externally know very little about. Systems like SETI, Sibyl, RePhil, SmartASS. These were all production ML systems that used fairly straightforward and conventional ML combined with innovative distributed computing and large-scale infrastructure to grow Google's product usage significantly over the past 20 years. For example here's a paper 10 years old now: https://static.googleusercontent.com/media/research.google.com/en/us/pubs/archive/41159.pdf https://static.googleusercontent.com/media/research.google.c... and another close to 10 years old now: https://research.google/pubs/pub43146/ https://research.google/pubs/pub43146/ The learning they expose in those papers came from the previous 10 years of operating SmartASS. However, SmartASS and sibyl weren't really what external ML people wanted- it was just fairly boring "increase watch time by identifying what videos people wioll click on" and "increase mobile app installs" or "show the ads people are likely to click on". It really wasn't until vincent vanhoucke stuffed a bunch of GPUs into a desktop and demonstrated scalable and dean/ng built their cat detector NN that google started being really active in deep learning. That was around 2010-2012.

lossolo 3y ago

> we've been doing this ai stuff since you (other AI companies) were little babies Actually, they kind of did. What's interesting is that they still only match GPT-4's version but don't propose any architectural breakthroughs. From an architectural standpoint, not much has changed since 2017. The 'breakthroughs', in terms of moving from GPT to GPT-4, included: adding more parameters (GPT-2/3/4), fine-tuning base models following instructions (RLHF), which is essentially structured training (GPT-3.5), and multi-modality, which involves using embeddings from different sources in the same latent space, along with some optimizations that allowed for faster inference and training. Increasing evidence suggests that AGI will not be attainable solely using LLMs/transformers/current architecture, as LLMs can't extrapolate beyond the patterns in their training data (according to a paper from DeepMind last month): "Together our results highlight that the impressive ICL abilities of high-capacity sequence models may be more closely tied to the coverage of their pretraining data mixtures than inductive biases that create fundamental generalization capabilities."[1] 1. https://arxiv.org/abs/2311.00871 https://arxiv.org/abs/2311.00871

alaskamiller 3y ago
In short: a chat bot is not AI.

schleck8 3y ago

They always do this, every time they get to mention AI. It appears somewhat desperate imo.

ugh123 3y ago

> One observation: Sundar's comments in the main video seem like he's trying to communicate "we've been doing this ai stuff since you (other AI companies) Sundar has been saying this repeatedly since Day 0 of the current AI wave. It's almost cliche for him at this point.

dragonwriter 3y ago
And he's going to keep saying it to tell investors why they should believe Google will eventually catch up in product until Google does catch up in product and he doesn't need to say it anymore. Or until Google gives up on the space, or he isn't CEO, if either of those come first, which I wouldn't rule out.
xnx 3y ago
Sundar announced his intentions to lead Google as an "AI first" company in May 2017: https://blog.google/technology/ai/making-ai-work-for-everyone/ https://blog.google/technology/ai/making-ai-work-for-everyon...

ac1spkrbox 3y ago

“Any man who must say ‘I am the king’ is no true King”

DonHopkins 3y ago
Any man who must say "I won't be a dictator, except for day one" will be a permanent dictator. https://eu.usatoday.com/story/news/politics/elections/2023/12/05/trump-dictator-day-one/71816204007/ https://eu.usatoday.com/story/news/politics/elections/2023/1...

irthomasthomas 3y ago

They played the same tune at that panel with Sam Altman the night before he was fired. https://youtu.be/ZFFvqRemDv8 https://youtu.be/ZFFvqRemDv8 He mentions Transformers - fine. Then he says that we've all been using Google AI for so long with Google Translate.

jonplackett 3y ago

I find this video really freaky. It’s like Gemini is a baby or very young child and also a massively know it all adult that just can’t help telling how clever it is and showing off its knowledge. People speak of the uncanny valley in terms of appearance. I am getting this from Gemini. It’s sort of impressive but feels freaky at the same time. Is it just me?

nolist_policy 3y ago
https://www.youtube.com/watch?v=PJgo3BBgWDA https://www.youtube.com/watch?v=PJgo3BBgWDA
kromem 3y ago
No, there's an odd disconnect between the impressiveness of the multimodal capabilities vs the juvenile tone and insights compared to something like GPT-4 that's very bizarre in application. It is a great example of what I've been finding a growing concern as we double down on Goodhart's Law with the "beats 30 out of 32 tests compared to existing models." My guess is those tests are very specific to evaluations of what we've historically imagined AI to be good at vs comprehensive tests of human ability and competencies. So a broad general pretrained model might actually be great at sounding 'human' but not as good at logic puzzles, so you hit it with extensive fine tuning aimed at improving test scores on logic but no longer target "sounding human" and you end up with a model that is extremely good at what you targeted as measurements but sounds like a creepy toddler. We really need to stop being so afraid of anthropomorphic evaluation of LLMs. Even if the underlying processes shouldn't be anthropomorphized, the expressed results really should be given the whole point was modeling and predicting anthropomorphic training data. "Don't sound like a creepy soulless toddler and sound more like a fellow human" is a perfectly appropriate goal for an enterprise scale LLM, and we shouldn't be afraid of openly setting that as a goal.

jiggawatts 3y ago

That was pretty impressive… but do I have to be “that guy” and point out the error it made? It said rubber ducks float because they’re made of a material less dense than water — but that’s not true! Rubber is more dense than water. The ducky floats because it’s filled with air. If you fill it with water it’ll sink. Interestingly, ChatGPT 3.5 makes the same error, but GPT 4 nails it and explains the it’s the air that provides buoyancy. I had the same impression with Google’s other AI demos: cute but missing something essential that GPT 4 has.

zyxin 3y ago
I would've liked to see an explanation that includes the weight of water being displaced. That would also explain how a steel ship with an open top is also able to float.
scoot 3y ago
I spotted that too, but also, it didn't recognise the "bird" until it had feet, when it is supposedly better than a human expert. I don't doubt that the examples were cherry-picked, so if this is the best it can do, it's not very convincing.

choppaface 3y ago

> A better look would simply be to show instead of tell. Completely! Just tried Bard. No images and the responses it gave me were pretty poor. Today's launch is a weak poor product launch, looks mostly like a push to close out stuff for Perf and before everybody leaves for the rest of the December for vacation.

neop1x 3y ago

A simple REST API with a static token auth like OpenAI API would help. Previously when I tried Bard API it was refusing to accept token auth, requiring that terrible oauth flow so I gave up.

OJFord 3y ago

Incredible stuff, and yet TTS is still so robotic. Frankly I assume it must be deliberate at this point, or at least deliberate that nobody's worked on it because it's comparatively easy and dull? (The context awareness of the current breed of generative AI seems to be exactly what TTS always lacks, awkward syllables and emphasis, pronunciation that would be correct sometimes but not after that word, etc.)

jongjong 3y ago

Yes it sounds like a conspiracy theory about government and big tech working on advanced tech which has existed for decades but kept secret.

nothrowaways 3y ago

SOTA is made by an ex Google employee. So their argument still holds.