6 ms·
So, better than GPT4 according to the benchmarks? Looks very interesting. Technical paper: https://goo.gle/GeminiPaper https://goo.gle/GeminiPaper Some detail
by albertzeyer 3y ago
So, better than GPT4 according to the benchmarks? Looks very interesting.
Technical paper: https://goo.gle/GeminiPaper https://goo.gle/GeminiPaper
Some details:
- 32k context length
- efficient attention mechanisms (for e.g. multi-query attention (Shazeer, 2019))
- audio input via Universal Speech Model (USM) (Zhang et al., 2023) features
- no audio output? (Figure 2)
- visual encoding of Gemini models is inspired by our own foundational work on Flamingo (Alayrac et al., 2022), CoCa (Yu et al., 2022a), and PaLI (Chen et al., 2022)
- output images using discrete image tokens (Ramesh et al., 2021; Yu et al., 2022b)
- supervised fine tuning (SFT) and reinforcement learning through human feedback (RLHF)
I think these are already more details than what we got from OpenAI about GPT4, but on the other side, still only very little details.
- ilaksh 3y agoThat's for Ultra right? Which is an amazing accomplishment, but it sounds like I won't be able to access it for months. If I'm lucky.
- verdverm 3y agoThere was a waiting period for ChatGPT4 as well, particularly direct API access, and the WebUI had (has?) a paywall
- freedomben 3y agoYep, at this point I'd rather they hold their announcements until everybody can access it, not just the beautiful people. I'm excited and want to try it right now, and would actually use it for a PoC I have in mind, but in a few months the excitement will be gone.
- jakderrida 3y agoIt's to their detriment, also. Being told Gemini beats GPT-4 while withholding that what I'm trying out is not the model they're talking about would have me think they're full of crap. They'd be better off making it clear that this is not the one that surpasses GPT-4.
- throwitaway222 3y agoIt really is. OpenAI has the Apple model of release - when it's announced the laptop is in you freaking hands 3 days later. Google announces vaporware that's never going to come out, or something that will be out in 5 months. It's frustrating and very bad for their image in the LLM space.
- EchoReflection 3y agoI wonder if the "release" was done in spite of dev knowledge that it isn't really ready. Like "screw it, we want to attract eyeballs even though we know it's premature"
- fl0ki 3y agoThis might be the best they can do to maintain any hope among nervous investors. That this may actually be the most rational play available to them would be incredibly sad.
- thatoneguy 3y agoAnd then six months later, it's gone because it's not on target for a $1B ARR and no one is getting promoted for something that's already launched.
- interloxia 3y agoThat wasn't my experience with OpenAI. Their availability has been a mixed bag.
- pixl97 3y agoJust because Apple offers a particular laptop, doesn't mean they won't be sold out. If we want to complete the analogy.
- Maxion 3y agoYep, the announcement is quite cheeky. Ultra is out sometime next year, with GPT-4 level capability. Pro is out now (?) with ??? level capability.
- HlessClaudesman 3y ago??? Capability, sometime next year, welcome to the Gemini era.
- KaoruAoiShiho 3y agoPro benchmarks are here: https://storage.googleapis.com/deepmind-media/gemini/gemini_1_report.pdf https://storage.googleapis.com/deepmind-media/gemini/gemini_... Sadly it's 3.5 quality, :(
- Maxion 3y agoLol that's why it's hidden in a PDF. They basically announced GPT 3.5, then. Big woop, by the time Ultra is out GPT-5 is probably also out.
- satchlj 3y agoYup, it's all a performance for the investors
- Racing0461 3y ago+1. The investors are the customers of this release, not end users.
- dmix 3y agoIsn't having GPT 3.5 still a pretty big deal? Obviously they are behind but does anyone else offer that? 3.5 is still highly capable and Google investing a lot into making it multi modal combined with potential integration with their other products makes it quite valuable. Not everyone likes having to switch to ChatGPT for queries.
- onlyrealcuzzo 3y agoThe article says "next year" - so that could be as soon as January, right?
- behnamoh 3y agoI hate this "tierification" of products into categories: normal, pro, max, ultra Apple does this and it's obvious that they do it to use the "decoy effect" when customers want to shop. Why purchase a measly regular iPhone when you can spend a little more and get the Pro version? But when it comes to AI, this tierification only leads to disappointment—everyone expects the best models from the FAANGO (including OpenAI), no one expects Google or OpenAI to offer shitty models that underperform their flagships when you can literally run Llama 2 and Mistral models that you can actually own.
- giovannibonetti 3y agoI think the expensive ones are used when the customer is the user — e.g. ChatGPT Plus (personal) subscription — and the cheap ones when they are not — e.g. customer support service bots.
- pphysch 3y agoThis isn't "tierificaton" or even premiumization. That may come later. Large AI models have tight resources requirements. You physically can't use X billion parameters without ~X billion ~bytes of memory. It makes complete sense to have these 3 "tiers". You have a max capability option, a price-performance scaling option, and an edge compute option.
- dragonwriter 3y ago> Large AI models have tight resources requirements. You physically can't use X billion parameters without ~X billion ~bytes of memory. Well, X billion bits times the parameter bit size. For base models, those are generally 32-bit (so 4X bytes), though smaller quantizations ate possible and widely used for public models, and I would assume as a cost measure for closed hosted models as well.
- pphysch 3y agoHence ~
- 3y ago
- arnaudsm 3y agoI miss when ML scientific papers had actual science in them. Now they all feel like ads.
- behnamoh 3y agoThat's because they're not "scientific papers", they're technical papers.
- HlessClaudesman 3y agoIt's aimed at the AI pontification industrial complex.
- miraculixx 3y agoIf it feals like a duck it is a duck. Duh(ck)
- Palmik 3y agoThe table is *highly* misleading. It uses different methodologies all over the place. For MMLU, it highlights the CoT @ 32 result, where Ultra beats GPT4, but it loses to GPT4 with 5-shot, for example. For GSM8K it uses Maj1@32 for Ultra and 5-shot CoT for GPT4, etc. Then also, for some reason, it uses different metrics for Ultra and Pro, making them hard to compare. What a mess of a "paper".
- Imnimo 3y agoIt really feels like the reason this is being released now and not months ago is that that's how long it took them to figure out the convoluted combination of different evaluation procedures to beat GPT-4 on the various benchmarks.
- deleted 3y ago[deleted]
- eurekin 3y agoThis, and also building the marketing website. It feels really desperate
- red-iron-pine 3y ago"we have no moat"
- visarga 3y agoEven not having a moat anymore, with their cash they might still be the biggest search provider 10 years from now. IBM still exists and is worth 146B. I wouldn't be surprised if Google still came out ok. Assuming they use unique data only they have to make a better LLM, then everyone is going to leech training examples from them bringing competition asymptotically closer, but never quite reaching. It's hard to copy-protect a model exposed to the public, as OpenAI is finding out. Many, many tasks can be executed on local GPUs today without paying a dime to OpenAI, there is no moat. AI likes to learn from other AIs. Give me a million hard problems solved step by step with GPT-5 and I can make Mistral much smarter. Everyone knows this dataset is going to leak in a few months.
- pcshah1996 3y agoSome discussion on twitter about misleading evaluation: https://twitter.com/a_a_cabrera/status/1732454328307511807 https://twitter.com/a_a_cabrera/status/1732454328307511807 (nitter: https://nitter.net/a_a_cabrera/status/1732454328307511807#m https://nitter.net/a_a_cabrera/status/1732454328307511807#m)
- yieldcrv 3y agoI wish Google’s UI would have the same chat interface as ChatGPT or even command line ones I’ve encountered
- deleted 3y ago[deleted]