6 ms·
Massive whoa if true from technical report "Studying the limits of Gemini 1.5 Pro's long-context ability, we find continued improvement in next-token predictio
by alphabetting 3y ago
Massive whoa if true from technical report
"Studying the limits of Gemini 1.5 Pro's long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>99%) up to at least 10M tokens"
https://storage.googleapis.com/deepmind-media/gemini/gemini_v1_5_report.pdf https://storage.googleapis.com/deepmind-media/gemini/gemini_...
- stavros 3y agoUntil I can talk to it, I care exactly zero.
- peterisza 3y agoyou can buy their stock if you think they'll make a lot of money with their tech
- HarHarVeryFunny 3y agoWell that's really the right question .. what can, and will, Google do with this that can move their corporate earnings needle in a meaningful way? Obviously they can sell API access and integrate it into their Google docs suite, as well as their new Project IDX IDE, but do any of these have potential to make a meaningful impact ? It's also not obvious how these huge models will fare against increasingly capable open source ones like Mixtral, perhaps especially since Google are confirming here that MoE is the path forward, which perhaps helps limit how big these models need to be.
- plaidfuji 3y agoIn the long run it could move the needle in enterprise market share of Workspace and GCP. They have a lot of room to grow and IMO have a far superior product to O365/Azure which could be exacerbated by strong AI products. Only problem is this sales cycle can take a decade or more, and Google hasn’t historically been patient or strategic about things like this.
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- megaman821 3y agoSo, will this outperform any RAG approach as long as the data fits inside the context window?
- saliagato 3y agobasically, yes. Pinecone? Dead. Azure AI Search? Dead. Quadrant? Dead.
- _boffin_ 3y agoPrompt token cost still a variable.
- ArcaneMoose 3y agoCost would still be a big concern
- TheGeminon 3y agoOutperform is dependent on the RAG approach (and this would be a RAG approach anyways, you can already do this with smaller context sizes). A simplistic one, probably, but dumping in data that you don't need dilutes the useful information, so I would imagine there would be at least _some_ degradation. But there is also the downside of "tuning" the RAG to return less tokens you will miss extra context that could be useful to the model.
- megaman821 3y agoDoesn't their needle/haystack benchmark seem to suggest there is almost no dilution? They pushed that demo out to 10M tokens.
- CuriouslyC 3y agoA perfect RAG system would probably outperform everything in a larger context due to prompt dilution, but in the real world putting everything in context will win a lot of the time. The large context system will also almost certainly be more usable due to elimination of retrieval latency. The large context system might lose on price/performance though.
- Workaccount2 3y ago10M tokens is absolutely jaw dropping. For reference, this is approximately thirty books of 500 pages each. Having 99% retrieval is nuts too. Models tend to unwind pretty badly as the context (tokens) grows. Put these together and you are getting into the territory of dumping all your company documents, or all your departments documents into a single GPT (or whatever google will call it) and everyone working with that. Wild.
- kranke155 3y agoSeems like Google caught up. Demis is again showing an incredible ability to lead a team to make groundbreaking work.
- huytersd 3y agoIf any of this is remotely true, not only did it catch up, it’s wiping the floor with how useful it can be compared to GPT4. Not going to make a judgement until I can actually try it out though.
- singularity2001 3y agoIn the demo videos gemini needs about a minute to answer long context questions. Which is better than reading thousands of pages yourself. But if it has to compete with classical search and skimming it might need some optimization.
- huytersd 3y agoThat’s a compute problem, something that involves just throwing money at the problem.
- a_wild_dandan 3y agoReplacing grep or `ctrl+F` with Gemini would be the user's fault, not Gemini's. If classical search for a job already a performant solution, use classical search. Save your tokens for jobs worthy of solving with a general intelligence!
- matsemann 3y agoCould you (or someone) explain what this means?
- FergusArgyll 3y agoThe input you give it can be very long. This can qualitatively change the experience. Imagine, for example, copy pasting the entire lord of the rings plus another 100 books you like and asking it to write a similar book...
- teaearlgraycold 3y agoI doubt it’s smart enough to write another (coherent, good) book based on 103 books. But you could ask it questions about the books and it would search and synthesize good answers.
- HarHarVeryFunny 3y agoI just googled it, and the LOTR trilogy apparently has a total of 480,000 words, which brings home how huge 1M is! It'd be fascinating to see how well Gemini could summarize the plot or reason about it. One point I'm unclear on is how these huge context sizes are implemented by the various models. Are any of them the actual raw "width of the model" that is propagated through it, or are these all hierarchical summarization and chunk embedding index lookup type tricks?
- mburns 3y agoFor another reference, Shakespeare’s complete works are ~885k words. The Encyclopedia Britannica is ~44M words.
- staticman2 3y agoReading Lord of the Rings, and writing a quality book in the same style, are almost wholly unrelated tasks. Over 150 million copies of Lord of the Rings have been sold, but few readers are capable of "writing a similar book" in terms of quality. There's no reason to think this would work well.
- famouswaffles 3y agoAnother whoa for me >Finally, we highlight surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person learning from the same content. Results - https://imgur.com/a/qXcVNOM https://imgur.com/a/qXcVNOM
- usaar333 3y agoI think this somewhat is mostly due to the ability to handle high context lengths better. Note how Claude 2.1 already highly outperforms GPT-4 on this task.
- a_wild_dandan 3y agoGPT-4V turbo outperforms Claude on long contexts, IIRC. Unless that's mistaken, I'd suspect a different explanation for that task.
- cchance 3y agoDid you watch the video of the Gemini 1.5 video recall after it processed the 44 minute video... holy shit
- deleted 3y ago[deleted]