10 ms·

The white paper is worth a read. The things that stand out to me are: 1. They don't talk about how they get to 10M token context 2. They don't talk about how

The white paper is worth a read. The things that stand out to me are: 1. They don't talk about how they get to 10M token context 2. They don't talk about how they get to 10M token context 3. The 10M context ability wipes out most RAG stack complexity immediately. (I imagine creating caching abilities is going to be important for a lot of long token chatting features now, though). This is going to make things much, much simpler for a lot of use cases. 4. They are pretty clear that 1.5 Pro is better than GPT-4 in general, and therefore we have a new LLM-as-judge leader, which is pretty interesting. 5. It seems like 1.5 Ultra is going to be highly capable. 1.5 Pro is already very very capable. They are running up against very high scores on many tests, and took a minute to call out some tests where they scored badly as mostly returning false negatives. Upshot, 1.5 Pro looks like it should set the bar for a bunch of workflow tasks, if we can ever get our hands on it. I've found 1.0 Ultra to be very capable, if a bit slow. Open models downstream should see a significant uptick in quality using it, which is great. Time to dust out my coding test again, I think, which is: "here is a tarball of a repository. Write a new module that does X". I really want to know how they're getting to 10M context, though. There are some intriguing clues in their results that this isn't just a single ultra-long vector; for instance, their audio and video "needle" tests, which just include inserting an image that says "the magic word is: xxx", or an audio clip that says the same thing, have perfect recall across up to 10M tokens. The text insertion occasionally fails. I'd speculate that this means there is some sort of compression going on; a full video frame with text on it is going to use a lot more tokens than the text needle.

deleted 3y ago
[deleted]
CharlieDigital 3y ago
> The 10M context ability wipes out most RAG stack complexity immediately. Remains to be seen. Large contexts are not always better. For starters, it takes longer to process. But secondly, even with RAG and the large context of GPT4 Turbo, providing it a more relevant and accurate context always yields better output. What you get with RAG is faster response times and more accurate answers by pre-filtering out the noise.
- behnamoh 3y ago
  Don't forget that Gemini also has access to the internet, so a lot of RAGging becomes pointless anyway.
  - beppo 3y ago
    Internet search is a form of RAG, though. 10M tokens is very impressive, but you're not fitting a database, let alone the entire internet into a prompt anytime soon.
    - behnamoh 3y ago
      You shouldn't fit an entire database in the context anyway. btw, 10M tokens is 78 times more context window than the newest GPT-4-turbo (128K). In a way, you don't need 78 GPT-4 API calls, only one batch call to Gemini 1.5.
      - rvnx 3y ago
        Well it's nice, just sad nobody can use it
      - cchance 3y ago
        I don't get this why is it people think that you need to put an entire database in the short-term memory of the AI to be useful? When you work with a DB are you memorizing the entire f*cking database, no, you know the summaries of it and how to access and use it. People also seem to forget that the average is 1b words that are read by people in their entire LIFETIME, and at 10m, with nearly 100% recall thats pretty damn amazing, i'm pretty sure I don't have perfect recall of 10m words myself lol
        Qwero 3y ago
        It increases the use cases. It can also be a good alternative for fine-tuning. And the use case of a code base is a good example: if the ai understands the whole context, it can do basically everything. Let me pay 5€ for a android app rewritten into iOS.
deleted 3y ago
[deleted]
cs702 3y ago
> 1. They don't talk about how they get to 10M token context > 2. They don't talk about how they get to 10M token context Yes. I wonder if they're using a "linear RNN" type of model like Linear Attention, Mamba, RWKV, etc. Like Transformers with standard attention, these models train efficiently in parallel, but their compute is O(N) instead of O(N²), so in theory they can be extended to much longer sequences much efficiently. They have shown a lot of promise recently at smaller model sizes. Does anyone here have any insight or knowledge about the internals of Gemini 1.5?
- sebzim4500 3y ago
  The fact they are getting perfect recall with millions of tokens rules out any of the existing linear attention methods.
  - cs702 3y ago
    I wouldn't be so sure perfect recall rules out linear RNNs, because I haven't seen any conclusive data on their ability to recall. Have you?
- candiodari 3y ago
  They do give a hint: "This includes making Gemini 1.5 more efficient to train and serve, with a new Mixture-of-Experts (MoE) architecture." One thing you could do with MoE is giving each expert different subsets of the input tokens. And that would definitely do what they claim here: it would allow search. You want to find where someone said "the password is X" in a 50 hour audio file, this would be perfect. If your question is "what is the first AND last thing person X said" ... it's going to suck badly. Anything that requires taking 2 things into account that aren't right next to eachother is just not going to work.
  - declaredapple 3y ago
    > One thing you could do with MoE is giving each expert different subsets of the input tokens. Don't MoE's route tokens to experts after the attention step? That wouldn't solve the n^2 issue the attention step has. If you split the tokens before the attention step, that would mean those tokens would have no relationship to each other - it would be like inferring two prompts in parallel. That would defeat the point of a 10M context
usaar333 3y ago
> They are pretty clear that 1.5 Pro is better than GPT-4 in general, and therefore we have a new LLM-as-judge leader, which is pretty interesting. They try to push that, but it's not the most convincing. Look at Table 8 for text evaluations (math, etc.) - they don't even attempt a comparison with GPT-4. GPT-4 is higher than any Gemini model on both MMLU and GSM8K. Gemini Pro seems slightly better than GPT-4 original in Human Eval (67->71). Gemini Pro does crush naive GPT-4 on math (though not with code interpreter and this is the original model). All in 1.5 Pro seems maybe a bit better than 1.0 Ultra. Given that in the wild people seem to find GPT-4 better for say coding than Gemini Ultra, my current update is Pro 1.5 is about equal to GPT-4. But we'll see once released.
- cchance 3y ago
  I mean i don't see GPT4 watching a 44 minute movie and being able to exactly pinpoint a guy taking a paper out of his pocket..
- panarky 3y ago
  > people seem to find GPT-4 better for say coding than Gemini Ultra For my use cases, Gemini Ultra performs significantly better than GPT-4. My prompts are long and complex, with a paragraph or two about the general objective followed by 15 to 20 numbered requirements. Often I'll include existing functions the new code needs to work with, or functions that must be refactored to handle the new requirements. I took 20 prompts that I'd run with GPT-4 and fed them to Gemini Ultra. Gemini gave a clearly better result in 16 out of 20 cases. Where GPT-4 might miss one or two requirements, Gemini usually got them all. Where GPT-4 might require multiple chat turns to point out its errors and omissions and tell it to fix them, Gemini often returned the result I wanted in one shot. Where GPT-4 hallucinated a method that doesn't exist, or had been deprecated years ago, Gemini used correct methods. Where GPT-4 called methods of third-party packages it assumed were installed, Gemini either used native code or explicitly called out the dependency. For the 4 out of 20 prompts where Gemini did worse, one was a weird rejection where I'd included an image in the prompt and Gemini refused to work with it because it had unrecognizable human forms in the distance. Another was a simple bash script to split a text file, and it came up with a technically correct but complex one-liner, while GPT-4 just used split with simple options to get the same result. For now I subscribe to both. But I'm using Gemini for almost all coding work, only checking in with GPT-4 when Gemini stumbles, which isn't often. If I continue to get solid results I'll drop the GPT-4 subscription.
swalsh 3y ago
"The 10M context ability wipes out most RAG stack complexity immediately." I'm skeptical, my past experience is just becaues the context has room to stuff whatever you want in it, the more you stuff in the context the less accurate your results are. There seems to be this balance of providing enough that you'll get high quality answers, but not too much that the model is overwhelmed. I think a large part of developing better models is not just a better architectures that support larger and larger context sizes, but also capable models that can properly leverage that context. That's the test for me.
- HereBePandas 3y ago
  They explicitly address this in page 11 of the report. Basically perfect recall for up to 1M tokens; way better than GPT-4.
  - westoncb 3y ago
    I don't think recall really addresses it sufficiently: the main issue I see is answers getting "muddy". Like it's getting pulled in too many directions and averaging.
    - a_wild_dandan 3y ago
      I'd urge caution in extending generalizations about "muddiness" to a new context architecture. Let's use the thing first.
      - westoncb 3y ago
        I'm not saying it applies to the new architecture, I'm saying that's a big issue I've observed in existing models and that so far we have no info on whether it's solved in the new one (i.e. accurate recall doesn't imply much in that regard).
        westoncb 3y ago
        Would be awesome if it is solved but seems like a much deeper problem tbh.
freedomben 3y ago
Is 10M token context correct? The blog post I see 1M but I'm not sure if these are different things Edit: Ah, I see, it's 1M reliably in production, up to 10M in research: > Through a series of machine learning innovations, we’ve increased 1.5 Pro’s context window capacity far beyond the original 32,000 tokens for Gemini 1.0. We can now run up to 1 million tokens in production. > This means 1.5 Pro can process vast amounts of information in one go — including 1 hour of video, 11 hours of audio, codebases with over 30,000 lines of code or over 700,000 words. In our research, we’ve also successfully tested up to 10 million tokens.
- huytersd 3y ago
  I know how I’m going to evaluate this model. Upload my codebase and ask it to “find all the bugs”.
- p1esk 3y ago
  How could one hour of video fit in 1M tokens? 1 hour at 30fps is 3600*30=100k frames. Each frame is converted in 256 tokens. So either they are not processing each frame, or each frame is converted into fewer tokens.
  - KTibow 3y ago
    The model can probably perform fine at 1 frame per second (3600*256=921600 tokens), and they could probably use some sort of compression.
tbruckner 3y ago
How do you know it isn't RAG?
tveita 3y ago
> The 10M context ability wipes out most RAG stack complexity immediately. The video queries they show take around 1 minute each, this probably burns a ton of GPU. I appreciate how clearly they highlight that the video is sped up though, they're clearly trying to avoid repeating the "fake demo" fiasco from the original Gemini videos.
theGnuMe 3y ago
For #1 and #2 it is some version of mixture of experts. This is mentioned in the blog post. So each expert only sees a subset of the tokens. I imagine they have some new way to route tokens to the experts that probably computes a global context. One scalable way to compute a global context is by a state space model. This would act as a controller and route the input tokens to the MoEs. This can be computed by convolution if you make some simplifying assumptions. They may also still use transformers as well. I could be wrong but there are some Mamba-MoEs papers that explore this idea.
resouer 3y ago
> The 10M context ability wipes out most RAG stack complexity immediately. This may not be true. My experience of the complexity of RAG lays in how to properly connect to various unstructured data sources and perform data transformation pipeline for large scale data set (which means GB, TB or even PB). It's in the critical path rather a "nice to have", because the quality of data and the pipeline is a major factor for the final generated the result. i.e., in RAG, the importance of R >>> G.
jorvi 3y ago
I just hope at some point we get access to mostly uncensored models. Both GPT-4 and Gemini are extremely shackled, and a slightly inferior model that hasn’t been hobbled by a very restricting preprompt would handily outperform them.
- ShamelessC 3y ago
  You can customize the system prompt with ChatGPT or via the completions API, just fyi.
ren_engineer 3y ago
RAG would still be useful for cost savings assuming they charge per token, plus I'm guessing using the full-context length would be slower than using RAG to get what you need for a smaller prompt
- nostrebored 3y ago
  This is going to be the real differentiator. HN is very focused on technical feasibility (which remains to be seen!), but in every LLM opportunity, the CIO/CFO/CEO are going to be concerned with the cost modeling. The way that LLMs are billed now, if you can densely pack the context with relevant information, you will come out ahead commercially. I don't see this changing with the way that LLM inference works. Maybe this changes with managed vector search offerings that are opaque to the user. The context goes to a preprocessing layer, an efficient cache understands which parts haven't been embedded (new bloom filter use case?), embeds the other chunks, and extracts the intent of the prompt.
  - mediaman 3y ago
    Agreed with this. The leading ability AI (in terms of cognitive power) will, generally, cost more per token than lower cognitive power AI. That means that at a given budget you can choose more cognitive power with fewer tokens, or less cognitive power with more tokens. For most use cases, there's no real point in giving up cognitive power to include useless tokens that have no hope of helping with a given question. So then you're back to the question of: how do we reduce the number of tokens, so that we can get higher cognitive power? And that's the entire field of information retrieval, which is the most important part of RAG.
  - golol 3y ago
    The way that LLMs are billed now, if you can densely pack the context with relevant information, you will come out ahead commercially. I don't see this changing with the way that LLM inference works. Really? Because to my understanding the compute necessary to generate a token grows linearly with the context, and doesn't the OpenAI billing reflect that by seperating prompt and output tokens?
cchance 3y ago
The youtube video of the Multimodal analysis of a video is insane, imagine feeding in movies or tv shows and being able to autosummary or find information about them dynamically, how the hell is all this possible already? AI is moving insanely fast.
- vineyardmike 3y ago
  > imagine feeding in movies or tv shows Google themselves have such a huge footprint of various businesses, that they alone would be an amazing customer for this, never mind all the other cool opportunities from third parties... Imagine that they can ingest the entirety of YouTube and then dump that into Google Search's index AND use it to generate training data for their next LLM. Imagine that they can hook it up to your security cameras (Nest Cam), and then ask questions about what happened last night. Imagine that you can ask Gemini how to do something (eg. fix appliance), and it can go and look up a YouTube video on how to accomplish that ask, and explain it to you. Imagine that it can apply summarization and descriptions to every photo AND video in your personal Google Photos library. You can ask it to find a video of your son's first steps, or a graduation/diploma walk for your 3rd child (by name) and it can actually do that. Imagine that Google Meet video calls can have the entire convo itself fed into an LLM (live?), instead of just a transcription. You can have an AI assistant there with you that can interject and discuss, based on both the audio and video feed.
  - anhner 3y ago
    I'd love to see that applied to the Google ecosystem, the question is - why haven't they already done this?
    - is_true 3y ago
      IMO, they aren't sure how to monetize it, Google is run by the ads team. Problem is they are jeopardizing their moat. Google is still in a great position, they have the knowledge and lots of data to pull this off. They just have to take the risk of losing some ad revenue for a while.
    - 3y ago
zitterbewegung 3y ago
RAG doesn’t go away at 10 Million tokens if you do esoteric sources like shodan API queries.
kylerush 3y ago
I assume using this large of a context window instead of RAG would mean the consumption of many orders of magnitude more GPU.
karmasimida 3y ago
Even 1m tokens eliminate the need for RAG, unless it is for cost.
- sroussey 3y ago
  Or accuracy
- 7734128 3y ago
  1 million might sound like a lot, but it's only a few megabytes. I would want RAG, somehow, to be able to process gigabytes or terabytes of material in a streaming fashion.
  - karmasimida 3y ago
    RAG will not change how many tokens LLM can produce at once. Longer context on the other hand, could put some RAG use cases to sleep, if your instructions are like, literally a manual long, then there is no need for rag.
    - 7734128 3y ago
      I think RAG could be used that do that. If you have a one time retrieval in the beginning, basically amending the prompt, then I agree with you. But there are projects (classmate doing his masters thesis project as one implementation of this) that retrieves once every few tokens and make the retrieved information available to the generation somehow. That would not take a toll on the context window.
localhost 3y ago
RE: RAG - they haven't released pricing, but if input tokens are priced at GPT-4 levels - $0.01/1K then sending 10M tokens will cost you $100.
- s-macke 3y ago
  If you think the current APIs will stay that way, then you're right. But when they start offering dedicated chat instances or caching options, you could be back in the penny region. You probably need a couple GB to cache a conversation. That's not so easy at the moment because you have to transfer that data to and from the GPUs and store the data somewhere.
  - localhost 3y ago
    The tokens need to be fed into the model along with the prompt and this takes time. Naive attention is O(N^2). They probably use at least flash attention, and likely something more exotic to their hardware. You'll notice in their video [1] that they never show the prompts running interactively. This is for a roughly 800K context. They claim that "the model took around 60s to respond to each of these prompts". This is not really usable as an interactive experience. I don't want to wait 1 minute for an answer each time I have a question. [1] https://www.youtube.com/watch?v=SSnsmqIj1MI https://www.youtube.com/watch?v=SSnsmqIj1MI
    - rfoo 3y ago
      GP's point is you can cache the state after the model processed the super long context but before it ingests your prompt. If you are going to ask "then why don't OpenAI do it now", the answer is it takes a lot of storage (and IO) so it may not be worth it for shorter context, it adds significant complexity to the entire serving stack, and is incoherent with how OpenAI originally imagined where the "custom-ish" LLM serving game is going to - they bet on finetuning and dedicated instances, instead of long context. The tradeoff can be reflected in the API and pricing, LLM APIs don't have to be like OpenAI's. What if you have an endpoint to generate a "cache" of your context (or really, a prefix of your prompt), billed as usual per token, then you can use your prompt prefix for a fixed price no matter how long it is?
TweedBeetle 3y ago
Regarding how they’re getting to 10M context, I think it’s possible they are using the new SAMBA architecture. Here’s the paper: https://arxiv.org/abs/2312.00752 https://arxiv.org/abs/2312.00752 And here’s a great podcast episode on it: https://www.cognitiverevolution.ai/emergency-pod-mamba-memory-and-the-ssm-moment/ https://www.cognitiverevolution.ai/emergency-pod-mamba-memor...
- LightMachine 3y ago
  As a Brazilian, I approve that choice. Vambora amigos!
renonce 3y ago
> They don't talk about how they get to 10M token context I don't know how either but maybe https://news.ycombinator.com/item?id=39367141 https://news.ycombinator.com/item?id=39367141 Anyway I mean, there is plenty of public research on this so it's probably just a matter of time for everyone else to catch up
- albertzeyer 3y ago
  Why do you think this specific variant (RingAttention)? There are so many different variants for this. As far as I know, the problem in most cases is that while the context length might be high in theory, the actual ability to use it is still limited. E.g. recurrent networks even have infinite context, but they actually only use 10-20 frames as context (longer only in very specific settings; or maybe if you scale them up).
  - renonce 3y ago
    There are ways to test the neural network’s ability to recall from a very long sequence. For example, if you insert a random sentence like “X is Sam Altman” somewhere in the text, will the model be able to answer the question “Who is X?”, or maybe somewhat indirectly “Who is X (in another language)” or “Which sentence was inserted out of context?” “Which celebrity was mentioned in the text?” Anyways the ability to generalize to longer context length is evidenced by such tests. If every token of the model’s output is able to answer questions in such a way that any sentence from the input would be taken into account, this gives evidence that the full context window indeed matters. Currently I find Claude 2 to perform very well on such tasks, so that sets my expectation of how a language model with an extremely long context window should look like.
AaronFriel 3y ago
There will always be more data that could be relevant than fits in a context window, and especially for multi-turn conversations, huge contexts incur huge costs. GPT-4 Turbo, using its full 128k context, costs around $1.28 per API call. At that pricing, 1m tokens is $10, and 10m tokens is an eye-watering $100 per API call. Of course prices will go down, but the price advantage of working with less will remain.
- 7734128 3y ago
  Would the price really increase linearly? Isn't the demands on compute and memory increasing steeper than that as a function of context length?
- elorant 3y ago
  I don't see a problem with this pricing. At 1m tokens you can upload the whole proceedings of a trial and ask it to draw an analysis. Paying $10 for that sounds like a steal.
  - AaronFriel 3y ago
    Of course, if you get exactly the answer you want in the first reply.
  - staticman2 3y ago
    While it's hard to say what's possible on the cutting edge, historically models tend to get dumber as the context size gets bigger. So you'd get a much more intelligent analysis of a 10,000 token excerpt of the trial than a million token complete transcript of the trial. I have not spent the money testing big token sizes in GPT 4 turbo, but it would not surprise me if it gets dumber. Think of it this way, if the model is limited to 3,000 token replies, if an analysis would require a more detailed response than 3,000 tokens, it cannot provide it, it'll just give you insufficient information. What it'll probably do is ignore parts of the trial transcript because it can't analyze all that information in 3,000 tokens. And asking a followup question is another million tokens.
  - ithkuil 3y ago
qwerty_clicks 3y ago
FYI, MM is the standard for million. 10MM not 10M I’m reading all these comments confused as heck why you are excited about 10M tokens
- MichaelNolan 3y ago
  Maybe for accountants, but for everyone else a single M is much more common.
a_vanderbilt 3y ago
After their giant fib with the Gemini video a few weeks back I'm not believing anything til I see it used by actual people. I hope it's that much better than GPT-4, but I'm not holding my breath there isn't an asterisk or trick hiding somewhere.
nborwankar 3y ago
Re RAG aren’t you ignoring the fact that no one wants to put confidential company data into such LLM’s. Private RAG infrastructure remains a need for the same reason that privacy of data of all sorts remains a need. Huge context solves the problem for large open source context material but that’s only part of the picture.
outside1234 3y ago
It takes 60 seconds to process all of that context in their three.js demo, which is, I will say, not super interactive. So there is still room for RAG and other faster alternatives to narrow the context.
aubanel 3y ago
> They are pretty clear that 1.5 Pro is better than GPT-4 in general, and therefore we have a new LLM-as-judge leader, which is pretty interesting I fully disagree, they compare Gemini 1.5 Pro and GPT4 only on context length, not on other tasks where they compare it only to other Gemini which is a strange self-own. I'm convinced that if they do not show the results against GPT4/Claude, it is because they do not look good.
kristjansson 3y ago
For other's reference, the paper: https://storage.googleapis.com/deepmind-media/gemini/gemini_v1_5_report.pdf https://storage.googleapis.com/deepmind-media/gemini/gemini_...
joshsabol46 3y ago
> The 10M context ability wipes out most RAG stack complexity immediately. RAG is needed for the same reason you don't `SELECT *` all of your queries.
nestorD 3y ago
Regarding the 10M tokens context, RingAttention has been shown [0] recently (by researchers, not ML engineers in a FAANG) to be able to scale to comparable (1M) context sizes (it does take work and a lot of GPUs). [0]: https://news.ycombinator.com/item?id=39367141 https://news.ycombinator.com/item?id=39367141
- jebarker 3y ago
  > researchers, not ML engineers in a FAANG Why did you point out this distinction?
  - nestorD 3y ago
    It means they have significantly less means (to get a lot of GPUs letting them scale up in context length) and are likely less well-versed in optimization (which also helps with scaling up)[0]. I believe those two things together are likely enough to explain the difference between a 1M context length and a 10M context length. [0]: Which is not looking down on that particular research team, the vast majority of people have less means and optimization know-how than Google.
  - vineyardmike 3y ago
    Probably to indicate that its research and not productized?
bschne 3y ago
> The 10M context ability wipes out most RAG stack complexity immediately. 1. People mention accuracy issues with longer contexts 2. People mention processing time issues with longer contexts 3. Something people haven't mentioned in this thread is cost -- even thought prompt tokens are usually cheaper than generated tokens, and Gemini seems to be cheaper than GPT-4, putting a whole knowledge base or 80-page document in the context is going to make every time you run that prompt quite expensive
lqcfcjx 3y ago
This might be a stupid question - even if there's no quality degradation from 10M context, will it be extremely slow in reference?
Havoc 3y ago
>3. The 10M context ability wipes out most RAG stack complexity immediately. I'd imagine RAG would still be much more efficient computationally
oblio 3y ago
What's RAG?
- girvo 3y ago
  Retrieval augmented generation. > Retrieval Augmented Generation (RAG) is a technique where the capabilities of a large language model (LLM) are augmented by retrieving information from other systems and inserting them into the LLM’s context window via a prompt. (stolen from: https://github.com/psychic-api/rag-stack https://github.com/psychic-api/rag-stack)
- ohmyiv 3y ago
  Retrieval Augmented Generation. In basic terms, it optimizes output of LLMs by using additional external data sources before answering queries. (That actually might be too basic of a description) Here: https://blogs.nvidia.com/blog/what-is-retrieval-augmented-generation/ https://blogs.nvidia.com/blog/what-is-retrieval-augmented-ge...
  - ssd532 3y ago
    Is it same as embedding? Is embedding an RAG method?
    - scotty79 3y ago
      I don't think so. I think embedding is just converting token string into its numeric representation. Numeric representations of semantically similar token strings are close geometrically. RAG is training AI to be a guy who read a lot of books. He doesn't know all of them in the context of this conversation you are having with him, but he sort of remembers where he read about the thing you are talking about and he has a library behind him into which he can reach and cite what he read verbatim thus introducing it into the context of your conversation. I might be wrong though. I'm a newb.
DebtDeflation 3y ago
>The 10M context ability wipes out most RAG stack complexity immediately From a technology standpoint, maybe. From an economics standpoint, it seems like it would be quite expensive to jam the entire corpus into every single prompt.
shostack 3y ago
Wake me when I can get access without handing over my texts and contacts. I opened the Gemini app on Android and that onerous privacy policy was the first experience. Worse, I didn't seem able to move past accepting giving Google the ability to hoover up my data to disable that in the settings so I just gave up and went back to ChatGPT where I at least generally have control over the data I give it.
tomaskafka 3y ago
"I really want to know how they're getting to 10M context, though." My $5 says it's a RAG or a similar technique (hierarchical RAG comes to mind), just like all other large context LLMs.

deleted 3y ago

[deleted]

CharlieDigital 3y ago

> The 10M context ability wipes out most RAG stack complexity immediately. Remains to be seen. Large contexts are not always better. For starters, it takes longer to process. But secondly, even with RAG and the large context of GPT4 Turbo, providing it a more relevant and accurate context always yields better output. What you get with RAG is faster response times and more accurate answers by pre-filtering out the noise.

behnamoh 3y ago
Don't forget that Gemini also has access to the internet, so a lot of RAGging becomes pointless anyway.
- beppo 3y ago
  Internet search is a form of RAG, though. 10M tokens is very impressive, but you're not fitting a database, let alone the entire internet into a prompt anytime soon.
  - behnamoh 3y ago
    You shouldn't fit an entire database in the context anyway. btw, 10M tokens is 78 times more context window than the newest GPT-4-turbo (128K). In a way, you don't need 78 GPT-4 API calls, only one batch call to Gemini 1.5.
    - rvnx 3y ago
      Well it's nice, just sad nobody can use it
    - cchance 3y ago
      I don't get this why is it people think that you need to put an entire database in the short-term memory of the AI to be useful? When you work with a DB are you memorizing the entire f*cking database, no, you know the summaries of it and how to access and use it. People also seem to forget that the average is 1b words that are read by people in their entire LIFETIME, and at 10m, with nearly 100% recall thats pretty damn amazing, i'm pretty sure I don't have perfect recall of 10m words myself lol
      - Qwero 3y ago
        It increases the use cases. It can also be a good alternative for fine-tuning. And the use case of a code base is a good example: if the ai understands the whole context, it can do basically everything. Let me pay 5€ for a android app rewritten into iOS.

deleted 3y ago

[deleted]

cs702 3y ago

> 1. They don't talk about how they get to 10M token context > 2. They don't talk about how they get to 10M token context Yes. I wonder if they're using a "linear RNN" type of model like Linear Attention, Mamba, RWKV, etc. Like Transformers with standard attention, these models train efficiently in parallel, but their compute is O(N) instead of O(N²), so in theory they can be extended to much longer sequences much efficiently. They have shown a lot of promise recently at smaller model sizes. Does anyone here have any insight or knowledge about the internals of Gemini 1.5?

sebzim4500 3y ago
The fact they are getting perfect recall with millions of tokens rules out any of the existing linear attention methods.
- cs702 3y ago
  I wouldn't be so sure perfect recall rules out linear RNNs, because I haven't seen any conclusive data on their ability to recall. Have you?
candiodari 3y ago
They do give a hint: "This includes making Gemini 1.5 more efficient to train and serve, with a new Mixture-of-Experts (MoE) architecture." One thing you could do with MoE is giving each expert different subsets of the input tokens. And that would definitely do what they claim here: it would allow search. You want to find where someone said "the password is X" in a 50 hour audio file, this would be perfect. If your question is "what is the first AND last thing person X said" ... it's going to suck badly. Anything that requires taking 2 things into account that aren't right next to eachother is just not going to work.
- declaredapple 3y ago
  > One thing you could do with MoE is giving each expert different subsets of the input tokens. Don't MoE's route tokens to experts after the attention step? That wouldn't solve the n^2 issue the attention step has. If you split the tokens before the attention step, that would mean those tokens would have no relationship to each other - it would be like inferring two prompts in parallel. That would defeat the point of a 10M context

usaar333 3y ago

> They are pretty clear that 1.5 Pro is better than GPT-4 in general, and therefore we have a new LLM-as-judge leader, which is pretty interesting. They try to push that, but it's not the most convincing. Look at Table 8 for text evaluations (math, etc.) - they don't even attempt a comparison with GPT-4. GPT-4 is higher than any Gemini model on both MMLU and GSM8K. Gemini Pro seems slightly better than GPT-4 original in Human Eval (67->71). Gemini Pro does crush naive GPT-4 on math (though not with code interpreter and this is the original model). All in 1.5 Pro seems maybe a bit better than 1.0 Ultra. Given that in the wild people seem to find GPT-4 better for say coding than Gemini Ultra, my current update is Pro 1.5 is about equal to GPT-4. But we'll see once released.

cchance 3y ago
I mean i don't see GPT4 watching a 44 minute movie and being able to exactly pinpoint a guy taking a paper out of his pocket..
panarky 3y ago
> people seem to find GPT-4 better for say coding than Gemini Ultra For my use cases, Gemini Ultra performs significantly better than GPT-4. My prompts are long and complex, with a paragraph or two about the general objective followed by 15 to 20 numbered requirements. Often I'll include existing functions the new code needs to work with, or functions that must be refactored to handle the new requirements. I took 20 prompts that I'd run with GPT-4 and fed them to Gemini Ultra. Gemini gave a clearly better result in 16 out of 20 cases. Where GPT-4 might miss one or two requirements, Gemini usually got them all. Where GPT-4 might require multiple chat turns to point out its errors and omissions and tell it to fix them, Gemini often returned the result I wanted in one shot. Where GPT-4 hallucinated a method that doesn't exist, or had been deprecated years ago, Gemini used correct methods. Where GPT-4 called methods of third-party packages it assumed were installed, Gemini either used native code or explicitly called out the dependency. For the 4 out of 20 prompts where Gemini did worse, one was a weird rejection where I'd included an image in the prompt and Gemini refused to work with it because it had unrecognizable human forms in the distance. Another was a simple bash script to split a text file, and it came up with a technically correct but complex one-liner, while GPT-4 just used split with simple options to get the same result. For now I subscribe to both. But I'm using Gemini for almost all coding work, only checking in with GPT-4 when Gemini stumbles, which isn't often. If I continue to get solid results I'll drop the GPT-4 subscription.

swalsh 3y ago

"The 10M context ability wipes out most RAG stack complexity immediately." I'm skeptical, my past experience is just becaues the context has room to stuff whatever you want in it, the more you stuff in the context the less accurate your results are. There seems to be this balance of providing enough that you'll get high quality answers, but not too much that the model is overwhelmed. I think a large part of developing better models is not just a better architectures that support larger and larger context sizes, but also capable models that can properly leverage that context. That's the test for me.

HereBePandas 3y ago
They explicitly address this in page 11 of the report. Basically perfect recall for up to 1M tokens; way better than GPT-4.
- westoncb 3y ago
  I don't think recall really addresses it sufficiently: the main issue I see is answers getting "muddy". Like it's getting pulled in too many directions and averaging.
  - a_wild_dandan 3y ago
    I'd urge caution in extending generalizations about "muddiness" to a new context architecture. Let's use the thing first.
    - westoncb 3y ago
      I'm not saying it applies to the new architecture, I'm saying that's a big issue I've observed in existing models and that so far we have no info on whether it's solved in the new one (i.e. accurate recall doesn't imply much in that regard).
      - westoncb 3y ago
        Would be awesome if it is solved but seems like a much deeper problem tbh.

freedomben 3y ago

Is 10M token context correct? The blog post I see 1M but I'm not sure if these are different things Edit: Ah, I see, it's 1M reliably in production, up to 10M in research: > Through a series of machine learning innovations, we’ve increased 1.5 Pro’s context window capacity far beyond the original 32,000 tokens for Gemini 1.0. We can now run up to 1 million tokens in production. > This means 1.5 Pro can process vast amounts of information in one go — including 1 hour of video, 11 hours of audio, codebases with over 30,000 lines of code or over 700,000 words. In our research, we’ve also successfully tested up to 10 million tokens.

huytersd 3y ago
I know how I’m going to evaluate this model. Upload my codebase and ask it to “find all the bugs”.
p1esk 3y ago
How could one hour of video fit in 1M tokens? 1 hour at 30fps is 3600*30=100k frames. Each frame is converted in 256 tokens. So either they are not processing each frame, or each frame is converted into fewer tokens.
- KTibow 3y ago
  The model can probably perform fine at 1 frame per second (3600*256=921600 tokens), and they could probably use some sort of compression.

tbruckner 3y ago

How do you know it isn't RAG?

tveita 3y ago

> The 10M context ability wipes out most RAG stack complexity immediately. The video queries they show take around 1 minute each, this probably burns a ton of GPU. I appreciate how clearly they highlight that the video is sped up though, they're clearly trying to avoid repeating the "fake demo" fiasco from the original Gemini videos.

theGnuMe 3y ago

For #1 and #2 it is some version of mixture of experts. This is mentioned in the blog post. So each expert only sees a subset of the tokens. I imagine they have some new way to route tokens to the experts that probably computes a global context. One scalable way to compute a global context is by a state space model. This would act as a controller and route the input tokens to the MoEs. This can be computed by convolution if you make some simplifying assumptions. They may also still use transformers as well. I could be wrong but there are some Mamba-MoEs papers that explore this idea.

resouer 3y ago

> The 10M context ability wipes out most RAG stack complexity immediately. This may not be true. My experience of the complexity of RAG lays in how to properly connect to various unstructured data sources and perform data transformation pipeline for large scale data set (which means GB, TB or even PB). It's in the critical path rather a "nice to have", because the quality of data and the pipeline is a major factor for the final generated the result. i.e., in RAG, the importance of R >>> G.

jorvi 3y ago

I just hope at some point we get access to mostly uncensored models. Both GPT-4 and Gemini are extremely shackled, and a slightly inferior model that hasn’t been hobbled by a very restricting preprompt would handily outperform them.

ShamelessC 3y ago
You can customize the system prompt with ChatGPT or via the completions API, just fyi.

ren_engineer 3y ago

RAG would still be useful for cost savings assuming they charge per token, plus I'm guessing using the full-context length would be slower than using RAG to get what you need for a smaller prompt

nostrebored 3y ago
This is going to be the real differentiator. HN is very focused on technical feasibility (which remains to be seen!), but in every LLM opportunity, the CIO/CFO/CEO are going to be concerned with the cost modeling. The way that LLMs are billed now, if you can densely pack the context with relevant information, you will come out ahead commercially. I don't see this changing with the way that LLM inference works. Maybe this changes with managed vector search offerings that are opaque to the user. The context goes to a preprocessing layer, an efficient cache understands which parts haven't been embedded (new bloom filter use case?), embeds the other chunks, and extracts the intent of the prompt.
- mediaman 3y ago
  Agreed with this. The leading ability AI (in terms of cognitive power) will, generally, cost more per token than lower cognitive power AI. That means that at a given budget you can choose more cognitive power with fewer tokens, or less cognitive power with more tokens. For most use cases, there's no real point in giving up cognitive power to include useless tokens that have no hope of helping with a given question. So then you're back to the question of: how do we reduce the number of tokens, so that we can get higher cognitive power? And that's the entire field of information retrieval, which is the most important part of RAG.
- golol 3y ago
  The way that LLMs are billed now, if you can densely pack the context with relevant information, you will come out ahead commercially. I don't see this changing with the way that LLM inference works. Really? Because to my understanding the compute necessary to generate a token grows linearly with the context, and doesn't the OpenAI billing reflect that by seperating prompt and output tokens?

cchance 3y ago

The youtube video of the Multimodal analysis of a video is insane, imagine feeding in movies or tv shows and being able to autosummary or find information about them dynamically, how the hell is all this possible already? AI is moving insanely fast.

vineyardmike 3y ago
> imagine feeding in movies or tv shows Google themselves have such a huge footprint of various businesses, that they alone would be an amazing customer for this, never mind all the other cool opportunities from third parties... Imagine that they can ingest the entirety of YouTube and then dump that into Google Search's index AND use it to generate training data for their next LLM. Imagine that they can hook it up to your security cameras (Nest Cam), and then ask questions about what happened last night. Imagine that you can ask Gemini how to do something (eg. fix appliance), and it can go and look up a YouTube video on how to accomplish that ask, and explain it to you. Imagine that it can apply summarization and descriptions to every photo AND video in your personal Google Photos library. You can ask it to find a video of your son's first steps, or a graduation/diploma walk for your 3rd child (by name) and it can actually do that. Imagine that Google Meet video calls can have the entire convo itself fed into an LLM (live?), instead of just a transcription. You can have an AI assistant there with you that can interject and discuss, based on both the audio and video feed.
- anhner 3y ago
  I'd love to see that applied to the Google ecosystem, the question is - why haven't they already done this?
  - is_true 3y ago
    IMO, they aren't sure how to monetize it, Google is run by the ads team. Problem is they are jeopardizing their moat. Google is still in a great position, they have the knowledge and lots of data to pull this off. They just have to take the risk of losing some ad revenue for a while.
  - 3y ago

zitterbewegung 3y ago

RAG doesn’t go away at 10 Million tokens if you do esoteric sources like shodan API queries.

kylerush 3y ago

I assume using this large of a context window instead of RAG would mean the consumption of many orders of magnitude more GPU.

karmasimida 3y ago

Even 1m tokens eliminate the need for RAG, unless it is for cost.

sroussey 3y ago
Or accuracy
7734128 3y ago
1 million might sound like a lot, but it's only a few megabytes. I would want RAG, somehow, to be able to process gigabytes or terabytes of material in a streaming fashion.
- karmasimida 3y ago
  RAG will not change how many tokens LLM can produce at once. Longer context on the other hand, could put some RAG use cases to sleep, if your instructions are like, literally a manual long, then there is no need for rag.
  - 7734128 3y ago
    I think RAG could be used that do that. If you have a one time retrieval in the beginning, basically amending the prompt, then I agree with you. But there are projects (classmate doing his masters thesis project as one implementation of this) that retrieves once every few tokens and make the retrieved information available to the generation somehow. That would not take a toll on the context window.

localhost 3y ago

RE: RAG - they haven't released pricing, but if input tokens are priced at GPT-4 levels - $0.01/1K then sending 10M tokens will cost you $100.

s-macke 3y ago
If you think the current APIs will stay that way, then you're right. But when they start offering dedicated chat instances or caching options, you could be back in the penny region. You probably need a couple GB to cache a conversation. That's not so easy at the moment because you have to transfer that data to and from the GPUs and store the data somewhere.
- localhost 3y ago
  The tokens need to be fed into the model along with the prompt and this takes time. Naive attention is O(N^2). They probably use at least flash attention, and likely something more exotic to their hardware. You'll notice in their video [1] that they never show the prompts running interactively. This is for a roughly 800K context. They claim that "the model took around 60s to respond to each of these prompts". This is not really usable as an interactive experience. I don't want to wait 1 minute for an answer each time I have a question. [1] https://www.youtube.com/watch?v=SSnsmqIj1MI https://www.youtube.com/watch?v=SSnsmqIj1MI
  - rfoo 3y ago
    GP's point is you can cache the state after the model processed the super long context but before it ingests your prompt. If you are going to ask "then why don't OpenAI do it now", the answer is it takes a lot of storage (and IO) so it may not be worth it for shorter context, it adds significant complexity to the entire serving stack, and is incoherent with how OpenAI originally imagined where the "custom-ish" LLM serving game is going to - they bet on finetuning and dedicated instances, instead of long context. The tradeoff can be reflected in the API and pricing, LLM APIs don't have to be like OpenAI's. What if you have an endpoint to generate a "cache" of your context (or really, a prefix of your prompt), billed as usual per token, then you can use your prompt prefix for a fixed price no matter how long it is?

TweedBeetle 3y ago

Regarding how they’re getting to 10M context, I think it’s possible they are using the new SAMBA architecture. Here’s the paper: https://arxiv.org/abs/2312.00752 https://arxiv.org/abs/2312.00752 And here’s a great podcast episode on it: https://www.cognitiverevolution.ai/emergency-pod-mamba-memory-and-the-ssm-moment/ https://www.cognitiverevolution.ai/emergency-pod-mamba-memor...

LightMachine 3y ago
As a Brazilian, I approve that choice. Vambora amigos!

renonce 3y ago

> They don't talk about how they get to 10M token context I don't know how either but maybe https://news.ycombinator.com/item?id=39367141 https://news.ycombinator.com/item?id=39367141 Anyway I mean, there is plenty of public research on this so it's probably just a matter of time for everyone else to catch up

albertzeyer 3y ago
Why do you think this specific variant (RingAttention)? There are so many different variants for this. As far as I know, the problem in most cases is that while the context length might be high in theory, the actual ability to use it is still limited. E.g. recurrent networks even have infinite context, but they actually only use 10-20 frames as context (longer only in very specific settings; or maybe if you scale them up).
- renonce 3y ago
  There are ways to test the neural network’s ability to recall from a very long sequence. For example, if you insert a random sentence like “X is Sam Altman” somewhere in the text, will the model be able to answer the question “Who is X?”, or maybe somewhat indirectly “Who is X (in another language)” or “Which sentence was inserted out of context?” “Which celebrity was mentioned in the text?” Anyways the ability to generalize to longer context length is evidenced by such tests. If every token of the model’s output is able to answer questions in such a way that any sentence from the input would be taken into account, this gives evidence that the full context window indeed matters. Currently I find Claude 2 to perform very well on such tasks, so that sets my expectation of how a language model with an extremely long context window should look like.

AaronFriel 3y ago

There will always be more data that could be relevant than fits in a context window, and especially for multi-turn conversations, huge contexts incur huge costs. GPT-4 Turbo, using its full 128k context, costs around $1.28 per API call. At that pricing, 1m tokens is $10, and 10m tokens is an eye-watering $100 per API call. Of course prices will go down, but the price advantage of working with less will remain.

7734128 3y ago
Would the price really increase linearly? Isn't the demands on compute and memory increasing steeper than that as a function of context length?
elorant 3y ago
I don't see a problem with this pricing. At 1m tokens you can upload the whole proceedings of a trial and ask it to draw an analysis. Paying $10 for that sounds like a steal.
- AaronFriel 3y ago
  Of course, if you get exactly the answer you want in the first reply.
- staticman2 3y ago
  While it's hard to say what's possible on the cutting edge, historically models tend to get dumber as the context size gets bigger. So you'd get a much more intelligent analysis of a 10,000 token excerpt of the trial than a million token complete transcript of the trial. I have not spent the money testing big token sizes in GPT 4 turbo, but it would not surprise me if it gets dumber. Think of it this way, if the model is limited to 3,000 token replies, if an analysis would require a more detailed response than 3,000 tokens, it cannot provide it, it'll just give you insufficient information. What it'll probably do is ignore parts of the trial transcript because it can't analyze all that information in 3,000 tokens. And asking a followup question is another million tokens.
- ithkuil 3y ago

qwerty_clicks 3y ago

FYI, MM is the standard for million. 10MM not 10M I’m reading all these comments confused as heck why you are excited about 10M tokens

MichaelNolan 3y ago
Maybe for accountants, but for everyone else a single M is much more common.

a_vanderbilt 3y ago

After their giant fib with the Gemini video a few weeks back I'm not believing anything til I see it used by actual people. I hope it's that much better than GPT-4, but I'm not holding my breath there isn't an asterisk or trick hiding somewhere.

nborwankar 3y ago

Re RAG aren’t you ignoring the fact that no one wants to put confidential company data into such LLM’s. Private RAG infrastructure remains a need for the same reason that privacy of data of all sorts remains a need. Huge context solves the problem for large open source context material but that’s only part of the picture.

outside1234 3y ago

It takes 60 seconds to process all of that context in their three.js demo, which is, I will say, not super interactive. So there is still room for RAG and other faster alternatives to narrow the context.

aubanel 3y ago

> They are pretty clear that 1.5 Pro is better than GPT-4 in general, and therefore we have a new LLM-as-judge leader, which is pretty interesting I fully disagree, they compare Gemini 1.5 Pro and GPT4 only on context length, not on other tasks where they compare it only to other Gemini which is a strange self-own. I'm convinced that if they do not show the results against GPT4/Claude, it is because they do not look good.

kristjansson 3y ago

For other's reference, the paper: https://storage.googleapis.com/deepmind-media/gemini/gemini_v1_5_report.pdf https://storage.googleapis.com/deepmind-media/gemini/gemini_...

joshsabol46 3y ago

> The 10M context ability wipes out most RAG stack complexity immediately. RAG is needed for the same reason you don't `SELECT *` all of your queries.

nestorD 3y ago

Regarding the 10M tokens context, RingAttention has been shown [0] recently (by researchers, not ML engineers in a FAANG) to be able to scale to comparable (1M) context sizes (it does take work and a lot of GPUs). [0]: https://news.ycombinator.com/item?id=39367141 https://news.ycombinator.com/item?id=39367141

jebarker 3y ago
> researchers, not ML engineers in a FAANG Why did you point out this distinction?
- nestorD 3y ago
  It means they have significantly less means (to get a lot of GPUs letting them scale up in context length) and are likely less well-versed in optimization (which also helps with scaling up)[0]. I believe those two things together are likely enough to explain the difference between a 1M context length and a 10M context length. [0]: Which is not looking down on that particular research team, the vast majority of people have less means and optimization know-how than Google.
- vineyardmike 3y ago
  Probably to indicate that its research and not productized?

bschne 3y ago

> The 10M context ability wipes out most RAG stack complexity immediately. 1. People mention accuracy issues with longer contexts 2. People mention processing time issues with longer contexts 3. Something people haven't mentioned in this thread is cost -- even thought prompt tokens are usually cheaper than generated tokens, and Gemini seems to be cheaper than GPT-4, putting a whole knowledge base or 80-page document in the context is going to make every time you run that prompt quite expensive

lqcfcjx 3y ago

This might be a stupid question - even if there's no quality degradation from 10M context, will it be extremely slow in reference?

Havoc 3y ago

>3. The 10M context ability wipes out most RAG stack complexity immediately. I'd imagine RAG would still be much more efficient computationally

oblio 3y ago

What's RAG?

girvo 3y ago
Retrieval augmented generation. > Retrieval Augmented Generation (RAG) is a technique where the capabilities of a large language model (LLM) are augmented by retrieving information from other systems and inserting them into the LLM’s context window via a prompt. (stolen from: https://github.com/psychic-api/rag-stack https://github.com/psychic-api/rag-stack)
ohmyiv 3y ago
Retrieval Augmented Generation. In basic terms, it optimizes output of LLMs by using additional external data sources before answering queries. (That actually might be too basic of a description) Here: https://blogs.nvidia.com/blog/what-is-retrieval-augmented-generation/ https://blogs.nvidia.com/blog/what-is-retrieval-augmented-ge...
- ssd532 3y ago
  Is it same as embedding? Is embedding an RAG method?
  - scotty79 3y ago
    I don't think so. I think embedding is just converting token string into its numeric representation. Numeric representations of semantically similar token strings are close geometrically. RAG is training AI to be a guy who read a lot of books. He doesn't know all of them in the context of this conversation you are having with him, but he sort of remembers where he read about the thing you are talking about and he has a library behind him into which he can reach and cite what he read verbatim thus introducing it into the context of your conversation. I might be wrong though. I'm a newb.

DebtDeflation 3y ago

>The 10M context ability wipes out most RAG stack complexity immediately From a technology standpoint, maybe. From an economics standpoint, it seems like it would be quite expensive to jam the entire corpus into every single prompt.

shostack 3y ago

Wake me when I can get access without handing over my texts and contacts. I opened the Gemini app on Android and that onerous privacy policy was the first experience. Worse, I didn't seem able to move past accepting giving Google the ability to hoover up my data to disable that in the settings so I just gave up and went back to ChatGPT where I at least generally have control over the data I give it.

tomaskafka 3y ago

"I really want to know how they're getting to 10M context, though." My $5 says it's a RAG or a similar technique (hierarchical RAG comes to mind), just like all other large context LLMs.