4 ms·

It's still up for debate as to whether long context windows are worth it. It's also not a cheap way to solve a problem. Clearly hardware vendors love LLMs. But

It's still up for debate as to whether long context windows are worth it. It's also not a cheap way to solve a problem. Clearly hardware vendors love LLMs. But it's just a highly inefficient approach for a lot of problems.

CharlesW 3y ago
> It's still up for debate as to whether long context windows are worth it. As someone who's run into problems related to short context windows quite often, can you explain what "worth it" means? Also, when you say "not cheap", what alternative do you have in mind?
- esafak 3y ago
  Continuous fine tuning so you don't have to pass everything in the context with every query would definitely yield a better UX.
  - littlestymaar 3y ago
    Afaik the few-shots learning abilities of LLMs are much better than what you can achieve with fine tuning when you only have tiny samples. Of course it is true for real long context but it's not clear if its going to work with the sparse context hacks intended to keep memory size low. Having the ability to keep the same coding session forever without having the LLM forgetting all the time and making the same mistakes over and over would be a game changer.
  - nl 3y ago
    I'd argue the fine-tuned UX is often worse. People like RAG-based solutions because you can include references in your answer very easily (eg, Perplexity, or see the DAnswer "internal search" product launched today on HN). That is extremely hard to make work reliably from a fine-tuned model.
    - esafak 3y ago
      I mean UX from the perspective of the person doing the search, not the engineer. I don't dispute that fine tuning is harder to implement.
      - nl 3y ago
        I mean from the perspective of the person doing the search too. They often want to see the reference - for example the LLM is constructing an answer about some company policy question it is good to include references to the actual company policy documents by URL. This is easy using RAG, but very hard to do reliably just using a fine tuned LLM. See Perplexity for a great example of this at web scale.
        esafak 3y ago
        This is true. If LLMs recalled perfectly their purveyors would get into legal trouble :)
- dmezzetti 3y ago
  Retrieval Augmented Generation aka RAG with a relevant context. Long contexts continue to have the lost in the middle syndrome.
  - nl 3y ago
    Gemini doesn't have the "lost in the middle syndrome"
    - dmezzetti 3y ago
      We don't have enough data on that yet to know for sure. It was just released this week. But nonetheless, sending 2M tokens to a LLM isn't an efficient way to solve most problems.
      - int_19h 3y ago
        LLMs in general are likely to be a very inefficient way to solve many problems they're currently being used for. But so long as we don't have something better, brute force remains a valid approach.
      - nl 3y ago
        > We don't have enough data on that yet to know for sure. It was just released this week. I think there is sufficient evidence to think it works. A bunch of people who do have access have put it through a bunch of real-world exercises to test this, and while GPT4 128K and Claude 200K both fail Gemini is passing. I agree sending 2M tokens to a LLM is costly at the moment and that RAG has a bunch of advantages in some circumstances.
pixl97 3y ago
A cheap way to solve a problem has always been throwing more compute power at it. Figuring out new algorithms is monumentally more difficult than more hardware, and even then when more efficient algorithms are found, we throw more hardware at it and get a million times more done.
- szundi 3y ago
  Billions of dollars spent on hardware kind of makes this rational to try
tkellogg 3y ago
It depends what problem you're solving. If it's a high frequency request, like a chat response, it's far too inefficient. Most web APIs would consider it bad practice to read 2MB of data on every request, even worse when you consider all the LLM computation. Instead, use RAG and pull targeted info out of some sort of low-latency database. However, caching might be a sweet spot for these multi-modal and large context LLMs. Take a bunch of documents and perform reasoning tasks to distill the knowledge down into something like a knowledge graph, to be used in RAG.
wongarsu 3y ago
Having them available is still hugely beneficial, even if they end up too expensive to use in most production use-cases. They would still be valuable for prototyping, where fast iteration makes it possible to learn more about the problem you are solving and whether it is even worth solving. They are also valuable for iteration-and-distillation approaches, where you can use the data generated from an expensive model to train a cheaper model.
Fripplebubby 3y ago
The cool thing about this paper is, actually it is a (relatively) cheap way to solve this particular problem (LLM on 2048k window), because they are using pre-trained models like LLaMA2 and Mistral and extending them to 2048k windows using a novel technique, rather than training a model from scratch with 2048k tokens which would be prohibitively expensive to mere mortals.
ogogmad 3y ago
It seems like a very long input to an LLM can act as additional training data. A possible future is that LLMs will be used to bootstrap AIs that will be "trained" by feeding them a giant prompt at the start. Might end the era of "supervised" learning, "reinforcement" learning, numerical methods and gradient descent -- those techniques are awkward and procrustean. Imagine what you could do if you could improve a neural network - maybe to AGI level - just by talking to it in English? So it looks like a VERY good idea. Who gets the gist of what I'm saying?
- mschuster91 3y ago
  > Imagine what you could do if you could improve a neural network - maybe to AGI level - just by talking to it in English? You'd get a repeat of Microsoft Tay, the short-lived 2016 chatbot that had to be taken down after not even a full day because 4chan managed to turn it into a full-blown hate spreader in hours [1]. Before something like this can be reasonably released to the Internet at large, we need to find out how to teach the "ingestion" part how to judge the input that it's being presented with... basically, AI school. The same way we teach our children that it's not OK to steal other people's stuff, to not be the one first throwing punches or to not discriminate against other people, or that there are reasonably trustworthy media and absolutely untrustworthy, we need to teach AIs. And we'd also need to figure out ways to teach an AI basic, hard truths: the Holocaust happened, the Earth is a globe not a pizza, the Earth is not hollow, and the moon landings were real. At the moment, we're half-ass attempting that by annotated training data (and this is the true moat of OpenAI, not the weights or prompts!), but even as we invest literally millions of hours of training compute time, an average high schooler that has been trained for 18 years can reasonably pass the above-mentioned criteria to be a productive member of society. [1] https://en.wikipedia.org/wiki/Tay_(chatbot) https://en.wikipedia.org/wiki/Tay_(chatbot)

CharlesW 3y ago

> It's still up for debate as to whether long context windows are worth it. As someone who's run into problems related to short context windows quite often, can you explain what "worth it" means? Also, when you say "not cheap", what alternative do you have in mind?

esafak 3y ago
Continuous fine tuning so you don't have to pass everything in the context with every query would definitely yield a better UX.
- littlestymaar 3y ago
  Afaik the few-shots learning abilities of LLMs are much better than what you can achieve with fine tuning when you only have tiny samples. Of course it is true for real long context but it's not clear if its going to work with the sparse context hacks intended to keep memory size low. Having the ability to keep the same coding session forever without having the LLM forgetting all the time and making the same mistakes over and over would be a game changer.
- nl 3y ago
  I'd argue the fine-tuned UX is often worse. People like RAG-based solutions because you can include references in your answer very easily (eg, Perplexity, or see the DAnswer "internal search" product launched today on HN). That is extremely hard to make work reliably from a fine-tuned model.
  - esafak 3y ago
    I mean UX from the perspective of the person doing the search, not the engineer. I don't dispute that fine tuning is harder to implement.
    - nl 3y ago
      I mean from the perspective of the person doing the search too. They often want to see the reference - for example the LLM is constructing an answer about some company policy question it is good to include references to the actual company policy documents by URL. This is easy using RAG, but very hard to do reliably just using a fine tuned LLM. See Perplexity for a great example of this at web scale.
      - esafak 3y ago
        This is true. If LLMs recalled perfectly their purveyors would get into legal trouble :)
dmezzetti 3y ago
Retrieval Augmented Generation aka RAG with a relevant context. Long contexts continue to have the lost in the middle syndrome.
- nl 3y ago
  Gemini doesn't have the "lost in the middle syndrome"
  - dmezzetti 3y ago
    We don't have enough data on that yet to know for sure. It was just released this week. But nonetheless, sending 2M tokens to a LLM isn't an efficient way to solve most problems.
    - int_19h 3y ago
      LLMs in general are likely to be a very inefficient way to solve many problems they're currently being used for. But so long as we don't have something better, brute force remains a valid approach.
    - nl 3y ago
      > We don't have enough data on that yet to know for sure. It was just released this week. I think there is sufficient evidence to think it works. A bunch of people who do have access have put it through a bunch of real-world exercises to test this, and while GPT4 128K and Claude 200K both fail Gemini is passing. I agree sending 2M tokens to a LLM is costly at the moment and that RAG has a bunch of advantages in some circumstances.

pixl97 3y ago

A cheap way to solve a problem has always been throwing more compute power at it. Figuring out new algorithms is monumentally more difficult than more hardware, and even then when more efficient algorithms are found, we throw more hardware at it and get a million times more done.

szundi 3y ago
Billions of dollars spent on hardware kind of makes this rational to try

tkellogg 3y ago

It depends what problem you're solving. If it's a high frequency request, like a chat response, it's far too inefficient. Most web APIs would consider it bad practice to read 2MB of data on every request, even worse when you consider all the LLM computation. Instead, use RAG and pull targeted info out of some sort of low-latency database. However, caching might be a sweet spot for these multi-modal and large context LLMs. Take a bunch of documents and perform reasoning tasks to distill the knowledge down into something like a knowledge graph, to be used in RAG.

wongarsu 3y ago

Having them available is still hugely beneficial, even if they end up too expensive to use in most production use-cases. They would still be valuable for prototyping, where fast iteration makes it possible to learn more about the problem you are solving and whether it is even worth solving. They are also valuable for iteration-and-distillation approaches, where you can use the data generated from an expensive model to train a cheaper model.

Fripplebubby 3y ago

The cool thing about this paper is, actually it is a (relatively) cheap way to solve this particular problem (LLM on 2048k window), because they are using pre-trained models like LLaMA2 and Mistral and extending them to 2048k windows using a novel technique, rather than training a model from scratch with 2048k tokens which would be prohibitively expensive to mere mortals.

ogogmad 3y ago

It seems like a very long input to an LLM can act as additional training data. A possible future is that LLMs will be used to bootstrap AIs that will be "trained" by feeding them a giant prompt at the start. Might end the era of "supervised" learning, "reinforcement" learning, numerical methods and gradient descent -- those techniques are awkward and procrustean. Imagine what you could do if you could improve a neural network - maybe to AGI level - just by talking to it in English? So it looks like a VERY good idea. Who gets the gist of what I'm saying?

mschuster91 3y ago
> Imagine what you could do if you could improve a neural network - maybe to AGI level - just by talking to it in English? You'd get a repeat of Microsoft Tay, the short-lived 2016 chatbot that had to be taken down after not even a full day because 4chan managed to turn it into a full-blown hate spreader in hours [1]. Before something like this can be reasonably released to the Internet at large, we need to find out how to teach the "ingestion" part how to judge the input that it's being presented with... basically, AI school. The same way we teach our children that it's not OK to steal other people's stuff, to not be the one first throwing punches or to not discriminate against other people, or that there are reasonably trustworthy media and absolutely untrustworthy, we need to teach AIs. And we'd also need to figure out ways to teach an AI basic, hard truths: the Holocaust happened, the Earth is a globe not a pizza, the Earth is not hollow, and the moon landings were real. At the moment, we're half-ass attempting that by annotated training data (and this is the true moat of OpenAI, not the weights or prompts!), but even as we invest literally millions of hours of training compute time, an average high schooler that has been trained for 18 years can reasonably pass the above-mentioned criteria to be a productive member of society. [1] https://en.wikipedia.org/wiki/Tay_(chatbot) https://en.wikipedia.org/wiki/Tay_(chatbot)