4 ms·

> It's still up for debate as to whether long context windows are worth it. As someone who's run into problems related to short context windows quite often, ca

> It's still up for debate as to whether long context windows are worth it. As someone who's run into problems related to short context windows quite often, can you explain what "worth it" means? Also, when you say "not cheap", what alternative do you have in mind?

esafak 3y ago
Continuous fine tuning so you don't have to pass everything in the context with every query would definitely yield a better UX.
- littlestymaar 3y ago
  Afaik the few-shots learning abilities of LLMs are much better than what you can achieve with fine tuning when you only have tiny samples. Of course it is true for real long context but it's not clear if its going to work with the sparse context hacks intended to keep memory size low. Having the ability to keep the same coding session forever without having the LLM forgetting all the time and making the same mistakes over and over would be a game changer.
- nl 3y ago
  I'd argue the fine-tuned UX is often worse. People like RAG-based solutions because you can include references in your answer very easily (eg, Perplexity, or see the DAnswer "internal search" product launched today on HN). That is extremely hard to make work reliably from a fine-tuned model.
  - esafak 3y ago
    I mean UX from the perspective of the person doing the search, not the engineer. I don't dispute that fine tuning is harder to implement.
    - nl 3y ago
      I mean from the perspective of the person doing the search too. They often want to see the reference - for example the LLM is constructing an answer about some company policy question it is good to include references to the actual company policy documents by URL. This is easy using RAG, but very hard to do reliably just using a fine tuned LLM. See Perplexity for a great example of this at web scale.
      - esafak 3y ago
        This is true. If LLMs recalled perfectly their purveyors would get into legal trouble :)
dmezzetti 3y ago
Retrieval Augmented Generation aka RAG with a relevant context. Long contexts continue to have the lost in the middle syndrome.
- nl 3y ago
  Gemini doesn't have the "lost in the middle syndrome"
  - dmezzetti 3y ago
    We don't have enough data on that yet to know for sure. It was just released this week. But nonetheless, sending 2M tokens to a LLM isn't an efficient way to solve most problems.
    - int_19h 3y ago
      LLMs in general are likely to be a very inefficient way to solve many problems they're currently being used for. But so long as we don't have something better, brute force remains a valid approach.
    - nl 3y ago
      > We don't have enough data on that yet to know for sure. It was just released this week. I think there is sufficient evidence to think it works. A bunch of people who do have access have put it through a bunch of real-world exercises to test this, and while GPT4 128K and Claude 200K both fail Gemini is passing. I agree sending 2M tokens to a LLM is costly at the moment and that RAG has a bunch of advantages in some circumstances.

esafak 3y ago

Continuous fine tuning so you don't have to pass everything in the context with every query would definitely yield a better UX.

littlestymaar 3y ago
Afaik the few-shots learning abilities of LLMs are much better than what you can achieve with fine tuning when you only have tiny samples. Of course it is true for real long context but it's not clear if its going to work with the sparse context hacks intended to keep memory size low. Having the ability to keep the same coding session forever without having the LLM forgetting all the time and making the same mistakes over and over would be a game changer.
nl 3y ago
I'd argue the fine-tuned UX is often worse. People like RAG-based solutions because you can include references in your answer very easily (eg, Perplexity, or see the DAnswer "internal search" product launched today on HN). That is extremely hard to make work reliably from a fine-tuned model.
- esafak 3y ago
  I mean UX from the perspective of the person doing the search, not the engineer. I don't dispute that fine tuning is harder to implement.
  - nl 3y ago
    I mean from the perspective of the person doing the search too. They often want to see the reference - for example the LLM is constructing an answer about some company policy question it is good to include references to the actual company policy documents by URL. This is easy using RAG, but very hard to do reliably just using a fine tuned LLM. See Perplexity for a great example of this at web scale.
    - esafak 3y ago
      This is true. If LLMs recalled perfectly their purveyors would get into legal trouble :)

dmezzetti 3y ago

Retrieval Augmented Generation aka RAG with a relevant context. Long contexts continue to have the lost in the middle syndrome.

nl 3y ago
Gemini doesn't have the "lost in the middle syndrome"
- dmezzetti 3y ago
  We don't have enough data on that yet to know for sure. It was just released this week. But nonetheless, sending 2M tokens to a LLM isn't an efficient way to solve most problems.
  - int_19h 3y ago
    LLMs in general are likely to be a very inefficient way to solve many problems they're currently being used for. But so long as we don't have something better, brute force remains a valid approach.
  - nl 3y ago
    > We don't have enough data on that yet to know for sure. It was just released this week. I think there is sufficient evidence to think it works. A bunch of people who do have access have put it through a bunch of real-world exercises to test this, and while GPT4 128K and Claude 200K both fail Gemini is passing. I agree sending 2M tokens to a LLM is costly at the moment and that RAG has a bunch of advantages in some circumstances.