5 ms·

Why would you want to retrain it from scratch every day? Stable Diffusion doesn't do that either.

paxys 4y ago
Because things happen every day. If ChatGPT wants to compete with Google, staying up to date with recent events is the minimum bar.
- Ajedi32 4y ago
  You wouldn't need to re-train from scratch for that, just fine-tune on the new data sources. I don't think constant re-training is the optimal strategy for that use-case anyway. Bing does it by letting the LLM search a more traditional web index to find the information it needs.
  - paxys 4y ago
    Okay but someone has to do the fine tuning. The code has to be updated. Parts of the training have to be redone. All of this has costs. It isn't a "do it once and forget about it" task that it is being touted as in this thread.
    - coldtea 4y ago
      >The code has to be updated I'm pretty sure this is not how an LLM works. >It isn't a "do it once and forget about it" task that it is being touted as in this thread. That's neither here, nor there. Training the LLM itself is not a "do it multiple times per day if you want to compete with Google" thing as it has been stated in this subthread.
      - baobabKoodaa 4y ago
        > > The code has to be updated > I'm pretty sure this is not how an LLM works. You can say that about any software. "You can use this software perfectly well without ever updating it." Sure, you can do that, but typically people have lots of reasons to update software. LLM isn't magic in this sense. An LLM does not mysteriously update its own code if you just wish hard enough. If you want to continue the development of the LLM then you need to make changes to the code, just like with any other software.
- simonw 4y ago
  That's not necessary. Look at how a Bing works: it's a LLM which can trigger searches, and then gets fed the search results back to it as part of the prompt. I wrote about one way to implement that pattern here: https://simonwillison.net/2023/Jan/13/semantic-search-answers/ https://simonwillison.net/2023/Jan/13/semantic-search-answer...
- coldtea 4y ago
  That's not what the training is about. Things happen everyday, but languages and words and their associations don't change in any measurable way every day... This is not like web crawling...
baobabKoodaa 4y ago
Well maybe not every day, but having a short feedback loop and the ability to run your code multiple times with different variations is generally considered to be a prerequisite for software development. If you actually want to keep developing the model, you need the funding to be able to train it more than once.
- coldtea 4y ago
  >but having a short feedback loop and the ability to run your code multiple times with different variations is generally considered to be a prerequisite for software development This is not "software development" in general, this is LLM training. It's not like you're building some regular app, api, or backend.
  - baobabKoodaa 4y ago
    If you are claiming that training a LLM literally only one time is enough and there is no need to train it more than once, you are wrong. The researchers who created OPT didn't go into a basement for 12 months, then come out, train their model once, hit publish, and go to coffee. That is a fantasy. Likewise, if a CS student wants to dabble in this research, they need the ability to train more than once. I'm not gonna engage in a rhetorical argument about whether this should be called "software development" or "LLM development" or something else. That's unrelated to the question of how much training is required.
    - coldtea 4y ago
      >If you are claiming that training a LLM literally only one time is enough and there is no need to train it more than once, you are wrong. No, I'm rather claiming that what you claimed is wrong in the context of LLM training: "Well maybe not every day, but having a short feedback loop and the ability to run your code multiple times with different variations is generally considered to be a prerequisite for software development". LLM training is not the same as writing a program and "running your code with different variations". For LLM you don't need to quickly rerun everything with some new corpus - it would be nice, but it's neither a prerequisite, not even crucial for any current use. Hell, it's not even a "prerequisite" in programming, just good to have. Tons of great programs have been written with very slow build times, without quick edit/compile/build/run cycles.

Because things happen every day. If ChatGPT wants to compete with Google, staying up to date with recent events is the minimum bar.

Ajedi32 4y ago
You wouldn't need to re-train from scratch for that, just fine-tune on the new data sources. I don't think constant re-training is the optimal strategy for that use-case anyway. Bing does it by letting the LLM search a more traditional web index to find the information it needs.
- paxys 4y ago
  Okay but someone has to do the fine tuning. The code has to be updated. Parts of the training have to be redone. All of this has costs. It isn't a "do it once and forget about it" task that it is being touted as in this thread.
  - coldtea 4y ago
    >The code has to be updated I'm pretty sure this is not how an LLM works. >It isn't a "do it once and forget about it" task that it is being touted as in this thread. That's neither here, nor there. Training the LLM itself is not a "do it multiple times per day if you want to compete with Google" thing as it has been stated in this subthread.
    - baobabKoodaa 4y ago
      > > The code has to be updated > I'm pretty sure this is not how an LLM works. You can say that about any software. "You can use this software perfectly well without ever updating it." Sure, you can do that, but typically people have lots of reasons to update software. LLM isn't magic in this sense. An LLM does not mysteriously update its own code if you just wish hard enough. If you want to continue the development of the LLM then you need to make changes to the code, just like with any other software.
simonw 4y ago
That's not necessary. Look at how a Bing works: it's a LLM which can trigger searches, and then gets fed the search results back to it as part of the prompt. I wrote about one way to implement that pattern here: https://simonwillison.net/2023/Jan/13/semantic-search-answers/ https://simonwillison.net/2023/Jan/13/semantic-search-answer...
coldtea 4y ago
That's not what the training is about. Things happen everyday, but languages and words and their associations don't change in any measurable way every day... This is not like web crawling...

baobabKoodaa 4y ago

Well maybe not every day, but having a short feedback loop and the ability to run your code multiple times with different variations is generally considered to be a prerequisite for software development. If you actually want to keep developing the model, you need the funding to be able to train it more than once.

coldtea 4y ago
>but having a short feedback loop and the ability to run your code multiple times with different variations is generally considered to be a prerequisite for software development This is not "software development" in general, this is LLM training. It's not like you're building some regular app, api, or backend.
- baobabKoodaa 4y ago
  If you are claiming that training a LLM literally only one time is enough and there is no need to train it more than once, you are wrong. The researchers who created OPT didn't go into a basement for 12 months, then come out, train their model once, hit publish, and go to coffee. That is a fantasy. Likewise, if a CS student wants to dabble in this research, they need the ability to train more than once. I'm not gonna engage in a rhetorical argument about whether this should be called "software development" or "LLM development" or something else. That's unrelated to the question of how much training is required.
  - coldtea 4y ago
    >If you are claiming that training a LLM literally only one time is enough and there is no need to train it more than once, you are wrong. No, I'm rather claiming that what you claimed is wrong in the context of LLM training: "Well maybe not every day, but having a short feedback loop and the ability to run your code multiple times with different variations is generally considered to be a prerequisite for software development". LLM training is not the same as writing a program and "running your code with different variations". For LLM you don't need to quickly rerun everything with some new corpus - it would be nice, but it's neither a prerequisite, not even crucial for any current use. Hell, it's not even a "prerequisite" in programming, just good to have. Tons of great programs have been written with very slow build times, without quick edit/compile/build/run cycles.